feat(profiling): Add Perfetto trace format support by markushi · Pull Request #5659 · getsentry/relay

markushi · 2026-02-25T08:17:21Z

This PR adds the ability to ingest binary Perfetto traces (.pftrace) and convert them into the existing Sample v2 JSON format used by Sentry's profiling pipeline. This targets Android
profiling, where Perfetto is the native tracing format.

Key Changes

New feature flag organizations:continuous-profiling-perfetto — gates all Perfetto processing.
Compound item format via meta_length — profile chunk items can now carry [JSON metadata][binary blob], split at meta_length. This is how Perfetto traces arrive: metadata describes the
profile, the binary blob is the raw .pftrace.
Perfetto → Sample v2 conversion — new relay-profiling/src/perfetto/ module (proto definitions + conversion logic) parses binary Perfetto traces and produces the existing Sample v2
JSON format.
Raw profile passthrough to Kafka — after expansion, the original Perfetto binary is preserved alongside the expanded JSON. Two new fields on ProfileChunkKafkaMessage: raw_profile and
raw_profile_content_type.
New fields on existing structs:
- Frame.package — library/container path for native/Java frames
- ProfileMetadata.content_type — carries "perfetto" through the pipeline
- DebugImage::native_image() constructor — creates debug images from Perfetto mapping data

This PR is part of a stack:

feat(profiling): Add Perfetto trace format support (this PR)
feat(profiling): Add pipeline workflow to perfetto profiling #5932
feat(profiling): Route raw profiles through objectstore service #6025

Add support for ingesting binary Perfetto traces (.pftrace) as profile chunks. The SDK sends an envelope with a ProfileChunk metadata item paired with a ProfileChunkData item containing the raw Perfetto protobuf. Relay decodes the Perfetto trace, extracts CPU profiling samples (PerfSample and StreamingProfilePacket), converts them to the internal Sample v2 format, and forwards both the expanded JSON and the original binary blob to Kafka for downstream processing. Key changes: - New `perfetto` module in relay-profiling for protobuf decoding and conversion to Sample v2 - New `ProfileChunkData` envelope item type for binary profile payloads - Pairing logic to associate ProfileChunk metadata with ProfileChunkData - Raw profile blob preserved through to Kafka for further processing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…tto-profiling-support

# Conflicts: # relay-server/src/envelope/item.rs

markushi · 2026-03-19T07:51:06Z

@sentry review

…fer main thread Separate the shared `strings` intern table into distinct `function_names`, `mapping_paths`, and `build_ids` tables matching the Perfetto spec where each InternedData field has its own ID namespace. Also infer the main thread name when tid equals pid and no explicit name is provided. Co-Authored-By: Claude <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: First timestamp delta skipped in StreamingProfilePacket processing
- Removed the 'i > 0' guard so that timestamp_delta_us[0] is now correctly applied to the first sample's timestamp.

Or push these changes by commenting:

@cursor push ab5f3ec796

Preview (ab5f3ec796)

diff --git a/relay-profiling/src/perfetto/mod.rs b/relay-profiling/src/perfetto/mod.rs
--- a/relay-profiling/src/perfetto/mod.rs
+++ b/relay-profiling/src/perfetto/mod.rs
@@ -229,9 +229,7 @@
             Some(Data::StreamingProfilePacket(spp)) => {
                 let mut ts = packet.timestamp.unwrap_or(0);
                 for (i, &cs_iid) in spp.callstack_iid.iter().enumerate() {
-                    if i > 0
-                        && let Some(&delta) = spp.timestamp_delta_us.get(i)
-                    {
+                    if let Some(&delta) = spp.timestamp_delta_us.get(i) {
                         // `delta` is i64 (can be negative for out-of-order samples).
                         // Casting to u64 wraps negative values, which is correct because
                         // `wrapping_add` of a wrapped negative value subtracts as expected.

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

Replace `get().is_none()` with `!contains_key()` to satisfy clippy and add a CHANGELOG entry for the Perfetto interning changes. Co-Authored-By: Claude <noreply@anthropic.com>

The first delta in timestamp_delta_us was skipped due to an i > 0 guard, but per the Perfetto spec the first sample's timestamp should be TracePacket.timestamp + timestamp_delta_us[0]. Update tests to use non-zero first deltas to verify the fix. Co-Authored-By: Claude <noreply@anthropic.com>

…kDescriptor StreamingProfilePacket samples were all assigned tid=0 because the code never resolved the trusted_packet_sequence_id → TrackDescriptor → ThreadDescriptor chain. This collapsed multi-thread profiles into a single thread. Now we build a seq_id→tid mapping from TrackDescriptor packets and use it when processing StreamingProfilePacket samples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…al state resets The two-pass architecture resolved all samples against the final state of the intern tables. If a trace contained an incremental state reset that reused an interning ID, samples collected before the reset would silently resolve to the wrong function names. This merges the two passes into a single pass that resolves callstacks immediately using the current intern table state, and extracts debug images inline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

process_compound_item was parsing the entire metadata JSON into a serde_json::Value tree just to read the content_type field, and split_item_payload was doing the same on the expanded JSON (potentially hundreds of KB). Instead, surface content_type from the already- deserialized ProfileMetadata via ExpandedPerfettoChunk, and hardcode "perfetto" in split_item_payload since compound items are always validated as perfetto by process_compound_item. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The previous commit moved expand_perfetto before the content_type check, which would waste work on non-perfetto payloads and produce confusing errors. Restore the early content_type check using a minimal serde struct that only deserializes the one field, and remove the unused content_type field from ExpandedPerfettoChunk. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace the hard-coded "perfetto" content type in split_item_payload with a lightweight deserialization that reads only the content_type field from the metadata JSON. This avoids a silent coupling that would produce incorrect values if compound items support other formats. Also remove a dead item.set_platform() call that wrote to an item immediately replaced by a new one. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

jjbayer

@Dav1dde and I will go over open comments to see what still needs to be done before we merge this.

jjbayer · 2026-06-10T11:11:52Z

+    }
+
+    let image_size = mapping.end.unwrap_or(0).saturating_sub(image_addr);
+    let image_vmaddr = mapping.load_bias.unwrap_or(0);


DebugImage::image_vmaddr can be None, but here we map None to Some(0). Is this semantically correct? Same question for image_addr.

Good point! Yes, all perfetto props are optional. mapping.start.unwrap_or(0) or mapping.end.unwrap_or(0) would be incorrect.

message Mapping { ... optional uint64 start = 4; optional uint64 end = 5; optional uint64 load_bias = 6; ... }

jjbayer · 2026-06-10T11:23:48Z

+fn collect_debug_image(
+    mapping_id: u64,
+    tables: &InternTables,
+    seen_images: &mut HashSet<(String, u64)>,


I feel like it should be possible to represent seen images as references to path vectors instead of allocated Strings, potentially in a newtype with a custom hash or equality implementation.

sentry · 2026-06-11T14:11:47Z

+    if image_addr.is_some_and(|image_addr| !seen_images.insert((code_file.clone(), image_addr))) {
+        return None;
+    }


Bug: The debug image deduplication check is bypassed when mapping.start is None because is_some_and short-circuits, which can lead to duplicate images in the final profile.
_{Severity: MEDIUM}

Suggested Fix

Modify the deduplication logic to correctly handle cases where the image address is None. One approach is to use unwrap_or(0) on image_addr when creating the key for the seen_images set. This ensures that mappings without a start address are still consistently deduplicated based on their code_file and a default address.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: relay-profiling/src/perfetto/mod.rs#L417-L419 Potential issue: In `relay-profiling/src/perfetto/mod.rs`, the deduplication logic for debug images is bypassed when a mapping's `start` address is `None`. The code uses `image_addr.is_some_and(...)` which short-circuits when `image_addr` is `None`, preventing the `seen_images.insert()` call. According to the proto schema, `mapping.start` is optional and can be `None`. If a trace contains a mapping without a start address that is referenced multiple times, a new `DebugImage` will be created and added to the profile for each reference, resulting in duplicate entries and a bloated output profile.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit dd8d044. Configure here.}

cursor · 2026-06-11T15:48:40Z

+
+    if expanded.payload.len() > ctx.config.max_profile_size() {
+        return Err(relay_profiling::ProfileError::ExceedSizeLimit.into());
+    }


Compound profile size limit bypass

Medium Severity

After Perfetto expansion, max_profile_size is applied only to the expanded JSON bytes, not to the rebuilt compound payload that also appends the original binary blob. A chunk within the ingest limit can become roughly twice the configured cap once JSON grows and the raw trace is preserved.

^{Reviewed by Cursor Bugbot for commit dd8d044. Configure here.}

sentry · 2026-06-11T15:51:25Z

+            Some(Data::ClockSnapshot(cs)) if clock_offset_ns.is_none() => {
+                clock_offset_ns = extract_clock_offset(&cs);
+            }


Bug: The code only attempts to calculate the clock offset from the first ClockSnapshot. If this snapshot is incomplete, subsequent snapshots are ignored, causing the entire trace conversion to fail.
_{Severity: HIGH}

Suggested Fix

Modify the logic to iterate through all ClockSnapshot packets until a valid offset can be calculated, instead of only trying with the first one. Alternatively, collect and merge clock information across multiple snapshots before attempting to calculate the offset.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: relay-profiling/src/perfetto/mod.rs#L251-L253 Potential issue: The trace conversion logic only attempts to extract the clock offset from the first `ClockSnapshot` packet encountered. The `extract_clock_offset` function requires both `CLOCK_REALTIME` and `CLOCK_BOOTTIME` to be present in the same snapshot. If the first snapshot is incomplete (lacking one of these clocks), `clock_offset_ns` remains `None`. Because of the `if clock_offset_ns.is_none()` guard, all subsequent `ClockSnapshot` packets are then ignored, even if they contain the necessary information. This leads to a failure when the code later requires a valid `clock_offset_ns`, causing the entire trace processing to fail for valid Perfetto traces that happen to start with an incomplete clock snapshot.

sentry · 2026-06-11T19:50:18Z

+        let idx = *ctx.frame_index.entry(key).or_insert_with(|| {
+            let next_idx = ctx.frames.len();
+            ctx.frames.push(frame);
+            next_idx
+        });


Bug: The total number of unique frames accumulated in ctx.frames is not bounded, which could allow a malicious trace to cause excessive memory usage.
_{Severity: MEDIUM}

Suggested Fix

Introduce a hard limit on the total number of unique frames that can be stored in ctx.frames. When the limit is reached, either stop processing new frames or return an error to prevent further memory allocation. This would align with existing protections against adversarial input like MAX_SAMPLES.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: relay-profiling/src/perfetto/mod.rs#L380-L384 Potential issue: The code processes Perfetto traces and accumulates unique stack frames in the `ctx.frames` vector. While there are limits on the number of samples and packets, there is no corresponding limit on the total number of unique frames. An adversarial trace could be crafted with a large number of unique frames, causing the `ctx.frames` vector to grow excessively. This could lead to memory exhaustion and a denial-of-service, as the memory usage is not properly constrained against malicious input. The presence of other bounds suggests this omission was an oversight.

markushi and others added 11 commits February 25, 2026 09:16

style(profiling): Format Perfetto module with cargo fmt

1687276

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix(profiling): Remove private intra-doc link in expand_perfetto

2cbf111

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/master' into feat/markushi/perfe…

9ee2661

…tto-profiling-support

Merge branch 'master' into feat/markushi/perfetto-profiling-support

9d4693c

# Conflicts: # relay-server/src/envelope/item.rs

Merge branch 'master' into feat/markushi/perfetto-profiling-support

4407d9e

Extend profile chunk to support blob attachment via meta_length

e67fa58

Improve code generation script and docs

c4fa16e

Guard perfetto processing by feature flag

1e6df1f

Address some more PR comments, add missing tests

0a58b24

Merge branch 'master' into feat/markushi/perfetto-profiling-support

f729460

sentry Bot reviewed Mar 19, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

markushi and others added 2 commits March 23, 2026 09:01

Merge branch 'master' into feat/markushi/perfetto-profiling-support

685ec0a

markushi marked this pull request as ready for review March 23, 2026 08:57

markushi requested a review from a team as a code owner March 23, 2026 08:57

cursor Bot reviewed Mar 23, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs

markushi and others added 2 commits March 23, 2026 10:27

fix(profiling): Fix clippy lint and add changelog entry

5c6e457

Replace `get().is_none()` with `!contains_key()` to satisfy clippy and add a CHANGELOG entry for the Perfetto interning changes. Co-Authored-By: Claude <noreply@anthropic.com>

cursor Bot reviewed Mar 27, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

markushi and others added 2 commits April 1, 2026 08:59

Merge branch 'master' into feat/markushi/perfetto-profiling-support

06467d1

sentry Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

cursor Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread relay-server/src/processing/profile_chunks/mod.rs Outdated

markushi and others added 2 commits April 1, 2026 10:36

cursor Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread relay-server/src/processing/profile_chunks/mod.rs Outdated

Comment thread relay-server/src/processing/profile_chunks/process.rs Outdated

markushi added 4 commits May 8, 2026 09:47

Use uuid::Uuid::from_bytes_le, avoid uncessary hex/string roundtrip

afb382f

Apply style suggestions

0990673

Add doc for new pub struct DebugImage

aecdb7c

Fix unneeded rename

ac8dc49

cursor Bot reviewed May 8, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs

Comment thread relay-server/src/processing/profile_chunks/process.rs

Merge branch 'master' into feat/markushi/perfetto-profiling-support

9945807

sentry Bot reviewed Jun 3, 2026

View reviewed changes

Comment thread relay-profiling/src/debug_image.rs

markushi mentioned this pull request Jun 3, 2026

feat(profiling): Route raw profiles through objectstore service #6025

Open

5 tasks

markushi and others added 2 commits June 3, 2026 11:48

Fix Changelog

d057317

Merge branch 'master' into feat/markushi/perfetto-profiling-support

b46f029

sentry-warden Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread relay-profiling/src/lib.rs

jjbayer reviewed Jun 10, 2026

View reviewed changes

jjbayer added 2 commits June 11, 2026 15:56

fix: Manually parse trace packets

d34106a

comment

a6fd768

cursor Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

keep optional optional

c5ae574

cursor Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs

sentry Bot reviewed Jun 11, 2026

View reviewed changes

jjbayer added 2 commits June 11, 2026 16:31

lint

bef7f17

restore local unwrap_or

dd8d044

cursor Bot reviewed Jun 11, 2026

View reviewed changes

sentry Bot reviewed Jun 11, 2026

View reviewed changes

sentry-warden Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs

jjbayer added 2 commits June 11, 2026 20:13

Merge branch 'master' into feat/markushi/perfetto-profiling-support

be742c9

fix: upper bound for frame_ids

5fd1156

sentry-warden Bot reviewed Jun 11, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs

bound on path segments

bef7314

jjbayer assigned jjbayer and unassigned markushi Jun 11, 2026

sentry Bot reviewed Jun 11, 2026

View reviewed changes

Conversation

markushi commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markushi commented Mar 19, 2026

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjbayer left a comment

Choose a reason for hiding this comment

Uh oh!

jjbayer Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

markushi Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjbayer Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sentry Bot Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 11, 2026

Choose a reason for hiding this comment

Compound profile size limit bypass

Uh oh!

sentry Bot Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sentry Bot Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

markushi commented Feb 25, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

markushi Jun 11, 2026 •

edited

Loading