Skip to content

ci: add memtrack walltime benchmarks to CI - #537

Open
not-matthias wants to merge 2 commits into
mainfrom
cod-3093-memtrack-data-format-improvements
Open

not-matthias wants to merge 2 commits into
mainfrom
cod-3093-memtrack-data-format-improvements

Conversation

@not-matthias

Copy link
Copy Markdown
Member

Adds a memtrack-benchmarks CI job that runs codspeed-memtrack track against a few representative workloads (ls, dd, tar), each with RSS-only and RSS+rmap variants, to catch regressions in memtrack's own overhead.

Runs both walltime (execution time) and memory (memtrack's own peak RSS/allocations while tracking each workload) modes against the same crates/memtrack/codspeed.yml config.

@codspeed

codspeed Bot commented Sep 16, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks
🆕 12 new benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 WallTime memtrack track tar N/A 9.7 s N/A
🆕 WallTime memtrack track tar (with physical) N/A 9.8 s N/A
🆕 WallTime memtrack track dd N/A 1.3 s N/A
🆕 WallTime memtrack track dd (with physical) N/A 1.5 s N/A
🆕 WallTime memtrack track ls N/A 1.2 s N/A
🆕 WallTime memtrack track ls (with physical) N/A 1.2 s N/A
🆕 Memory memtrack track tar N/A 50.5 MB N/A
🆕 Memory memtrack track tar (with physical) N/A 50.9 MB N/A
🆕 Memory memtrack track dd N/A 50.9 MB N/A
🆕 Memory memtrack track dd (with physical) N/A 51.1 MB N/A
🆕 Memory memtrack track ls N/A 50.9 MB N/A
🆕 Memory memtrack track ls (with physical) N/A 51 MB N/A

Comparing cod-3093-memtrack-data-format-improvements (ae11946) with main (88c994e)

Open in CodSpeed

@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch 2 times, most recently from 596e932 to 5f67554 Compare September 16, 2026 10:29
@not-matthias
not-matthias changed the base branch from main to cod-3492-memtrack-classic-tracepoints-cannot-attach-inside-the September 16, 2026 10:29
Base automatically changed from cod-3492-memtrack-classic-tracepoints-cannot-attach-inside-the to main September 16, 2026 10:49
@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch 2 times, most recently from b7adccd to 3efb21e Compare September 16, 2026 14:49
@not-matthias
not-matthias marked this pull request as ready for review September 16, 2026 14:51
@greptile-apps

greptile-apps Bot commented Sep 16, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The PR appears safe to merge with no outstanding correctness or repository-rule issues.

Summary

This PR adds CI coverage for memtrack overhead and optimizes its event pipeline by sending events in batches.

  • Runs RSS-only and physical-memory variants of representative ls, dd, and tar workloads in walltime and memory modes.
  • Uses the dedicated codspeed-macro runner for walltime measurements and pins both memtrack and the CodSpeed runner to the tested revision.
  • Batches ring-buffer events before channel delivery and updates downstream consumers accordingly.
  • Adds regression tests for partial-batch flushing during timeout, explicit drain, and shutdown.
  • Adds channel and worker-scaling benchmarks for the event encoder.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
    PR[PR revision] --> Matrix{Benchmark mode}
    Matrix -->|walltime| Macro[codspeed-macro]
    Matrix -->|memory| Ubuntu[ubuntu-latest]
    Macro --> Setup[Install and configure memtrack]
    Ubuntu --> Setup
    Setup --> Workloads[Run ls, dd, and tar variants]
    Workloads --> Poller[eBPF ring-buffer poller]
    Poller --> Batch[Batch events]
    Batch --> Encode[Encode memtrack artifacts]
    Encode --> Results[CodSpeed benchmark results]
Loading

Reviews (2) · Last reviewed commit: "test(memtrack): bench the channel and wo..."

Comment thread .github/workflows/ci.yml Outdated
Comment thread crates/memtrack/src/ebpf/poller.rs
@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch from 21ce920 to 829da2f Compare September 16, 2026 16:17

@GuillaumeLagrange GuillaumeLagrange left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

olgtm, good idea!

Comment thread .github/workflows/ci.yml
Comment thread crates/memtrack/codspeed.yml Outdated
Comment thread crates/runner-shared/benches/memtrack_writer.rs Outdated

Copy link
Copy Markdown
Contributor

Memtrack walltime benchmarks took 19m to run, maybe we can shard something before introducing a 20m delay in the CI pipeline? 😇

The poll thread sent one item per channel message, and std's mpsc allocates a
31-slot block per 31 messages, so a run that captured 704k events also
allocated 22.7k blocks purely to hand them over.

The callback now fills a shared 1024-item buffer and sends it whole. Partial
batches are flushed after every poll, after the drain-path consume before its
ack, and on the shutdown consume, so `drain()` keeps promising that all
pending entries sit in the channel once it returns.

Consumers take `Vec<T>` and flatten: the encode pipeline keeps its
`IntoIterator<Item = MemtrackEvent>` contract unchanged.
@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch from 829da2f to ae11946 Compare September 18, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants