perf: micro-bench suite + continuous profile capture for production - #861
Open
GrapeBaBa wants to merge 24 commits into
Open
perf: micro-bench suite + continuous profile capture for production#861GrapeBaBa wants to merge 24 commits into
GrapeBaBa wants to merge 24 commits into
Conversation
…aseline - New bench/stf_bench.zig: two variants benchmarking state transition via SSZ-roundtrip clone + apply_raw_block / apply_transition (validateResult=false) - Updated bench/common/fixtures.zig: replaced std.fs.Dir (removed in Zig 0.16) with std.Io.Dir.cwd().readFileAlloc for fixture loading - Registered bench-stf step in build.zig following the lean xmss pattern - Captured docs/perf/baselines/stf.txt: ~151µs apply_raw_block, ~102µs apply_transition - Appended stf profile note to docs/perf/profiles/notes.md
…y cost Re-recorded with -Cdebuginfo=line-tables-only -Cstrip=none and atos-resolved. Breakdown per worker thread: Poseidon1 internal 30%, MDS karatsuba 22%, Poseidon1 external 14%, Poseidon1Compress 12%, Keccak p1600 7%, sumcheck 4%, quintic ext 3%, ShakePRF 2%, other 6%. Combined Poseidon = 78%. Conclusion: aggregation cost lives entirely in Plonky3+leansig FFI; no zeam-side fix can move it. Real levers are upstream (Poseidon1->Poseidon2 ~2x, FFT-based MDS, GPU prover) or protocol-level (extend inclusion deadline).
…duction
Replaces the -Dslot-probes instrumentation with attach-mode profiling
tooling. No source modifications are needed in production: perf (Linux)
and samply (macOS) attach by PID and sample continuously at ~0.5% CPU
overhead, rotating chunks for async upload to object storage.
Probe call sites at chain.{onBlock,produceBlock,onGossip,onInterval}
and forkchoice.{aggregate,acceptNewAttestations}Unlocked emitted "over
budget" log warnings as a tail signal. The same information is already
exposed by existing Prometheus histograms (lean_committee_signatures_
aggregation_time_seconds, zeam_chain_onblock_duration_seconds, etc.),
so the probe layer was redundant once a continuous-capture pipeline
exists alongside metrics.
Removed:
- pkgs/utils/src/slot_probe.zig (RealProbe/NoopProbe module)
- build.zig -Dslot-probes option
- Probe call sites in chain.zig and forkchoice.zig
Added:
- scripts/profile-continuous.sh — attach perf/samply to a running zeam,
19Hz default, 200MB chunk rotation (Linux) or 5-min chunks (macOS),
designed for systemd-managed always-on capture
- scripts/profile-slice.sh — extract a time-window flamegraph from a
rotated chunk after correlating a Prometheus event to wall-clock
- docs/perf/README.md — replaces "Slot-budget probes" section with the
continuous-capture workflow
GrapeBaBa
changed the title
Neither is source. baselines/*.txt are zbench stdout snapshots tied to one machine (Apple M5) — they noise-diff every perf-touching PR and nothing in code/CI consumes them. profiles/notes.md was a free-text findings log; the same content belongs in PR descriptions or commit messages where it surfaces during review rather than rotting in a file. Profile-artifact gitignore patterns (*.samply.json, *.perf.data, *.svg) moved from docs/perf/profiles/.gitignore to the root .gitignore — no tracked files anywhere in the repo conflict, so the narrow scope was unnecessary. README updated: replaces "Refreshing a baseline" with a one-liner on diffing your own before/after runs locally; drops baselines/ and notes.md references.
Three scripts collapsed into one entry point with subcommands: profile.sh bench <name> one-shot bench profile.sh attach <pattern> [out-dir] continuous attach profile.sh slice <chunk> [start end] [out.svg] flamegraph slice The bench and attach paths really are different shapes (run-binary vs attach-PID-with-rotation) so they stay as separate code branches inside one file. slice was a 3-line pipe wrapped in argument-parsing ceremony; it now lives next to its peers rather than as its own tiny script. Env-var tunables (PROFILE_FREQ_HZ etc.) apply to `attach` only and are documented in the usage banner.
…macOS Two cross-platform fixes found while running profile.sh bench smoke end-to-end on macOS: * bench mode now passes --save-only to samply. Without it samply opens a local web UI after recording and blocks until the browser tab is closed — breaks any non-interactive use (CI, scripted profiling). The JSON file is still written; load it later with `samply load <file>` or upload to https://profiler.firefox.com/. * slice mode now detects samply JSON inputs and macOS hosts up front and prints an actionable message instead of failing with "perf not on PATH". samply JSON is self-contained — Firefox Profiler does time-window slicing interactively in its UI, no CLI needed.
Cosmetic fix: 'attach <pattern> /tmp/foo/' was producing chunk paths like '/tmp/foo//samply-<ts>.json' (double slash). Harmless to the FS but ugly in logs and uploader keys. Found by end-to-end attach validation against a real mockNetwork zeam on macOS (3 chunks written cleanly, 25 threads, ~6-7k samples each, SIGTERM trap shut samply down cleanly with all chunks flushed).
Resolved conflicts: - pkgs/spectest/src/lib.zig: keep both 'fixtures' helper struct (from this branch) and 'refAllDeclsRecursive' Zig 0.16 compat (from main, PR #715) - pkgs/spectest/src/runner/state_transition_runner.zig: keep both decodeBlock (this branch) and parseAggregationBits/parseAttestationData (from main) Submodule pointers (lean-quickstart, leanSpec) adopted from main.
GrapeBaBa
marked this pull request as ready for review
May 13, 2026 08:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Investigation-driven micro-bench suite for the four CPU-critical paths the #803 threading-refactor doc named as "must not run on the libxev IO thread": XMSS PROD verify, STF, signature aggregation, forkchoice + SSZ. Plus in-process slot-budget probes for live-load measurement.
What's new
bench/directory with 5 binaries (smoke, xmss, stf, aggregation, forkchoice-ssz), each invokable viazig build bench-<name>or all together viazig build bench. All builtReleaseFastregardless of-Doptimize=....scripts/profile.shwrappingsamply(macOS) /perf+ flamegraph (linux) on the bench binaries. Profile artifacts gitignored underdocs/perf/profiles/; findings logged innotes.md.pkgs/utils/src/slot_probe.zigbehind a-Dslot-probesbuild option. Default off →NoopProbecompiles away to zero cost (verified: binary size identical before/after). 6 probe call sites:chain.zig(onBlock,produceBlock,onGossip,onInterval) andforkchoice.zig(aggregateUnlocked,acceptNewAttestationsUnlocked).docs/perf/baselines/— text-format zbench reports for each target;git diffis the regression detector.bench/README.md+docs/perf/README.md— running benches, profiling, refreshing baselines, slot probes.Headline findings (from
docs/perf/profiles/notes.md)hashsig_glue), already NEON-vectorized and parallelized across 10 rayon workers. Not optimizable at the Zig level — confirms BeamNode threading model refactor — 8-point plan (supersedes #798-#802) #803's "must move off IO thread" framing.hash_tree_rootcost — ~30% of per-block STF work.AutoHashMapputs +protoArrayindex inserts. Optimization avenue: arena allocation / lock-free structures, not faster math.Out of scope (documented as follow-ups)
lean-quickstartblocksByRoot, gossip)compute_aggregated_signatures(fixture too large for single-task scope),fc_acceptNewAttestations(needs multi-validator state buildup),hash_tree_root_cached_miss/hit(cache lacks force-miss API; STF bench covers it indirectly)Test Plan
zig build bench— all 5 targets produce zbench reports (smoke + xmss + stf + aggregation + forkchoice-ssz)zig build test --summary all— full suite passes with probes OFF (default)zig build test -Dslot-probes=true --summary all— full suite passes with probes ONzig build stress-quick— passes both probe modes (no panics / UAFs / deadlocks)cargo fmt --check+cargo clippy -D warnings+zig fmt --check .— all cleandocs/perf/baselines/docs/perf/profiles/notes.md(4 entries)