Skip to content

Make bench-e2e hardfork flags optional#5613

Merged
klkvr merged 5 commits into
mainfrom
centaur/1781123881-20394-31463
Jun 11, 2026
Merged

Make bench-e2e hardfork flags optional#5613
klkvr merged 5 commits into
mainfrom
centaur/1781123881-20394-31463

Conversation

@decofe

@decofe decofe commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • Derive Tempo hardfork names from crates/node/tests/assets/test-genesis.json instead of maintaining a hardcoded TEMPO_HARDFORKS list.
  • Pick up new hardfork config keys like t8Time automatically for latest-tempo-hardfork and hardfork genesis args.
  • Stop defaulting bench-e2e workflow_dispatch / workflow_call hardfork inputs to T6; leave them empty so bench-e2e.nu uses its existing defaults.

Validation

  • nu helper check: hardforks resolve through T8 and latest-tempo-hardfork returns T8
  • nu bench-e2e.nu e2e --help
  • uv run --with pyyaml python YAML parse check for .github/workflows/bench-e2e.yml
  • git diff --check

@decofe decofe force-pushed the centaur/1781123881-20394-31463 branch 5 times, most recently from 7b93e75 to a051222 Compare June 10, 2026 21:03
@decofe decofe force-pushed the centaur/1781123881-20394-31463 branch from a051222 to bfedcec Compare June 10, 2026 21:04
@decofe

decofe commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

cc @klkvr

❌ Benchmark failed. View logs

@decofe

decofe commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

cc @klkvr

⚪ Benchmark complete: No Difference View job

⚪ Bench Comparison: No Difference

Refs: main vs centaur/1781123881-20394-31463
Criteria: 95% run-bootstrap CI must clear floor; cells show delta (+/-CI/floor).

Configuration

  • Derek command: derek bench mode=e2e preset=tip20 duration=90 bloat=100 tps=50000 accounts=1000 max-concurrent-requests=100 baseline=main feature=centaur/1781123881-20394-31463 baseline-hardfork="" feature-hardfork="" gas-limit=1000000000 run-pairs=3 otlp=true metrics=false no-cache=false force-bloat=false
  • Bloat: 100000 MiB
  • Preset: tip20
  • Target TPS: 50000
  • Duration: 90s
  • Run pairs: 3
  • Baseline blocks: 483
  • Feature blocks: 486

Tempo Metrics

Metric Baseline Feature Delta
TPS Mean 21196 21191 -0.02% ⚪ (+/-1.09/floor 0.55)
Gas Throughput [Mgas/s] 1077.5 1077.2 -0.03% ⚪ (+/-1.06/floor 0.50)
Block Time Mean [ms] 555.4 554.0 -0.25% ⚪ (+/-0.55/floor 0.40)
Block Time P50 [ms] 556.0 554.0 -0.36% ⚪ (+/-0.51/floor 0.70)
Block Time P90 [ms] 594.0 590.0 -0.67% ⚪ (+/-0.95/floor 0.70)
Block Time P99 [ms] 618.0 614.0 -0.65% ⚪ (+/-1.78/floor 1.60)
Serialized Block Size / Tx P50 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)
Serialized Block Size / Tx P90 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)
Serialized Block Size / Tx P99 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)

Builder

Metric Baseline Feature Delta
Gas Throughput [Mgas/s] 2540.9 2537.6 -0.13% ⚪ (+/-0.88/floor 0.95)
P50 [ms] 234.5 234.3 -0.09% ⚪ (+/-0.31/floor 0.45)
P90 [ms] 245.1 246.1 +0.41% ⚪ (+/-0.64/floor 0.90)
P99 [ms] 303.1 309.5 +2.11% ⚪ (+/-3.35/floor 1.25)
Builder details
Metric Baseline Feature Delta
Finish P50 [ms] 16.7 16.9 +1.20%
Finish P90 [ms] 23.8 24.6 +3.36%
Finish P99 [ms] 31.8 34.9 +9.75%
Pool Fetch P50 [ms] 6.9 8.9 +28.99%
Pool Fetch P90 [ms] 21.2 21.5 +1.42%
Pool Fetch P99 [ms] 26.7 29.3 +9.74%
Included Tx Exec P50 [ms] - -
Included Tx Exec P90 [ms] - -
Included Tx Exec P99 [ms] - -
Invalid Tx Exec P50 [ms] - -
Invalid Tx Exec P90 [ms] - -
Invalid Tx Exec P99 [ms] - -
Invalid Tx Attempts P50 0.0 0.0 0.00%
Invalid Tx Attempts P90 0.0 0.0 0.00%
Invalid Tx Attempts P99 0.0 0.0 0.00%
Invalid Tx Skips 107 0 -100.00%
Nonce Too Low Skips 0 0 0.00%
Serialized Block Size P50 [KiB] 2899.2 2893.0 -0.21%
Serialized Block Size P90 [KiB] 3274.7 3245.0 -0.91%
Serialized Block Size P99 [KiB] 3521.5 3546.5 +0.71%
Fill Overhead P50 [ms] - -
Fill Overhead P90 [ms] - -
Fill Overhead P99 [ms] - -
Fill Idle P50 [ms] 0.0 0.0 0.00%
Fill Idle P90 [ms] 39.0 39.0 +0.00%
Fill Idle P99 [ms] 147.0 153.0 +4.08%

Validator

Metric Baseline Feature Delta
Gas Throughput [Mgas/s] 2498.6 2503.0 +0.18% ⚪ (+/-0.68/floor 0.65)
P50 [ms] 241.8 241.1 -0.29% ⚪ (+/-0.79/floor 1.55)
P90 [ms] 269.4 265.2 -1.56% ⚪ (+/-1.33/floor 1.55)
P99 [ms] 283.3 284.9 +0.56% ⚪ (+/-2.64/floor 2.05)

Observability

shekhirin
shekhirin previously approved these changes Jun 11, 2026
View reviewed changes
0xrusowsky
0xrusowsky previously approved these changes Jun 11, 2026
View reviewed changes
@klkvr klkvr dismissed stale reviews from 0xrusowsky and shekhirin via 31647f0 June 11, 2026 10:26
@decofe

decofe commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

cc @klkvr

⚪ Benchmark complete: No Difference View job

⚪ Bench Comparison: No Difference

Refs: main vs centaur/1781123881-20394-31463
Criteria: 95% run-bootstrap CI must clear floor; cells show delta (+/-CI/floor).

Configuration

  • Derek command: derek bench mode=e2e preset=tip20 duration=90 bloat=100 tps=50000 accounts=1000 max-concurrent-requests=100 baseline=main feature=centaur/1781123881-20394-31463 baseline-hardfork="" feature-hardfork="" gas-limit=1000000000 run-pairs=3 otlp=true metrics=false no-cache=false force-bloat=false
  • Bloat: 100000 MiB
  • Preset: tip20
  • Target TPS: 50000
  • Duration: 90s
  • Run pairs: 3
  • Baseline blocks: 478
  • Feature blocks: 478

Tempo Metrics

Metric Baseline Feature Delta
TPS Mean 21890 21756 -0.61% ⚪ (+/-0.36/floor 0.55)
Gas Throughput [Mgas/s] 1112.8 1106.0 -0.61% ⚪ (+/-0.34/floor 0.50)
Block Time Mean [ms] 559.0 558.9 -0.02% ⚪ (+/-0.36/floor 0.40)
Block Time P50 [ms] 560.0 559.0 -0.18% ⚪ (+/-0.63/floor 0.70)
Block Time P90 [ms] 605.0 603.0 -0.33% ⚪ (+/-0.83/floor 0.70)
Block Time P99 [ms] 624.0 629.0 +0.80% ⚪ (+/-0.85/floor 1.60)
Serialized Block Size / Tx P50 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)
Serialized Block Size / Tx P90 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)
Serialized Block Size / Tx P99 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)

Builder

Metric Baseline Feature Delta
Gas Throughput [Mgas/s] 2628.7 2609.5 -0.73% ⚪ (+/-0.44/floor 0.95)
P50 [ms] 234.1 234.7 +0.26% ⚪ (+/-0.43/floor 0.45)
P90 [ms] 250.2 251.7 +0.60% ⚪ (+/-1.40/floor 0.90)
P99 [ms] 300.9 300.6 -0.10% ⚪ (+/-1.81/floor 1.25)
Builder details
Metric Baseline Feature Delta
Finish P50 [ms] 15.7 16.5 +5.10%
Finish P90 [ms] 23.6 23.7 +0.42%
Finish P99 [ms] 30.7 31.3 +1.95%
Pool Fetch P50 [ms] 9.0 9.5 +5.56%
Pool Fetch P90 [ms] 21.9 22.1 +0.91%
Pool Fetch P99 [ms] 29.4 29.4 +0.00%
Included Tx Exec P50 [ms] - -
Included Tx Exec P90 [ms] - -
Included Tx Exec P99 [ms] - -
Invalid Tx Exec P50 [ms] - -
Invalid Tx Exec P90 [ms] - -
Invalid Tx Exec P99 [ms] - -
Invalid Tx Attempts P50 0.0 0.0 0.00%
Invalid Tx Attempts P90 0.0 0.0 0.00%
Invalid Tx Attempts P99 0.0 0.0 0.00%
Invalid Tx Skips 0 134 n/a
Nonce Too Low Skips 0 0 0.00%
Serialized Block Size P50 [KiB] 3013.1 2996.0 -0.57%
Serialized Block Size P90 [KiB] 3426.7 3387.4 -1.14%
Serialized Block Size P99 [KiB] 3584.8 3623.5 +1.08%
Fill Overhead P50 [ms] - -
Fill Overhead P90 [ms] - -
Fill Overhead P99 [ms] - -
Fill Idle P50 [ms] 0.0 0.0 0.00%
Fill Idle P90 [ms] 51.0 61.0 +19.61%
Fill Idle P99 [ms] 143.0 138.0 -3.50%

Validator

Metric Baseline Feature Delta
Gas Throughput [Mgas/s] 2606.7 2591.0 -0.60% ⚪ (+/-0.42/floor 0.65)
P50 [ms] 242.5 243.2 +0.29% ⚪ (+/-1.04/floor 1.55)
P90 [ms] 275.3 277.0 +0.62% ⚪ (+/-0.96/floor 1.55)
P99 [ms] 287.4 293.0 +1.95% ⚪ (+/-1.83/floor 2.05)

Observability

@klkvr

klkvr commented Jun 11, 2026

Copy link
Copy Markdown
Member

derek bench baseline-hardfork=t6 feature-hardfork=t8

@decofe

decofe commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

❌ Invalid bench command

Unknown hardfork(s): T8. Must be one of: T0, T1, T1A, T1B, T1C, T2, T3, T4, T5, T6.

Usage: @decofe bench [mode=MODE] [chain=mainnet|testnet] [blocks=N] [warmup=N] [run-pairs=N] [preset=NAME] [duration=N] [bloat=1|10|100] [tps=N] [accounts=N] [max-concurrent-requests=N] [gas-limit=N|NM|NG|NT] [baseline=REF] [feature=REF] [baseline-hardfork=T0|T1|T1A|T1B|T1C|T2|T3|T4|T5|T6] [feature-hardfork=T0|T1|T1A|T1B|T1C|T2|T3|T4|T5|T6] [baseline-features="FEATURES"] [feature-features="FEATURES"] [txgen-ref=REF] [samply] [otlp] [valscope] [force-bloat] [no-cache] [no-slack] [tracy=MODE] [tracy-seconds=N] [tracy-offset=N] [baseline-args="ARGS"] [feature-args="ARGS"] [bench-args="ARGS"] [bench-env="VARS"] [baseline-env="VARS"] [feature-env="VARS"]

@decofe

decofe commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

cc @klkvr

⚪ Benchmark complete: No Difference View job

⚪ Bench Comparison: No Difference

Refs: main vs centaur/1781123881-20394-31463
Criteria: 95% run-bootstrap CI must clear floor; cells show delta (+/-CI/floor).

Configuration

  • Derek command: derek bench mode=e2e preset=tip20 duration=90 bloat=100 tps=50000 accounts=1000 max-concurrent-requests=100 baseline=main feature=centaur/1781123881-20394-31463 baseline-hardfork=t6 feature-hardfork=t8 gas-limit=1000000000 run-pairs=3 otlp=true metrics=false no-cache=false force-bloat=false
  • Bloat: 100000 MiB
  • Preset: tip20
  • Target TPS: 50000
  • Duration: 90s
  • Run pairs: 3
  • Baseline blocks: 478
  • Feature blocks: 478

Tempo Metrics

Metric Baseline Feature Delta
TPS Mean 21546 21472 -0.34% ⚪ (+/-0.51/floor 0.55)
Gas Throughput [Mgas/s] 1095.3 1091.5 -0.35% ⚪ (+/-0.49/floor 0.50)
Block Time Mean [ms] 558.1 558.8 +0.13% ⚪ (+/-0.36/floor 0.40)
Block Time P50 [ms] 559.0 559.0 +0.00% ⚪ (+/-0.42/floor 0.70)
Block Time P90 [ms] 594.0 597.0 +0.51% ⚪ (+/-0.67/floor 0.70)
Block Time P99 [ms] 622.0 622.0 +0.00% ⚪ (+/-1.26/floor 1.60)
Serialized Block Size / Tx P50 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)
Serialized Block Size / Tx P90 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)
Serialized Block Size / Tx P99 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)

Builder

Metric Baseline Feature Delta
Gas Throughput [Mgas/s] 2612.3 2613.6 +0.05% ⚪ (+/-0.66/floor 0.95)
P50 [ms] 232.1 232.1 +0.00% ⚪ (+/-0.40/floor 0.45)
P90 [ms] 242.9 243.6 +0.29% ⚪ (+/-0.54/floor 0.90)
P99 [ms] 288.0 291.8 +1.32% ⚪ (+/-5.81/floor 1.25)
Builder details
Metric Baseline Feature Delta
Finish P50 [ms] 17.8 17.7 -0.56%
Finish P90 [ms] 24.8 24.8 +0.00%
Finish P99 [ms] 33.1 33.3 +0.60%
Pool Fetch P50 [ms] 8.5 8.5 +0.00%
Pool Fetch P90 [ms] 21.0 20.9 -0.48%
Pool Fetch P99 [ms] 31.6 25.9 -18.04%
Included Tx Exec P50 [ms] - -
Included Tx Exec P90 [ms] - -
Included Tx Exec P99 [ms] - -
Invalid Tx Exec P50 [ms] - -
Invalid Tx Exec P90 [ms] - -
Invalid Tx Exec P99 [ms] - -
Invalid Tx Attempts P50 0.0 0.0 0.00%
Invalid Tx Attempts P90 0.0 0.0 0.00%
Invalid Tx Attempts P99 0.0 0.0 0.00%
Invalid Tx Skips 0 0 0.00%
Nonce Too Low Skips 0 0 0.00%
Serialized Block Size P50 [KiB] 2956.0 2942.1 -0.47%
Serialized Block Size P90 [KiB] 3309.5 3322.5 +0.39%
Serialized Block Size P99 [KiB] 3567.4 3634.5 +1.88%
Fill Overhead P50 [ms] - -
Fill Overhead P90 [ms] - -
Fill Overhead P99 [ms] - -
Fill Idle P50 [ms] 0.0 0.0 0.00%
Fill Idle P90 [ms] 8.0 16.0 +100.00%
Fill Idle P99 [ms] 122.0 126.0 +3.28%

Validator

Metric Baseline Feature Delta
Gas Throughput [Mgas/s] 2501.3 2486.9 -0.57% ⚪ (+/-0.45/floor 0.65)
P50 [ms] 245.1 246.0 +0.37% ⚪ (+/-0.52/floor 1.55)
P90 [ms] 272.3 272.5 +0.07% ⚪ (+/-1.37/floor 1.55)
P99 [ms] 292.5 292.2 -0.10% ⚪ (+/-2.76/floor 2.05)

Observability

@decofe

decofe commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

cc @klkvr

🚀 Benchmark started! View job

⏳ Status: Running benchmark phase feature-1 (1/6)...

Config: mode: e2e, preset: tip20, duration: 90s, bloat: 100 GiB, tps: 50000, accounts: 1000, max-concurrent-requests: 100, gas-limit: 1000000000, run-pairs: 3, baseline: main, feature: centaur/1781123881-20394-31463, txgen-ref: default, baseline-hardfork: t6, feature-hardfork: t8, otlp: enabled, valscope: disabled

@decofe

decofe commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

cc @klkvr

❌ Benchmark failed. View logs

@decofe

decofe commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

cc @klkvr

⚠️ Benchmark complete: Mixed Results View job

⚠️ Bench Comparison: Mixed Results

Refs: main vs centaur/1781123881-20394-31463
Criteria: 95% run-bootstrap CI must clear floor; cells show delta (+/-CI/floor).

Configuration

  • Derek command: derek bench mode=e2e preset=tip20 duration=90 bloat=100 tps=50000 accounts=1000 max-concurrent-requests=100 baseline=main feature=centaur/1781123881-20394-31463 baseline-hardfork=t6 feature-hardfork=t8 gas-limit=1000000000 run-pairs=3 otlp=true metrics=false no-cache=false force-bloat=false
  • Bloat: 100000 MiB
  • Preset: tip20
  • Target TPS: 50000
  • Duration: 90s
  • Run pairs: 3
  • Baseline blocks: 483
  • Feature blocks: 481

Tempo Metrics

Metric Baseline Feature Delta
TPS Mean 21294 22677 +6.49% ✅ (+/-0.90/floor 0.55)
Gas Throughput [Mgas/s] 1082.5 998.7 -7.74% ❌ (+/-0.86/floor 0.50)
Block Time Mean [ms] 554.4 558.0 +0.65% ⚪ (+/-0.60/floor 0.40)
Block Time P50 [ms] 557.0 559.0 +0.36% ⚪ (+/-0.48/floor 0.70)
Block Time P90 [ms] 588.0 598.0 +1.70% ❌ (+/-0.60/floor 0.70)
Block Time P99 [ms] 622.0 628.0 +0.96% ⚪ (+/-2.36/floor 1.60)
Serialized Block Size / Tx P50 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)
Serialized Block Size / Tx P90 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)
Serialized Block Size / Tx P99 [B/tx] 251.1 251.1 +0.00% ⚪ (+/-0.00/floor 0.70)

Builder

Metric Baseline Feature Delta
Gas Throughput [Mgas/s] 2591.7 2426.1 -6.39% ❌ (+/-1.06/floor 0.95)
P50 [ms] 231.4 228.0 -1.47% ✅ (+/-0.17/floor 0.45)
P90 [ms] 240.0 243.7 +1.54% ❌ (+/-0.28/floor 0.90)
P99 [ms] 269.7 268.9 -0.30% ⚪ (+/-6.17/floor 1.25)
Builder details
Metric Baseline Feature Delta
Finish P50 [ms] 19.6 26.5 +35.20%
Finish P90 [ms] 25.8 38.1 +47.67%
Finish P99 [ms] 34.0 48.5 +42.65%
Pool Fetch P50 [ms] 9.2 8.2 -10.87%
Pool Fetch P90 [ms] 20.9 22.7 +8.61%
Pool Fetch P99 [ms] 29.5 29.0 -1.69%
Included Tx Exec P50 [ms] - -
Included Tx Exec P90 [ms] - -
Included Tx Exec P99 [ms] - -
Invalid Tx Exec P50 [ms] - -
Invalid Tx Exec P90 [ms] - -
Invalid Tx Exec P99 [ms] - -
Invalid Tx Attempts P50 0.0 0.0 0.00%
Invalid Tx Attempts P90 0.0 0.0 0.00%
Invalid Tx Attempts P99 0.0 0.0 0.00%
Invalid Tx Skips 0 87 n/a
Nonce Too Low Skips 0 0 0.00%
Serialized Block Size P50 [KiB] 2945.5 3105.3 +5.43%
Serialized Block Size P90 [KiB] 3255.1 3514.4 +7.97%
Serialized Block Size P99 [KiB] 3514.7 4115.2 +17.09%
Fill Overhead P50 [ms] - -
Fill Overhead P90 [ms] - -
Fill Overhead P99 [ms] - -
Fill Idle P50 [ms] 0.0 0.0 0.00%
Fill Idle P90 [ms] 0.0 17.0 n/a
Fill Idle P99 [ms] 88.0 96.0 +9.09%

Validator

Metric Baseline Feature Delta
Gas Throughput [Mgas/s] 2495.0 2314.0 -7.25% ❌ (+/-0.52/floor 0.65)
P50 [ms] 242.0 242.9 +0.37% ⚪ (+/-1.16/floor 1.55)
P90 [ms] 264.5 266.6 +0.79% ⚪ (+/-1.42/floor 1.55)
P99 [ms] 279.2 286.5 +2.61% ⚪ (+/-5.86/floor 2.05)

Observability

@tempo-voight-kampff

tempo-voight-kampff Bot commented Jun 11, 2026

Copy link
Copy Markdown

Hi @klkvr — your review approval was detected by Voight-Kampff but no live Voight-Kampff agent connection received it, so Voight-Kampff sent a push fallback. The push was not approved before the approval window closed. Your review did not count toward this PR.

Re-submit the review to send a fresh approval request. If Voight-Kampff is running and connected, you will get a prompt on your device.

@klkvr klkvr enabled auto-merge June 11, 2026 16:22
@klkvr klkvr added this pull request to the merge queue Jun 11, 2026
Merged via the queue into main with commit fc2ef15 Jun 11, 2026
48 of 53 checks passed
@klkvr klkvr deleted the centaur/1781123881-20394-31463 branch June 11, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@klkvr klkvr klkvr approved these changes

@shekhirin shekhirin shekhirin left review comments

@0xrusowsky 0xrusowsky 0xrusowsky left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@decofe @klkvr @shekhirin @0xrusowsky