feat(1474): expose ledger-emitted events from the node by m2ux · Pull Request #1849 · midnightntwrk/midnight-node

m2ux · 2026-07-08T08:57:47Z

Summary

Expose the ledger's per-transaction events as additive Substrate runtime events so the indexer and other consumers read them from the node instead of re-executing every transaction to reconstruct them.

🐛 Issue 📐 Engineering

Motivation

When the node applies a transaction the ledger emits a stream of events describing what happened, and the node immediately discards it. Every consumer that needs those events — the indexer above all — must re-run each transaction locally to reconstruct them, duplicating compute on every operator's machine and slowing time-to-visibility for wallets and explorers.

Changes

This change makes the node expose the events that the ledger already produces when it applies a transaction. The node now deposits those events as ordinary runtime events, carrying each one as the ledger's own self-describing tagged bytes, so consumers can read them straight from chain state. Both ordinary user transactions and system transactions emit their events, and the system-transaction event now carries the transaction hash so a consumer can line it up with the matching ledger events. Because deposited events live in per-block event storage that is cleared every block rather than in the gossiped block body, publishing them adds no ongoing traffic between peers.

Exposing a new event surface changes the runtime's public shape, so the work is built to stay safe across the window where a network is part-way through an upgrade. The ledger's host functions are versioned: the previous return shapes are preserved unchanged and the events travel on a new version alongside them, so an older runtime on a newer node still reads the old shape cleanly, while a newer runtime on an older node fails immediately and visibly rather than silently misreading data. The upgrade tooling now refuses to activate a new runtime until every validator is confirmed to be on a capable binary, enforcing a binaries-first rollout, and the release pipeline refuses to publish a changed runtime under an unchanged version number. The runtime version is raised to 2.2.0 to give the new surface a distinct identity, and the published metadata is regenerated so off-chain decoders can see and decode the new events.

The work also hardens two areas the new event stream interacts with. The cross-chain bridge previously cleared its pending accounting records as soon as it attempted a system transaction, even on failure, which could permanently lose a transfer's bookkeeping; those records are now cleared only when the transaction actually succeeds, so a failure leaves the transfer recoverable while still guaranteeing each one is processed at most once. And because emitting events is real work, both paths that deposit them now reserve block weight for each event so a large burst cannot quietly exceed a block's budget, with the accompanying benchmark anchored to the true per-block data ceiling rather than a small fraction of it.

🤖 AI Assistance

Assistant / Model: Claude Code / claude-opus-4-8 (1M context)
Context scope: mixed
Prompt classes: code-generation, test-writing, refactoring, docs
Provenance log: 08-provenance-log.md

📌 Submission Checklist

Changes are backward-compatible (or flagged if breaking)
Pull request description explains why the change is needed
Self-reviewed the diff
I have included a change file, or skipped for this reason: changes/runtime/added/expose-ledger-events.md
If the changes introduce a new feature, I have bumped the node minor version
Update documentation (if relevant)
No new todos introduced

🔱 Fork Strategy

Node Runtime Update
Node Client Update
Other
N/A

🗹 TODO before merging

Rebuild the runtime metadata for the new event variants
Sync the branch with main
Bump the node minor version for this new feature
Run clippy (workspace clippy with -D warnings is green; fmt is clean)
Run the full cargo test suite (green in CI)
Run the re-anchored 50 MB bench_block_full_of_events benchmark and fold the measured per-event cost into the placeholder weights
Add the bridge failure-branch test for the consume-on-success fix (the mock executor currently always succeeds); remaining partial-success / integration cases are in the test plan
Confirm with stakeholders that the indexer is the only service that needs to read these events directly from the node (asked on the issue thread)
Ready for review

Scaffold commit to open the draft PR for midnight-node#1474. Implementation follows per the work package plan. Signed-off-by: Mike Clay <mike.clay@shielded.io>

…ht-node#1474) Stop discarding the per-transaction Vec<Event<D>> that the ledger already produces. Ledger::apply_verified_transaction now returns the event stream alongside its applied-stage classification, and Ledger::apply_system_tx keeps the event vec from its success tuple instead of dropping it. Because versions/common/ compiles once per ledger version via module parameterization, this single source edit lands in v7, v8 and v9. The two Bridge call sites destructure the vec but discard it for now; T3 consumes it. Failure semantics are unchanged: TransactionResult::Failure still returns LedgerApiError::Transaction(Invalid(..)) and emits no events. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…eld (midnight-node#1474) Introduce the SCALE wire shape that carries ledger events across the host boundary, using the routing-header hybrid from the plan (section 3): - LedgerEventSource mirrors the ledger's EventSource (tag event-source[v1], stable across v7/v8/v9) field-for-field, so consumers can route on the (transaction_hash, logical_segment, physical_segment) triple. - LedgerEvent { source, content_tagged_bytes } keeps the variable/divergent EventDetails<D> payload opaque as the ledger's own tagged bytes. The tag is self-describing, so version divergence (event-details[v9] vs [v14], the v9 ContractLog.logged_item change) needs no node-side branching. Both host-return structs gain an additive events: Vec<LedgerEvent> field, appended last (design decision D1). Appending keeps the field decode-safe against historical replay: a decoder built with the field defaults it to an empty vec on bytes encoded before it existed, so no new #[version(N)] host function is required. Error plumbing: a new SerializationError::EventDetails variant and its SerializableError impl route tagged-serialise failures on EventDetails<D> through the existing LedgerApiError::Serialization channel. The two Bridge constructors get empty-vec placeholders; T3 populates them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…ts (midnight-node#1474) Consume the event stream that T1 surfaced in Bridge::apply_transaction and Bridge::apply_system_transaction. A shared build_ledger_events helper mirrors each EventSource into the SCALE LedgerEventSource header and tagged-serialises the EventDetails<D> payload into content_tagged_bytes, then populates the additive events field on the return struct. The helper is uniform across all three ledger versions: the tag is embedded in the serialised bytes, so the v9 divergence (event-details[v14], ContractLog.logged_item change) is absorbed opaquely with no per-version conversion code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…idnight (midnight-node#1474) Add Event::LedgerEvent(LedgerEvent) to the pallet-midnight event enum, appended last so the existing eight variants keep their positional indices. In send_mn_transaction, iterate the new result.events field and deposit one runtime event per ledger event (design decision D2), preserving per-event (pallet_index, variant_index) filtering for subxt / Polkadot.js consumers. The existing deposit sites are byte-identical. Emission is non-consensus narration and is not weighed (accept-unpriced, D3); the T7 benchmark is the guardrail for that decision. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…ght-node#1474) Apply the symmetric system-tx change (design decision D4). Add a sibling Event::LedgerEvent(LedgerEvent) variant to pallet-midnight-system, appended last so the existing SystemTransactionApplied index is unchanged. Both system-tx apply paths — the send_mn_system_transaction dispatchable and the MidnightSystemTransactionExecutor::execute_system_transaction trait impl — now carry the new result.events field out of the mut_ledger_state closure and deposit one runtime event per ledger event beside the existing SystemTransactionApplied deposit. This surfaces ParamChange, DustInitialUtxo and dust-generation events that were previously invisible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…dnight-node#1474) Add the worst-case event-emission benchmark that discharges GO-gate condition C1 / acceptance metric M8. It fills a block with up to MAX_BENCH_EVENTS ledger events, each carrying a deploy-heavy ~4 KiB opaque payload, so the total event volume (~1 MiB) tracks the ledger's per-block bytes_churned ceiling — the code-grounded worst-case magnitude from the comprehension artifact (DD11). The benchmark measures the runtime-side deposit cost (the state-trie write into frame_system::Events) that the accept-unpriced decision (D3) deliberately leaves unweighed. The human runs it and compares the reported weight against BlockWeights headroom; this converts the accept-unpriced choice into a decision backed by measurement. Compilation and execution are deferred to the human runner (cargo bench / runtime-benchmarks harness). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…node#1474) Add docs/ledger-events.md covering the new runtime-event surface: - What is emitted: the per-event LedgerEvent variant on pallet-midnight and its sibling on pallet-midnight-system, and the routing-header-plus-opaque-bytes wire shape. - Indexer-author guidance: consume from frame_system::Events via subscribeStorage / getStorage, and decode content_tagged_bytes with the matching ledger version's tagged_deserialize (the tag self-identifies the version). - Contract-author guidance: (address, entry_point) is the contract-event namespace; no separate node-side topic field. - Pricing: the accept-unpriced decision, why it is safe, and the bench_block_full_of_events guardrail plus the per-event-weight-term fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

Add changes/runtime/added/expose-ledger-events.md per the in-tree change-file convention (component-tagged runtime addition), describing the new LedgerEvent runtime-event surface and pointing at docs/ledger-events.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…node#1474) Trim agent-authored doc blocks and inline comments on the new ledger-event surface down to terse one-liners, matching the proportionality of the surrounding code. Comment-only; no logic change. Ref: #1474 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…1474) Add PR1849-TC-01/TC-02 payload round-trip and TC-11/TC-14 wire-type SCALE round-trip. Strengthen the apply_verified_transaction test helper to return and assert the emitted events instead of discarding them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

…de#1474) Add PR1849-TC-01 per-event deposit with tx-hash routing, TC-15 failed-tx emits nothing, TC-17 dry-run emits nothing, and TC-11 last-variant index check. Update the existing send_mn_transaction assertion for the new interleaved LedgerEvent records. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike Clay <mike.clay@shielded.io>

chatgpt-codex-connector · 2026-07-13T10:46:20Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

…es (midnight-node#1474) The event-stream work added `Vec<Event<D>>` to the return tuples of `apply_verified_transaction` and `apply_system_tx`, pushing both past clippy's type-complexity threshold under `-D warnings` on the Rust 1.95 toolchain (merged from main). The tuple shapes are intentional and read clearly at the call sites, so annotate both methods with `#[allow(clippy::type_complexity)]`. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mike Clay <mike.clay@shielded.io>

The ledger-event work deposits LedgerEvent from pallet-midnight and pallet-midnight-system, changing the runtime event surface, so the checked-in static metadata no longer matches the runtime (failing the metadata check and the toolkit's static codegen). Regenerate midnight_metadata.scale and the 2.0.0 snapshot via `subxt metadata -f bytes` (subxt 0.50.0) against the node image built for this branch. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mike Clay <mike.clay@shielded.io>

jina-simulation

Content removed.

Address the eight findings from the runtime review of PR #1849, plus the metadata regeneration the review flagged as a release follow-up. - Emit the ledger transaction hash (not the state root) in the deposited system-transaction event on both the governance dispatchable and the internal executor, so downstream consumers can correlate the system event with its ledger events by hash. - Version the ledger host functions for the new return shape: the apply structs revert to their events-free (pre-PR, byte-identical) form and gain WithEvents siblings carrying the trailing events field, so an old node binary and a new runtime negotiate the events-free shape while a new binary provides the events version. - Re-anchor the event-volume benchmark to the real 50 MB bytesChurned ceiling instead of the earlier ~1 MiB bound. - Consume pending cross-chain accounting state only on system-transaction success: the subminimal-transfer accumulator is retained and the approved-hash entry re-inserted when execution fails, preserving single-use protection on success. - Add a pre-activation validator-binary compatibility gate to the runtime upgrade flow, refusing to authorize when the node's active runtime spec version is below the required minimum, with an explicit opt-out. - Apply the advertised include/exclude service filters during image rollout. - Bump the runtime spec_version to 2.1.0 for the changed System.Events ABI and the new versioned host function, and add a release-workflow guard that fails when runtime-ABI source changed since the last runtime tag but spec_version did not increase. - Add an event-volume weight term to both deposit paths: a per-event weight in cnight-observation and a pre-dispatch allowance refined post-dispatch in midnight-system. Per-event figures are conservative placeholders pending the benchmark run. - Regenerate the static runtime metadata for the new event variants. The per-event weight benchmark, the failure-branch bridge test, clippy, and the full test suite are run by the reviewer/author; compilation was verified via a full node release build. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mike Clay <mike.clay@shielded.io>

Address the eight findings from the runtime review of PR #1849, plus the metadata regeneration the review flagged as a release follow-up. - Emit the ledger transaction hash (not the state root) in the deposited system-transaction event on both the governance dispatchable and the internal executor, so downstream consumers can correlate the system event with its ledger events by hash. - Version the ledger host functions for the new return shape: the apply structs revert to their events-free (pre-PR, byte-identical) form and gain WithEvents siblings carrying the trailing events field on a new host-function version. WASM always calls the latest version its imports declare, so there is no negotiate-down: a new runtime on an old binary fails at WASM instantiation on the missing import rather than silently truncating a decode. The versioned split guards the reverse direction (an old events-free runtime on a new binary decodes the events-free shape); the pre-activation validator-binary gate is what keeps the mixed-version upgrade window safe by enforcing binaries-first rollout. - Re-anchor the event-volume benchmark to the real 50 MB bytesChurned ceiling instead of the earlier ~1 MiB bound. - Consume pending cross-chain accounting state only on system-transaction success: the subminimal-transfer accumulator is retained and the approved-hash entry re-inserted when execution fails, preserving single-use protection on success. - Add a pre-activation validator-binary compatibility gate to the runtime upgrade flow, refusing to authorize when the node's active runtime spec version is below the required minimum, with an explicit opt-out. - Apply the advertised include/exclude service filters during image rollout. - Bump the runtime spec_version to 2.1.0 for the changed System.Events ABI and the new versioned host function, and add a release-workflow guard that fails when runtime-ABI source changed since the last runtime tag but spec_version did not increase. - Add an event-volume weight term to both deposit paths: a per-event weight in cnight-observation and a pre-dispatch allowance refined post-dispatch in midnight-system. Per-event figures are conservative placeholders pending the benchmark run. - Regenerate the static runtime metadata for the new event variants. The per-event weight benchmark, the failure-branch bridge test, clippy, and the full test suite are run by the reviewer/author; compilation was verified via a full node release build. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mike Clay <mike.clay@shielded.io>

Reconcile the runtime version to 2.2.0: main (pallet-babe, session-keys migration) and this branch (expose ledger events) had each independently bumped to 2.1.0, so the combined runtime takes the next distinct identity (spec_version 002_002_000; node and metadata crates 2.2.0). Regenerate the static runtime metadata (midnight_metadata_2.2.0.scale) for the merged ABI and wire the V2_2_0 runtime version into the metadata crate and toolkit fetcher. Merged tree verified: full node release build and clippy/fmt across the workspace both pass. Signed-off-by: Mike Clay <mike.clay@shielded.io>

Sync with the latest main (pallet_authorship, serde_with bump, reserve-contracts pin). Regenerate the 2.2.0 runtime metadata for the merged ABI. Runtime stays at spec_version 002_002_000 — main did not bump its version for pallet_authorship, so this branch keeps its distinct 2.2.0 identity above main's 2.1.0. Merged tree compiles (full node release build). Signed-off-by: Mike Clay <mike.clay@shielded.io>

…at/1474-ledger-events-v2 Signed-off-by: Mike Clay <mike.clay@shielded.io>

…2.2.0 Satisfies the docs-link test (docs/ledger-events.md was unlinked in the root README) and the static openrpc sync test (docs/openrpc.json version lagged the 2.2.0 runtime bump; no RPC method surface change). Signed-off-by: Mike Clay <mike.clay@shielded.io>

…f envelope RB8's pre-dispatch event allowance for send_mn_system_transaction was anchored to the per-block 50 MB churn ceiling (~12k events), giving the call a ~50 MB proof-size weight that no governance motion can bound. The update-ledger-parameters E2E failed with FederatedAuthority::MotionWeightBoundTooLow. Cap the per-call worst case at 200 events so the pre-dispatch weight fits the ~1 MB motion proof envelope; the post-dispatch actual weight still charges the real event count. Signed-off-by: Mike Clay <mike.clay@shielded.io>

chore(1474): initialize work package branch

7bad928

Scaffold commit to open the draft PR for midnight-node#1474. Implementation follows per the work package plan. Signed-off-by: Mike Clay <mike.clay@shielded.io>

m2ux self-assigned this Jul 8, 2026

m2ux linked an issue Jul 8, 2026 that may be closed by this pull request

Node should expose events emitted by the Ledger #1474

Open

m2ux and others added 11 commits July 8, 2026 12:17

This comment has been minimized.

Sign in to view

m2ux added the 2026-q3-sow? label Jul 10, 2026

Merge branch 'main' into feat/1474-ledger-events-v2

ae03405

m2ux marked this pull request as ready for review July 13, 2026 10:46

m2ux requested a review from a team as a code owner July 13, 2026 10:46

This comment was marked as outdated.

Sign in to view

jina-simulation Bot reviewed Jul 14, 2026

View reviewed changes

Comment thread pallets/midnight-system/src/lib.rs

Comment thread ledger/src/common/types.rs Outdated

Comment thread pallets/midnight/src/benchmarking.rs Outdated

midnightntwrk deleted a comment from jina-simulation Bot Jul 15, 2026

m2ux force-pushed the feat/1474-ledger-events-v2 branch from ece99e7 to 0ea9a90 Compare July 17, 2026 05:39