feat: anchor RPC block queries by number and hash - #25204
Open
spalladino wants to merge 11 commits into
Open
Conversation
Behind a load balancer a client can sync to block N+1 through one node and then anchor follow-up queries against another node still at block N, which fails them immediately even though the block lands a moment later. The node now waits a bounded time for the anchor to arrive before answering: a block number exactly one ahead of the proposed tip waits up to RPC_UNSEEN_BLOCK_BY_NUMBER_WAIT_MS (default twice the block duration), an unknown block hash or archive root waits up to RPC_UNSEEN_BLOCK_BY_HASH_WAIT_MS (default 3s), and tags never wait. Setting either to 0 restores the previous fail-fast behavior. Outcomes are unchanged, only delayed: on a miss callers still throw or return undefined exactly as before. A cap of 100 simultaneous holds bounds resource use, beyond which misses fail fast. The world-state retry loop only holds off on its first attempt so a budget is never multiplied by the attempt count.
- Document that a wait budget is approximate: the poll loop sleeps a full interval before re-checking the deadline, so the actual wait can overshoot by up to one poll interval plus block-source read latency. - Replace wall-clock upper bounds with block-source call-count assertions wherever the real signal is "did not hold", and loosen the two upper bounds that are genuinely needed. Lower bounds are deterministic and stay. - Add config-mapping tests covering the unset by-number wait, the by-hash default, and an explicit 0. - Add elapsedMs to the structured context of the arrived / gave-up log lines.
A client can now name the block a query is anchored to by both its number and its hash. The hash still pins the fork, so resolution is unchanged on a hit; on a miss the number tells the node whether the anchor is the block right after its tip — a client that raced one block ahead through another node, worth waiting for — or a block it should already hold, which means the anchor is stale or reorged away and is failed immediately instead of after a blind hash wait. When the block does arrive its hash is compared against the anchor, so a real fork is reported as a miss rather than answered from the wrong chain. The combined form lives only at the RPC boundary: the hold-off reduces it to a single-selector query before anything reads the block source, since the archiver resolves `number` in preference to `hash` and would drop the fork pin. To keep that invariant every anchored read routes through the hold-off first, including the initial one, and unit tests assert the block source only ever receives queries naming a block one way. The wire schema also gains a catch-all object variant, so an object carrying a key this version does not know about has it stripped rather than being rejected, while objects naming two different blocks are still refused. Log queries are unified with the rest: `referenceBlock` widens from a block hash to any block parameter, the node resolves it (with the same hold-off) and rewrites the query with the concrete hash, and the log store's own in-transaction anchor check stays hash-based, rejecting the forms the node is expected to have resolved. PXE sends the anchored form wherever it anchors a query on a block header it holds.
The PXE read cache keyed an anchored entry by the block hash alone, so a later query naming the same hash at the wrong height would have been served that entry instead of reaching the node and being rejected as the bad request it is. An anchored entry now carries its height in the key. A logs anchor that names a block by number, tag, or archive root has no hash to hand the log store, so a miss on one is now reported by the node rather than delegated to a store error phrased for a caller that sent a hash. Anchors that do name a hash are still delegated untouched, leaving the store's in-transaction check authoritative. The reason given for that delegation was also wrong: the genesis anchor a client syncs from is resolvable through the block source, which returns synthetic metadata for it, so the rewrite covers genesis too. The utility oracle's anchor fast path knows the anchor's height, so its queries now name it: the callback takes the block parameter to query at, which is the anchored form at the anchor and a bare hash for the arbitrary historical blocks a contract asks about. Since the logs query schemas are shared with the archiver's API, a direct archiver client is schema-authorized to send anchors the log store rejects; the logs source interface now documents that only hash-bearing forms are served there.
spalladino
commented
Aug 12, 2026
Comment on lines
+388
to
+394
| const hash = blockParameterHash(referenceBlock); | ||
| if (hash === undefined) { | ||
| throw new Error( | ||
| `Log query referenceBlock ${inspectBlockParameter(referenceBlock)} does not name a block hash. Block numbers, ` + | ||
| `tags and archive roots are resolved to a hash by the node before the query reaches the log store.`, | ||
| ); | ||
| } |
Contributor
Author
There was a problem hiding this comment.
Can we solve this with narrower types rather than a throw?
spalladino
commented
Aug 12, 2026
| * syncs from before it has seen a block — raises the error it always did. Any other form has no hash to delegate, | ||
| * so the miss is reported here. | ||
| */ | ||
| async #resolveLogsReferenceBlock<T extends LogsQueryBase>(query: T): Promise<T> { |
Contributor
Author
There was a problem hiding this comment.
Can we move this method somewhere else, rather than top-level in the node server.ts? Maybe we need another module, like a NodeLogsProvider that handles this and the getters, similar to the NodeBlockProvider?
spalladino
commented
Aug 12, 2026
Comment on lines
+128
to
+133
| if (query.number !== tip + 1) { | ||
| this.log.verbose(`Not holding off query for unseen anchor block, its height is not next after the tip`, { | ||
| blockParameter, | ||
| tip, | ||
| }); | ||
| return undefined; |
Contributor
Author
There was a problem hiding this comment.
I'm thinking: let's say there has just been a prune with a new block added, and this node hasn't yet seen it, so the client queries for an old block number, but with a different hash. Shouldn't we wait for the 3s to see if the prune kicks in and we get this block?
- Add `getBlock` to `UnseenBlockHoldOff`, which polls the cheap block-data read and then reads the full block pinned by the resolved hash, and drop `NodeBlockProvider`'s private hold-off helper. - Add `getBlockNumber` to `UnseenBlockHoldOff` so `NodeWorldStateQueries` resolves a query to a block number through the hold-off instead of unwrapping block data itself. - Resolve the world-state query once before the sync-retry loop instead of threading a first-attempt flag into the resolution: retries re-resolve without holding off, so a resolution miss now propagates without further attempts and a client never waits more than one budget.
…ff anchored miss below tip - Type the logs-source getters with `ResolvedLogsQuery`, whose `referenceBlock` is a bare block hash, and drop the runtime throw the log store used to enforce that. The node rewrites a hash-bearing miss to its bare hash so the store's in-transaction check stays authoritative, and the wire schemas keep accepting every block parameter. - Move the logs getters and their anchor resolution out of `server.ts` into a `NodeLogsProvider` module, sharing the node's hold-off instance, with its tests in `node_logs_provider.test.ts`. - Wait on the hash budget for an anchored query whose hash is unknown at a height that is not next after the tip: a prune this node has not applied yet can put the client's block at a height it already holds, which is exactly the skew the hold-off exists to absorb.
- Resolve `getBlockHashMembershipWitness`'s reference block through `#resolveBlockNumberAndHash` like every other world-state query and drop `#resolveBlockNumber`, which leaves `UnseenBlockHoldOff.getBlockNumber` unused: removed. - Give `UnseenBlockHoldOff` a single private read-with-hold-off path parameterized by the read to perform, which `getBlockData` and `getBlock` fill in. A held query is now polled on the read the caller asked for, instead of polling metadata and then reading the block back by the resolved hash.
A client anchors on the genesis block before it has synced any block, as a PXE does for its first tagged-log queries. The block is synthetic, so a source that does not answer for it now never will and waiting only delays the answer by a whole by-hash budget.
- Probe and poll an anchored query on block metadata instead of on the read the caller asked for, and verify both the hash and the number it names before reading once. A held `getBlock` request no longer reconstructs the arriving block on every poll only to fail the anchor check, and a hash the block source resolves to another fork (its hash resolution is not transactional) is reported as a miss instead of being served. - Skip the hold-off for an anchored query naming the genesis block hash, as the normalized by-hash path does. - Report an unresolved anchored logs anchor from the node instead of handing its bare hash to the logs source, which resolves the hash alone and would serve logs past the height the client claimed. A genesis anchor is still handed on, since only the logs source can confirm a block that never lands in a store. - Narrow the archiver transport test's mock to the resolved logs query and have it assert the anchor arrives as a block hash.
A lower bound of the same duration as the arrival delay races the clock that delay is scheduled on, and was observed failing at 199.8ms against a 200ms bound. The extra block-source reads prove the wait just as well.
Same clock race as the normalized arrival tests: a lower bound of the same duration as the arrival delay races the clock that delay is scheduled on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of 2 of A-1688. Part 1 (#25203, v5 line) taught the node to briefly hold an RPC query whose anchor block it has not seen yet; this forward-ports that and lets a client name its anchor precisely, by both block number and hash.
Context
Behind a load balancer a client can sync to block N+1 through one node and then anchor follow-up queries — tx construction, note sync — against a different node that has only seen block N. Failing those queries aborts a whole client flow over a skew that resolves in a second or two.
Part 1 can only guess from a bare hash: a hash carries no height, so "one block ahead" and "reorged away" are indistinguishable and both wait out a short budget. Sending the number alongside the hash removes the guess.
Approach
BlockParametergains a{ number, hash }form. The hash still resolves the query, so the fork it pins is honored exactly as a bare hash is; the number only decides what a miss means.numberin preference tohash, so an anchored query reaching it would answer by height and drop the fork pin; the hold-off reduces the anchor to a single-selector query before anything reads the block source, and every anchored read — including the first, non-miss one — goes through it. Unit tests spy on the block source and assert it only ever receives queries naming a block one way.LogsQueryBase.referenceBlockwidens from a block hash to aBlockParameter, so note-sync queries take the same anchored form and get the same hold-off. The node resolves the anchor and rewrites the query with the concrete hash; the archiver's in-transaction check stays hash-based and authoritative.API changes
BlockParameteraccepts{ number, hash }on every endpoint that takes one (world-state witnesses,getPublicStorageAt,getContract,getBlock/getBlockData,findLeavesIndexes), andreferenceBlockon the logs-by-tags queries accepts anyBlockParameterrather than only a block hash.The v6 PXE emits the anchored form unconditionally. A v5 server rejects it with a 400, which is accepted: v6 clients are assumed in sync with v6 servers, and cross-major client/server pairings are out of scope.
Two env vars, carried over from #25203, bound the wait; setting either to 0 restores the previous fail-fast behavior:
RPC_UNSEEN_BLOCK_BY_NUMBER_WAIT_MS— max wait when the anchor is the block right after the node's tip. Defaults to twice the block duration.RPC_UNSEEN_BLOCK_BY_HASH_WAIT_MS— max wait when the anchor is an unknown block hash or archive root, which carries no height. Defaults to 3s.Fixes A-1721