Problem
There is no documented way to test a handler for an event the local dev chain cannot emit.
README.md covers only the dev-chain flow — docker compose up, yarn create:local,
yarn deploy:local — which runs against livepeer/geth-with-livepeer-protocol:streamflow. That
image is pinned well before recent protocol upgrades, so any new event simply never fires. Most of
test/test.ts is it.skip'd on top of that, so the suite is a no-regression check rather than a
way to validate new indexing logic.
The only practical alternative is to replay against Arbitrum One from a start block just before a
known real event. That works well — it validated #253 in seconds rather than the ~12h a full resync
takes — but it is written down nowhere, and several of its failure modes are silent.
Hit concretely while testing #253 (reward caller delegation).
What tripped us up
All verified while getting the replay working:
-
yarn deploy:local silently clobbers the manifest. It runs yarn prepare:development first,
regenerating subgraph.yaml as network: development. Deploying that to a node configured for
Arbitrum fails with network not supported by registrar: no network development found on chain ethereum, which does not point at the cause. The fix is npx graph deploy — graph is not
global, so it needs npx.
-
The dev and Arbitrum stacks collide on ports 8000/8020/8030. The second up -d appears to
succeed while the first stack still holds the ports. Needs an explicit docker compose down
first.
-
Env var naming does not line up, and fails silently. .env.template documents RPC_URL and
GRAPH_NETWORK. Neither is read by any compose file: the dev docker-compose.yml hardcodes
ethereum: "development:http://geth:8545", and the Arbitrum one reads ARBITRUM_RPC_URL. With
nothing set, interpolation yields ethereum: 'arbitrum-one:' — an empty URL. The stack starts,
reports health: healthy, and indexes nothing. Worth collapsing to one name and using
${RPC_URL:?...} so this errors upfront instead of looking like a slow sync.
-
A narrow start block produces wrong round numbers. Starting mid-history creates a fresh
Protocol with a zeroed lastRoundLengthUpdateStartBlock, so createOrLoadRound degenerates to
blockNum / roundLength. Observed round 4021 where the contract reported 4285. This is an
expected artifact, not a bug — but it looks exactly like one, so it has to be signposted. Round
correctness has to be verified by a full sync or a graft.
-
A true archive RPC is required. Handlers eth_call at the historical block. A non-archive
endpoint fails with missing trie node ... state is not available and graph-node retries
forever — stuck, while still reporting health: healthy.
Proposal
Document the Arbitrum replay flow, covering both use cases:
- reproducing an indexing crash at a known bad block, and
- verifying a new handler against known-good on-chain events — currently undocumented, and the
only option whenever the pinned geth image predates the protocol change being indexed.
Content should include the working command sequence, the traps above with their symptoms (each one
presents as something other than its cause), how to read fatalError from
http://localhost:8030/graphql, and when to reach for grafting instead because the bug depends on
accumulated state.
Also worth committing the Arbitrum compose stack itself, since the flow depends on it and it is
currently local-only.
Note
A local, uncommitted docs/debugging.md already covers much of this — including the round-shift
artifact and the yarn deploy:local clobber — and is a good starting point. Its setup steps are
stale though: they predate the Arbitrum compose file, so they point at docker compose up -d (the
dev stack, which cannot index Arbitrum) and at env vars nothing consumes. Committing it as-is would
ship a recipe that does not work.
Related: #252 (ABI sync), #253 (the PR this surfaced from).
Problem
There is no documented way to test a handler for an event the local dev chain cannot emit.
README.mdcovers only the dev-chain flow —docker compose up,yarn create:local,yarn deploy:local— which runs againstlivepeer/geth-with-livepeer-protocol:streamflow. Thatimage is pinned well before recent protocol upgrades, so any new event simply never fires. Most of
test/test.tsisit.skip'd on top of that, so the suite is a no-regression check rather than away to validate new indexing logic.
The only practical alternative is to replay against Arbitrum One from a start block just before a
known real event. That works well — it validated #253 in seconds rather than the ~12h a full resync
takes — but it is written down nowhere, and several of its failure modes are silent.
Hit concretely while testing #253 (reward caller delegation).
What tripped us up
All verified while getting the replay working:
yarn deploy:localsilently clobbers the manifest. It runsyarn prepare:developmentfirst,regenerating
subgraph.yamlasnetwork: development. Deploying that to a node configured forArbitrum fails with
network not supported by registrar: no network development found on chain ethereum, which does not point at the cause. The fix isnpx graph deploy—graphis notglobal, so it needs
npx.The dev and Arbitrum stacks collide on ports 8000/8020/8030. The second
up -dappears tosucceed while the first stack still holds the ports. Needs an explicit
docker compose downfirst.
Env var naming does not line up, and fails silently.
.env.templatedocumentsRPC_URLandGRAPH_NETWORK. Neither is read by any compose file: the devdocker-compose.ymlhardcodesethereum: "development:http://geth:8545", and the Arbitrum one readsARBITRUM_RPC_URL. Withnothing set, interpolation yields
ethereum: 'arbitrum-one:'— an empty URL. The stack starts,reports
health: healthy, and indexes nothing. Worth collapsing to one name and using${RPC_URL:?...}so this errors upfront instead of looking like a slow sync.A narrow start block produces wrong round numbers. Starting mid-history creates a fresh
Protocolwith a zeroedlastRoundLengthUpdateStartBlock, socreateOrLoadRounddegenerates toblockNum / roundLength. Observed round4021where the contract reported4285. This is anexpected artifact, not a bug — but it looks exactly like one, so it has to be signposted. Round
correctness has to be verified by a full sync or a graft.
A true archive RPC is required. Handlers
eth_callat the historical block. A non-archiveendpoint fails with
missing trie node ... state is not availableand graph-node retriesforever — stuck, while still reporting
health: healthy.Proposal
Document the Arbitrum replay flow, covering both use cases:
only option whenever the pinned geth image predates the protocol change being indexed.
Content should include the working command sequence, the traps above with their symptoms (each one
presents as something other than its cause), how to read
fatalErrorfromhttp://localhost:8030/graphql, and when to reach for grafting instead because the bug depends onaccumulated state.
Also worth committing the Arbitrum compose stack itself, since the flow depends on it and it is
currently local-only.
Note
A local, uncommitted
docs/debugging.mdalready covers much of this — including the round-shiftartifact and the
yarn deploy:localclobber — and is a good starting point. Its setup steps arestale though: they predate the Arbitrum compose file, so they point at
docker compose up -d(thedev stack, which cannot index Arbitrum) and at env vars nothing consumes. Committing it as-is would
ship a recipe that does not work.
Related: #252 (ABI sync), #253 (the PR this surfaced from).