Skip to content

Document local Arbitrum replay for testing handlers the dev chain can't exercise #254

Description

@rickstaa

Problem

There is no documented way to test a handler for an event the local dev chain cannot emit.

README.md covers only the dev-chain flow — docker compose up, yarn create:local,
yarn deploy:local — which runs against livepeer/geth-with-livepeer-protocol:streamflow. That
image is pinned well before recent protocol upgrades, so any new event simply never fires. Most of
test/test.ts is it.skip'd on top of that, so the suite is a no-regression check rather than a
way to validate new indexing logic.

The only practical alternative is to replay against Arbitrum One from a start block just before a
known real event. That works well — it validated #253 in seconds rather than the ~12h a full resync
takes — but it is written down nowhere, and several of its failure modes are silent.

Hit concretely while testing #253 (reward caller delegation).

What tripped us up

All verified while getting the replay working:

  1. yarn deploy:local silently clobbers the manifest. It runs yarn prepare:development first,
    regenerating subgraph.yaml as network: development. Deploying that to a node configured for
    Arbitrum fails with network not supported by registrar: no network development found on chain ethereum, which does not point at the cause. The fix is npx graph deploygraph is not
    global, so it needs npx.

  2. The dev and Arbitrum stacks collide on ports 8000/8020/8030. The second up -d appears to
    succeed while the first stack still holds the ports. Needs an explicit docker compose down
    first.

  3. Env var naming does not line up, and fails silently. .env.template documents RPC_URL and
    GRAPH_NETWORK. Neither is read by any compose file: the dev docker-compose.yml hardcodes
    ethereum: "development:http://geth:8545", and the Arbitrum one reads ARBITRUM_RPC_URL. With
    nothing set, interpolation yields ethereum: 'arbitrum-one:' — an empty URL. The stack starts,
    reports health: healthy, and indexes nothing. Worth collapsing to one name and using
    ${RPC_URL:?...} so this errors upfront instead of looking like a slow sync.

  4. A narrow start block produces wrong round numbers. Starting mid-history creates a fresh
    Protocol with a zeroed lastRoundLengthUpdateStartBlock, so createOrLoadRound degenerates to
    blockNum / roundLength. Observed round 4021 where the contract reported 4285. This is an
    expected artifact, not a bug — but it looks exactly like one, so it has to be signposted. Round
    correctness has to be verified by a full sync or a graft.

  5. A true archive RPC is required. Handlers eth_call at the historical block. A non-archive
    endpoint fails with missing trie node ... state is not available and graph-node retries
    forever — stuck, while still reporting health: healthy.

Proposal

Document the Arbitrum replay flow, covering both use cases:

  • reproducing an indexing crash at a known bad block, and
  • verifying a new handler against known-good on-chain events — currently undocumented, and the
    only option whenever the pinned geth image predates the protocol change being indexed.

Content should include the working command sequence, the traps above with their symptoms (each one
presents as something other than its cause), how to read fatalError from
http://localhost:8030/graphql, and when to reach for grafting instead because the bug depends on
accumulated state.

Also worth committing the Arbitrum compose stack itself, since the flow depends on it and it is
currently local-only.

Note

A local, uncommitted docs/debugging.md already covers much of this — including the round-shift
artifact and the yarn deploy:local clobber — and is a good starting point. Its setup steps are
stale though: they predate the Arbitrum compose file, so they point at docker compose up -d (the
dev stack, which cannot index Arbitrum) and at env vars nothing consumes. Committing it as-is would
ship a recipe that does not work.

Related: #252 (ABI sync), #253 (the PR this surfaced from).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions