Add explorer NATS publishing and backfill service#404
Open
Hebilicious wants to merge 54 commits into
Open
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…-indexing-impl # Conflicts: # src/app/zeko/sequencer/run.ml
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Contributor
Author
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a NATS-based publishing infrastructure for the Zeko sequencer to support explorer and indexing systems. Key additions include a standalone explorer backfill service for republishing historical data, a shared NATS event module, and updated sequencer hooks for emitting transaction, finality, and health events. The PR also includes comprehensive Gherkin-based integration and end-to-end tests. Feedback identifies a critical compilation issue regarding the use of Uuid_unix without proper library dependencies, a recommendation to use safer TCP connection handling in health checks to prevent resource leaks, and a concern about the lack of concurrency limits for background backfill jobs which could lead to resource exhaustion.
There was a problem hiding this comment.
Pull request overview
Adds an explorer-facing NATS/JetStream publishing path to the Zeko sequencer (gated behind --nats-url), plus a standalone backfill service that can replay historical DA diffs into the same NATS subjects. The PR also updates the OCaml dependency set for yojson 2.x and GraphQL 0.14 compatibility, and introduces dedicated CI workflows and Gherkin-based acceptance coverage for the new explorer surface.
Changes:
- Introduce shared explorer event helpers (
Explorer_events) and wire sequencer publishing (transactions/finality/health) behind--nats-url. - Add a standalone backfill service with GraphQL query/mutation + GraphQL-SSE subscriptions, plus integration/E2E Gherkin coverage.
- Upgrade/adjust GraphQL wrapper + Yojson usage to support new dependency versions and CI setup changes.
Reviewed changes
Copilot reviewed 44 out of 45 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/lib/mina_graphql/types.ml | Adds GraphQL default-value support hooks; updates Rosetta scalar/default JSON handling. |
| src/lib/logproc_lib/interpolator.ml | Updates Yojson pretty/compact formatting for Yojson 2.x API changes. |
| src/lib/graphql_wrapper/ocaml_graphql_server_tests/test_schema.ml | Adjusts test schema object field construction for GraphQL 0.14 API. |
| src/lib/graphql_wrapper/ocaml_graphql_server_tests/error_test.ml | Adjusts nullable/error tests for GraphQL 0.14 API. |
| src/lib/graphql_wrapper/ocaml_graphql_server_tests/abstract_test.ml | Adjusts abstract/interface tests for GraphQL 0.14 API. |
| src/lib/graphql_wrapper/graphql_wrapper.ml | Extends Arg wrapper to support default values by converting defaults to GraphQL const values. |
| src/lib/graphql_lib/consensus/graphql_scalars.ml | Updates schema object construction style for GraphQL 0.14 API. |
| src/lib/fields_derivers_graphql/fields_derivers_graphql.ml | Updates derived schema object creation to match GraphQL 0.14 API expectations. |
| src/lib/cli_lib/commands.ml | Switches JSON input parsing to Yojson.Safe.seq_from_channel and improves empty-input handling. |
| src/app/zeko/sequencer/tests/testing_ledger/gql.ml | Updates schema object construction style for GraphQL 0.14 API. |
| src/app/zeko/sequencer/tests/test_spec.ml | Threads optional nats_url into sequencer test spec generator. |
| src/app/zeko/sequencer/tests/sequencer_test.ml | Ensures tests call Sequencer.create with ?nats_url explicitly. |
| src/app/zeko/sequencer/tests/run-sequencer-test.sh | Adds explorer-e2e mode, pins RabbitMQ image, and waits for queue consumers. |
| src/app/zeko/sequencer/run.ml | Adds --nats-url CLI option and passes it into sequencer runtime creation. |
| src/app/zeko/sequencer/README.md | Documents NATS publishing subjects and the new backfill service/test entry points. |
| src/app/zeko/sequencer/lib/zeko_sequencer.ml | Wires optional NATS client/sink into sequencer lifecycle; publishes transaction/finality/health events. |
| src/app/zeko/sequencer/lib/gql.ml | Updates schema object construction style for GraphQL 0.14 API. |
| src/app/zeko/sequencer/lib/dune | Adds explorer_events and NATS client libraries to sequencer lib deps. |
| src/app/zeko/sequencer/explorer/tests/README.md | Documents explorer Gherkin integration + broker-backed + E2E suites. |
| src/app/zeko/sequencer/explorer/tests/features/sequencer-hooks.feature | Adds acceptance scenarios for replay genesis classification. |
| src/app/zeko/sequencer/explorer/tests/features/nats-integration.feature | Adds acceptance scenarios for NATS round-trips and JetStream dedupe. |
| src/app/zeko/sequencer/explorer/tests/features/event-contract.feature | Adds acceptance scenarios for event payload/header contracts and error behavior. |
| src/app/zeko/sequencer/explorer/tests/features/e2e.feature | Adds end-to-end scenarios asserting sequencer publish + backfill replay to NATS. |
| src/app/zeko/sequencer/explorer/tests/features/backfill-api.feature | Adds acceptance scenarios for backfill GraphQL API and SSE progress streams. |
| src/app/zeko/sequencer/explorer/tests/feature_parser.ml | Adds helper to load/validate Gherkin scenario inventory from feature files. |
| src/app/zeko/sequencer/explorer/tests/explorer_nats_gherkin_tests.ml | Adds real-NATS integration harness (publish/subscribe + JetStream dedupe assertion). |
| src/app/zeko/sequencer/explorer/tests/explorer_gherkin_tests.ml | Adds in-process harness for payload contract + GraphQL/SSE behavior. |
| src/app/zeko/sequencer/explorer/tests/explorer_e2e_gherkin_tests.ml | Adds slow E2E harness that boots sequencer + backfill against real services and NATS. |
| src/app/zeko/sequencer/explorer/tests/dune | Adds new test libraries/executables and wires NATS scenarios into runtest. |
| src/app/zeko/sequencer/explorer/MESSAGE_CONTRACT.md | Documents the JetStream stream + subject + payload/header contract. |
| src/app/zeko/sequencer/explorer/explorer-indexing-plan.md | Adds design/implementation plan for explorer publishing + backfill approach. |
| src/app/zeko/sequencer/explorer/explorer_events.ml | Implements shared subjects, payload encoding, JetStream setup, and publish helpers. |
| src/app/zeko/sequencer/explorer/explorer_backfill_sse.ml | Implements GraphQL-SSE transport for backfill subscription streaming. |
| src/app/zeko/sequencer/explorer/explorer_backfill_service.ml | Implements backfill job state machine, NATS replay publishing, and progress subscriber fanout. |
| src/app/zeko/sequencer/explorer/explorer_backfill_server.ml | Boots HTTP server with /graphql and /graphql/stream routing. |
| src/app/zeko/sequencer/explorer/explorer_backfill_graphql.ml | Defines GraphQL schema for backfill jobs, health, and progress subscriptions. |
| src/app/zeko/sequencer/explorer/dune | Adds explorer libraries and standalone backfill server executable. |
| src/app/zeko/da_layer/lib/core.ml | Minor comment whitespace/format adjustment. |
| src/app/cli/src/init/graphql_internal.ml | Switches websocket transport module to Graphql_websocket. |
| opam.export | Updates dependencies (GraphQL 0.14, Yojson 2.0.0, NATS clients, Merlin). |
| .prototools | Adds OCaml toolchain pinning via prototools. |
| .github/workflows/explorer-gherkin.yaml | Adds dedicated Explorer Gherkin workflow with NATS service container. |
| .github/workflows/explorer-e2e.yaml | Adds dedicated Explorer E2E workflow running the slower harness. |
| .github/workflows/ci.yaml | Refactors CI to use shared composite setup action and builds explorer target. |
| .github/actions/setup-zeko-ci/action.yaml | Introduces shared CI bootstrap action (opam/toolchain/caches). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds explorer/indexing support to the Zeko sequencer:
--nats-urlReviewer Guide
Most new implementation is isolated under
src/app/zeko/sequencer/explorer/.Important new files:
explorer_events.ml: shared NATS subjects, JetStream setup, payload encoding, and publish helpersexplorer_backfill_service.ml: in-memory backfill job state, DA diff replay, and NATS publish handlingexplorer_backfill_graphql.ml: GraphQL schema for backfill jobs and healthexplorer_backfill_sse.ml: GraphQL-SSE transport for backfill progressexplorer_backfill_server.ml: standalone HTTP server boot for/graphqland/graphql/streamMESSAGE_CONTRACT.md: explorer-facing message contract for consumerstests/features/*.feature: Gherkin scenario inventory for behavior introduced by this PRtests/run-explorer-e2e-test.sh: dedicated slow E2E harness for explorer scenarios that need the sequencer service stackExisting Files Changed And Why
Sequencer runtime integration:
src/app/zeko/sequencer/run.ml: adds the--nats-urlCLI option and passes it intoSequencer.create.src/app/zeko/sequencer/lib/dune: addsexplorer_events,nats-client, andnats-client-asyncdependencies needed by the sequencer library.src/app/zeko/sequencer/lib/zeko_sequencer.ml: wires the optional NATS client/sink into sequencer lifecycle and publishes transaction, finality, and health events at the existing DA enqueue, proof, commit, sync, and heartbeat points.src/app/zeko/sequencer/tests/test_spec.ml: threads optionalnats_urlthrough the existing test-spec helper so the E2E Gherkin harness can create a sequencer with NATS enabled.src/app/zeko/sequencer/tests/run-sequencer-test.sh: keeps its original CLI and sequencer-test role; remaining changes are limited to RabbitMQ version pinning, safer early cleanup, and waiting for prover queue consumers before running the existing sequencer tests.src/app/zeko/sequencer/README.md: adds a short pointer to--nats-url, the backfill service, and the dedicated explorer test README/contract docs.Dependency compatibility required by published NATS packages:
opam.export: adds publishednats-clientandnats-client-async; updates the dependency set toyojson.2.0.0and GraphQL 0.14-compatible packages required for that resolver result..prototools: pins the repo-local OCaml toolchain metadata to OCaml 4.14.0.src/lib/logproc_lib/interpolator.ml: adapts to the Yojson 2.xto_stringAPI.src/lib/cli_lib/commands.ml: replaces deprecated stream-based Yojson input withseq_from_channeland preserves empty-input handling.src/app/cli/src/init/graphql_internal.ml: uses the GraphQL 0.14 websocket module name.src/lib/graphql_wrapper/**,src/lib/mina_graphql/types.ml,src/lib/fields_derivers_graphql/**,src/lib/graphql_lib/consensus/graphql_scalars.ml,src/app/zeko/sequencer/lib/gql.ml, andsrc/app/zeko/sequencer/tests/testing_ledger/gql.ml: minimal GraphQL 0.14 API/default-value compatibility updates needed by the dependency change.CI and test wiring:
.github/actions/setup-zeko-ci/action.yaml: extracts the existing CI setup into a shared action so the new explorer workflows do not duplicate the full opam/bootstrap block..github/workflows/ci.yaml: uses the shared setup action and builds the explorer target with the existing Zeko packages..github/workflows/explorer-gherkin.yaml: fast Gherkin suite with a NATS service container..github/workflows/explorer-e2e.yaml: slower E2E Gherkin suite with a NATS service container, using the dedicated explorer E2E script.Test Coverage
event-contract.feature: transaction/finality/health payloads, dedup headers, and disabled publisher behaviornats-integration.feature: real NATS publish/subscribe and JetStream dedup behaviorbackfill-api.feature: genesis classification, invalid hash handling, job lookup, health, concurrency limit, and GraphQL-SSE progresssequencer-hooks.feature: sync replay genesis classificatione2e.feature: sequencer-applied transaction publishing and backfill replay from sequencer DA diffsValidation
Current-head workflows run after every push:
CI / Build & TestExplorer GherkinExplorer E2ELinear
Fixes ZK-509
Fixes ZK-510