feat: add modular authentication and single-DAG Starfish-RBC

[NaitsabesMue]

Summary

This PR now contains only three related pieces:

• modular block authentication with Ed25519, ML-DSA-44, ML-DSA-65, and experimental pairwise-MAC modes;
• the direct Starfish-RBC kernel/service for canonical header reliable broadcast; and
• starfish-rbc-single-dag, which embeds typed ECHO/READY references in ordinary Starfish blocks so consensus and RBC use one block identity and one DAG.

The superseded standalone carrier/shadow RBC-DAG has been removed completely, including its second clock, reducer/projection planes, WAL/receipt recovery, synchronization transport, vote-QC experiment, metrics, tests, and design document. This reduces the PR from 53 files and 57,396 added lines to 35 files and 12,688 added lines.

Signature-free single-DAG lower bound

Finite benchmarks can enable --starfish-rbc-single-dag-echo-qc-fast-path. A receiver then delivers an exact header after locally authenticating quorum ordinary DAG blocks carrying ECHO, while still emitting the normal READY reference. It adds no public signature, portable QC object, full MAC witness vector, standalone phase message, or extra fast-path communication.

This flag has an explicit Byzantine-totality caveat. Pairwise-MAC testimony is receiver-local: quorum intersection and per-sender slot locks preserve integrity and uniqueness, but Byzantine ECHO senders may selectively reveal testimony so one honest receiver obtains quorum while another never does. The flag is therefore restricted to bounded testbed runs and must not be presented as production-safe RBC. The default V3 path continues to require quorum READY.

Two attempted portable-certificate variants were removed: aggregate BLS evidence violated the signature-free requirement, while complete per-recipient MAC witness vectors grew with the committee and overloaded the 40-validator local testbed.

Why the retained files are needed

• Authentication, committee/configuration, block encoding, relay, and storage changes implement content-stable block references with independently selectable authentication.
• The direct RBC kernel/service and network messages provide authenticated INIT/ECHO/READY, exact header/payload recovery, sender locks, and clean-DAG admission.
• Core, DagState, committer, linearizer, and synchronizer changes keep dirty blocks available for RBC progress while excluding them from consensus evidence and committed output.
• CLI/orchestrator/local-dryrun changes make the authentication and single-DAG modes runnable; per-message byte metrics support the bandwidth comparisons.
• The two protocol documents state the testbed boundaries and the receiver-local totality limitation.

Validation

• starfish_rbc::tests: 46/46 passed.
• starfish_rbc_service::tests: 9/9 passed.
• Single-DAG DagState plus validator commit/synchronization: 3/3 passed.
• Broad starfish-core library suite excluding the separately exercised validator smoke matrix: 250/250 passed.
• starfish CLI tests: 4/4 passed.
• Remote-genesis protocol-instance test passed for direct and single-DAG Starfish-RBC.
• Strict Clippy passed for all targets in starfish-core, starfish, and orchestrator with -D warnings.
• Release build, formatting, and diff checks passed.
• A four-validator zero-latency receiver-local single-DAG smoke run completed cleanly after the cleanup.

Matched single-DAG measurements

The retained implementation previously produced the following matched local results at 1,000 offered tx/s for 20 seconds with MAC authentication and the fixed 50 ms V3 round limiter. Every run eventually committed exactly 20,000 submitted transactions. These are single-machine testbed measurements, not production or statistical claims.

Validators	Network	p50 E2E	Cutoff TPS	Eventual TPS	Outbound/node
10	0 ms	400.5 ms	982.5	1,000	0.79 MB/s
10	AWS RTT table	956.0 ms	956.3	1,000	0.61 MB/s
40	0 ms	390.2 ms	976.8	1,000	5.88 MB/s
40	AWS RTT table	968.2 ms	936.6	1,000	3.07 MB/s

The receiver-local lower bound meets the preliminary 1,000 ms p50 objective at both 10 and 40 validators, subject to the totality caveat above.

Remaining limitations

This is a research testbed. Production use still needs a complete asynchronous proof, durable restart/replay for embedded phase locks, bounded retirement and state transfer, broader Byzantine testing, peer-identity hardening, and resource/DoS analysis. The exact protocol and caveat are documented in docs/starfish-rbc-single-dag-v3.md.

[github-actions]

Package	Line Rate	Health
crates/orchestrator/src	24%	❌
crates/orchestrator/src/protocol	28%	❌
crates/starfish-core/src/consensus	70%	➖
crates/starfish/src	3%	❌
crates/starfish-core/src/starfish_rbc_dag	89%	✔
crates/orchestrator/src/client	7%	❌
crates/starfish-core/src/runtime	23%	❌
crates/starfish-core/src	58%	➖
crates/starfish-core/src/core_thread	21%	❌
Summary	50% (11682 / 23205)	➖

[polinikita]

I don't understand why many unrelated parts were modified

[NaitsabesMue]

Thanks, made changes, now hopefully removed all unnecessary changes.

[NaitsabesMue]

Also

• Removed the committed benchmark-result reports; results can instead be shown as a graph or short summary in the PR.
• Decoupled digital-signature selection from the consensus protocol. block_authentication now selects Ed25519 (default), ML-DSA-44, or ML-DSA-65 for every protocol.
• Removed the separate *-ml-dsa-* protocol variants.
• Kept the MAC variants separate and clearly marked them as experimental lower-bound benchmarks. MAC vectors are not a drop-in signature replacement and currently provide no protocol safety/liveness guarantee without an additional quorum-authentication/RBC mechanism.
• Removed unrelated changes and kept the modular block-authentication plumbing needed for these experiments.

[polinikita]

I would not add the experiment results as they will need to be reevaluated after every small modification of a protocol, which sometimes happens (e.g. starfish-speed is very sensitive to the soft_timeout constant). In addition, I would not make standalone protocols with just different identification fields. So, my recommendation:

• remove benchmark results as a text - a graph or short summary inside PR would fit better
• add a configurable parameter for selecting an authentication method that works with every protocol in the repo. By default, the current Ed25519 scheme. This allows for more consistent changes across the protocol.
• make the pr more focused without touching too many components where it is not necessary.

[NaitsabesMue]

Latency checkpoint pushed in 27bb05f.

This simplifies and accelerates the existing no-extra-communication carrier path:

• validates/decodes each locally broadcast carrier once, then reuses its exact reference across per-peer mailbox admission;
• caches RBC phase stake totals instead of recomputing sender-set stake on every transition;
• counts only the reducer's bounded future window for the buffered-carrier metric instead of rescanning authenticated history;
• piggybacks pending phases on the producer's imminent application carrier, with a bounded committee-scaled deadline (100 ms at n=10; 25 ms at n=40);
• scales the still-bounded application callback/cache window with committee fan-in (64 at n=10; 120 at n=40, global cap 128).

Fresh matched 20 s / 1,000 offered tx/s results:

Committee / network	p50 E2E	Pipeline frontier applied	Verdict
n=10 / 0 ms	339.8 ms	326.8 ms	VALID, skew 1
n=10 / AWS RTT	993.0 ms	946.9 ms	VALID, skew 0
n=40 / 0 ms	1527.7 ms	1409.2 ms	VALID, skew 2
n=40 / AWS RTT	3169.3 ms	2818.5 ms	VALID, skew 1

The n=10 AWS run reaches the preliminary sub-1 s objective without additional protocol communication, synchronization, relaxed thresholds, or extra parents. The n=40 result remains dominated by committee-scale RBC phase/projection work rather than Core application.

Validation: 54 service tests, 30 shadow/recovery tests, 44 RBC model tests, 15 bridge/mailbox tests, strict Clippy -D warnings, formatting/diff checks, and core/binary checks all pass.

Correction to the earlier PR description: the application callback/cache bound is no longer fixed at 64 for n=40; it is committee-scaled and remains explicitly capped at 128.