refactor: add in-memory chainstate backends to speed up tests

[cylewitruk-stacks]

Description

Refactor chainstate persistence behind pluggable storage backends and migrate the Stackslib test suite to in-memory storage where disk persistence is not under test.

This introduces:

• ChainStatePersistence and composable backends for the chainstate index, Clarity state, block store, and Nakamoto staging blocks.
• Disk-backed, isolated in-memory, and shared in-memory chainstate implementations.
• In-memory SQLite support for the burnchain, MARF, Clarity, sortition, and staging-block databases.
• Snapshot/fork support for shared ephemeral chainstates.
• Generic chainstate plumbing through the coordinator, miner, networking, RPC, signer, and node layers.
• Updated chainstate, networking, RPC, PoX, and block-construction tests using the new ephemeral backends.
• Fixes for test isolation and flakes exposed by running the suite concurrently.

Production nodes continue to use the disk-backed implementation by default. The in-memory implementations are restricted to tests and builds using the testing feature.

On a local M4 Mac, the full workspace test suite now completes in approximately 4m18s, down from roughly 15–18 minutes.

Applicable issues

• N/A

Additional info (benefits, drawbacks, caveats)

Benefits:

• Removes most filesystem I/O from Stackslib tests.
• Reduces full-suite wall-clock time by roughly 70–76%.
• Preserves tests that explicitly exercise disk-backed behavior.
• Makes storage selection explicit and allows future backends to be composed from individual persistence components.

Caveats:

• This is a broad internal type-system change because chainstate consumers now carry the selected persistence backend.
• Test namespaces must remain unique when using shared in-memory SQLite databases.

Checklist

• Test coverage for new or modified code paths
• For new Clarity features or consensus changes, add property tests (see
  docs/property-testing.md)
• Changelog fragment(s) or "no changelog" label added (see
  changelog.d/README.md). If this PR breaks
  anything for node operators or users, or requires them to manually do
  anything (such as adjust a setting), use the breaking category.
• Required documentation changes (e.g.,
  rpc/openapi.yaml for RPC endpoints,
  event-dispatcher.md for new events)
• New clarity functions have corresponding PR in clarity-benchmarking repo

[hstove-stacks]

@cylewitruk-stacks can you add a fix for event_dispatcher::tests::test_http_delivery_blocks_once_queue_is_full (failing in CI)? Thanks!

[cylewitruk-stacks]

@cylewitruk-stacks can you add a fix for event_dispatcher::tests::test_http_delivery_blocks_once_queue_is_full (failing in CI)? Thanks!

@hstove-stacks Yup, the bots had already tackled that failure with the following explanation:

---

The test was rewritten to remove fixed sleeps and wall-clock assumptions:

• The mock HTTP server now waits on explicit channel signals.
• The first three dispatches run in a guarded thread and must enqueue within 5 seconds.
• The fourth dispatch must remain blocked while the queue is full.
• Releasing the first HTTP request must unblock the fourth dispatch.
• Remaining requests are released individually before draining the worker.
• The observer timeout is now longer than the test guard, ensuring failures surface through the test’s own diagnostic instead of HTTP retries.

This makes the queue behavior deterministic and avoids failures caused by slower CI scheduling.

---

Which I just pushed in b52a704

[coveralls]

Coverage Report for CI Build 32702726478

Coverage increased (+1.6%) to 88.213%

Details

• Coverage increased (+1.6%) from the base build.
• Patch coverage: Could not be determined — this PR's diff is too large for GitHub to return (406 error at GitHub).
• 7972 coverage regressions across 121 files.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

7972 previously-covered lines in 121 files lost coverage.

Top 10 Files by Coverage Loss	Lines Losing Coverage	Coverage
stackslib/src/chainstate/stacks/db/blocks.rs	726	90.27%
stackslib/src/net/p2p.rs	693	76.98%
stackslib/src/chainstate/burn/db/sortdb.rs	479	91.31%
stacks-node/src/neon_node.rs	388	83.13%
stackslib/src/chainstate/nakamoto/mod.rs	355	84.77%
stackslib/src/net/download/epoch2x.rs	347	60.76%
stackslib/src/net/relay.rs	335	74.82%
stackslib/src/burnchains/burnchain.rs	246	72.52%
stackslib/src/chainstate/stacks/miner.rs	221	84.54%
stackslib/src/clarity_vm/database/marf.rs	220	65.32%

---

Coverage Stats

Relevant Lines:	230851
Covered Lines:	203640
Line Coverage:	88.21%
Coverage Strength:	19937036.33 hits per line

---

💛 - Coveralls