exec: the shard execution layer, pinned to pyquarkchain at three levels - #48
Draft
syntrust wants to merge 24 commits into
Draft
exec: the shard execution layer, pinned to pyquarkchain at three levels#48syntrust wants to merge 24 commits into
syntrust wants to merge 24 commits into
Conversation
- Add TokenBalances type with sorted list encoding compatible with pyquarkchain - Add StateAccount.MntBalances field and QKC 6-element RLP codec (replaces generated gen_account_rlp.go with hand-written EncodeRLP/DecodeRLP) - Add uint32 RLP encoding helpers for token IDs - Update genesis hashes to reflect QKC 6-element account encoding - Add comprehensive tests: roundtrip, pyquarkchain encode/decode compatibility
Comment 1: SlimAccount only held {Nonce,Balance,Root,CodeHash}, so the
slim-RLP path used by SlimAccountRLP (stateupdate.go account updates/
origins), FullAccount (pathdb rollback in triedb/pathdb/execute.go) and
flatReader.Account (snapshot flat read) silently dropped MntBalances and
FullShardKey. A QKC account served from any of those paths came back with
MntBalances=nil / FullShardKey=0 and re-committed a corrupted account,
forking the trie root.
Extend SlimAccount with:
- MntBal []byte (rlp optional) = TokenBalances.SerializeToBytes()
- FullShardKey uint32 (rlp optional)
MntBal uses the []byte serialization (TokenBalances holds an unexported
map, not RLP-struct-encodable) and preserves the nil-vs-empty distinction
so the 0x80 / 0x8200c0 trie encoding stays byte-stable across the slim
round-trip. Unlike the trie qkcAccountRLP.TokenBal, the QKC default
balance is NOT merged into MntBal — slim keeps it in the Balance field.
Both fields are rlp optional so pre-MNT snapshots still decode. Because
FullAccount now reconstructs both fields, the pathdb rollback path is
covered without further changes.
Extend TestSlimRLPRoundTripEquivalence with fullShardKey / MNT-only /
MNT+QKC+shard cases (direct-QKC-encode == via-slim-encode).
Comment 3: remove the rlpgen go:generate directive. StateAccount now uses
the hand-written QKC codec (EncodeRLP/DecodeRLP in state_account_qkc.go);
regenerating gen_account_rlp.go would reintroduce a conflicting standard
4-field codec that drops MntBalances / FullShardKey. Replaced the
directive with a NOTE explaining why it must stay removed.
Comment 2 (empty() must consider MntBalances) is resolved downstream on
feature/mnt-state (stateObject.empty() checks IsBlankMnt(), covered by
TestEmptyAccountWithMntNotPruned); core/state is not part of this branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…reum genesis hashes
Fixes found while reviewing the MNT account encoding: - EncodeRLP no longer dereferences a nil *TokenBalances. When Balance is nil (or zero) and MntBalances is non-nil but empty, mergeQKCTokenBalances returns nil, and the old three-branch switch fell through to calling SerializeToBytes on that nil receiver. Collapsing the switch into a single nil-guarded path removes the whole class of gap. - DecodeRLP rejects a non-empty optional field instead of silently dropping it. pyquarkchain's _Account always writes b"" there, so a non-empty value could only come from a foreign encoder, and discarding it would change the bytes on re-encode. - Dropped the dead rlp:"optional" tag on StateAccount.MntBalances. The type has a hand-written codec so the tag never applied, and as written it was invalid (an optional field followed by the non-optional FullShardKey), which would break any future codec built for this struct. - Removed qkc/common/uint32_rlp.go: qkc/common/special_rlp.go now provides Uint32 after it moved down from qkc/types. Deliberately unchanged: the zero-valued-token-balance encoding is non-idempotent (first encode 0x00c0, re-encode empty) because pyquarkchain's TokenBalances.serialize tests len(_balances) before filtering zero balances. Canonicalizing it would fork the account trie root. Pinned by TestStateAccountEmptyBalancesPythonGolden. SlimAccountRLP keeps panicking on a serialization error, matching the surrounding geth convention. Adds TestStateAccountEncodeBalanceMntCombinations, covering Balance (nil/zero/non-zero) against MntBalances (nil/empty/zero-valued/non-zero) and pinning the wire TokenBal for each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gen_exec_golden.py drives pyquarkchain's own EvmState and exports what the Go execution layer has to reproduce: state-level cases (direct mutations -> post state root plus per-account reads) and message-level cases (apply_transaction / apply_xshard_deposit -> root, receipts, gas counters, produced deposits). The generator reads the singularity configs this repo ships rather than a pyquarkchain checkout's, so the vectors are bound to our own artifacts. Its first two cases are the two networks' genesis allocations and it refuses to write anything unless they reproduce the roots already pinned in minor_genesis_golden.json -- without that calibration a mismatch cannot be told apart from a case description that never reached EvmState. Block-level vectors need a root block body and follow separately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Go counterpart of quarkchain/evm/state.py over geth's trie and triedb: StateDB, stateObject and a journal. geth's core/state is not reused because balances are a table indexed by token id rather than a scalar, the address's full shard key lives in the leaf, existence and deletion follow QuarkChain's own is_blank rule, and the journal has to cover direct overwrites of the storage trie's root that geth has no entry for. Commit keeps the convention commitGenesisAlloc already proves: storage tries are flushed as roots in their own right, before the account trie that names them, and the account trie is committed with collectLeaf=false, because triedb/hashdb builds its references by decoding account leaves as geth's four-field StateAccount. The package doc records why that costs no consensus correctness and how to get reference counting back. Three behaviors are the opposite of what reading state.py suggests, and are implemented against the golden vectors rather than against the source: reverting reset_balances does not restore balances, because pyquarkchain journals the restore onto a misspelled attribute (state.py:195); a balance zeroed in place stays in the map and serializes to an empty pair list, a different leaf from an account that never held the token; and neither reset_balances nor reset_storage marks the account touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four fixes, all with a reproduction behind them. Account trie reads and writes went through geth's Must* accessors, which log a missing node and carry on returning nothing. A corrupt database therefore read as an absent account, the writes made against it were dropped, and Commit still reported success with the root unchanged -- exactly what the package doc promised could not happen. Fault injection reproduces it: delete the leaves under a committed root, and a 1000 balance reads as 0 with Error() nil. Both tries now use the plain trie with explicit key hashing, which is what pyquarkchain's SecureTrie does anyway, and every read and write reports its errors. TestCorruptTrieFailsLoudly covers it. State-level vectors ran their operations against an uncommitted allocation, so every allocated account was still touched when the operations began. That shape only occurs while genesis is being built and it changes the answers, since commit skips an account nothing touched. Allocations are now committed first, and each vector records the root they commit to. Note that this alone would have quietly disarmed revert_does_not_restore_reset_balances -- with an untouched account both a faithful and a naive implementation agree -- so that case now credits the account before its snapshot, and a mutation test confirms it still fails a naive journal. Message-level cases now declare whether they pin a success or a rejection. A blanket except turned any failure -- a drifted API, a broken constructor, a crash in the success path -- into a valid "rejected" vector that consumers would then assert forever. The vectors also record their oracle: the pyquarkchain commit and a digest of each module that decides execution. Generation refuses when one of those modules is uncommitted; --allow-dirty proceeds and names them in the output. The check is scoped to those modules on purpose, so an unrelated dirty script does not train the operator to pass the flag by habit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
syntrust
marked this pull request as ready for review
August 11, 2026 02:47
Carry QuarkChain multi-native-token (MNT) balances through the state layer so accounts can hold per-token balances alongside the native Ether balance. - state_object_qkc.go / statedb_qkc.go: MNT balance accessors on stateObject and StateDB (Get/Add/Sub/Set), guarding the per-account token count against the MNT limit. - journal.go: journal entries for MNT balance changes so they revert correctly with snapshots. - state_object.go: deep-copy MntBalances in newObject and on copy, so a mutation through one object cannot leak into another. - reader.go: decode MntBalances and FullShardKey out of the slim account served by the flat reader. - database_mpt.go / database_ubt.go: re-encode AccountsOrigin from slim-RLP to full QKC account RLP before handing it to pathdb, which needs it to match the trie leaf format for history verification. - pathdb/execute.go: decode AccountsOrigin as a full QKC StateAccount instead of slim-RLP, matching the invariant the commit paths now establish. mnt_test.go covers the accessors, journal revert, and copy isolation. The in-package test adaptations (state_test.go, statedb_fuzz_test.go) follow the account encoding change.
Carry QuarkChain multi-native-token (MNT) balances through the state layer so accounts can hold per-token balances alongside the native Ether balance. - state_object_qkc.go / statedb_qkc.go: MNT balance accessors on stateObject and StateDB (Get/Add/Sub/Set), guarding the per-account token count against the MNT limit. - journal.go: journal entries for MNT balance changes so they revert correctly with snapshots. - state_object.go: deep-copy MntBalances in newObject and on copy, so a mutation through one object cannot leak into another. - reader.go: decode MntBalances and FullShardKey out of the slim account served by the flat reader. - database_mpt.go / database_ubt.go: re-encode AccountsOrigin from slim-RLP to full QKC account RLP before handing it to pathdb, which needs it to match the trie leaf format for history verification. - pathdb/execute.go: decode AccountsOrigin as a full QKC StateAccount instead of slim-RLP, matching the invariant the commit paths now establish. mnt_test.go covers the accessors, journal revert, and copy isolation. The in-package test adaptations (state_test.go, statedb_fuzz_test.go) follow the account encoding change.
# Conflicts: # core/state/mnt_test.go # core/state/reader.go # core/state/state_object_qkc.go # core/state/statedb_qkc.go # triedb/pathdb/execute.go
The generator now drives ShardState.run_block as well as EvmState, so the vectors come at three granularities: direct state mutations, one transaction or one cross-shard deposit, and a whole minor block on a shard built from its own genesis with a root chain alongside. A block-level case carries the shard's GENESIS.ALLOC, the serialized root blocks the shard saw, the deposit lists its neighbours sent, and each block in order. The allocation is injected as the shard's own, so a consumer that applies it and nothing else must land on the genesis state root -- the same self-check the genesis cases make one level up. Two things stood between the script and a clean pyquarkchain checkout, and both are handled from the outside so no oracle module is patched. cluster_config resolves the local hostname at import time, which raises on a machine whose hostname does not resolve, so gethostbyname is pinned to loopback before that import. And add_block publishes subscription notifications through asyncio.create_task, so main() runs inside asyncio.run. The oracle is pinned to master 75f8d7e1 and generation no longer needs --allow-dirty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A shard walks its cross-shard cursor over root blocks, and the cursor's position is (root block height, index into that block's minor header list, index into that minor block's deposit list). The middle coordinate makes the header list consensus data rather than an index, so RootBlock carries it in order, along with the tracking data the header's size accounting includes. The merkle root over the header list is exposed but never recomputed on assembly: a block read off the wire keeps the root it arrived with, so a mismatch stays visible to validation instead of being papered over. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The MNT balance layer landed the leaf format and the per-token journal. What it did not have is the rest of the account lifetime, which decides whether a leaf exists at all and which shard key it carries. A zero-valued entry is a state of its own: an account that once held a token and was drained serializes to 00c0, a different leaf from one that never held it. balanceUpdateCount already recorded that presence, but its lifetime was geth's, not pyquarkchain's. A reverted balance change no longer forgets the entry, because the upstream undo writes the previous value back into the map (state.py:166) and leaves the key; only reset_balances removes it, so the counter is now cleared there instead. Read-back stopped recovering the marker from a 00c0 leaf, because the map is rebuilt from a pair list that carries no zeros (state.py:104) -- reproducing it would look faithful and fork the root of the first block to touch such an account. The slim encoding follows the trie for the same reason, so the snapshot and the trie cannot answer the same account read differently. Existence is decided once per block, over everything the block touched. That is geth's Finalise over the journal's dirty set, provided a block calls it exactly once, at the end -- an invariant now written down at the top of statedb_qkc.go, along with why the two halves line up when it holds. del_account and reset_balances are built on it: del_account strips the account rather than marking it self-destructed, so an account that self-destructs and is paid again later in the same block stays alive, as it does upstream; reset_balances journals nothing and dirties nothing, matching an undo that upstream writes to a misspelled attribute (state.py:195). The shard key freezes when an address is first *read*, not first written -- get_and_cache_account stamps a blank account with the state's current key and caches it for the block (state.py:387). An address read by one transaction and first written by the next therefore keeps the earlier key. It is recorded in a side table so getStateObject keeps returning nil for an absent account. Writes that exceed the token limit, name the wrong token, or overflow now go through setError, so Commit refuses a root instead of logging and carrying on with a silently dropped write. The limit itself gets a sentinel, ErrTokenTrieUnsupported, so the execution layer can tell a profile boundary from a database fault. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The account leaf, the per-token balances, the journal and the emptiness rule now live in geth's core/state and core/types, so this package no longer needs a trie, a state object or a journal of its own. What is left is what quarkchain/evm/state.py has beyond geth's StateDB: the execution context. EvmState embeds StateDB and shadows the handful of methods whose QuarkChain meaning differs -- balances take a token id, nonce and code writes drop a tracing reason QuarkChain has no use for, and logs are a flat per-transaction list rather than a map keyed by transaction hash. The rest of geth reaches through the embedded field unchanged. The context is STATE_DEFAULTS (state.py:45): the gas counters, the receipts and produced deposits, the block fee tokens, and the block-level parameters a message can read -- gas limit, height, coinbase, difficulty, timestamp, shard key, cursor and the previous headers BLOCKHASH walks. Snapshot copies all of it and revert puts it back, because upstream snapshots the whole of STATE_DEFAULTS and only full_shard_key actually moves during a block. Refunds are not kept here: the interpreter's SSTORE and SELFDESTRUCT write to the embedded StateDB's journalled counter, so a second one would be read by nobody and revert differently. Logs use StateDB's per-transaction list with a watermark, so the frame that produced them can be identified without a second list to keep in step. The state-level vectors gain three cases for the account semantics landed alongside: the shard key freezing at first read, a reverted balance change leaving a zero entry behind, and that entry being lost across a read-back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The message layer, not the interpreter: opcodes, gas table and tracing stay geth's. The profile is off unless EVM.QKC is set, and every divergence is behind that check, so an Ethereum caller runs unchanged. _apply_msg and create_contract are reimplemented rather than approximated, because their failure modes are observable. A call whose value cannot move burns the frame's gas instead of returning it (messages.py:656). A non-default token handed to code that never asked which token it holds fails afterwards, so a contract written before native tokens existed cannot be tricked into treating them as QKC -- and unlike the transfer failure it keeps its remaining gas. The POSW disallow map bars a recent block producer from spending into its locked stake, checked before anything is moved. Contract addresses are keccak(rlp([sender, fullShardKey, nonce]))[12:], and a top-level create does not advance the creator's nonce. BALANCE and SELFDESTRUCT read the chain's default token rather than the scalar (messages.py:582); on the shipped configs those coincide, so the golden vectors cannot see the difference and a shard config with a different default token is what covers it. SELFDESTRUCT records the account on the message's own list instead of marking it destroyed, since QuarkChain strips the account at the end of the message and lets the end-of-block sweep decide whether it survives; the refund's de-duplication reads that list. Five precompiles sit at 0x…514b4300 01 through 05. Three re-enter the message layer -- the current token id, a transfer in another token, and deploying a system contract -- so they take the EVM and the whole message rather than a byte slice. All five are gated on a strict timestamp comparison, and the two native-token ones listen only to ENABLE_NON_RESERVED_NATIVE_TOKEN_TIMESTAMP (env.py:63-76); the general switch decides only when its own system contract may be deployed. One boundary is reported rather than executed: CREATE2's word charge can drive pyquarkchain's gas counter negative when the memory needs no growing, which mem_extend would otherwise have caught. The frame flags it and the block is abandoned, instead of guessing at an out-of-gas. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_block and everything under it: transaction validation, application, the cross-shard cursor, POSW, and the comparison against what a block claims. Process runs on an isolated copy of the parent state opened at the parent's root, so a rejected block leaves the caller's state alone. The cross-shard half runs first and whatever it leaves unspent of its allowance goes back to the in-shard gas limit; that ordering is consensus, not sequencing. The cursor is a three-level position -- root block height, index into that block's minor header list, index into that minor block's deposit list -- and a root block missing at or below the block's own root tip is an error rather than an end of stream, because a node that skipped those deposits would compute a root every node holding the data rejects. A deposit is credited directly before the EVM switch and executed as a message after it; the two paths are mutually exclusive and both are pinned. Native tokens are executed rather than gated. A transaction paying gas in another token buys its rate from the general native token manager, has the reserve moved on its behalf, and gets its refund and burn split by the contract's refund rate; validation applies its balance and gas-limit rules per token. The rate is quoted by running the contract under a snapshot that is then rolled back, as upstream does, so the quote leaves nothing behind. The root chain POSW contract's runtime bytecode is embedded, since it is reachable throughout the supported window. Two boundaries are reported instead of executed, and both abandon the block: an account that would hold more non-zero tokens than the leaf's list encoding takes, which is a state this fork cannot write; and a transaction whose gas counter would go below zero, which is a transaction pyquarkchain cannot execute -- no node can mine or accept a block holding one, so executing it cleanly here would make this the only client that accepts such a block. ENABLE_TX_TIMESTAMP and TX_WHITELIST_SENDERS were in the configs but unread; they are the early-mainnet gate that let only whitelisted senders transact. Block seal, difficulty and the structural checks around run_block are not here: they belong to validate_block. POSW's staking decay is likewise absent on purpose -- the disallow map uses the undecayed stake, and the decay only ever reaches the seal difficulty. The message-level vectors grow to 41 cases and the block-level file lands with 11. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
syntrust
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The QuarkChain execution layer — everything a minor block does to state — pinned by golden vectors generated from pyquarkchain at three granularities.
Test
🤖 Generated with Claude Code