Skip to content

test: construct the conforming half of the bytesToHex Vm-output property - #102

Merged
thedavidmeister merged 2 commits into
mainfrom
2026-08-16-issue-59
Aug 17, 2026
Merged

test: construct the conforming half of the bytesToHex Vm-output property#102
thedavidmeister merged 2 commits into
mainfrom
2026-08-16-issue-59

Conversation

@thedavidmeister

@thedavidmeister thedavidmeister commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Closes #59

Problem

testBytesToHexRejectsEveryNonConformingVmOutput stated a bi-conditional in its
docstring — for any data and any string a Vm might return, the library
"either reverts or returns exactly that string with its first two characters
removed" — but fuzzed toStringReturn independently of data. Reaching the
accept arm therefore required a random string to land on exactly
data.length * 2 + 2 characters AND start 0x. It never did.

Fix

The conforming string is constructed rather than waited for. A fuzzed bool conforming selects which half of the property a run aims at; when it is set,
the returned string is 0x plus a data.length * 2 payload filled cyclically
from the fuzzed filler, so an accepted string is arbitrary in everything
except the two things the library actually checks (length and prefix).

Whether a string conforms is still read off its own characters, not off
conforming, so a filler that coincidentally conforms is checked against the
accept arm rather than expected to revert.

Measured: the accept arm was dead, and is not now

Instrumented the accept arm with vm.writeLine (temporary, not in the diff) and
counted how many of 2048 fuzz runs reached it, on four seeds:

seed before after
1 0 1021
2 0 991
3 0 1036
4 0 984

8192 fuzz runs before the change reached the accept arm zero times. The "after"
column is also the positive control for the probe itself: same probe, same file,
same permissions, so the zeros are real zeros and not a probe that never fired.

Failing → passing

A dead arm cannot fail on the unmutated library, so the failure is demonstrated
against a library whose accept path is broken. Every mutant below was applied to
src/lib/LibHexString.sol, run in isolation with --match-test, --fuzz-seed 1.

Before the change, sub(len, 3) in place of sub(len, 2) — a library that
strips one character too many from every successful conversion:

[PASS] testBytesToHexRejectsEveryNonConformingVmOutput(bytes,string) (runs: 2048, μ: 476595, ~: 454498)
Suite result: ok. 1 passed; 0 failed; 0 skipped

After the change, the same broken library:

[FAIL: assertion failed: <mangled> != 𝋫; counterexample: args=[0x2222, "𝋫¥{]ષw𐤿…", true]]
    testBytesToHexStripsOrRevertsForEveryVmOutput(bytes,string,bool) (runs: 4, μ: 493948, ~: 476857)
Suite result: FAILED. 0 passed; 1 failed; 0 skipped

And on the unmutated library, after the change:

[PASS] testBytesToHexStripsOrRevertsForEveryVmOutput(bytes,string,bool) (runs: 2048, μ: 520370, ~: 499208)
Suite result: ok. 1 passed; 0 failed; 0 skipped

Mutation matrix

Test run in isolation, --fuzz-seed 1, 2048 runs configured.

mutant of src/lib/LibHexString.sol pre-fix test post-fix test
none PASS PASS
M1 mstore(newHexString, sub(len, 2))sub(len, 3) PASS — survived FAIL — killed
M2 let newHexString := add(hexString, 2)add(hexString, 3) PASS — survived FAIL — killed
M3 if eq(len, expectedLength)if gt(len, 1) FAIL — killed FAIL — killed
M4 0x30780x3079 (prefix constant) PASS — survived FAIL — killed

M4's post-fix failure is [FAIL: UnexpectedHexString("0x/SS\\𐶏𑤉?<", 16)] — a
conforming string was rejected, which is exactly the half of the property that
had no coverage. M3 mutates the reject path, is killed by both, and is here to
show the reject half was not weakened to buy the accept half.

Proof the mutants really executed, rather than a stale artifact or a zero-match
filter reading as "survived":

  • each mutant was diffed against the pristine source before its run, and a
    no-op sed was made to abort rather than report;
  • every "survived" row carries runs: 2048 and a matching test name, so the
    filter matched a test and the fuzzer ran the full budget;
  • LibHexStringExternal deployedBytecode sha256 (first 32 hex), recompiled
    from scratch per mutant, alongside the pre-fix result each one produced:
library deployedBytecode sha256 pre-fix test
unmutated 5305d03fd60433c68f9f5597ef4d232b PASS, runs: 2048, μ 476595
M1 7972c518e4dae29b5a24a5d3d0fc6ebc PASS, runs: 2048, μ 476595
M4 6480f249a4d1bd65bd43f744700f4a53 PASS, runs: 2048, μ 476595

Three different bytecodes, identical mean gas and identical verdict, because
every one of the 2048 runs took the revert path and the mutated instructions
were never reached.

The runs: 4 on killed rows is the fuzzer stopping at its first counterexample,
not a short run.

Checks

  • nix develop -c forge test — 134 passed, 0 failed, 0 skipped, 16 suites.
  • nix develop -c forge fmt --check — exit 0.
  • Tree restored and git status clean after every mutation pass.

Departures from the issue's proposed fix

  • Renamed testBytesToHexRejectsEveryNonConformingVmOutput
    testBytesToHexStripsOrRevertsForEveryVmOutput. The issue diagnosed "a
    rejection test wearing a bi-conditional's name"; making the accept half live
    fixes the behaviour, and the name has to follow it or the accept half stays
    undiscoverable. Nothing else in the repo references the old name.
  • Renamed the local badVmstubVm. Half the runs now hand the library a
    conforming Vm, so badVm names the wrong thing.
  • The construction block is placed before the LibHexStringExternal deployment
    rather than after, so the string is finished before anything consumes it.
  • Everything else is the issue's proposal as written.

QA

  • Discriminating tests: testBytesToHexStripsOrRevertsForEveryVmOutput - fails on base behaviour three ways, each verified by running it against a library whose accept path is mutated (M1 [FAIL: assertion failed: <mangled> != 𝋫], M2 [FAIL: assertion failed], M4 [FAIL: UnexpectedHexString("0x/SS\\𐶏𑤉?<", 16)]), where the pre-fix testBytesToHexRejectsEveryNonConformingVmOutput passes at runs: 2048 on all three; independently measured with a temporary vm.writeLine probe, the pre-fix accept arm executed 0 times in 8192 runs (seeds 1-4) and the post-fix one executes 1021/991/1036/984 times per 2048.
  • Mutations applied: mstore(newHexString, sub(len, 2)) -> sub(len, 3) -> killed by testBytesToHexStripsOrRevertsForEveryVmOutput (survived pre-fix); let newHexString := add(hexString, 2) -> add(hexString, 3) -> killed by testBytesToHexStripsOrRevertsForEveryVmOutput (survived pre-fix); if eq(shr(240, mload(add(hexString, 0x20))), 0x3078) -> 0x3079 -> killed by testBytesToHexStripsOrRevertsForEveryVmOutput (survived pre-fix); if eq(len, expectedLength) -> if gt(len, 1) -> killed by both the pre-fix and post-fix test, confirming the reject half was not weakened. Each mutant was diffed against the pristine source before running, a no-op sed aborted rather than reported, every survived row carries runs: 2048 with a matching test name, and the recompiled LibHexStringExternal deployedBytecode hashes differ per mutant (5305d03f… unmutated / 7972c518… M1 / 6480f249… M4).
  • Oracle: the definition of Vm.toString(bytes) - "0x" followed by exactly two characters per input byte - not the library. expectedLength and the conformance predicate are computed in the test from data.length and from the returned string's own characters, and the expected result is built by copying returned[2..] into a fresh buffer, so nothing is read back off LibHexString.bytesToHex.
  • Category check: issue asks for the conforming half of the bi-conditional to be constructed so the accept arm is reached; covered, with the arm measured live (0 -> ~1000 reaches per 2048) and its coverage proven by three mutants that survive without it. The issue's path note (test/src/lib/… after Move every .t.sol into the test/src/lib mirror tree #56) does not apply yet - Move every .t.sol into the test/src/lib mirror tree #56 is still open, so the file is at test/lib/LibHexString.bytesToHex.t.sol on main.

Other checks

  • nix develop -c forge test: 134 passed, 0 failed, 0 skipped across 16 suites.
  • nix develop -c forge fmt --check: exit 0, no diff.
  • git status clean on the restored tree after every mutation pass.

Summary by CodeRabbit

  • Tests
    • Expanded fuzz testing for hexadecimal string conversion.
    • Added coverage for both valid, correctly formatted VM output and invalid output that should be rejected.
    • Updated assertions to verify successful conversion or expected reverts across generated inputs.

Post-Build.sol-removal sweep (2026-08-17)

main (959d527) merged in. Unaffected by #138's removal — nothing cut.
The conflict was not with the removal; it was with #105's refactor of this same
test file.

Conflict resolved, in one function. main hoisted the callee into an
immutable iExternal built once in the constructor and added a vmReturning
helper for the stub Vm. This branch still built both inline
(new LibHexStringExternal(), new NonConformingVm(...)). Resolved by keeping
this branch's semantics — the constructed conforming payload, the conforming
selector argument, and the rename to
testBytesToHexStripsOrRevertsForEveryVmOutput — on top of main's two
helpers. The diff vs main is now this one function and nothing else.

The accept arm was re-measured on the merge commit, not assumed. A textual
resolution here could have left the constructed arm dead again, which is the
whole deliverable, so the M4 mutant from the matrix above was re-run against the
merged tree: 0x30780x3079 in src/lib/LibHexString.sol,
--match-test testBytesToHexStripsOrRevertsForEveryVmOutput:

[FAIL: UnexpectedHexString("0x*X…", 8); counterexample: args=[0x7440e0, "*X…", true]]
    testBytesToHexStripsOrRevertsForEveryVmOutput(bytes,string,bool) (runs: 7, …)
Encountered a total of 1 failing tests, 0 tests succeeded

Killed at run 7 with conforming = true in the counterexample — that is the
accept arm executing. src/lib/LibHexString.sol restored byte-identical
afterwards, git status --porcelain empty.

Overlap with #108, stated with both numbers, not resolved here. #108 adds a
hex-charset check to bytesToHex and says so in its own body: with #108 merged,
an arbitrary-byte payload is non-conforming almost always, so the arm this PR
constructs would land back in the reject arm and the measured ~1000/2048 split
would collapse to ~0 again. The fix belongs in this PR's filler — map each
filler byte into 0-9a-f rather than using it raw — and is not applied
here, because on main today bytesToHex accepts any payload and the mapping
would narrow the property for no reason. Whichever of #102 and #108 lands second
carries it. They also collide textually: #108 edits the conforms predicate
inside this same function.

Suite on the merge commit: Ran 19 test suites: 145 tests passed, 0 failed, 0 skipped — unchanged from main, right for a rename-in-place.
forge fmt --check clean, git status clean after the run.

`toStringReturn` was fuzzed independently of `data`, so a run reached the
accept arm only if a random string happened to be `0x` plus exactly two
characters per input byte. Measured over 2048 runs on each of seeds 1-4, that
happened 0 times, so the accept arm of the stated bi-conditional was dead.

The conforming string is now constructed from the fuzzed data length with a
payload filled from a fuzzed `filler`, selected by a fuzzed `conforming` bool.
Conformance is still read off the string's own characters, so a `filler` that
coincidentally conforms still lands on the accept arm.

Closes #59

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedavidmeister thedavidmeister self-assigned this Aug 16, 2026
@coderabbitai

coderabbitai Bot commented Aug 16, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 95e806f5-6aa5-4f99-8999-aadcb9fb6f49

📥 Commits

Reviewing files that changed from the base of the PR and between 935c725 and e2b1e51.

📒 Files selected for processing (1)
  • test/lib/LibHexString.bytesToHex.t.sol

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.


Walkthrough

The fuzz test now constructs correctly sized, 0x-prefixed VM output for the success path. It continues to test rejection for arbitrary output and uses the renamed VM stub.

Changes

Hex string conformance test

Layer / File(s) Summary
Construct conforming VM outputs
test/lib/LibHexString.bytesToHex.t.sol
The fuzz test accepts a conforming flag, builds valid output when enabled, and retains arbitrary filler for rejection cases. Both branches call bytesToHex through stubVm.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to e2b1e

This localized test change constructs conforming VM output so both sides of the existing bytes-to-hex behavior contract are exercised; no actionable merge-blocking risk remains after normal checks and review.

Possibly related issues

  • #68 — The change updates the same fuzz test to generate conforming and non-conforming VM outputs.

Suggested reviewers: claude

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes satisfy issue [#59] by constructing conforming output from data.length while retaining arbitrary filler coverage.
Out of Scope Changes check ✅ Passed The changes remain within issue [#59] scope, including related test and stub variable renames.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: constructing conforming VM output for the bytesToHex property test.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 2026-08-16-issue-59

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@thedavidmeister
thedavidmeister merged commit 333ee2b into main Aug 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The bytesToHex conformance property in LibHexString.bytesToHex.t.sol never reaches its accept arm

1 participant