Skip to content

audit declares when its dependency half is not the --head version - #28

Merged
modusensus merged 2 commits into
mainfrom
fix/audit-worktree-version-declaration
Oct 9, 2026
Merged

modusensus merged 2 commits into
mainfrom
fix/audit-worktree-version-declaration

Conversation

@modusensus

@modusensus modusensus commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

audit's two halves read from different places. history takes the commits named by --base/--head; deps takes the manifest currently checked out at the repository root. Nothing in the output said so — measured on the rustls run, a clone parked on main yielded 284 dependency facts for a range whose tag holds 366, and the report looked entirely normal.

This is the CLI follow-up to the rustls case study, where the split was documented (docs/case-study-rust.md §3, plus an AGENTS.md pitfall row). The maintainer chose declaration over re-measurement; this PR is that decision.

What it does

One new entry in the dependency half of coverage.notEvaluated, carrying a machine-readable code:

code when
manifest-not-head the manifest that was measured differs in content from the one at --head, or does not exist there at all
manifest-unverified the probe itself could not run

Exit codes are unchanged, deliberately: audit's rule is that a missing or partial criterion cannot fail a run the history half answered (src/cli.js:524), and a declared gap is that kind of entry. Declaring also does not suppress the dependency facts that were measured — they stay in the report, and now say what version they describe.

The criterion is content, and that was the review's finding

The first commit compared commit shas (HEAD vs --head) and added a separate git status check for uncommitted edits. An independent review of that version found it wrong in both directions; both cases reproduce against the real CLI:

A) lockfile gitignored and never committed, HEAD == --head
   codes: []                        ← one dependency fact measured, zero declarations
B) HEAD is a PR merge commit, its lockfile byte-identical to --head's
   codes: ["worktree-not-head"]     ← fires on every run

B is not cosmetic. With notEvaluated non-empty, no gate may conclude pass (fact-contract rule 1), so every finding citing a dependency fact was forced to not_evaluated — and the remedy the skill teaches, "check out an end of the range", is impossible on a merge ref. A is the opposite failure: content that belongs to no commit, measured silently, reading as clean.

Content comparison fixes both with one criterion: git ls-tree answers whether the measured manifest exists at --head, git diff --name-only <head> -- <path> answers whether it matches. Using git diff also keeps core.autocrlf checkouts from reporting a difference that exists only in line endings — the phantom-diff failure this repository already records for LICENSE.

manifest-unverified exists because the earlier version passed allowFailure to both git calls, so a git error produced silence: the direction that inverts "missing data is never clean".

Why declare rather than read the blob at --head

Reading git show <head>:Cargo.lock would make the facts genuinely range-pinned, but every dependency fact's evidence pointer is a path:line a reader is told to re-run (euthyna deps --lockfile '<path>' --dep '<name>'). Pointing at a blob would break that channel — the file on disk would not contain what the report claims. Making it honest meant changing the evidence format across the contract, both skill editions and both docs/ copies. That is a much larger change than the failure it fixes.

Tests

Ten new/rewritten, building real git repositories as the suite already does:

  • content differs from --head → declared, names the commit, keeps the facts, does not change the exit code
  • manifest absent at --head (gitignored, never committed) → declared
  • merge commit with lockfile content identical to --head → declares nothing, with assertions that the fixture's two manifests really are identical and really are different commits
  • manifest exactly matching --head → declares nothing
  • uncommitted edit away from --head content → declared
  • --head omitted → declares nothing
  • the decision table as unit tests of the exported pure function, including the fail-closed branch
  • the English reason renders under --lang en and the Chinese one does not

The limit flagged in the first version of this PR is now closed by measurement, not argument: the merge-commit and matching-manifest guards were written against the sha implementation and failed (that was their RED). Two "asserts nothing is declared" tests that fail under the wrong criterion cannot be vacuous.

docs/fact-contract.md and docs/fact-contract-zh.md gain rule 4 (code is machine-readable, appears only on --json, never a verdict); both fact-producers.md §七 bullets and the AGENTS.md pitfall row describe the content criterion and say plainly that a declaration is not a measurement. Both language pairs were edited together, so test/docs-pair.test.js and test/skill-mirror.test.js are what prove the two languages did not drift.

Suite

node --test → tests 246 / pass 245 / fail 0 / skipped 1 (the deliberate non-Windows case).

Summary by Sourcery

Declare dependency-version gaps in audit by comparing the measured manifest's content with --head and fail closed when that verification is unavailable.

New Features:

  • Declare when the dependency manifest measured by audit is absent from or differs in content from the --head version, using machine-readable manifest-not-head and manifest-unverified codes while retaining measured dependency facts.

Bug Fixes:

  • Prevent false declarations caused by comparing commit identities instead of manifest content, including merge commits with matching manifests and gitignored manifests with no committed counterpart.
  • Fail closed when manifest verification cannot be performed instead of silently treating the probe as successful.

Enhancements:

  • Compare manifest content against --head while remaining compatible with working-tree evidence paths and line-ending normalization.
  • Document the audit version-alignment behavior, declaration semantics, and machine-readable fact-contract codes in both language editions and agent guidance.

Documentation:

  • Add the optional machine-readable code contract for JSON-only not-evaluated entries in English and Chinese fact-contract documentation.
  • Update fact-producer references and repository guidance to explain that audit declarations identify gaps but do not remeasure dependencies.

Tests:

  • Add integration and unit coverage for matching, differing, absent, uncommitted, merge-commit, omitted-head, and failed-probe manifest cases.
  • Verify localized audit reasons and synchronization of the English and Chinese documentation and skill copies.

audit's two halves read from different places: history takes the commits named
by --base/--head, deps takes the manifest checked out in --repo. Measured on the
rustls run, a clone parked on main produced 284 dependency facts for a range
whose tag holds 366, and nothing in the output said so.

The fix declares rather than corrects: a dependency entry under criteria-not-
evaluated, with a machine-readable code (worktree-not-head, or
manifest-uncommitted for an edited-but-uncommitted manifest, which a commit
comparison cannot see). Choosing declaration over reading the blob at --head was
deliberate — the evidence pointers would stop naming files a reader can re-run.

A declared gap cannot fail a run the history half answered, so the exit-code
contract is unchanged. Tests build real repositories and assert both the presence
and the absence of the declaration; the English string is pinned separately,
because an unexercised half of a t() call is still production code.
@sourcery-ai

sourcery-ai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Sorry @modusensus, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 5 days and 3 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@sourcery-ai

sourcery-ai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Reviewer's Guide

The PR makes audit honest about its dependency half reading the currently checked-out manifest rather than the resolved --head tree. It adds machine-readable worktree-not-head and manifest-uncommitted declarations without suppressing facts or changing exit codes, documents the contract and operational guidance in both languages, and validates the behavior with real Git-repository integration tests.

Sequence diagram for audit dependency version declarations

sequenceDiagram
    participant CLI as audit CLI
    participant Git as Git repository
    participant Deps as Dependency collector
    participant Report as Audit report

    CLI->>Git: revParse(toplevel, HEAD)
    CLI->>Deps: collectDependencyFacts(...)
    Deps-->>CLI: dependency facts and subject.lockfile
    CLI->>Git: git(status --porcelain -- manifest)
    alt worktree HEAD differs from resolved --head
        CLI->>Report: add dependency code worktree-not-head
    end
    alt manifest has uncommitted changes
        CLI->>Report: add dependency code manifest-uncommitted
    end
    CLI->>Report: retain measured facts and unchanged exit code
Loading

File-Level Changes

Change Details Files
Declare when dependency facts are measured from a worktree version or manifest content that does not match the requested audit range.
  • Resolve and compare the checked-out HEAD with the resolved --head commit, adding worktree-not-head when they differ.
  • Inspect the detected manifest with git status and add manifest-uncommitted when it has uncommitted changes.
  • Keep measured dependency facts and existing exit-code behavior unchanged while adding localized English and Chinese reasons.
src/cli.js
Extend the fact contract and contributor guidance with machine-readable audit-gap codes and version-alignment instructions.
  • Document the optional code field and the two dependency gap codes in both language versions of the contract.
  • Update both skill editions and AGENTS.md to explain that audit halves read different versions and require rerunning at a range endpoint.
docs/fact-contract.md
docs/fact-contract-zh.md
.agents/skills/euthyna-en/references/fact-producers.md
.agents/skills/euthyna/references/fact-producers.md
AGENTS.md
Add integration coverage for version mismatch, dirty manifests, omission of --head, and localized rendering.
  • Build real repositories with changing npm lockfiles and verify declarations, commit identifiers, preserved dependency facts, and unchanged exit status.
  • Verify clean --head worktrees and omitted --head produce no declaration.
  • Verify English output uses the English reason and does not emit the Chinese text.
test/cli.test.js
test/i18n.test.js
test/helpers.js

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@modusensus

Copy link
Copy Markdown
Contributor Author

@sourcery-ai review

@sourcery-ai

sourcery-ai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Sorry @modusensus, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 5 days and 3 hours by commenting @sourcery-ai review. Upgrade to get a review now.

…out sha

An independent review of the first version found the criterion wrong in both
directions, and both cases reproduce against the real CLI:

- a PR merge commit whose package-lock.json is byte-identical to --head's got
  worktree-not-head on every run. Not cosmetic: with a non-empty notEvaluated no
  gate may pass (fact-contract rule 1), so every finding citing a dependency fact
  was forced to not_evaluated, and the remedy the skill teaches — check out an end
  of the range — is impossible on a merge ref.
- a lockfile that is gitignored and was never committed produced dependency facts
  with no declaration at all, even though its content belongs to no commit.

The criterion is now content: git ls-tree decides whether the measured manifest
exists at --head, git diff --name-only -- <head> decides whether it matches, and
either way the entry is one code (manifest-not-head) rather than two. Using git
diff also keeps core.autocrlf checkouts from reporting a difference that exists
only in line endings. A probe that fails is declared as manifest-unverified: a
git error is not evidence of a match, which is the direction the allowFailure
version got backwards.

The decision moved into an exported pure function so the fail-closed branch is
reachable from a test; nothing else here can make git fail on demand. Suite:
246 tests, 245 pass, 0 fail, 1 skipped.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sourcery assessment

Approved.

@modusensus
modusensus merged commit cdece0f into main Oct 9, 2026
10 checks passed
@modusensus
modusensus deleted the fix/audit-worktree-version-declaration branch October 9, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant