feat(cli): opt-in discovery of an author-shipped baseline (#278)#286
Open
rodboev wants to merge 1 commit into
Open
feat(cli): opt-in discovery of an author-shipped baseline (#278)#286rodboev wants to merge 1 commit into
rodboev wants to merge 1 commit into
Conversation
Discover a co-located .skillspector-baseline.yaml and apply it only when the consumer opts in with --use-shipped-baseline, reporting provenance on stderr. Detection without opt-in leaves findings and the risk score untouched and never parses the file. Explicit --baseline still wins. Closes NVIDIA#278 Signed-off-by: Rod Boev <rod.boev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A skill distributed with a vetted
.skillspector-baseline.yamlgets no benefit from it today:skillspector scan <skill>ignores the file unless every consumer finds it and passes--baselineby hand, so consumers of distributed skills see the author's known false positives raw. Loading it automatically would be worse: a baseline shipped by the skill author is third-party configuration from the party being audited, and applying it silently would let it suppress findings, lower the risk score, and flip the non-zero-exit install gate in a consumer's scan.This makes discovery explicit-trust:
scandetects a co-located baseline and reports it, applies it only when the consumer opts in with--use-shipped-baseline, and prints provenance when it does.Closes #278
Behavior
.skillspector-baseline.yamlat the top level of the scanned directory produces a stderr notice naming the file and the opt-in flag. Findings, risk score, and exit code are unchanged, and the file is never parsed; discovery is a pure existence check.--use-shipped-baseline: the discovered baseline goes through the same loading and validation as--baselineand is applied; a stderr provenance line records that an author-shipped baseline at that path was applied.--show-suppressedlists what it suppressed and why through the existing rendering. A malformed shipped baseline exits 2, same as an explicit one; it is never silently skipped.--baseline PATHalways wins: discovery is skipped entirely when an explicit baseline is given, including the missing-file exit-2 path.--format jsonand--format sarifstdout stays machine-parseable.Discovery is limited to the canonical
.skillspector-baseline.yaml, the nameskillspector baselinewrites by default..ymland.jsonbaselines stay usable through explicit--baseline; keeping auto-discovery to one filename keeps the author-controlled trust surface small and avoids any precedence question.Diff Notes
src/skillspector/suppression.py: newdiscover_baseline()that returns the co-located.skillspector-baseline.yamlpath orNonewithout reading the file. Top level of the scanned directory only.src/skillspector/cli.py:--use-shipped-baselineflag onscan, a discovery gate in the single-directory path, and a stderr console for the detection notice, provenance line, and a note when opt-in is given but nothing is shipped. The discovered path flows through the existing baseline plumbing, so suppression semantics and report rendering are unchanged.tests/unit/test_suppression.py: discovery unit tests (canonical file, absence, non-directory input, directory-named-like-the-baseline, noncanonical.yml/.jsonsiblings ignored, nested files ignored).tests/unit/test_cli.py: end-to-end tests (all--no-llm) for detect-without-apply, opt-in apply with provenance andsuppressed: trueentries, explicit--baselineprecedence over a shipped suppress-all baseline, JSON and SARIF stdout parseability with a discovered baseline in a directory whose name has a space and a non-ASCII character, no-op opt-in when nothing canonical is shipped,--show-suppressedreason rendering, exit 2 on a malformed shipped baseline under opt-in, and an untouched recursive path.Scope
--baselinethreading there is tracked separately ([BUG] Recursive multi-skill scans silently ignore --baseline and --show-suppressed #201/fix(cli): baseline suppression in recursive multi-skill scans (#201) #205).Verification
uv run pytest tests/unit/test_cli.py tests/unit/test_suppression.py -q→ 47 passed (33 pre-existing + 14 new)uv run ruff check src/ tests/→ All checks passed.uv run ruff format --check src/ tests/→ 144 files already formatted