Skip to content

fix(detectors): stop FileDetector scoring an unexamined file as a pass - #2080

Open
CTWalk wants to merge 1 commit into
NVIDIA:mainfrom
CTWalk:fix/filedetector-unexamined-unscored
Open

fix(detectors): stop FileDetector scoring an unexamined file as a pass#2080
CTWalk wants to merge 1 commit into
NVIDIA:mainfrom
CTWalk:fix/filedetector-unexamined-unscored

Conversation

@CTWalk

@CTWalk CTWalk commented Aug 17, 2026

Copy link
Copy Markdown

Summary

FileDetector.detect() ends with

test_result = self._test_file(local_filename.text)
yield test_result if test_result is not None else 0.0

while _test_file is declared -> Union[None, float] (detectors/base.py:316)
and 0.0 is the no-hit end of the scale. A None meaning "I could not examine
this file" is reported as "this file is not a hit".

FileIsExecutable._test_file returns that None when self.magic is None,
which happens whenever python-magic imports but the system libmagic is
missing: _load_deps catches the ImportError and logs it. On such a machine
the detector never reads a byte and every file it is handed comes back as not an
executable.

Yielding the None instead lets the evaluator count it in nones and print
SKIP rather than PASS. That is the call #2029 already made for the other
branch of this same function, and it matches StringDetector's
[None] * len(all_outputs) and HFDetector's graceful_fail. The edit is in
detectors/base.py because the coercion is there; a subclass cannot undo it.
FileDetector's docstring already documented the format-mismatch None, so it
gains one sentence for this one.

Two things to flag rather than leave to review. analyze/analyze_log.py:60 and
analyze/qual_review.py:82 compare scores with >= / > and raise TypeError
on a None; that is pre-existing, since 20 detector modules already emit None
and this makes fileformats the 21st, so it wants its own fix. And _test_file
could raise instead, but the base signature already permits None and the whole
detector layer uses that channel.

Reproduction

On a machine without libmagic, the condition your suite already guards for with
skipif(magic is None):

$ python -m pytest tests/detectors/test_detectors_fileformats.py -q
before: 17 passed, 6 skipped
after:  18 passed, 6 skipped

test_fileisexectuable_nonexec was one of the cases that passed without the
detector running
: it asserts [0.0] on a text file, _test_file returned
None, and else 0.0 supplied the expected answer. Rather than guard it with
skipif and lose the case, it now expects [0.0] where libmagic is present and
[None] where it is not, so it runs everywhere and asserts something true in
both. Nothing that ran before is skipped now.

The new test monkeypatches magic = None, so it too runs either way. Reverting
the one-line fix and keeping both tests gives 2 failed.

Why this is not a duplicate

Verification

  • python -m pytest tests/detectors/ -q: 744 passed, 34 skipped against
    743 passed, 34 skipped on an unpatched tree. Same 13 errors either way,
    all in test_detectors_judge.py and unrelated to this change
  • python -m pytest tests/detectors/test_detectors_fileformats.py -q:
    18 passed, 6 skipped, up from 17 passed, 6 skipped
  • Fix reverted with both tests kept: 2 failed, so they pin the behaviour
  • black 25.1.0 clean on the lines touched
  • python -m pytest tests/ in full, on the submitter's machine

AI assistance

AI assistance was used in preparing this change, per AGENTS.md. The submitter
has reviewed every changed line, run the tests above, and can defend the change.

FileDetector.detect() coerced a None from _test_file to 0.0, the no-hit end of
the scale, while _test_file is declared -> Union[None, float]. FileIsExecutable
returns that None when libmagic is missing, so every file it was handed came
back as not an executable without being examined.

Yield the None instead, so the evaluator counts it in nones and reports SKIP.
This is the call NVIDIA#2029 made for the format-mismatch branch of the same function.

test_fileisexectuable_nonexec passed without the detector running; it now
expects [0.0] where libmagic is present and [None] where it is not, so it runs
in both environments.

Co-authored-by: Claude
Signed-off-by: CTWalk <100585900+CTWalk@users.noreply.github.com>
@CTWalk
CTWalk force-pushed the fix/filedetector-unexamined-unscored branch from 34c6acc to 0a93ee1 Compare August 18, 2026 01:57

@cmcnosky cmcnosky left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At head 0a93ee1c, the change preserves the None returned by FileIsExecutable._test_file() when the magic dependency is unavailable. At base 7ff1f277, FileDetector.detect() coerces that result to 0.0, causing ZeroToleranceEvaluator to report PASS; at the head it remains unscored, producing SKIP with zero passes, zero failures, one unscored result, and zero evaluated results.

The base-class edit is in the right place because that is where the distinction between “not examined” and “examined, no hit” was lost. FileIsExecutable is the only current in-tree _test_file() implementation that returns None; external subclasses using the documented Union[None, float] contract will also preserve their unscored result.

Verification at the exact head on Python 3.12.13 and macOS arm64:

  • python -m pytest tests/detectors/ -q — 757 passed, 34 skipped

  • python -m pytest tests/detectors/test_detectors_base.py tests/detectors/test_detectors_fileformats.py tests/evaluators/test_base.py -q — 51 passed, 6 skipped

  • python -m pytest tests/evaluators/ -q — 58 passed

  • git diff --check 7ff1f277..0a93ee1c — clean

No code-review blockers found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants