Skip to content

fix(probes,generators): raise GarakException on unpicklable parallel worker payload - #2075

Open
glatinone wants to merge 3 commits into
NVIDIA:mainfrom
glatinone:fix/issue-361-parallel-pickling-error
Open

fix(probes,generators): raise GarakException on unpicklable parallel worker payload#2075
glatinone wants to merge 3 commits into
NVIDIA:mainfrom
glatinone:fix/issue-361-parallel-pickling-error

Conversation

@glatinone

Copy link
Copy Markdown

Summary of Changes

Probe._execute_all (garak/probes/base.py) and Generator.generate (garak/generators/base.py) each catch pickle.PicklingError around their multiprocessing.Pool.imap_unordered call, alongside the existing OSError/errno-24 handling, and re-raise it as a GarakException with actionable guidance to reduce parallel_attempts/parallel_requests to 1 — mirroring the existing "Parallelisation limit hit" pattern already in both functions.

Root Cause & Performance Impact

Both parallel_attempts (probes) and parallel_requests (generators) submit work to a multiprocessing.Pool via imap_unordered. Submitting a task pickles the bound method together with self (the probe/generator instance). If that instance holds an attribute that cannot be pickled — e.g. a dynamically loaded plugin object whose defining module can't be re-imported by name in the worker process — Pool._handle_tasks raises a raw _pickle.PicklingError from deep inside multiprocessing internals, exactly as shown in the two tracebacks in #361 (import of module 'core.py' failed, import of module 'summarize_document.py' failed).

This isn't a performance issue so much as an abrupt, unhelpful failure mode: the run aborts with an internal multiprocessing traceback instead of a clear, actionable error — the maintainer's own framing on the issue was "what should we do? at least explode /politely/, for now." This PR scopes strictly to that: converting the raw internal traceback into a clear GarakException. Auto de-parallelization (mentioned as a follow-up in the issue) is intentionally left out of scope.

Test Coverage & Verification

  • tests/generators/test_generators.py::test_parallel_requests_unpicklable_attribute_raises_garak_exception — reproduces the generate() path (parallel_requests), using a module-level stub function whose __module__ is corrupted to force a genuine pickle.PicklingError (import of module '...' failed) matching the issue's exact failure shape, rather than relying on a local-closure AttributeError.
  • tests/test_internal_structures.py::test_execute_all_unpicklable_probe_raises_garak_exception — same technique for the Probe._execute_all path (parallel_attempts).
  • Both tests were verified to fail against the pre-fix code with the exact _pickle.PicklingError: Can't pickle ...: import of module '...' failed traceback from fail more helpfully when parallelism not supported #361, and to pass after the fix (asserting GarakException is raised instead).
  • Full suite: pytest — 1489 passed (only the 2 directly touched test files plus tests/test_config.py -k parallel were run against both pre- and post-fix code to isolate signal). One pre-existing failure (test_probe_metadata[probes.audio.AudioAchillesHeel], missing optional soundfile/librosa extras) and 3 pre-existing errors (openai_compat_mocks fixture not resolving when test_openai.py is invoked alongside certain other test paths) were confirmed present identically on main before this change — unrelated to this diff.
  • black --check and pylint show no new issues introduced by this diff (both tools flag two pre-existing, unrelated formatting/lint items on main in the touched files, outside the lines this PR changes).

Closes #361

…worker payload (NVIDIA#361)

When parallel_attempts/parallel_requests is enabled and a probe or generator
instance holds a non-picklable attribute (e.g. a plugin dynamically loaded
from a file path multiprocessing cannot re-import in the worker process),
Pool.imap_unordered raises a raw _pickle.PicklingError from deep inside
multiprocessing's internals instead of failing cleanly.

Catch pickle.PicklingError alongside the existing OSError/errno-24 handling
in Probe._execute_all and Generator.generate, and re-raise as a GarakException
with actionable guidance (reduce parallel_attempts/parallel_requests to 1),
mirroring the existing "Parallelisation limit hit" pattern.

Scope stays to "explode politely" per the issue; auto de-parallelization is
explicitly left as separate future work.

Signed-off-by: glatinone <93207632+glatinone@users.noreply.github.com>
@glatinone
glatinone force-pushed the fix/issue-361-parallel-pickling-error branch from 2febf1a to 33c88de Compare August 16, 2026 12:34
@jmartin-tech jmartin-tech self-assigned this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fail more helpfully when parallelism not supported

2 participants