fix(probes,generators): raise GarakException on unpicklable parallel worker payload - #2075
Open
glatinone wants to merge 3 commits into
Open
fix(probes,generators): raise GarakException on unpicklable parallel worker payload#2075glatinone wants to merge 3 commits into
glatinone wants to merge 3 commits into
Conversation
…worker payload (NVIDIA#361) When parallel_attempts/parallel_requests is enabled and a probe or generator instance holds a non-picklable attribute (e.g. a plugin dynamically loaded from a file path multiprocessing cannot re-import in the worker process), Pool.imap_unordered raises a raw _pickle.PicklingError from deep inside multiprocessing's internals instead of failing cleanly. Catch pickle.PicklingError alongside the existing OSError/errno-24 handling in Probe._execute_all and Generator.generate, and re-raise as a GarakException with actionable guidance (reduce parallel_attempts/parallel_requests to 1), mirroring the existing "Parallelisation limit hit" pattern. Scope stays to "explode politely" per the issue; auto de-parallelization is explicitly left as separate future work. Signed-off-by: glatinone <93207632+glatinone@users.noreply.github.com>
glatinone
force-pushed
the
fix/issue-361-parallel-pickling-error
branch
from
August 16, 2026 12:34
2febf1a to
33c88de
Compare
Signed-off-by: glatinone <93207632+glatinone@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
Probe._execute_all(garak/probes/base.py) andGenerator.generate(garak/generators/base.py) each catchpickle.PicklingErroraround theirmultiprocessing.Pool.imap_unorderedcall, alongside the existingOSError/errno-24 handling, and re-raise it as aGarakExceptionwith actionable guidance to reduceparallel_attempts/parallel_requeststo 1 — mirroring the existing "Parallelisation limit hit" pattern already in both functions.Root Cause & Performance Impact
Both
parallel_attempts(probes) andparallel_requests(generators) submit work to amultiprocessing.Poolviaimap_unordered. Submitting a task pickles the bound method together withself(the probe/generator instance). If that instance holds an attribute that cannot be pickled — e.g. a dynamically loaded plugin object whose defining module can't be re-imported by name in the worker process —Pool._handle_tasksraises a raw_pickle.PicklingErrorfrom deep insidemultiprocessinginternals, exactly as shown in the two tracebacks in #361 (import of module 'core.py' failed,import of module 'summarize_document.py' failed).This isn't a performance issue so much as an abrupt, unhelpful failure mode: the run aborts with an internal multiprocessing traceback instead of a clear, actionable error — the maintainer's own framing on the issue was "what should we do? at least explode /politely/, for now." This PR scopes strictly to that: converting the raw internal traceback into a clear
GarakException. Auto de-parallelization (mentioned as a follow-up in the issue) is intentionally left out of scope.Test Coverage & Verification
tests/generators/test_generators.py::test_parallel_requests_unpicklable_attribute_raises_garak_exception— reproduces thegenerate()path (parallel_requests), using a module-level stub function whose__module__is corrupted to force a genuinepickle.PicklingError(import of module '...' failed) matching the issue's exact failure shape, rather than relying on a local-closureAttributeError.tests/test_internal_structures.py::test_execute_all_unpicklable_probe_raises_garak_exception— same technique for theProbe._execute_allpath (parallel_attempts)._pickle.PicklingError: Can't pickle ...: import of module '...' failedtraceback from fail more helpfully when parallelism not supported #361, and to pass after the fix (assertingGarakExceptionis raised instead).pytest— 1489 passed (only the 2 directly touched test files plustests/test_config.py -k parallelwere run against both pre- and post-fix code to isolate signal). One pre-existing failure (test_probe_metadata[probes.audio.AudioAchillesHeel], missing optionalsoundfile/librosaextras) and 3 pre-existing errors (openai_compat_mocksfixture not resolving whentest_openai.pyis invoked alongside certain other test paths) were confirmed present identically onmainbefore this change — unrelated to this diff.black --checkandpylintshow no new issues introduced by this diff (both tools flag two pre-existing, unrelated formatting/lint items onmainin the touched files, outside the lines this PR changes).Closes #361