`hyperopt` fails when run in parallel on macOS

[CaptSolo]

When annif hyperopt is run on macOS in parallel it fails with the following error message:

annif hyperopt nllsh-mllm-en --trials 2 --jobs 2 txt-validation-en

[...]

[W 2026-01-01 19:07:17,060] Trial 1 failed with parameters: {'min_samples_leaf': 29, 'max_leaf_nodes': 1402, 'max_samples': 0.7878340014898446} because of the following error: TypeError("'NoneType' object is not subscriptable").
Traceback (most recent call last):
  File "/Users/captsolo/Documents/Code/LNB_Annif_1.4/.venv/lib/python3.12/site-packages/optuna/study/_optimize.py", line 201, in _run_trial
    value_or_values = func(trial)
                      ^^^^^^^^^^^
  File "/Users/captsolo/Documents/Code/LNB_Annif_1.4/.venv/lib/python3.12/site-packages/annif/backend/hyperopt.py", line 67, in _objective_wrapper
    return cls.objective(trial, cls.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/captsolo/Documents/Code/LNB_Annif_1.4/.venv/lib/python3.12/site-packages/annif/backend/mllm.py", line 59, in objective
    model.fit(args["train_x"], args["train_y"])
              ~~~~^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

Full error message:

• annif_error_msg.log

Related:

• https://groups.google.com/g/annif-users/c/b3twYAn33mg

[CaptSolo]

Here is the analysis of this problem using Claude Code:

The problem is with how the multiprocessing pool is initialized. The init method is called in the main process (line 133 in hyperopt.py), but when worker processes are spawned, they don't inherit the class variable args because each process has its own memory space.

The problem is in Annif/annif/backend/hyperopt.py:143-152

When the multiprocessing pool is created, it doesn't pass an initializer function to set up the worker processes. This means that while objective_args is prepared and init() is called in the main process (line 133), the worker processes don't have access to cls.args.

I will check if its suggested changes fix the problem.

[CaptSolo]

Changing line 144 in annif/backend/hyperopt.py from:

with pool_class(jobs) as pool:

... to:
with pool_class(jobs, initializer=self._objective.init, initargs=(objective_args,)) as pool:

... fixed the issue (on macOS). I hope it does not break Annif for other operating systems.

[osma]

Hi @CaptSolo , thanks for reporting the issue.

The analysis is correct; this happens because on macOS, the multiprocessing module uses spawn instead of fork as the default start method for worker processes. The Annif code implicitly relies on fork semantics - that worker subprocesses will inherit global state, including static variables such as cls.args in this case. This is very efficient when those data structures are large, which is often the case with Annif backends. That's why we have implemented this solution (see also BaseWorker in parallel.py). But it is apparently a bit fragile. And now I see (on the multiprocessing page linked above) that on Python 3.14+, the default start method has been changed from fork to forkserver also for Linux, which I think would break this functionality, so we may have to do something in the near future for Linux as well...

The fix that you suggested should work, but I think that it may hurt performance and possibly increase overall memory usage because the initialization will now have to be done separately for each worker. There is another way to fix the problem; change MP_START_METHOD to "fork" in parallel.py and see if that helps.

I think it would be helpful if you could do some benchmarking on macOS to compare the different solutions in terms of total RAM usage and performance (total wall clock time & CPU time). I would like to know the relative performance of running a significant sized hyperopt job (a hundred or more trials) using:

1. single-process (--jobs 1)
2. multi-process (--jobs equal to your number of CPU cores) using your initializer fix
3. multi-process using MP_START_METHOD="fork"

Can you try these and report back the results? Then we would have a better idea of how to fix this, not just for macOS, but potentially other OSes as well.

[mjsuhonos]

Results for the above cases:

vocabulary: wikicore-scitecheng (930020)
hyperopt docs: wikicore-scitechend-test (65111)

1. Single-process (baseline)

time uv run annif hyperopt --metric F1@5 --trials 100 wikicore_en_ensemble_scitecheng   fulltext/scitecheng-test.tsv
---
Got best F1@5 score 0.6758 with:
---
sources=wikicore_en_scitecheng_core:0.1282,wikicore_en_scitecheng_class_science:0.2809,wikicore_en_scitecheng_class_technology:0.1298,wikicore_en_scitecheng_occupation_science:0.2802,wikicore_en_scitecheng_occupation_engineering:0.1810
---
     3363.56 real      3176.83 user       181.55 sys
         12617580544  maximum resident set size
             2080029  page reclaims
                5944  voluntary context switches
              466792  involuntary context switches
           293231280  instructions retired
           100682177  cycles elapsed
            20480408  peak memory footprint

2. multi-process using patched code

time uv run annif hyperopt --metric F1@5 --trials 100 -j 4 wikicore_en_ensemble_scitecheng   fulltext/scitecheng-test.tsv
---
Got best F1@5 score 0.6758 with:
---
sources=wikicore_en_scitecheng_core:0.1403,wikicore_en_scitecheng_class_science:0.2713,wikicore_en_scitecheng_class_technology:0.1360,wikicore_en_scitecheng_occupation_science:0.2829,wikicore_en_scitecheng_occupation_engineering:0.1696
---
     3880.11 real      5169.06 user      6893.50 sys
         12635750400  maximum resident set size
          1261907891  page reclaims
              479091  voluntary context switches
           209227476  involuntary context switches
           305307304  instructions retired
           111322340  cycles elapsed
            20955544  peak memory footprint

3. multi-process using MP_START_METHOD="fork"

time uv run annif hyperopt --metric F1@5 --trials 100 -j 4 wikicore_en_ensemble_scitecheng   fulltext/scitecheng-test.tsv
---
Got best F1@5 score 0.6758 with:
---
sources=wikicore_en_scitecheng_core:0.0903,wikicore_en_scitecheng_class_science:0.1805,wikicore_en_scitecheng_class_technology:0.0891,wikicore_en_scitecheng_occupation_science:0.4626,wikicore_en_scitecheng_occupation_engineering:0.1775
---
     1642.09 real      4209.76 user      1495.90 sys
         12627869696  maximum resident set size
           298998027  page reclaims
               10730  voluntary context switches
            35426599  involuntary context switches
           286795911  instructions retired
           106214266  cycles elapsed
            21103024  peak memory footprint

So using fork is the right solution on MacOS. This is using an M4 w/32GB, 10 CPUs. A couple of points worth noting:

• I had to use j = 4 even on a 10-CPU machine; only 4 cores are 'performance cores', the rest just add thermal load; also:
• this dataset is large, so each thread consumes 8GB of RAM; 8x5 > 32 so there is some swapping in the data, but I think the alternatives are still characterized accurately
• I was hoping fork would take advantage of copy-on-write behaviour to limit memory use, but it appears this isn't the case; not sure why

To add:

I noticed that the hyperopt results aren't always the same with multiprocessing:

wikicore_en_scitecheng_core:0.1282
wikicore_en_scitecheng_class_science:0.2809
wikicore_en_scitecheng_class_technology:0.1298
wikicore_en_scitecheng_occupation_science:0.2802
wikicore_en_scitecheng_occupation_engineering:0.1810

wikicore_en_scitecheng_core:0.1403
wikicore_en_scitecheng_class_science:0.2713
wikicore_en_scitecheng_class_technology:0.1360
wikicore_en_scitecheng_occupation_science:0.2829
wikicore_en_scitecheng_occupation_engineering:0.1696

wikicore_en_scitecheng_core:0.0903
wikicore_en_scitecheng_class_science:0.1805
wikicore_en_scitecheng_class_technology:0.0891
wikicore_en_scitecheng_occupation_science:0.4626
wikicore_en_scitecheng_occupation_engineering:0.1775

The first two converge on roughly the same weights, but the third is measurably off on 4/5 weights. I noticed similar behaviour with the threshold_ensemble as well, though in that case, both multiprocessed results were similar, and the single-process result was different.