diff --git a/docs/source/api.rst b/docs/source/api.rst index 4c8a5999a..01f22822f 100644 --- a/docs/source/api.rst +++ b/docs/source/api.rst @@ -564,6 +564,8 @@ Utils set_log_level setup_seed set_download_dir + set_download_provider + get_download_provider make_process_pipelines Benchmark diff --git a/docs/source/whats_new.rst b/docs/source/whats_new.rst index 615a7da3c..c8ad8d45e 100644 --- a/docs/source/whats_new.rst +++ b/docs/source/whats_new.rst @@ -41,6 +41,9 @@ Enhancements - Skip zip extraction in :class:`moabb.datasets.GuttmannFlury2025` when files are already extracted, with ``/scratch`` fallback for NFS filesystems on compute nodes (by `Bruno Aristimunha`_) - Re-enable auto-execution of the Riemannian Artifact Rejection tutorial (``examples/advanced_examples/plot_riemannian_artifact_rejection.py``) now that pyRiemann 0.11 is on PyPI with per-potato metrics and ``method_combination`` support on ``PotatoField`` (by `Bruno Aristimunha`_) - Use NEMAR as the default download source for datasets with an assigned ``nemar_id``, while preserving existing dataset-specific downloaders as a fallback (by `Bruno Aristimunha`_). +- :meth:`moabb.datasets.base.BaseDataset.download` now fetches NEMAR's ``sourcedata/`` — the **original pre-BIDS distribution**, byte-identical to what the upstream host serves and stored under the same upstream filenames — instead of the deposit's BIDS copy. The BIDS copy is a re-encoding whose events and session/run labels differ from what each dataset's own loader produces, so substituting it would silently change results rather than only change where the bytes come from; ``sourcedata/`` changes the source without touching the science. Loading is unaffected: ``get_data`` still runs each dataset's own parser. Falls back to the dataset's upstream downloader when a deposit publishes no ``sourcedata/`` (by `Bruno Aristimunha`_). +- Serve the **original, pre-BIDS distribution** from NEMAR through the new :meth:`moabb.datasets.base.BaseDataset.sourcedata_path`. NEMAR deposits republish the files exactly as the authors distributed them under ``sourcedata/``, keeping the upstream filenames, so NEMAR can now stand in for upstream hosts that are slow, rate-limited, behind a bot gate, or retired. Datasets may set ``nemar_sourcedata_include`` (e.g. ``"sourcedata/subject_{subject:02d}.*"``) to fetch a single subject instead of the whole tree; without it the full ``sourcedata/`` is downloaded, because that tree keeps the upstream layout rather than a BIDS one. Backed by :func:`moabb.datasets.download.nemar_sourcedata_dl` (by `Bruno Aristimunha`_). +- Add :func:`moabb.set_download_provider` / :func:`moabb.get_download_provider` to pin where MOABB fetches data from: ``"auto"`` (default — NEMAR first, upstream fallback), ``"nemar"`` (NEMAR only; a failure is raised rather than silently falling back to a host the caller opted out of), or ``"upstream"`` (never use NEMAR). Also settable per run via the ``MOABB_DOWNLOAD_PROVIDER`` environment variable, which takes precedence over the stored config (by `Bruno Aristimunha`_). - Add :class:`moabb.datasets.preprocessing.EuclideanAlignment`, a trial-level Euclidean Alignment transformer (He & Wu 2020; Junqueira et al. 2024) that whitens each trial by the inverse square root of the Euclidean mean covariance to remove per-domain covariance shift before a (deep) model sees the data. Inductive and leakage-free by default (``fit`` learns the reference from training trials, ``transform`` re-applies it to unseen trials); ``fit_transform`` gives the transductive, per-recording form. Accepts an :class:`mne.BaseEpochs` or an ``(n_trials, n_channels, n_times)`` ndarray, uses a shrinkage covariance estimator (``"lwf"``) for robustness, and adds no new dependency (``pyriemann >= 0.11`` is already required). Distinct from :class:`pyriemann.transfer.TLCenter`, which recenters covariance *matrices* (:gh:`1108` by `Bruno Aristimunha`_). - Add an ``n_jobs`` parameter to :meth:`moabb.paradigms.base.BaseParadigm.get_data` and :meth:`moabb.datasets.base.BaseDataset.get_data` to load and preprocess subjects in parallel with :class:`joblib.Parallel`. Per-subject processing (reading, filtering, resampling, epoching) is independent, so this gives a near-linear speedup on datasets with many subjects, with identical numerical results. moabb's own patches to the shared BIDS cache files (``participants.tsv``/``.json``, ``dataset_description.json``) now take the mne-bids cross-process file lock, so parallel caching stays consistent (:gh:`1124` by `Bruno Aristimunha`_). - Drive cross-validation folds with any stock scikit-learn cross-validator passed as ``cv_class``, controlled by a ``groups`` argument — a metadata column name, a list of column names (compound key, e.g. ``["subject", "session"]``), or a callable ``metadata -> array`` — together with callable ``cv_kwargs`` resolved against the metadata (e.g. ``cv_class=PredefinedSplit`` with a ``test_fold`` callable to target a single fold). ``groups`` is exposed on :class:`moabb.evaluations.WithinSessionEvaluation`, :class:`moabb.evaluations.WithinSubjectEvaluation`, :class:`moabb.evaluations.CrossSessionEvaluation` and :class:`moabb.evaluations.CrossSubjectEvaluation` and threaded to their splitters; each splitter keeps its default grouping (``"subject"`` / ``"session"`` / labels) when ``groups`` is ``None``. :class:`moabb.evaluations.splitters.CrossDatasetSplitter` gains ``groups`` (its ``group_column`` argument is now a deprecated alias) (:gh:`1104` by `Bruno Aristimunha`_). diff --git a/moabb/__init__.py b/moabb/__init__.py index 562793913..0a2694574 100644 --- a/moabb/__init__.py +++ b/moabb/__init__.py @@ -2,4 +2,11 @@ __version__ = "1.5.0dev0" from .benchmark import benchmark -from .utils import make_process_pipelines, set_download_dir, set_log_level, setup_seed +from .utils import ( + get_download_provider, + make_process_pipelines, + set_download_dir, + set_download_provider, + set_log_level, + setup_seed, +) diff --git a/moabb/datasets/base.py b/moabb/datasets/base.py index dec5acb47..d3118500d 100644 --- a/moabb/datasets/base.py +++ b/moabb/datasets/base.py @@ -29,8 +29,9 @@ _interface_map, get_bids_root, ) -from moabb.datasets.download import NemarDownloadError, nemar_dl +from moabb.datasets.download import NemarDownloadError, nemar_dl, nemar_sourcedata_dl from moabb.datasets.preprocessing import FixedPipeline, SetRawAnnotations +from moabb.utils import get_download_provider if TYPE_CHECKING: @@ -926,12 +927,21 @@ def download( This function is only useful to download all the dataset at once. + When the dataset declares a :attr:`nemar_id` and the download provider + is not ``"upstream"``, the files come from NEMAR's ``sourcedata/`` -- + the original pre-BIDS distribution, byte-identical to what the upstream + host serves. On any NEMAR failure this falls back to the dataset's own + downloader with a warning (unless the provider is pinned to + ``"nemar"``). See :meth:`sourcedata_path` and + :func:`moabb.set_download_provider`. Parameters ---------- subject_list : list of int | None List of subjects id to download, if None all subjects - are downloaded. + are downloaded. On the NEMAR path each subject is resolved through + the deposit's ``sourcedata_provenance.json``; deposits enriched + before that manifest recorded subjects fetch the whole tree. path : None | str Location of where to look for the data storing location. If None, the environment variable or config parameter @@ -944,33 +954,52 @@ def download( update_path : bool | None If True, set the MNE_DATASETS_(dataset)_PATH in mne-python config to the given path. If None, the user is prompted. + Not used on the NEMAR path. accept: bool - Accept licence term to download the data, if any. Default: False + Accept licence term to download the data, if any. Default: False. + Only relevant to the dataset's own downloader; NEMAR mirrors are + already public. verbose : bool, str, int, or None If not None, override default verbose level (see :func:`mne.verbose`). """ if subject_list is None: subject_list = self.subject_list + provider = get_download_provider() + if provider == "nemar" and self.nemar_id is None: + raise NemarDownloadError( + f"Download provider is pinned to 'nemar' but {self.code} declares " + "no nemar_id, so it cannot be fetched from NEMAR. Use " + "moabb.set_download_provider('auto') to allow the upstream " + "downloader for datasets NEMAR does not mirror." + ) + # Prefer NEMAR's `sourcedata/` -- the ORIGINAL pre-BIDS distribution, + # byte-identical to what the upstream host serves and stored under the + # same upstream filenames. Deliberately not the deposit's BIDS copy: + # that is a re-encoding whose events and session/run labels differ from + # what each dataset's own loader produces, so substituting it would + # silently change results rather than just change where bytes come from. + if self.nemar_id is not None and provider != "upstream": + try: + self._download_nemar_sourcedata( + subject_list=subject_list, + path=path, + force_update=force_update, + verbose=verbose, + ) + return + except NemarDownloadError as exc: + if provider == "nemar": + raise + warnings.warn( + f"Could not download {self.code} sourcedata from NEMAR " + f"({self.nemar_id}); falling back to the dataset data_path " + f"downloader. Original error: {exc}", + RuntimeWarning, + stacklevel=2, + ) + for subject in subject_list: - if self.nemar_id is not None: - try: - self._download_nemar( - subject=subject, - path=path, - force_update=force_update, - update_path=update_path, - verbose=verbose, - ) - continue - except NemarDownloadError as exc: - warnings.warn( - f"Could not download {self.code} from NEMAR ({self.nemar_id}); " - "falling back to the dataset data_path downloader. " - f"Original error: {exc}", - RuntimeWarning, - stacklevel=2, - ) # check if accept is needed sig = signature(self.data_path) if "accept" in [str(p) for p in sig.parameters]: @@ -1050,6 +1079,77 @@ def _download_nemar( **(self.nemar_bids_filters or {}), ) + def _download_nemar_sourcedata( + self, subject_list=None, path=None, force_update=False, verbose=None + ): + """Fetch the original pre-BIDS distribution for a set of subjects. + + Each subject is resolved through the deposit's provenance manifest + rather than by pattern-matching the layout, which across the catalogue + is genuinely arbitrary -- subjects appear as letters, as ``sub1``, as + ``S10_Session_1``, split across two path components, or not at all. + + Raises + ------ + moabb.datasets.download.NemarDownloadError + If nemar-py is unavailable, a download fails, or the deposit + publishes no ``sourcedata/``. + """ + for subject in subject_list: + self.sourcedata_path( + subject=subject, path=path, force_update=force_update, verbose=verbose + ) + + def sourcedata_path(self, subject=None, path=None, force_update=False, verbose=None): + """Get the dataset's *original* pre-BIDS distribution from NEMAR. + + Where :meth:`data_path` fetches the original files from the upstream + host, this fetches the copy NEMAR mirrors under ``sourcedata/``. The + files keep their upstream names, so the two are interchangeable in + content -- but NEMAR stays reachable when the upstream host is slow, + rate-limited, behind a bot gate, or retired. + + Parameters + ---------- + subject : int | str | None + Restrict the download to one subject, resolved through the + deposit's ``sourcedata_provenance.json``. Deposits enriched before + that manifest recorded subjects fall back to the whole tree with a + warning. When ``None`` the whole tree is fetched. + path : None | str + Base path where MOABB stores datasets. + force_update : bool + Re-fetch even when a local copy is present. + verbose : bool, str, int, or None + If not None, override default verbose level. + + Returns + ------- + str + Local path to the ``sourcedata`` directory. + + Raises + ------ + ValueError + If the dataset declares no ``nemar_id``. + moabb.datasets.download.NemarDownloadError + If nemar-py is unavailable, the download fails, or the deposit + publishes no ``sourcedata/``. + """ + if self.nemar_id is None: + raise ValueError( + f"{self.code} declares no nemar_id, so its original distribution " + "cannot be fetched from NEMAR." + ) + return nemar_sourcedata_dl( + self.nemar_id, + self.code, + path=path, + force_update=force_update, + subject=subject, + verbose=verbose, + ) + def _get_single_subject_data_from_nemar(self, subject, path=None): """Download and load one subject from the NEMAR BIDS dataset.""" try: diff --git a/moabb/datasets/download.py b/moabb/datasets/download.py index 6ffbc0f07..6b22da162 100644 --- a/moabb/datasets/download.py +++ b/moabb/datasets/download.py @@ -8,6 +8,7 @@ import logging import os import os.path as osp +import warnings from pathlib import Path from urllib.parse import urlparse @@ -302,6 +303,185 @@ def nemar_dl( return str(target_dir) +#: Name of the manifest every enriched deposit ships inside ``sourcedata/``. +SOURCEDATA_PROVENANCE = "sourcedata/sourcedata_provenance.json" + + +def _sourcedata_files_for_subject(target_dir, nemar_id, subject, force_update): + """Return the ``sourcedata/`` paths belonging to one subject, or ``None``. + + Reads the deposit's ``sourcedata_provenance.json``, which records every + file's upstream name, size, SHA-256 and -- for deposits enriched after the + subject field was added -- which subject it came from. + + This is deliberately data-driven rather than pattern-driven. ``sourcedata/`` + preserves whatever layout the authors published, and across the catalogue + that is genuinely arbitrary: subjects appear as ``sub-A`` (letters), + ``subject_01``, ``sub1``, ``S10_Session_1``, ``session1/s1/sess01_subj01`` + (subject split across two components), ``subject_01_PC`` / ``_VR`` (two + files per subject) -- and in at least one deposit the path carries no + subject at all, just an upstream file id. No glob template spans that, so + asking the deposit is the only approach that generalises. + + Returns + ------- + list of str | None + Include patterns for that subject, or ``None`` when the provenance + predates the subject field, in which case the caller should fall back + to fetching the whole tree. + """ + provenance = target_dir / SOURCEDATA_PROVENANCE + if force_update or not provenance.is_file(): + try: + nemar.download( + dataset=nemar_id, + target_dir=target_dir, + scope="sourcedata", + include=SOURCEDATA_PROVENANCE, + trust_existing=not force_update, + ) + except (NemarError, OSError, ConnectionError, TimeoutError, ValueError) as exc: + raise NemarDownloadError( + f"Could not read the sourcedata manifest for {nemar_id}." + ) from exc + try: + record = json.loads(provenance.read_text(encoding="utf-8")) + except (OSError, ValueError) as exc: + raise NemarDownloadError( + f"NEMAR dataset {nemar_id} has an unreadable sourcedata manifest." + ) from exc + + entries = record.get("files") or [] + if not any("subject" in entry for entry in entries): + return None + wanted = str(subject) + files = [entry["file"] for entry in entries if str(entry.get("subject")) == wanted] + if not files: + raise NemarDownloadError( + f"NEMAR dataset {nemar_id} lists no sourcedata for subject " + f"{subject!r}. Known subjects: " + f"{sorted({str(e.get('subject')) for e in entries if e.get('subject')})}." + ) + return [f"sourcedata/{name}" for name in files] + + +@verbose +def nemar_sourcedata_dl( + nemar_id, + dataset_code, + path=None, + force_update=False, + subject=None, + include=None, + verbose=None, +): + """Download a NEMAR dataset's ``sourcedata/`` and return its local root. + + ``sourcedata/`` holds the **original, pre-BIDS distribution** as published + by the dataset authors, stored under the upstream filenames. It is + therefore a drop-in mirror of whatever :meth:`BaseDataset.data_path` would + otherwise fetch from the upstream host — useful when that host is slow, + rate-limited, behind a bot gate, or gone. + + Parameters + ---------- + nemar_id : str + NEMAR dataset identifier. + dataset_code : str + MOABB dataset code used to choose the local dataset directory. + path : None | str + Base path where MOABB stores datasets. + force_update : bool + Re-fetch even when a local copy is already present. + subject : int | str | None + Restrict the download to one subject, resolved through the deposit's + ``sourcedata_provenance.json`` rather than by guessing at the layout. + Falls back to fetching the whole tree, with a warning, when the deposit + was enriched before that manifest recorded subjects. + include : str | list of str | None + Explicit path glob(s) relative to the dataset root, for callers that + know the layout. Overrides ``subject``. Without either, the whole + ``sourcedata/`` tree is downloaded. + verbose : bool, str, int, or None + If not None, override default verbose level. + + Returns + ------- + str + Local path to the downloaded ``sourcedata`` directory. + + Raises + ------ + NemarDownloadError + If nemar-py is unavailable, the download fails, or the deposit + publishes no ``sourcedata/`` at all. + """ + # Keyed on the NEMAR id, not the MOABB code: one deposit backs several + # classes (nm000132 is shared by all seven ErpCore2021 subclasses, nm000250 + # by four Dreyer2023 ones), and a per-code path would download the same + # tree once per class. + root = Path(get_dataset_path(dataset_code, path)) + target_dir = root / "NEMAR" / nemar_id + + if include is None and subject is not None: + include = _sourcedata_files_for_subject( + target_dir, nemar_id, subject, force_update + ) + if include is None: + # warnings.warn, not mne.utils.warn: the latter emits a given + # warning once per session, which makes it order-dependent for + # anything asserting on it -- and matches what base.py already does + # for the sibling upstream fallback. + warnings.warn( + f"NEMAR dataset {nemar_id} was enriched before its sourcedata " + "manifest recorded subjects, so one subject cannot be selected; " + "downloading the whole sourcedata/ tree instead.", + RuntimeWarning, + stacklevel=2, + ) + + kwargs = {"include": include} if include is not None else {} + sourcedata_dir = target_dir / "sourcedata" + what = f"sourcedata/ matching {include!r}" if include else "sourcedata/" + missing = ( + f"NEMAR dataset {nemar_id} published no {what} -- the original " + "distribution is not mirrored there." + ) + try: + nemar.download( + dataset=nemar_id, + target_dir=target_dir, + scope="sourcedata", + trust_existing=not force_update, + **kwargs, + ) + except (NemarError, OSError, ConnectionError, TimeoutError, ValueError) as exc: + # nemar-py raises SelectionError (a NemarError) when the scope or the + # include glob matches nothing, so "this deposit has no sourcedata/" and + # "this subject is not in it" both arrive here rather than as a + # successful empty download. + raise NemarDownloadError(missing) from exc + + # Ask about the subset this call requested, not the whole tree. `download()` + # calls this once per subject into a shared directory, so "the directory has + # something in it" is true from the second subject on even when this + # subject's fetch matched nothing -- and a before/after diff of the tree + # would walk every file twice per subject to find that out. + if include is not None: + patterns = [include] if isinstance(include, str) else include + fetched = any(next(target_dir.glob(p), None) is not None for p in patterns) + else: + # The manifest does not count: a deposit that publishes only its + # provenance has no original distribution to offer. + manifest = target_dir / SOURCEDATA_PROVENANCE + fetched = sourcedata_dir.is_dir() and any( + entry != manifest for entry in sourcedata_dir.rglob("*") + ) + if not fetched: + raise NemarDownloadError(missing) + return str(sourcedata_dir) + + # This function is from https://github.com/cognoma/figshare (BSD-3-Clause) def fs_issue_request(method, url, headers, data=None, binary=False): """Wrapper for HTTP request. diff --git a/moabb/tests/test_datasets.py b/moabb/tests/test_datasets.py index 2b4058ae4..c588d69a0 100644 --- a/moabb/tests/test_datasets.py +++ b/moabb/tests/test_datasets.py @@ -292,26 +292,29 @@ def test_all_datasets_have_valid_nemar_id(self, dataset): assert re.fullmatch(NEMAR_ID_PATTERN, nemar_id) def test_download_prefers_nemar(self, monkeypatch, tmp_path): + """``download()`` takes the original distribution from NEMAR. + + It fetches ``sourcedata/`` rather than the deposit's BIDS copy: the + BIDS copy is a re-encoding whose events and session/run labels differ + from what each dataset's own loader produces. + """ dataset = FakeDataset(n_subjects=1) dataset.nemar_id = "nm000001" calls = [] - def nemar_dl(*args, **kwargs): + def nemar_sourcedata_dl(*args, **kwargs): calls.append((args, kwargs)) - return str(tmp_path / "nemar") + return str(tmp_path / "nemar" / "sourcedata") - monkeypatch.setattr("moabb.datasets.base.nemar_dl", nemar_dl) + monkeypatch.setattr( + "moabb.datasets.base.nemar_sourcedata_dl", nemar_sourcedata_dl + ) dataset.download(subject_list=[1], path=tmp_path) assert calls == [ ( ("nm000001", dataset.code), - { - "path": tmp_path, - "force_update": False, - "subject": "1", - "verbose": None, - }, + {"path": tmp_path, "force_update": False, "subject": 1, "verbose": None}, ) ] @@ -320,7 +323,7 @@ def test_download_falls_back_from_nemar(self, monkeypatch, tmp_path): dataset.nemar_id = "nm000001" fallback_calls = [] - def nemar_dl(*args, **kwargs): + def nemar_sourcedata_dl(*args, **kwargs): raise NemarDownloadError("NEMAR unavailable") def data_path( @@ -328,7 +331,9 @@ def data_path( ): fallback_calls.append((subject, path, force_update, update_path, verbose)) - monkeypatch.setattr("moabb.datasets.base.nemar_dl", nemar_dl) + monkeypatch.setattr( + "moabb.datasets.base.nemar_sourcedata_dl", nemar_sourcedata_dl + ) monkeypatch.setattr(dataset, "data_path", data_path) with pytest.warns(RuntimeWarning, match="falling back"): diff --git a/moabb/tests/test_download.py b/moabb/tests/test_download.py index 482b5c5ad..bc29f530c 100644 --- a/moabb/tests/test_download.py +++ b/moabb/tests/test_download.py @@ -1,9 +1,11 @@ """Tests to ensure that datasets download correctly using pytest.""" import inspect +import json import os import sys -from pathlib import Path +from pathlib import Path, PurePath +from types import SimpleNamespace import mne import pytest @@ -13,7 +15,9 @@ import moabb.datasets.brandl2020 as brandl import moabb.datasets.download as dl import moabb.datasets.romani_bf2025_erp as romani +from moabb.datasets import base as base_module from moabb.datasets.bbci_eeg_fnirs import BaseShin2017 +from moabb.datasets.fake import FakeDataset from moabb.datasets.utils import dataset_list from moabb.utils import set_download_dir @@ -55,7 +59,10 @@ def test_nemar_download_equivalence(dl_data, tmp_path, dataset_class): f"{dataset_class.__name__} has no NEMAR dataset ID" ) subject = dataset.subject_list[0] - dataset.download(subject_list=[subject], path=tmp_path) + # `download()` now fetches NEMAR's `sourcedata/` (the original pre-BIDS + # distribution), so drive the BIDS copy explicitly to keep covering the + # BIDS loader below. + dataset._download_nemar(subject=subject, path=tmp_path) bids_root = tmp_path / f"MNE-{dataset.code.lower()}-data" / dataset.nemar_id description = bids_root / "dataset_description.json" @@ -357,6 +364,317 @@ def fake_fetch_dataset( assert all("config_key" not in call["dataset_params"] for call in calls) +# -------------------------------------------------------------------------- +# sourcedata/: NEMAR as a provider for the ORIGINAL, pre-BIDS distribution +# -------------------------------------------------------------------------- + + +class _FakeNemar: + """Stand-in for the ``nemar`` module that records calls and writes files. + + A recording function would be enough for the argument-forwarding checks, + but the "published no sourcedata/" guard inspects the target directory, so + the fake has to actually put files on disk. It honours ``include`` so a + per-subject call that matches nothing behaves like the real client. + """ + + def __init__(self, files=("sourcedata/subject_01.mat",), subjects=None): + self.files = files + # {relative path -> subject}; None means the deposit predates the + # subject field, which is the state every already-enriched deposit is + # in today. + self.subjects = subjects + self.calls = [] + + def download(self, **kwargs): + self.calls.append(kwargs) + target = Path(kwargs["target_dir"]) + include = kwargs.get("include") + patterns = [include] if isinstance(include, str) else include + for relative in self._served(): + if patterns is not None and not any( + PurePath(relative).match(p) for p in patterns + ): + continue + destination = target / relative + destination.parent.mkdir(parents=True, exist_ok=True) + if relative == dl.SOURCEDATA_PROVENANCE: + destination.write_text(json.dumps(self._provenance())) + else: + destination.write_bytes(b"x") + + def _served(self): + return (dl.SOURCEDATA_PROVENANCE, *self.files) + + def _provenance(self): + entries = [] + for relative in self.files: + name = relative[len("sourcedata/") :] + entry = {"file": name, "bytes": 1, "sha256": "0" * 64} + if self.subjects is not None: + entry["subject"] = self.subjects[relative] + entries.append(entry) + return {"dataset": "nm000341", "files": entries} + + +@pytest.fixture +def fake_nemar(monkeypatch): + """Install a :class:`_FakeNemar` in place of the real client.""" + + def _install(files=("sourcedata/subject_01.mat",), subjects=None): + fake = _FakeNemar(files, subjects) + monkeypatch.setattr(dl, "nemar", fake) + return fake + + return _install + + +@pytest.fixture +def nemar_calls(monkeypatch, tmp_path): + """Record what ``base`` asks of NEMAR, for both entry points. + + Both are patched so a test can assert the BIDS copy was *not* fetched, + which is the whole point of preferring ``sourcedata/``. + """ + calls = SimpleNamespace(sourcedata=[], bids=[], path=tmp_path) + monkeypatch.setattr(base_module, "get_download_provider", lambda: "auto") + monkeypatch.setattr( + base_module, + "nemar_sourcedata_dl", + lambda *a, **k: calls.sourcedata.append(k) or str(tmp_path), + ) + monkeypatch.setattr( + base_module, "nemar_dl", lambda *a, **k: calls.bids.append(k) or str(tmp_path) + ) + return calls + + +@pytest.mark.parametrize( + ("kwargs", "expected", "absent"), + [ + pytest.param( + {}, + {"dataset": "nm000341", "scope": "sourcedata"}, + ("include",), + id="sourcedata-scope-without-include", + ), + pytest.param( + {"include": "sourcedata/subject_01.*"}, + {"include": "sourcedata/subject_01.*"}, + (), + id="include-forwarded", + ), + pytest.param( + {"force_update": True}, + {"trust_existing": False}, + (), + id="force-update-distrusts-existing", + ), + ], +) +def test_nemar_sourcedata_dl_forwards_arguments( + tmp_path, fake_nemar, kwargs, expected, absent +): + """The scope must be ``sourcedata`` -- ``raw`` would fetch the BIDS copy. + + ``include`` is omitted rather than passed as ``None``, which would filter + everything out instead of selecting everything. + """ + fake = fake_nemar() + + dl.nemar_sourcedata_dl("nm000341", "Cattan2019-PHMD", path=tmp_path, **kwargs) + + call = fake.calls[0] + assert expected.items() <= call.items() + assert not set(absent) & set(call) + + +def test_nemar_sourcedata_dl_returns_the_sourcedata_directory(tmp_path, fake_nemar): + fake_nemar() + + root = dl.nemar_sourcedata_dl("nm000341", "Cattan2019-PHMD", path=tmp_path) + + assert Path(root).name == "sourcedata" + assert Path(root).is_dir() + + +@pytest.mark.parametrize( + ("files", "include", "match"), + [ + pytest.param((), None, "no sourcedata/", id="deposit-has-none"), + pytest.param( + ("sourcedata/subject_01.mat",), + "sourcedata/subject_99.*", + "no sourcedata/", + id="subject-not-in-deposit", + ), + ], +) +def test_nemar_sourcedata_dl_raises_when_nothing_was_fetched( + tmp_path, fake_nemar, files, include, match +): + """An empty result must fail here, not surface later as an empty cache. + + The second case matters because ``download()`` calls this once per subject + into a shared directory, so "the directory is non-empty" is true from the + second subject on even when this subject matched nothing. + """ + fake_nemar(files) + + with pytest.raises(dl.NemarDownloadError, match=match): + dl.nemar_sourcedata_dl("nm000115", "Zhou2016", path=tmp_path, include=include) + + +def test_sourcedata_path_requires_a_nemar_id(): + """A dataset NEMAR does not mirror cannot serve its original files.""" + dataset = FakeDataset() + dataset.nemar_id = None + + with pytest.raises(ValueError, match="declares no nemar_id"): + dataset.sourcedata_path() + + +@pytest.mark.parametrize( + ("subject", "expected"), + [ + pytest.param(1, ["sourcedata/sub-A/eeg/a.eeg"], id="letters"), + pytest.param(2, ["sourcedata/session1/s2/x.mat"], id="nested-session-dir"), + pytest.param( + 3, + ["sourcedata/S3_Session_1.mat", "sourcedata/S3_Session_2.mat"], + id="two-files-one-subject", + ), + ], +) +def test_sourcedata_selects_a_subject_from_the_provenance( + tmp_path, fake_nemar, subject, expected +): + """Selection is data-driven, so any upstream layout works. + + These three shapes are all real: ``sub-A`` (nm000193 labels subjects with + letters), ``session1/s2/`` (nm000273 splits the subject across two path + components), and two files per subject (nm000339). No glob template spans + them, and at least one deposit encodes no subject in the path at all -- + which is why the deposit is asked instead of guessed at. + """ + files = { + "sourcedata/sub-A/eeg/a.eeg": "1", + "sourcedata/session1/s2/x.mat": "2", + "sourcedata/S3_Session_1.mat": "3", + "sourcedata/S3_Session_2.mat": "3", + } + fake = fake_nemar(tuple(files), subjects=files) + + dl.nemar_sourcedata_dl("nm000341", "Cattan2019-PHMD", path=tmp_path, subject=subject) + + # First call fetches the manifest, second fetches that subject's files. + assert fake.calls[0]["include"] == dl.SOURCEDATA_PROVENANCE + assert sorted(fake.calls[-1]["include"]) == sorted(expected) + + +def test_sourcedata_reports_an_unknown_subject_with_the_known_ones(tmp_path, fake_nemar): + """A subject the deposit does not carry must say so, and say what it has.""" + files = {"sourcedata/sub-A/eeg/a.eeg": "1"} + fake_nemar(tuple(files), subjects=files) + + with pytest.raises(dl.NemarDownloadError, match="lists no sourcedata for subject"): + dl.nemar_sourcedata_dl("nm000341", "Cattan2019-PHMD", path=tmp_path, subject=99) + + +def test_sourcedata_falls_back_to_the_whole_tree_without_subject_records( + tmp_path, fake_nemar +): + """Deposits enriched before the subject field still work, with a warning. + + Every deposit is in this state today, so this is the live path until the + manifests are regenerated -- it must degrade to the whole tree rather than + fail or silently fetch nothing. + """ + fake = fake_nemar(("sourcedata/subject_01.mat",), subjects=None) + + with pytest.warns(RuntimeWarning, match="before its sourcedata manifest"): + dl.nemar_sourcedata_dl("nm000341", "Cattan2019-PHMD", path=tmp_path, subject=1) + + assert "include" not in fake.calls[-1] + + +@pytest.mark.parametrize( + ("n_subjects", "subject_list", "expected"), + [ + pytest.param(2, None, [1, 2], id="all-subjects"), + pytest.param(3, [1, 2], [1, 2], id="explicit-subject-list"), + ], +) +def test_download_selects_sourcedata_not_the_bids_copy( + nemar_calls, n_subjects, subject_list, expected +): + """``download()`` pulls the ORIGINAL distribution, never the BIDS copy. + + The BIDS copy is a re-encoding whose events and session/run labels differ + from what each dataset's own loader produces, so substituting it would + change results rather than only change where the bytes come from. + """ + dataset = FakeDataset(n_subjects=n_subjects) + dataset.nemar_id = "nm000341" + + dataset.download(subject_list=subject_list, path=nemar_calls.path) + + assert [call.get("subject") for call in nemar_calls.sourcedata] == expected + assert nemar_calls.bids == [] + + +def test_download_provider_upstream_skips_nemar(nemar_calls, monkeypatch): + """``upstream`` must not touch NEMAR even when a nemar_id exists.""" + monkeypatch.setattr(base_module, "get_download_provider", lambda: "upstream") + dataset = FakeDataset(n_subjects=1) + dataset.nemar_id = "nm000341" + + dataset.download(path=nemar_calls.path) + + assert nemar_calls.sourcedata == [] + assert nemar_calls.bids == [] + + +def test_download_provider_nemar_does_not_fall_back(monkeypatch, tmp_path): + """Pinned to NEMAR, a failure surfaces instead of silently going upstream.""" + monkeypatch.setattr(base_module, "get_download_provider", lambda: "nemar") + + def _boom(*args, **kwargs): + raise dl.NemarDownloadError("nemar is down") + + monkeypatch.setattr(base_module, "nemar_sourcedata_dl", _boom) + dataset = FakeDataset(n_subjects=1) + dataset.nemar_id = "nm000341" + + with pytest.raises(dl.NemarDownloadError, match="nemar is down"): + dataset.download(path=tmp_path) + + +def test_download_provider_nemar_rejects_unmirrored_dataset(monkeypatch, tmp_path): + """Pinning to NEMAR must explain why an unmirrored dataset cannot work.""" + monkeypatch.setattr(base_module, "get_download_provider", lambda: "nemar") + dataset = FakeDataset(n_subjects=1) + dataset.nemar_id = None + + with pytest.raises(dl.NemarDownloadError, match="pinned to 'nemar'"): + dataset.download(path=tmp_path) + + +def test_download_falls_back_to_upstream_when_sourcedata_missing(monkeypatch, tmp_path): + """A deposit without sourcedata/ must not block the download.""" + monkeypatch.setattr(base_module, "get_download_provider", lambda: "auto") + + def _no_sourcedata(*args, **kwargs): + raise dl.NemarDownloadError("published no sourcedata/") + + monkeypatch.setattr(base_module, "nemar_sourcedata_dl", _no_sourcedata) + dataset = FakeDataset(n_subjects=1) + dataset.nemar_id = "nm000115" + + with pytest.warns(RuntimeWarning, match="falling back"): + dataset.download(path=tmp_path) + + # -- Brandl2020 / DepositOnce ------------------------------------------------- # # DepositOnce answers HTTP 200 with an HTML app shell for files it cannot serve, diff --git a/moabb/utils.py b/moabb/utils.py index 3573ac7f4..ddd5d3890 100644 --- a/moabb/utils.py +++ b/moabb/utils.py @@ -258,6 +258,100 @@ def set_download_dir(path): _clear_legacy_dataset_paths(old_path, path) +def _set_moabb_config(key, value, **kwargs): + """Write a MOABB-owned config key without MNE's unknown-key warning. + + MNE warns on every write to a key outside its own vocabulary, and offers no + flag to suppress it, so the filter is the only lever. Written once here + rather than at each call site. + """ + with warnings.catch_warnings(): + warnings.filterwarnings( + "ignore", + message=rf'Setting non-standard config type: "{re.escape(key)}"', + category=RuntimeWarning, + ) + set_config(key, value, **kwargs) + + +#: Where MOABB may fetch dataset files from. +#: +#: ``"auto"`` try NEMAR first for datasets that declare a ``nemar_id``, +#: fall back to the dataset's own upstream downloader. +#: ``"nemar"`` NEMAR only -- do not fall back to the upstream host. +#: ``"upstream"`` never use NEMAR; always use the dataset's own downloader. +DOWNLOAD_PROVIDERS = ("auto", "nemar", "upstream") + + +def set_download_provider(provider): + """Choose where MOABB fetches dataset files from. + + Many datasets are mirrored on NEMAR, which also republishes the original + pre-BIDS distribution under ``sourcedata/``. Pinning the provider to + ``"nemar"`` makes downloads reproducible and immune to upstream hosts that + are slow, rate-limited, behind a bot gate, or retired -- at the cost of + failing outright, rather than silently falling back, when NEMAR cannot + serve a dataset. + + Parameters + ---------- + provider : str | None + One of ``"auto"`` (default), ``"nemar"``, or ``"upstream"``. Passing + ``None`` restores the default. + + Raises + ------ + ValueError + If ``provider`` is not a recognised value. + + Notes + ----- + The choice is stored in the MNE config as ``MOABB_DOWNLOAD_PROVIDER`` and + can also be set for a single run via the environment variable of the same + name, which takes precedence. + """ + if provider is not None: + normalized = str(provider).lower() + if normalized not in DOWNLOAD_PROVIDERS: + raise ValueError( + f"Unknown download provider {provider!r}; " + f"expected one of {', '.join(DOWNLOAD_PROVIDERS)}." + ) + else: + normalized = None + _set_moabb_config("MOABB_DOWNLOAD_PROVIDER", normalized) + + +def get_download_provider(): + """Return the active download provider. + + The environment variable ``MOABB_DOWNLOAD_PROVIDER`` wins over the stored + MNE config value, so a single run can override a persisted preference. + An unrecognised value falls back to ``"auto"`` with a warning rather than + raising, so a stale config cannot break every download. + + Returns + ------- + str + One of :data:`DOWNLOAD_PROVIDERS`. + """ + # get_config consults os.environ first, so an env var set for one run wins + # over the stored preference without a second lookup here. + provider = get_config("MOABB_DOWNLOAD_PROVIDER") + if not provider: + return "auto" + normalized = str(provider).lower() + if normalized not in DOWNLOAD_PROVIDERS: + log.warning( + "Ignoring unknown MOABB_DOWNLOAD_PROVIDER %r; using 'auto'. " + "Expected one of %s.", + provider, + ", ".join(DOWNLOAD_PROVIDERS), + ) + return "auto" + return normalized + + def make_process_pipelines( processing: "BaseProcessing", dataset: "BaseDataset",