Serve dataset downloads from NEMAR's sourcedata/, and add a provider switch

docs/source/api.rst

@@ -564,6 +564,8 @@ Utils
     set_log_level
     setup_seed
     set_download_dir
+    set_download_provider
+    get_download_provider
     make_process_pipelines
 
 Benchmark

docs/source/whats_new.rst

@@ -41,6 +41,9 @@ Enhancements
 - Skip zip extraction in :class:`moabb.datasets.GuttmannFlury2025` when files are already extracted, with ``/scratch`` fallback for NFS filesystems on compute nodes (by `Bruno Aristimunha`_)
 - Re-enable auto-execution of the Riemannian Artifact Rejection tutorial (``examples/advanced_examples/plot_riemannian_artifact_rejection.py``) now that pyRiemann 0.11 is on PyPI with per-potato metrics and ``method_combination`` support on ``PotatoField`` (by `Bruno Aristimunha`_)
 - Use NEMAR as the default download source for datasets with an assigned ``nemar_id``, while preserving existing dataset-specific downloaders as a fallback (by `Bruno Aristimunha`_).
+- :meth:`moabb.datasets.base.BaseDataset.download` now fetches NEMAR's ``sourcedata/`` — the **original pre-BIDS distribution**, byte-identical to what the upstream host serves and stored under the same upstream filenames — instead of the deposit's BIDS copy. The BIDS copy is a re-encoding whose events and session/run labels differ from what each dataset's own loader produces, so substituting it would silently change results rather than only change where the bytes come from; ``sourcedata/`` changes the source without touching the science. Loading is unaffected: ``get_data`` still runs each dataset's own parser. Falls back to the dataset's upstream downloader when a deposit publishes no ``sourcedata/`` (by `Bruno Aristimunha`_).
+- Serve the **original, pre-BIDS distribution** from NEMAR through the new :meth:`moabb.datasets.base.BaseDataset.sourcedata_path`. NEMAR deposits republish the files exactly as the authors distributed them under ``sourcedata/``, keeping the upstream filenames, so NEMAR can now stand in for upstream hosts that are slow, rate-limited, behind a bot gate, or retired. Datasets may set ``nemar_sourcedata_include`` (e.g. ``"sourcedata/subject_{subject:02d}.*"``) to fetch a single subject instead of the whole tree; without it the full ``sourcedata/`` is downloaded, because that tree keeps the upstream layout rather than a BIDS one. Backed by :func:`moabb.datasets.download.nemar_sourcedata_dl` (by `Bruno Aristimunha`_).
+- Add :func:`moabb.set_download_provider` / :func:`moabb.get_download_provider` to pin where MOABB fetches data from: ``"auto"`` (default — NEMAR first, upstream fallback), ``"nemar"`` (NEMAR only; a failure is raised rather than silently falling back to a host the caller opted out of), or ``"upstream"`` (never use NEMAR). Also settable per run via the ``MOABB_DOWNLOAD_PROVIDER`` environment variable, which takes precedence over the stored config (by `Bruno Aristimunha`_).
 - Add :class:`moabb.datasets.preprocessing.EuclideanAlignment`, a trial-level Euclidean Alignment transformer (He & Wu 2020; Junqueira et al. 2024) that whitens each trial by the inverse square root of the Euclidean mean covariance to remove per-domain covariance shift before a (deep) model sees the data. Inductive and leakage-free by default (``fit`` learns the reference from training trials, ``transform`` re-applies it to unseen trials); ``fit_transform`` gives the transductive, per-recording form. Accepts an :class:`mne.BaseEpochs` or an ``(n_trials, n_channels, n_times)`` ndarray, uses a shrinkage covariance estimator (``"lwf"``) for robustness, and adds no new dependency (``pyriemann >= 0.11`` is already required). Distinct from :class:`pyriemann.transfer.TLCenter`, which recenters covariance *matrices* (:gh:`1108` by `Bruno Aristimunha`_).
 - Add an ``n_jobs`` parameter to :meth:`moabb.paradigms.base.BaseParadigm.get_data` and :meth:`moabb.datasets.base.BaseDataset.get_data` to load and preprocess subjects in parallel with :class:`joblib.Parallel`. Per-subject processing (reading, filtering, resampling, epoching) is independent, so this gives a near-linear speedup on datasets with many subjects, with identical numerical results. moabb's own patches to the shared BIDS cache files (``participants.tsv``/``.json``, ``dataset_description.json``) now take the mne-bids cross-process file lock, so parallel caching stays consistent (:gh:`1124` by `Bruno Aristimunha`_).
 - Drive cross-validation folds with any stock scikit-learn cross-validator passed as ``cv_class``, controlled by a ``groups`` argument — a metadata column name, a list of column names (compound key, e.g. ``["subject", "session"]``), or a callable ``metadata -> array`` — together with callable ``cv_kwargs`` resolved against the metadata (e.g. ``cv_class=PredefinedSplit`` with a ``test_fold`` callable to target a single fold). ``groups`` is exposed on :class:`moabb.evaluations.WithinSessionEvaluation`, :class:`moabb.evaluations.WithinSubjectEvaluation`, :class:`moabb.evaluations.CrossSessionEvaluation` and :class:`moabb.evaluations.CrossSubjectEvaluation` and threaded to their splitters; each splitter keeps its default grouping (``"subject"`` / ``"session"`` / labels) when ``groups`` is ``None``. :class:`moabb.evaluations.splitters.CrossDatasetSplitter` gains ``groups`` (its ``group_column`` argument is now a deprecated alias) (:gh:`1104` by `Bruno Aristimunha`_).

moabb/__init__.py

@@ -2,4 +2,11 @@
 __version__ = "1.5.0dev0"
 
 from .benchmark import benchmark
-from .utils import make_process_pipelines, set_download_dir, set_log_level, setup_seed
+from .utils import (
+    get_download_provider,
+    make_process_pipelines,
+    set_download_dir,
+    set_download_provider,
+    set_log_level,
+    setup_seed,
+)

moabb/datasets/base.py

@@ -29,8 +29,9 @@
     _interface_map,
     get_bids_root,
 )
-from moabb.datasets.download import NemarDownloadError, nemar_dl
+from moabb.datasets.download import NemarDownloadError, nemar_dl, nemar_sourcedata_dl
 from moabb.datasets.preprocessing import FixedPipeline, SetRawAnnotations
+from moabb.utils import get_download_provider
 
 
 if TYPE_CHECKING:
@@ -926,12 +927,21 @@ def download(
 
         This function is only useful to download all the dataset at once.
 
+        When the dataset declares a :attr:`nemar_id` and the download provider
+        is not ``"upstream"``, the files come from NEMAR's ``sourcedata/`` --
+        the original pre-BIDS distribution, byte-identical to what the upstream
+        host serves. On any NEMAR failure this falls back to the dataset's own
+        downloader with a warning (unless the provider is pinned to
+        ``"nemar"``). See :meth:`sourcedata_path` and
+        :func:`moabb.set_download_provider`.
 
         Parameters
         ----------
         subject_list : list of int | None
             List of subjects id to download, if None all subjects
-            are downloaded.
+            are downloaded. On the NEMAR path each subject is resolved through
+            the deposit's ``sourcedata_provenance.json``; deposits enriched
+            before that manifest recorded subjects fetch the whole tree.
         path : None | str
             Location of where to look for the data storing location.
             If None, the environment variable or config parameter
@@ -944,33 +954,52 @@ def download(
         update_path : bool | None
             If True, set the MNE_DATASETS_(dataset)_PATH in mne-python
             config to the given path. If None, the user is prompted.
+            Not used on the NEMAR path.
         accept: bool
-            Accept licence term to download the data, if any. Default: False
+            Accept licence term to download the data, if any. Default: False.
+            Only relevant to the dataset's own downloader; NEMAR mirrors are
+            already public.
         verbose : bool, str, int, or None
             If not None, override default verbose level
             (see :func:`mne.verbose`).
         """
         if subject_list is None:
             subject_list = self.subject_list
+        provider = get_download_provider()
+        if provider == "nemar" and self.nemar_id is None:
+            raise NemarDownloadError(
+                f"Download provider is pinned to 'nemar' but {self.code} declares "
+                "no nemar_id, so it cannot be fetched from NEMAR. Use "
+                "moabb.set_download_provider('auto') to allow the upstream "
+                "downloader for datasets NEMAR does not mirror."
+            )
+        # Prefer NEMAR's `sourcedata/` -- the ORIGINAL pre-BIDS distribution,
+        # byte-identical to what the upstream host serves and stored under the
+        # same upstream filenames. Deliberately not the deposit's BIDS copy:
+        # that is a re-encoding whose events and session/run labels differ from
+        # what each dataset's own loader produces, so substituting it would
+        # silently change results rather than just change where bytes come from.
+        if self.nemar_id is not None and provider != "upstream":
+            try:
+                self._download_nemar_sourcedata(
+                    subject_list=subject_list,
+                    path=path,
+                    force_update=force_update,
+                    verbose=verbose,
+                )
+                return
+            except NemarDownloadError as exc:
+                if provider == "nemar":
+                    raise
+                warnings.warn(
+                    f"Could not download {self.code} sourcedata from NEMAR "
+                    f"({self.nemar_id}); falling back to the dataset data_path "
+                    f"downloader. Original error: {exc}",
+                    RuntimeWarning,
+                    stacklevel=2,
+                )
+
         for subject in subject_list:
-            if self.nemar_id is not None:
-                try:
-                    self._download_nemar(
-                        subject=subject,
-                        path=path,
-                        force_update=force_update,
-                        update_path=update_path,
-                        verbose=verbose,
-                    )
-                    continue
-                except NemarDownloadError as exc:
-                    warnings.warn(
-                        f"Could not download {self.code} from NEMAR ({self.nemar_id}); "
-                        "falling back to the dataset data_path downloader. "
-                        f"Original error: {exc}",
-                        RuntimeWarning,
-                        stacklevel=2,
-                    )
             # check if accept is needed
             sig = signature(self.data_path)
             if "accept" in [str(p) for p in sig.parameters]:
@@ -1050,6 +1079,77 @@ def _download_nemar(
             **(self.nemar_bids_filters or {}),
         )
 
+    def _download_nemar_sourcedata(
+        self, subject_list=None, path=None, force_update=False, verbose=None
+    ):
+        """Fetch the original pre-BIDS distribution for a set of subjects.
+
+        Each subject is resolved through the deposit's provenance manifest
+        rather than by pattern-matching the layout, which across the catalogue
+        is genuinely arbitrary -- subjects appear as letters, as ``sub1``, as
+        ``S10_Session_1``, split across two path components, or not at all.
+
+        Raises
+        ------
+        moabb.datasets.download.NemarDownloadError
+            If nemar-py is unavailable, a download fails, or the deposit
+            publishes no ``sourcedata/``.
+        """
+        for subject in subject_list:
+            self.sourcedata_path(
+                subject=subject, path=path, force_update=force_update, verbose=verbose
+            )
+
+    def sourcedata_path(self, subject=None, path=None, force_update=False, verbose=None):
+        """Get the dataset's *original* pre-BIDS distribution from NEMAR.
+
+        Where :meth:`data_path` fetches the original files from the upstream
+        host, this fetches the copy NEMAR mirrors under ``sourcedata/``. The
+        files keep their upstream names, so the two are interchangeable in
+        content -- but NEMAR stays reachable when the upstream host is slow,
+        rate-limited, behind a bot gate, or retired.
+
+        Parameters
+        ----------
+        subject : int | str | None
+            Restrict the download to one subject, resolved through the
+            deposit's ``sourcedata_provenance.json``. Deposits enriched before
+            that manifest recorded subjects fall back to the whole tree with a
+            warning. When ``None`` the whole tree is fetched.
+        path : None | str
+            Base path where MOABB stores datasets.
+        force_update : bool
+            Re-fetch even when a local copy is present.
+        verbose : bool, str, int, or None
+            If not None, override default verbose level.
+
+        Returns
+        -------
+        str
+            Local path to the ``sourcedata`` directory.
+
+        Raises
+        ------
+        ValueError
+            If the dataset declares no ``nemar_id``.
+        moabb.datasets.download.NemarDownloadError
+            If nemar-py is unavailable, the download fails, or the deposit
+            publishes no ``sourcedata/``.
+        """
+        if self.nemar_id is None:
+            raise ValueError(
+                f"{self.code} declares no nemar_id, so its original distribution "
+                "cannot be fetched from NEMAR."
+            )
+        return nemar_sourcedata_dl(
+            self.nemar_id,
+            self.code,
+            path=path,
+            force_update=force_update,
+            subject=subject,
+            verbose=verbose,
+        )
+
     def _get_single_subject_data_from_nemar(self, subject, path=None):
         """Download and load one subject from the NEMAR BIDS dataset."""
         try: