Skip to content

Add a score on the lowest performing subjects - #1133

Open
adityasingh2400 wants to merge 1 commit into
NeuroTechX:developfrom
adityasingh2400:feat/lowest-performing-subject-score
Open

Add a score on the lowest performing subjects#1133
adityasingh2400 wants to merge 1 commit into
NeuroTechX:developfrom
adityasingh2400:feat/lowest-performing-subject-score

Conversation

@adityasingh2400

Copy link
Copy Markdown

This adds moabb.analysis.meta_analysis.compute_lowest_subject_scores, the metric asked for in #733. It takes a results dataframe, averages a subject's sessions so every subject weighs the same, ranks the subjects of each (dataset, pipeline) pair, and averages the score over the lowest percentile percent of them. It returns one row per (dataset, pipeline) with the resulting score and the n_subjects it was computed on.

The motivation is the one in the issue. In ERP, SSVEP and c-VEP a good pipeline is already near a perfect score on most of the cohort, so a mean over all subjects is dominated by subjects that no longer separate one pipeline from another. The subjects a pipeline handles worst are where the differences still live, which is what the F1@20% metric in Gnassounou et al. 2025 reports and what the issue points at.

Two details worth flagging for review. The retained count is ceil(n_subjects * percentile / 100) and never drops below one, so a small cohort still yields the single worst subject rather than an empty selection. Ties at the cut-off are broken on the subject id after a stable sort, so the same input always produces the same selection. A percentile outside (0, 100] raises ValueError, and percentile=100 reduces to the plain per-subject mean.

Tested by a new TestLowestSubjectScores class in moabb/tests/test_analysis.py covering the worst-subject selection, the rounding up, the at-least-one floor, the percentile=100 identity against numpy.mean, the per-dataset and per-pipeline grouping, the invalid percentile, and one case where the subject holding the single worst session is not the worst subject on average, which pins the ordering to the per-subject mean rather than the raw rows. On develop these fail with AttributeError: module 'moabb.analysis.meta_analysis' has no attribute 'compute_lowest_subject_scores', and with the change the whole file passes at 24 tests. pre-commit run --files is clean on the four touched files.

Nothing existing is modified, so the results dataframe, analyze and the benchmark output are all unchanged. The function is added to the Statistics section of the API reference.

Fixes #733

compute_lowest_subject_scores ranks the subjects of every (dataset,
pipeline) pair and averages only the lowest percentile of them, which is
the family of metrics F1@20% belongs to.

ERP, SSVEP and c-VEP saturate close to a perfect score on most subjects,
so a mean over the whole cohort is dominated by subjects that no longer
separate pipelines. The subjects a pipeline handles worst are where the
remaining headroom is.

Fixes NeuroTechX#733
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Score on lowest performing subjects

1 participant