Add a score on the lowest performing subjects - #1133
Open
adityasingh2400 wants to merge 1 commit into
Open
Conversation
compute_lowest_subject_scores ranks the subjects of every (dataset, pipeline) pair and averages only the lowest percentile of them, which is the family of metrics F1@20% belongs to. ERP, SSVEP and c-VEP saturate close to a perfect score on most subjects, so a mean over the whole cohort is dominated by subjects that no longer separate pipelines. The subjects a pipeline handles worst are where the remaining headroom is. Fixes NeuroTechX#733
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds
moabb.analysis.meta_analysis.compute_lowest_subject_scores, the metric asked for in #733. It takes a results dataframe, averages a subject's sessions so every subject weighs the same, ranks the subjects of each (dataset, pipeline) pair, and averages the score over the lowestpercentilepercent of them. It returns one row per (dataset, pipeline) with the resultingscoreand then_subjectsit was computed on.The motivation is the one in the issue. In ERP, SSVEP and c-VEP a good pipeline is already near a perfect score on most of the cohort, so a mean over all subjects is dominated by subjects that no longer separate one pipeline from another. The subjects a pipeline handles worst are where the differences still live, which is what the F1@20% metric in Gnassounou et al. 2025 reports and what the issue points at.
Two details worth flagging for review. The retained count is
ceil(n_subjects * percentile / 100)and never drops below one, so a small cohort still yields the single worst subject rather than an empty selection. Ties at the cut-off are broken on the subject id after a stable sort, so the same input always produces the same selection. Apercentileoutside(0, 100]raisesValueError, andpercentile=100reduces to the plain per-subject mean.Tested by a new
TestLowestSubjectScoresclass inmoabb/tests/test_analysis.pycovering the worst-subject selection, the rounding up, the at-least-one floor, thepercentile=100identity againstnumpy.mean, the per-dataset and per-pipeline grouping, the invalid percentile, and one case where the subject holding the single worst session is not the worst subject on average, which pins the ordering to the per-subject mean rather than the raw rows. Ondevelopthese fail withAttributeError: module 'moabb.analysis.meta_analysis' has no attribute 'compute_lowest_subject_scores', and with the change the whole file passes at 24 tests.pre-commit run --filesis clean on the four touched files.Nothing existing is modified, so the results dataframe,
analyzeand the benchmark output are all unchanged. The function is added to the Statistics section of the API reference.Fixes #733