Skip to content

test_large_model_generation_is_not_quadratic trips its 20 s budget on CI runners (22.7 s, 22.2 s) #1479

Open

Description

@wshlavacek

python/tests/test_codegen_gen_scaling.py::test_large_model_generation_is_not_quadratic times generate_combined_from_model on a 40,000-reaction model against a 20 s wall-clock budget. Its docstring says the budget has ">20x headroom on a normal machine, so it trips only on a genuine re-quadratic regression, not on CI jitter". That is true locally (1.7 s on an M-series Mac), but not on GitHub's ubuntu runners:

Neither change touched codegen. Both runs went green on gh run rerun --failed. The shard is a required check, so each trip blocks a merge until someone reruns it.

On those runners the headroom is about 1x, not 20x, so the test is a coin flip on a slow machine.

Fix direction: measure scaling, not seconds. Time the same generator on a model a tenth the size, in the same process, and bound the ratio: linear is about 10, quadratic about 100, so a bound near 30 separates them on any runner. The alternative is to raise the budget, which only moves the coin flip.

Activity

  1. added
    testingTest suites, parity checks, nightlies and benchmarks: how bngsim is checked, not what it computes
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingTest suites, parity checks, nightlies and benchmarks: how bngsim is checked, not what it computes

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions