Separating Tests

[rjsparks]

(Bringing several individual conversations together)

The tests as we currently have them defined, especially with the addition of the v.Nu validator in feat/bs5 are taking far too long to run. We can pursue making them more efficient, but until we find a breakthrough on that front, we probably need to change how we are planning to use them.

The proposals I've heard so far intersect on something like this:

• We modify the test suite to make the html validation opt-in (--with-html-validation)
• We do a similar thing to make selenium testing opt-in (--with-selenium)
• We configure the automatically triggered test action on PRs to run without those flags.
• We provide a test action that can be run manually by maintainers on a PR that will include those flags.
• Release actions will automatically run with all of these options on.

Question: should we treat cypress the same way (--with-cypress-tests) or is it efficient enough to run in the default test suite Right now, there are so few cypress tests it's hard to know.

[NGPixel]

Because Cypress can imitate user interaction, some tests will inevitably include things like fake delays, wait for UI animations to complete, wait for page load, etc.

These types of slowdown will get more frequent as more tests are added...

[larseggert]

If we make the things in the bullet list separately runnable (via options), we can parallelize them into separate CI runs on GitHub.

[rjsparks]

it makes sense to parallelize the cypress and selenium tests from other things, but not as much the html validation from the rest of the tests (as the html validation runs all of those same tests anyhow).

[rjsparks]

This is over a year old - I'm going to close it - we will schedule work in the future (off the end of the horizon at the moment) to rework the test framework anyhow so we can run just the TestCase tests in parallel.