Prevent spurious breakage when tagging happens

[Geod24]

At the moment, we frequently hit the following issue:

• Buildkite builds a project
• Some change is made in the project (e.g. in .travis.yml)
• Nothing happens for a while
• A new release is tagged
• The Buildkite build is broken, stalling all auto-merge in DMD/druntime/phobos (although not blocking direct merge)
• The person around has to find a solution, which is either fixing buildkite, disabling the test, or fixing the lib + getting a patch release

As the number of project grows, so does this issue.

We could set as a requirement that a project being built via Buildkite also uses Buildkite itself as one of its CI, but that will obviously trigger more issues (especially since the logs aren't public).
Another idea would be to have a fallback to the previous version for the next X hours after a new release is created, but have dlang-bot create an issue in this repo tagging the maintainer and us.

Any thought @wilzbach ?

[MartinNowak]

Projects that regularly regularly release broken test-suites should be removed from the project-tester.
If OTOH projects regularly get broken by dmd/druntime/phobos changes, then those failures are legitimate and should rarely be worked around.

[Geod24]

Projects that regularly regularly release broken test-suites should be removed from the project-tester.

The thing is, it's not broken for them. They do a legitimate change to their project test suite. I've never seen a project release a broken release, it was always some change in the settings (e.g. installing a new dependency in their .travis.yml) that broke stuff.

[MartinNowak]

So you're talking about that angle. Indeed new dependencies might break builds, do you have a good example? I think the simplest way to solve this for now is by communication, there are not so many projects with external dependencies (as those hardly integrate with dub anyhow).

Also we've pre-installed quite a list of dependencies and I just linked to that in the README (https://github.com/dlang/ci/blob/master/README.md#pre-installed-dependencies-on-buildkite-agents).

[PetarKirov]

Why not simply require that all projects are build in containers? That way won't have any dependency management to do on our side, except for the container runtime (e.g. Docker, Windows Containers, etc.)

[Geod24]

@MartinNowak : The only breakages I've seen so far have been just this.
A dependency to the build gets added, but we don't have it.
E.g. we had meson builds being added in 2 cases ( #307 (comment) ), and now a rewritten test-suite.