Adjust batch update size

[yselkowitz]

Currently, we have set a threshold of 750 builds for a single update, above which (e.g. during a mass rebuild) the batch is split into multiple updates. AFAICS the largest single update we've had since then was 608 for the Python 3.15 upgrade so I wouldn't want to make that any smaller.

However, once we have passed that threshold and we're splitting anyway, then we could theoretically have a different (presumably smaller) split update size. IOW, up to 750 builds in one update, but if more, then split into groups of e.g. 500. Since we're not as concerned with testing things together, then maybe more (somewhat) smaller updates would be easier for bodhi/CI/QA to handle.

/cc @AdamWill @LecrisUT for their thoughts

[AdamWill]

yeah, that kinda makes sense to me.

[LecrisUT]

If rmdepcheck passes wit the split updates, then hopefully once the testing of Fedora-CI big updates is fixed, I expect the other test to be passing also. My concern would be if picking the subset of builds could lead to dependencies not being visible. But that should be covered by injecting the buildroot repos iiuc.

The other aspect could be bodhi itself slowing down. I don't know what's the cause of it, but it seems to be unrelated to the tests results, rather it slows down when the update is still in testing or below.

[yselkowitz]

Now that you mention it, rmdepcheck actually FAILED in part, because some packages had a >=NVR specific dep on glibc (btw what causes that?), so the updates that did not include glibc showed "errors" (false alarms, under the circumstances).

[yselkowitz]

As for the slowdown, it can take hours to get all the builds in each update signed, and until then the update remains in pending. (As a development branch with no gating, ELN updates move very quickly from testing to stable.)

[AdamWill]

tests are going to be 'at risk' any time we have to split an update, really, it doesn't make a huge difference whether we split it into 750 or 500 package chunks...honestly, the tests aren't as reliable for ELN as anything else because of the way they don't follow the original update groupings anyway. even with smaller update batches...

[sgallagher]

Splitting at 750 during the recent mass rebuild also had the unexpected -- but welcome -- side effect of staggering the load into CentOS Stream 11. Rather than hammering it with 2000 build requests at once, we had smaller hits a few hours apart (due to the time it takes to sign all the packages in the update).

Whatever strategy we choose here, I don't think we want to lose that by picking a too-low split.