We'll need to break up the builder tests and run them across two machines, because they span the full range of builds we're doing and there's no single machine that can do them all: Big Sur machines can't run older Xcodes and Catalina machines can't run 12.5.

A few issues that I've been battling with the last two days:

• we can't run the Linux tests the way I intended, because they require a pre-built docker image (we can't run them via swift test, at least not without major fiddling)
• getting a weird test failure running BuildMatrixTests.test_5_2_macosSpm on spi-prod
• test all is flakey, perhaps due to --parallel. Failure but success on re-run.

My hope was to have all 40 builds in BuildMatrixTests covered by respective builders as a confirmation for deploying a new builder but that doesn't seem to be feasible.

Might have to scrap this but I really want to avoid having to do manual testing of deployments again.

I'll need to think this through but there may be a way to test the actual binary and image on a branch before merging. That'd be the best way to confirm it's going to work.

Alright, I've got a solution now that'll work without too much fiddling and it'll actually ensure the builder we deploy is passing all builds across the full matrix. Here's a partial run (I still need to add the remaining tests):

https://gitlab.com/finestructure/swiftpackageindex-builder/-/pipelines/295239568

It's a bit repetitive to set up but I much prefer that being repetitive once and then it just runs instead of having to exercise those tests manually before each deploy.

There's some cleanup work left to do but the jobs have finally all passed across the whole build matrix:

https://gitlab.com/finestructure/swiftpackageindex-builder/-/pipelines/296053511

I'm pulling the "update the build matrix" point into a new issue so we can close this one here. It's probably going to take a couple of days for the builds to be caught up.