I’m wondering if there is a way to block the manual retry buttons on pipelines that are not the latest for their branch?
A scenario came up last week where two merges to the main branch happened relatively closely to each other.
We have concurrency groups set up to make sure they deploy in the correct order.
The first pipeline failed due to an intermittent error.
The second pipeline that was waiting for the first was then allowed to run and ran to completion deploying our latest Docker image into our Kubernetes clusters.
Then a user clicked the retry button in the failed step of the first (older) pipeline that then ran the deploy again, installing the slightly older Docker image into our clusters.
Hope you’ve been well - and a great question too Hope the Easter break has been great!
We haven’t had that asked previously to my knowledge - at the moment the way that you could potentially achieve this is through a custom hook; most applicable I believe would be at repository level where you’d fetch the latest builds, parse the latest one for its number and block the job from progressing any further if this value != the retried jobs’ BUILDKITE_BUILD_NUMBER. An approach like this would need an API token within the environment to use, but could work well in a case like this!
There is an option for disabling manual Rebuilds of all builds in a Pipelines’ settings (Pipeline view-> Settings → Builds and the Rebuilds option (as below), though that only applies to builds - not their individual jobs. For job-level retry restrictions however, something like above currently would be the way to go about it.
The intention isn’t to turn manual retries off since we want them.
But only to turn them off if the pipeline has been superseded by a later one.
So the Disable Rebuilds option you’ve shown there isn’t what we’d want.
Regarding your solution via a hook; where there are two merges and concurrency has the latest one waiting, would a hook as you’ve described, interrupt the currently running pipeline because it is no longer the latest one?
We would still like the older pipeline to run to completion before concurrency allows the newer one to run, but once that newer one has overtaken the older one (because the old one failed for example and concurrency allows the newer one to proceed), the older one shouldn’t be allowed to be retried any more.
Definitely understandable - that Disable Rebuilds button is applicable for builds - with every step (job) retried in their entirety of course, and not individually. The end result of course is these rebuilds invoking the resultant steps meeting criteria - which is a layer “above” where we require functionality here.
That logic above will work on the premise that the build numbers can only go up - and in the case of concurrency gating arranging FIFO ordering; it will hold. Technically this approach would be dependant on the API feeding back the absolute latest number. Once the newer build is created, the API will return it and thus with the logic above - if one were to re-run a job on the older build, it’s build number (BUILDKITE_BUILD_NUMBER) will still be on the older n-1 integer of the latest returned API - and for which you’d exit the job there within the plugin then based on the most appropriate lifecycle event. This’ll need to be before the command hook where the actual invocation will commence, in our case here, the image rollout.
Hope the Easter break has been great!
image rollout.