I’d love to be able to set a concurrency limit on a Pipeline, so that I can have X number of builds running at the same time at most, rather than just on a step by step basis.
Yes yes yes! We are going to have to make some hacky workaround where we wrap another build in a job or make our own locking mechanism. It seems like a critical feature to have to only allow one build per pipeline.
Hi folks,
Apologies for the late response here, but I believe I have good news!
An option which I believe will suit your needs, but I confess we’ve not been very good at explaining is what we call concurrency gating.
In short, the jobs which are part of a concurrency group within a given build which are eligible to run (i.e. not following a currently blocked block step, are not preceded by a failure) must all finish before another build’s jobs in that concurrency group are permitted to run.
This means you can make a no-op or some other preparatory step (we’re using echo commands, but what is run in the step doesn’t matter) to denote the start of the “gate”, and another to denote the end, and anything between those two steps which share a concurrency group must finish before the gate will be reopened to other work.
Now, I must concede that this is not, explicitly, a limit on concurrent builds, but it is functionally more or less equivalent without tying up other resources, and works today.
I hope this helps!
Hi jess,
Thanks for the quick reply. Sorry it’s taken me a couple of days to respond. I thought I had tested this but it turns out the little experiment I had done with concurrency gating hadn’t quite been showing my I thought it was. I have re-done the experiment and can confirm that it works as you describe (which of course you already know) and that is exactly what we need. Thank you again.
I found running the experiment really useful to understand how the concurrency model works as it was not immediately obvious to me. I’ve added the pipeline config I used below in case anyone else comes across this post with a similar confusion to me.
steps:
- label: "🚴 let's get started"
command: |
echo "Start @ $(date --rfc-3339=ns)"
- label: "🚦 Enter concurrency gate"
command: exit 0
concurrency: 1
concurrency_group: "testing-concurrency"
- label: "⏱️ Hold on there"
command: |
echo "Start @ $(date --rfc-3339=ns)"
sleep $(shuf -i 1-20 -n 1)
echo "End @ $(date --rfc-3339=ns)"
- label: "🚦 Leave concurrency gate"
command: exit 0
concurrency: 1
concurrency_group: "testing-concurrency"
- label: "🛑 All done"
command: |
echo "End @ $(date --rfc-3339=ns)"
I understand that this workaround solves most of my problems but it can not be called a nice solution IMHO. Any hope of a pipeline “native” solution?
Only one invocation of the pipeline should be running at any time… not just parts of it.
Hello, @Asmund! Welcome to the Buildkite Community!
Thank you for your feedback!
At the moment, the recommended way to ensure that only one build will be running on a given pipeline is to use the concurrency gates.
However, as an alternative, you could set up a cluster with a default queue containing just one agent. Then, you can assign this cluster to the pipeline in the Pipeline Settings. By doing this, only one job will be executed on the pipeline at any moment.
While this isn’t our top recommendation compared to concurrency gates, it will ensure that only one invocation of the pipeline is running at any time.
Best!
@karen.sawrey thank you for responding!
Or problem is that we end up having multiple release pipelines instances running in parallel. Leading to random concurrency problems.
Running the release in sequence would take > 4h. Parallel it is ~20min.
Please update this topic if Buildkite ever decide to implement proper concurrency control on pipeline level.
You’re welcome, @Asmund!
From what you’re describing, I’d really suggest you give concurrency gate a go. If you have any difficulties implementing that, you can reach out to us via support@buildkite.com for us to help troubleshoot your specific use case.
If down the line we do add some functionality that fits your description, it’ll surely be mentioned in the Buildkite changelog.
Best!
Karen
Hi @karen.sawrey
I did try the above suggestion and it did not work for me. What am I doing wrong?
Could it be that the “gate” do not work across a ‘wait’. If so could you provide an example with a ‘wait’ between?
---
agents:
queue: ci-blue
env:
BUILDKITE_TOKEN_PATH: /run/secrets/buildkite-api-token
steps:
- label: ":buildkite:"
key: ":init:"
concurrency: 1
concurrency_group: "Block-concurrent-releases"
command: "echo 'Initialising release pipeline ...'"
# this step generates a many build steps
- label: ":llama:"
command: ops/buildkite/llama-release.sh
# Wait for all previous steps to complete.
- wait: null
continue_on_failure: false
# After all release steps are executed correctly, create a commit
# with update to infrastructure configuration.
- label: ":git:"
key: ":git:"
concurrency: 1
concurrency_group: "Block-concurrent-releases"
command: ops/buildkite/git-release.sh
---
Hello again, @Asmund!
A small tweak should fix things. What’s missing here is the dependencies after opening the concurrency gate. So you can either add a wait step after opening it:
- label: ":buildkite:"
key: ":init:"
concurrency: 1
concurrency_group: "Block-concurrent-releases"
command: "echo 'Initialising release pipeline ...'"
- wait:
# this step generates a many build steps
- label: ":llama:"
command: ops/buildkite/llama-release.sh
Or you can add depends_on on the step opening the concurrency gate:
- label: ":buildkite:"
key: ":init:"
concurrency: 1
concurrency_group: "Block-concurrent-releases"
command: "echo 'Initialising release pipeline ...'"
# this step generates a many build steps
- label: ":llama:"
command: ops/buildkite/llama-release.sh
depends_on:
- ":init:"
Best!