Pipeline Concurrency Limit

I’d love to be able to set a concurrency limit on a Pipeline, so that I can have X number of builds running at the same time at most, rather than just on a step by step basis.

Yes yes yes! We are going to have to make some hacky workaround where we wrap another build in a job or make our own locking mechanism. It seems like a critical feature to have to only allow one build per pipeline.

Is there a workaround for this limitation?

Hi folks,

Apologies for the late response here, but I believe I have good news!

An option which I believe will suit your needs, but I confess we’ve not been very good at explaining is what we call concurrency gating.

In short, the jobs which are part of a concurrency group within a given build which are eligible to run (i.e. not following a currently blocked block step, are not preceded by a failure) must all finish before another build’s jobs in that concurrency group are permitted to run.

This means you can make a no-op or some other preparatory step (we’re using echo commands, but what is run in the step doesn’t matter) to denote the start of the “gate”, and another to denote the end, and anything between those two steps which share a concurrency group must finish before the gate will be reopened to other work.

Now, I must concede that this is not, explicitly, a limit on concurrent builds, but it is functionally more or less equivalent without tying up other resources, and works today.

I hope this helps!

1 Like

Hi jess,

Thanks for the quick reply. Sorry it’s taken me a couple of days to respond. I thought I had tested this but it turns out the little experiment I had done with concurrency gating hadn’t quite been showing my I thought it was. I have re-done the experiment and can confirm that it works as you describe (which of course you already know) and that is exactly what we need. Thank you again.

I found running the experiment really useful to understand how the concurrency model works as it was not immediately obvious to me. I’ve added the pipeline config I used below in case anyone else comes across this post with a similar confusion to me.

steps:
  - label: "🚴 let's get started"
    command: |
      echo "Start @ $(date --rfc-3339=ns)"

  - label: "🚦 Enter concurrency gate"
    command: exit 0
    concurrency: 1
    concurrency_group: "testing-concurrency"

  - label: "⏱️  Hold on there"
    command: |
      echo "Start @ $(date --rfc-3339=ns)"
      sleep $(shuf -i 1-20 -n 1)
      echo "End @ $(date --rfc-3339=ns)"

  - label: "🚦 Leave concurrency gate"
    command: exit 0
    concurrency: 1
    concurrency_group: "testing-concurrency"

  - label: "🛑 All done"
    command: |
      echo "End @ $(date --rfc-3339=ns)"