Dynamic Concurrency Group

General
1

In our CI/CD pipelines, we support feature branch deployment for each repository. During the deploy and test steps of feature branch CI/CD, we’d like to make sure that the deploy and test steps are only run one at a time, in order, for any push to a specific feature branch (not atomically across all builds for the associated repository).

This would seem to be a good use case for concurrency groups, but so far, we’ve been unable to figure out a simple way to dynamically construct the concurrency group label using execution time env vars or metadata. Ideally, what we’d like to do is something like concurrency_group: gate/$BUILDKITE_BRANCH. Is there a way to generate the concurrency_group at build/execution time?

2

Hello @TheoryOfLight

Welcome to our community.

I took a look at this and the based on your use case of running the steps one at a time and in order. Concurrency-groups would be the best option.

You can definitely interpolate these Env Vars directly. If its a custom env var, it can done in a pipeline upload in your script

So if you’re using a script to generate pipeline yaml it might look like
Example

src/test.sh

#!/bin/bash

DEPLOYMENT_ENV="stage"

buildkite-agent pipeline upload <<YAML
steps:
  - label: "start gate"
    command: echo "---> starting concurrency gate"
    concurrency_group: "gate/${DEPLOYMENT_ENV}"
YAML

And in the pipeline.yml

steps: 
  - command: 'echo hello world'
    label: 'Dynamic Using Branch'
    branches: 'main'
    concurrency: 1
    concurrency_group: 'gate/$BUILDKITE_BRANCH'
    
  - command: "./src/test.sh"
    label: 'Running test script'

But as long as it is set in the context of the buildkite-agent pipeline upload command, it should interpolate as expected.

Cheers

3

Thanks for the follow up. The reason for my post was that the second snippet you shared here for pipeline.yml didn’t work (the env var did not get interpolated), but when I dug deeper into this I found we use buildkite-agent pipeline upload --no-interpolation in the upload step, and instead use a templating library to preprocess the pipeline YAML, so I was able to get the dynamic concurrency group working with our templating system instead. Thanks again!

4

Thanks for the follow up. Glad you found a workaround for this

Cheers!