A way to get the index of a concurrency group?

We have a set of shared resources that we want to access in our build jobs, and concurrency groups seem like the ideal way to make sure that we don’t exceed our maximum number of simultaneous jobs.

However, ideally we’d like to be able to tell which job is currently running. For example, in this scenario:

- command: "tests.sh"
  concurrency: 5
  concurrency_group: "tests"

I would like the running build to know which number job it is out of the 5 possible jobs.

This functionality already exists more or less exactly as I would like it in the parallelism feature - you get two env vars that give you the number of total parallel jobs and which one of those jobs is represented by the current build:

BUILDKITE_PARALLEL_JOB=0
BUILDKITE_PARALLEL_JOB_COUNT=5

Is this possible and I’m not seeing it, or is this something we can get simply another way?

Hi @SeanR! Concurrency groups don’t have a start and end counter, like parallelism does. So I’m scratching my head to figure out how you’d get those sort of environment variables working :thinking:

This sounds sort of like a job for agent queues than the concurrency group perhaps, but it’s hard to know exactly.

With agent queues, and using a fixed number of agents, you can more easily tell which agents are busy with which jobs. But you won’t get that sort of info as environment variables like you showed.

Is there a way for the shared set of resources to tell the job above which one they are? What type of resources? If we have a little bit more info about the underlying problem you’re solving, maybe we can suggest alternative ideas.

You don’t by any chance want to combine the two? Something like this would work:

- command: "tests.sh"
  parallelism: 5
  concurrency: 2
  concurrency_group: "tests"

In that example, we’ll run the tests.sh script 5 times, but only 2 at a time. You’ll able to use the BUILDKITE_PARALLEL_JOB in that step as well.

@keithpitt Unfortunately, nope. We need to limit the concurrency level globally AND have a global index (i.e. if we have three builds running tests.sh concurrently we’d like each of them to get a unique index in (0, <num of restricted resource>] ).
Is there a timeline for implementing this or a workaround?

In this case, the shared resource are game consoles, and the software necessary to interact with those consoles is…well, let’s just say actively hostile to working in a CI environment (it basically makes tons of assumptions that are basically only ever true if the console resource is a single-user device sitting on someone’s desk)…

What we basically have to do is:

  1. Limit the number of concurrent builds requiring access to the hardware to X.
  2. Ensure that each running job knows which one of those limited resources are available to use.

Essentially having this index available would allow us to set it up such that:
Concurrent Job #1 uses console #1
Concurrent Job #2 uses console #2

Concurrent Job #N uses console #N

Effectively using the concurrency index as a mutex for a specific physical resource.

Unfortunately, in this use case, we don’t have a real way to check to see if the resource is in use before we try to use it, and if it is in use when we try, it may kill running tasks.

Basically what we’re looking for is some sort of shared state between all jobs in a concurrency group that allows us to know for certain that a specific resource is free.

I’d thought maybe something like

  label: "do a fancy thing on a console"
  #...
  needs:
    xbox-ones: 
      candidates: ["10.32.1.5", "10.32.1.6"]
    ps4s:
      candidates: ["10.32.2.5", "10.32.2.5"]

(since a job might need more than one of a limited resource).

(the choice of a dictionary key instead of those being naked arrays means named anchors can be used to DRY more easily)

Then, within the job, set environment variables according to which resource-id of each type was allocated.

I think we could probably achieve this within a pipeline via a plugin. However, I don’t really know how we might go about sharing resources like this between pipelines, though, without there being a central registry of these limited resources and usage locking/tracking.