Schedule policy of steps across nodes / VM's

Pipelines
1

hi,

We are planing to scale up the number of agents. Currently we have one large VM. We will now need to go to a multiple VM’s setup. What we need to know is if there is ANY logic to the scheduling of steps to nodes/VM’s. The logic used for scheduling steps to agents will shape the solution …

BR,
Åsmund

2

Hello @Asmund

Welcome to Buildkite Community,

Buildkite uses a combination of queuing and agent tags to schedule steps to agents.

Agent Tags: Agents can be tagged with arbitrary strings (e.g., role=database or aws=true). These tags are used to match jobs to the appropriate agents. You can specify required tags for each step in your pipeline.
Queueing: All jobs are placed into a queue, and agents will fetch the next available job they are eligible to run based on their tags and the job’s requirements. Note, agents can only be configured to listen on a single queue within a cluster. An unclustered agent can listen on a single queue or on multiple queues.
Prioritization: Buildkite doesn’t prioritize jobs by default. Jobs are picked up on a first-come, first-served basis by agents that meet the tag requirements.
Concurrency: The maximum number of jobs that can be run concurrently can be configured for each step. This helps in controlling the load on your agents.

This setup should help you manage which jobs are run on which VMs based on the tags and configurations you set, shaping your scaling solution accordingly.

Thanks,

3

Thanks for responding Ivanna,

Before I come back to the topic I need clarification on how Buildkite uses some terms. Could you please point me to the documentation on how Buildkite in use:

  • Pipeline before it is running.
  • Running pipeline
  • Job
  • Step a as it configured
  • A running step
  • Queued
  • A Build

I need to know the terms Buildkite uses to increase the likelihood being understood and understanding the answers. ;-) I am hoping there is some place the terms builkite uses are gathered in one place.

BR,
Åsmund

4

Hey @Asmund we have a glossary in the docs but it doesnt cover all those terms. So you can find other things at Defining your pipeline steps | Buildkite Documentation describing states which I think will help you understand the terms used.

If any remain, let us know and we can add them to the docs for future travellers as well

5

Thanks for the feedback.

I did search and failed to find if there is any possibility with a priority list on the step queue entry.

queue = tagpri1 || tagpri2

Or something like this.

Is this possible?
Are the plans to support this?

BR,
Åsmund

6

Hi @Asmund

Buildkite does not natively support a priority list within the step queue entry as described (e.g., queue = tagpri1 || tagpri2). However, you can achieve similar functionality by using a combination of Buildkite’s existing features such as agent queues and priority tags, along with custom scripting.

For instance:

use agent queues: you can create multiple agent queues in your Buildkite configuration, each representing a different priority level;

and

use dynamic pipeline to control which queue each step goes into based on your custom logic.

This approach allows you to effectively prioritize steps by assigning them to different queues based on your custom logic.

Thanks,