I’m getting around to trying to understand clusters and how to use them in our org.
I’m using buildkite-builder now to generate our pipelines. I have a dedicated set of hosts that just run the pipeline build step for every build in every pipeline we have. I was contemplating starting my foray in to clusters by clustering these agents.
But reading the docs, it seems like if a job starts on an agent in a cluster, it can’t generate more jobs in the build that leave that cluster. Right now all my agents are in an unclustered setup so this isn’t a problem.
If I clustered just my buildkite-builder agents am I going to run in to an issue with the steps that get generated or can they target another queue in another cluster?
Hope you’re well - and welcome to the Buildkite Forum!
The agents in Buildkite can be organized into clusters, which typically are used to group agents with similar configurations or capabilities. This helps in managing and scaling your infrastructure according to your needs.
When you use the Buildkite Builder to generate pipelines, it creates a series of steps that are executed by available agents. If all your agents are currently unclustered, like you mentioned above, this means that any agent can pick up any job from the queue, regardless of the type of job or the pipeline it belongs to.
However, when you start clustering agents, you’re effectively creating boundaries within which agents can operate. If a job starts on an agent within a specific cluster, it is generally expected to continue execution within that cluster. This design is primarily for efficiency and security reasons, allowing you to control where certain jobs run and ensuring that resources are optimized within clusters.
Buildkite does offer flexibility in job targeting using agent query rules, which include attributes like queue. Here is queue best practices. You can specify which queue a job should go to in the pipeline definition. If you have different clusters with agents listening on different queues, you could potentially design your generated pipeline steps to target specific queues that may reside in different clusters. Just to make sure that the right jobs land on the right agents.
Hope this information clears up any doubts you have!
Thanks @ivanna – that does clear things up for me.
We are currently using queue to isolate things. For example, the buildkite-builder agents listen to just one queue. And we just queue the buildkite-builder agents on that one queue.
Sounds like clusters offer a level of isolation I don’t yet need. Thanks!
Using queues for isolation, as you are currently doing, is less complex and more straightforward than using clusters for the same purpose. Clustering agents is more about scaling and managing large numbers of agents, particularly when they are spread across different environments or need to be segregated for security reasons, so keep in mind, that if your infrastructure grows or your needs change, clusters can offer another layer of organization and control, especially when needing to enforce stricter access controls and resource allocations.
Clustering agents is more about scaling and managing large numbers of agents, particularly when they are spread across different environments or need to be segregated for security reasons, so keep in mind, that if your infrastructure grows or your needs change, clusters can offer another layer of organization and control, especially when needing to enforce stricter access controls and resource allocations.