Allow agent targeting overrides through UI & API before running pipelines

Features Requests
1

Recently I find myself wanting to test my pipelines on different agents because I’m trying to replace some older agents. Currently the only way to do this is to manually modify the code where my pipelines are defined and change the targeting queue tag values.

But I think the ideal solution would be to have this supported in Buildkite somehow.

Here’s my ideal scenario:

  1. I trigger a build for a pipeline either through the UI or via API and specify a custom queue override (eg. with an env var like BUILDKITE_AGENT_META_DATA_QUEUE=something_new - but the specific env var or mechanism can be different).
  2. Buildkite recognizes this and overrides all agent targeting for all a) jobs in the build (including dynamically added ones) and b) all triggered pipelines from the build, regardless of what the targeting parameters are in each pipeline’s YAML/uploaded code.

Would this be feasible to have as a feature in Buildkite?

2

Sorry Buildkite doesn’t do that!

Another way to solve this would be to add a parameter to your pipeline YAML that can be overridden by an environment variable. For example, if this was your pipeline YAML:

steps: 
- command: true 
  agents: 
    queue: something

Modify it to use an environment variable with a default, like this:

steps: 
- command: true 
  agents: 
    queue: ${QUEUE_NAME:-something}

And then you can create a new pipeline pointing at the same pipeline YAML, but with QUEUE_NAME=something_new in the pipeline environment.

Would something like that work?

3

The problem is there are a lot of pipelines and many of them also dynamically generate build steps or trigger more pipelines. So I have to update all those code points to either use env var substitution and/or pass the env vars down to dynamically created builds and triggered pipelines. The sheer number of pipelines I’m working with makes this a very tedious task. Hence, it’d be much nicer if Buildkite was updated to provide some functionality like this.

4

That sounds like you’ve got a really interesting use case!

We’ve done migration like this in the past by editing the pipeline yamls, I’d love to hear more about your use case and the structure of your pipelines to understand why this feels like a tedious task?

5

So in my case there are several pipelines inside a large multi-project/service repo with an inconsistent blend of pipeline definitions. Some are in YAML, others use Python, and yet others use shell scripts to generate dynamic pipelines. There may be even more methods - I’m not familiar with the entire code base just yet… And there are several teams and engineers working on all the code simultaneously at any given point. Consolidation is good, but that doesn’t guarantee people who create new/modify their pipelines will remember to do so. This is why it’d be nicer if this could be done at the service level.

6

Hmmm, it sounds like you might need to work with each of the teams individually to get the agent migration done! Perhaps you can help them achieve more consistency?

We’re unlikely to add a feature to support overriding queue names with an environment variable provided from outside the agent environment, as that might introduce unexpected security issues for some organisations. We protect agent configuration options from direct modifications on buildkite.com as otherwise this could be a security vulnerability, for example this env variable.

If you are really keen, you could write a wrapper script for the buildkite agent, and pre-process pipelines before they are uploaded.