Dynamically select which agent to use

General
1

Hello, new to buildkite and keen to explore what it’s all capable of.
What I’m currently stuck on is trying to select which agent to used based on the branch that triggered the build.

So for example:
Branch name is mybranch-development, I want to cut the end of this branch name to deploy to the agent that has the tag Environment=development.

I tried the below, but I don’t seem to be able to get it working:

steps:
  - label: Run Ansible Script
    command:
     # - agent=${BUILDKITE_BRANCH##*-} (I tried this but got an error)
      - agent=$(echo $BUILDKITE_BRANCH | rev | cut -d- -f1 | rev) #(This should work to return just the word development)
      - echo $agent #(this outputs nothing, so the above variable doesn't appear to be saving or working)
      - my commands
    plugins:
      - docker#v3.1.0:
          image: "centos:7"
          always-pull: true
          propagate-environment: true
    agents:
      Environment: $agent #(this is where I would want to use the last part of my branch name )

The branch could be mybranch-staging or mybranch-production and I would want to use the staging or production agent respectively.

This would allow me to have one pipeline that runs all builds - the alternative would be to have three different pipelines that only runs based on the branch name being *-development, *-staging, etc… and then the agent is static for each.

2

Welcome! :wave:t2:

One way to solve that would be to use dynamic pipeline uploads and have a script generate your pipeline to upload. Your step would look like .buildkite/pipeline.sh | buildkite-agent pipeline upload, and then you’d have a .buildkite/pipeline.sh that looked something like this:

#!/bin/bash
set -euo pipefail
agent_branch_name="$(echo $BUILDKITE_BRANCH | rev | cut -d- -f1 | rev)"

cat <<YAML 
steps:
  - label: Run Ansible Script
    agents:
      queue: "${agent_branch_name}"
    command:
      - echo running on queue=${agent_branch_name} in docker
      - my commands
    plugins:
      - docker#v3.1.0:
          image: "centos:7"
          always-pull: true
          propagate-environment: true
YAML

I hope that helps!

3

Thanks, great answer.
Got me thinking and I did something similar:
My pipeline upload has the following steps:

steps:
  - command: 
      - "export AGENT_ENVIRONMENT=$${BUILDKITE_BRANCH##*-}"
      - "envsubst < .buildkite/pipeline.yml > .buildkite/pipeline_ready.yml"
      - "buildkite-agent pipeline upload .buildkite/pipeline_ready.yml"

Means that my pipeline.yaml file is still a plain yaml file but just has:

  agents:
      Environment: $AGENT_ENVIRONMENT

which the envsubst replaces.
Thanks for setting me off in the right direction.

4

I’ve overlooked something, but I don’t think there is a way around it - maybe you could confirm.

As per https://buildkite.com/docs/pipelines/defining-steps#dynamic-pipelines:

Because the pipeline upload step runs on your agent machine, you can generate pipelines dynamically using scripts from your source code.

This then means that I need to make use of an agent before even running the steps to run my pipeline.sh.
So in a situation where I have a DEV environment (and corresponding build agent) and a PROD environment (also with it’s own agent) - I would still need to decide which agent will be used for the initial step of running my pipeline.sh and uploading it for all jobs.

5

Ah right, yes you are correct. Another approach would be to have multiple pipelines for the same repository and then use branch filters in the pipeline repository setup.

For instance, you might have myproject which only runs the master branch, and then you might have myproject-dev which runs *-development branches.

You can see we use this technique in the agent to filter *-stable branches only.

Settings%20%C2%B7%20Agent%202019-04-29%2016-58-46
Settings · Agent 2019-04-29 16-58-46.png2442×1326 267 KB

Might that be a better fit?

6

Sorry for steering you in the wrong direction initially @mr_green, I got too excited about dynamic pipeline uploads :sweat_smile:

7

No problem at all, I learnt quite a bit going down dynamic pipeline route - and will definitely use it in other places. :wink:

1 Like