Any way to make all pipelines inherit common pipeline configuration?

Pipelines
1

Hi,
Question as in the title.

And ‘common pipeline configuration’ would be things like:

  • env: variables
  • YAML anchors. Would be even better if it’s possible to dynamically populate values. For example:
common:
  - aws_assume_deployer_role: &aws_role_dev
      aws-assume-role-with-web-identity:
        role-arn: arn:aws:iam::${BUILDKITE_AGENT_META_DATA_AWS_ACCOUNT_ID}:role/${BUILDKITE_PIPELINE_NAME}-deploy
  - vault: &secrets_dev
      PropellerAero/vault-secrets:
        auth:
          role: ${BUILDKITE_PIPELINE_NAME}-deployer
        path: app-secrets/${BUILDKITE_AGENT_META_DATA_ENVIRONMENT_NAME}/BUILDKITE_PIPELINE_NAME/deployment

Feel free to suggest different patterns that would accomplish the same goal: reducing boilerplate across many pipelines that adhere to some conventions.

2

Hey there, @alp_propeller!

There are definitely a couple of ways to approach this however one that I’d lean towards would be to use a preflight directory in the repo of the service, along side your .buildkite directory. The structure at the end will look something like this:

.
├── .buildkite
│   └── pipeline.yaml
└── preflight
    ├── preflight.yaml
    └── upload

The ./preflight/pipeline.yaml contains all of my env and anchors, I’m using Docker in this example.

env:
  NAME: "Ben"
  IMAGE: "node:current-alpine3.17"
docker: &docker
  docker#v5.6.0:
    image: ${IMAGE}

Note that you are welcome to use a common: heading for your anchors, I just chose not to.

My .buildkite/pipeline.yaml is pretty basic, has a single step which utilises the things in my preflight.yaml:

steps:
  - label: Hello, world!
    plugins:
      *docker
    command: |
      echo "Hello, $NAME"
      sleep 3

The upload file is where all of the functionality lives, it looks like this:

#!/bin/bash

set -eou pipefail

echo -e "$(cat ./preflight/preflight.yaml)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

buildkite-agent pipeline upload .buildkite/pipeline.yaml

What this does it, echos the current state of the preflight.yaml file and the pipeline.yaml file, joined, to the pipeline.yaml file, overwriting the current state with the new state.

It would also be possible to clone down a “central” repository, containing your common configuration and cat the contents of the configuration file through the command, in place of ./preflight/preflight.yaml in the echo:

echo -e "$(cat ./PATH/TO/CONFIG/FILE)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

Example:

#!/bin/bash

set -eou pipefail

git clone git@github.com:USERNAME/configs.git

echo -e "$(cat ./configs/preflight.yaml)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

buildkite-agent pipeline upload .buildkite/pipeline.yaml

Finally, the YAML steps for your pipeline would be:

steps:
  - command: "./preflight/upload"
    label: ":pipeline: Generate steps"

There are also other tools that can be used to concat the YAML files, such as yq, Yq - yq, however that isn’t installed on the agent by default so I wanted to ensure there was a way without relying on external tooling.

Hope that helps get you on your way!

Cheers,

Ben

1 Like
3

This approach looks promising, thanks Ben!

I was about to reply to this thread to ask the possibility of piping things other than steps through buildkite-agent pipeline upload, as the documentation uses the word ‘steps’ which makes it sound like it’d only be able to dynamically alter the steps, but I am reassured by your well detailed explanation.

Will play with this and report back if I hit a snag.

Cheers,
Alp

4

Awesome, let us now how you go @alp_propeller!

I’ve added in an example for cloning a remote repo, which might come in handy for you.

Good luck!

1 Like