Any way to make all pipelines inherit common pipeline configuration?

Pipelines
2

Hey there, @alp_propeller!

There are definitely a couple of ways to approach this however one that I’d lean towards would be to use a preflight directory in the repo of the service, along side your .buildkite directory. The structure at the end will look something like this:

.
├── .buildkite
│   └── pipeline.yaml
└── preflight
    ├── preflight.yaml
    └── upload

The ./preflight/pipeline.yaml contains all of my env and anchors, I’m using Docker in this example.

env:
  NAME: "Ben"
  IMAGE: "node:current-alpine3.17"
docker: &docker
  docker#v5.6.0:
    image: ${IMAGE}

Note that you are welcome to use a common: heading for your anchors, I just chose not to.

My .buildkite/pipeline.yaml is pretty basic, has a single step which utilises the things in my preflight.yaml:

steps:
  - label: Hello, world!
    plugins:
      *docker
    command: |
      echo "Hello, $NAME"
      sleep 3

The upload file is where all of the functionality lives, it looks like this:

#!/bin/bash

set -eou pipefail

echo -e "$(cat ./preflight/preflight.yaml)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

buildkite-agent pipeline upload .buildkite/pipeline.yaml

What this does it, echos the current state of the preflight.yaml file and the pipeline.yaml file, joined, to the pipeline.yaml file, overwriting the current state with the new state.

It would also be possible to clone down a “central” repository, containing your common configuration and cat the contents of the configuration file through the command, in place of ./preflight/preflight.yaml in the echo:

echo -e "$(cat ./PATH/TO/CONFIG/FILE)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

Example:

#!/bin/bash

set -eou pipefail

git clone git@github.com:USERNAME/configs.git

echo -e "$(cat ./configs/preflight.yaml)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

buildkite-agent pipeline upload .buildkite/pipeline.yaml

Finally, the YAML steps for your pipeline would be:

steps:
  - command: "./preflight/upload"
    label: ":pipeline: Generate steps"

There are also other tools that can be used to concat the YAML files, such as yq, Yq - yq, however that isn’t installed on the agent by default so I wanted to ensure there was a way without relying on external tooling.

Hope that helps get you on your way!

Cheers,

Ben

1 Like