# Any way to make all pipelines inherit common pipeline configuration?

**URL:** <https://forum.buildkite.community/t/any-way-to-make-all-pipelines-inherit-common-pipeline-configuration/2898>\
**Category:** Pipelines\
**Created:** [May 4, 2023, 10:25am UTC](https://forum.buildkite.community/t/any-way-to-make-all-pipelines-inherit-common-pipeline-configuration/2898 "2023-05-04T10:25:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![alp\_propeller](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@alp\_propeller](https://forum.buildkite.community/u/alp_propeller)\
**Post date:** [May 4, 2023, 10:25am UTC](https://forum.buildkite.community/t/any-way-to-make-all-pipelines-inherit-common-pipeline-configuration/2898/1 "2023-05-04T10:25:03Z")

</div>

Hi,  
Question as in the title.

And ‘common pipeline configuration’ would be things like:

- `env:` variables
- YAML anchors. Would be even better if it’s possible to dynamically populate values. For example:

```auto
common:
  - aws_assume_deployer_role: &aws_role_dev
      aws-assume-role-with-web-identity:
        role-arn: arn:aws:iam::${BUILDKITE_AGENT_META_DATA_AWS_ACCOUNT_ID}:role/${BUILDKITE_PIPELINE_NAME}-deploy
  - vault: &secrets_dev
      PropellerAero/vault-secrets:
        auth:
          role: ${BUILDKITE_PIPELINE_NAME}-deployer
        path: app-secrets/${BUILDKITE_AGENT_META_DATA_ENVIRONMENT_NAME}/BUILDKITE_PIPELINE_NAME/deployment

```

Feel free to suggest different patterns that would accomplish the same goal: reducing boilerplate across many pipelines that adhere to some conventions.

---

<div class="post-metadata">

**Author:** ![benmc](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/benmc/32/1159_2.png) [@benmc](https://forum.buildkite.community/u/benmc)\
**Post date:** [May 5, 2023, 1:26am UTC](https://forum.buildkite.community/t/any-way-to-make-all-pipelines-inherit-common-pipeline-configuration/2898/2 "2023-05-05T01:26:14Z")

</div>

Hey there, @alp_propeller!

There are definitely a couple of ways to approach this however one that I’d lean towards would be to use a `preflight` directory in the repo of the service, along side your `.buildkite` directory. The structure at the end will look something like this:

```auto
.
├── .buildkite
│ └── pipeline.yaml
└── preflight
    ├── preflight.yaml
    └── upload

```

The `./preflight/pipeline.yaml` contains all of my `env` and `anchors`, I’m using Docker in this example.

```yaml
env:
  NAME: "Ben"
  IMAGE: "node:current-alpine3.17"
docker: &docker
  docker#v5.6.0:
    image: ${IMAGE}

```

Note that you are welcome to use a `common:` heading for your anchors, I just chose not to.

My `.buildkite/pipeline.yaml` is pretty basic, has a single step which utilises the things in my `preflight.yaml`:

```yaml
steps:
  - label: Hello, world!
    plugins:
      *docker
    command: |
      echo "Hello, $NAME"
      sleep 3

```

The upload file is where all of the functionality lives, it looks like this:

```bash
#!/bin/bash

set -eou pipefail

echo -e "$(cat ./preflight/preflight.yaml)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

buildkite-agent pipeline upload .buildkite/pipeline.yaml

```

What this does it, `echos` the current state of the `preflight.yaml` file and the `pipeline.yaml` file, joined, to the `pipeline.yaml` file, overwriting the current state with the new state.

It would also be possible to `clone` down a “central” repository, containing your common configuration and `cat` the contents of the configuration file through the command, in place of `./preflight/preflight.yaml` in the `echo`:

```auto
echo -e "$(cat ./PATH/TO/CONFIG/FILE)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

```

Example:

```bash
#!/bin/bash

set -eou pipefail

git clone git@github.com:USERNAME/configs.git

echo -e "$(cat ./configs/preflight.yaml)\n$(cat .buildkite/pipeline.yaml)" > .buildkite/pipeline.yaml

buildkite-agent pipeline upload .buildkite/pipeline.yaml

```

Finally, the YAML steps for your pipeline would be:

```yaml
steps:
  - command: "./preflight/upload"
    label: ":pipeline: Generate steps"

```

There are also other tools that can be used to concat the YAML files, such as `yq`, [Yq - yq](https://mikefarah.gitbook.io/yq/v/v2.x), however that isn’t installed on the agent by default so I wanted to ensure there was a way without relying on external tooling.

Hope that helps get you on your way!

Cheers,

Ben

---

<div class="post-metadata">

**Author:** ![alp\_propeller](https://avatars.discourse-cdn.com/v4/letter/a/848f3c/32.png) [@alp\_propeller](https://forum.buildkite.community/u/alp_propeller)\
**Post date:** [May 5, 2023, 2:04am UTC](https://forum.buildkite.community/t/any-way-to-make-all-pipelines-inherit-common-pipeline-configuration/2898/3 "2023-05-05T02:04:09Z")

</div>

This approach looks promising, thanks Ben!

I was about to reply to this thread to ask the possibility of piping things other than `step`s through `buildkite-agent pipeline upload`, as the documentation uses the word ‘steps’ which makes it sound like it’d only be able to dynamically alter the `step`s, but I am reassured by your well detailed explanation.

Will play with this and report back if I hit a snag.

Cheers,  
Alp

---

<div class="post-metadata">

**Author:** ![benmc](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/benmc/32/1159_2.png) [@benmc](https://forum.buildkite.community/u/benmc)\
**Post date:** [May 5, 2023, 2:04am UTC](https://forum.buildkite.community/t/any-way-to-make-all-pipelines-inherit-common-pipeline-configuration/2898/4 "2023-05-05T02:04:59Z")

</div>

Awesome, let us now how you go @alp_propeller!

I’ve added in an example for cloning a remote repo, which might come in handy for you.

Good luck!

---

<div class="post-metadata">

**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\
**Post date:** [August 29, 2023, 5:39am UTC](https://forum.buildkite.community/t/any-way-to-make-all-pipelines-inherit-common-pipeline-configuration/2898/5 "2023-08-29T05:39:30Z")

</div>

Closed on August 29, 2023, 5:39am UTC


