Can I `include` one pipeline yaml in another?

Features Requests
1

I’d like to sort of include one pipeline in another one.
Currently I can add a step to the first pipeline forcing it to upload my second pipeline. eg
buildkite-agent pipeline upload mysecond-pipeline.yml
However with this I’d have to duplicate all definitions in the second yaml file as well
I want to avoid that and create more modular pipelines. Can this be done?

2

Hi @js-cb

Please can you explain in more detail what you mean by “I have to duplicate all definitions in the second yaml” ?

You can have first pipeline where one of the step is pipeline upload then have remaining steps in your first pipeline. Only the steps you want to be part of second pipeline you can include in pipeline definition that you are uploading.

Is that you need to run same steps in both pipelines ?

3

See this example. I tend to prefer cat over echo as it reads cleaner.

In your pipeline.yml file you need a Pipeline upload step:

steps:
  - label: "Yada"
    command: ".buildkite/pipeline.sh | buildkite-agent pipeline upload"

Then in your pipeline.sh file have something like this:

#!/bin/bash
set -eu

cat << EOF
steps:
  - label: "Tada"
    command: "npm start"
EOF
4

Example:

_agents: &agents
  queue: docker

I call that a definition.
And I use it here:

steps:

- label: 'Cache checkout and set metadata'
  agents: *agents

If I had the ability to include yaml files then I would not have to repeat these in all YAML files which i upload using the command I mentioned

5

Hey @js-cb!

The alternative that iammcgaber offered it’s a good one, and it’s what we recommend for these cases.

Best!

6

Firstly, thanks for the bash method iamcgaber.

Secondly, out of curiosity, has there been any progression around standardising this feature request?

Github Actions has a concept of sharing and reusing workflows, which supports both private and public repos.

So essentially you can have the situation:

some isolated body of steps:

# PATH/TO/WORKFLOW/COMPONENT.yml
name: workflow-componet
  description: test action
  inputs:
    test_input:
      description: key
      required: true
  runs:
    using: composite
    steps:
      - name: defined once, used everywhere
        shell: bash
        run: |
          echo "hello ${{inputs.test_input}}"

main pipeline/workflow

  - name: test-action
    uses: ./PATH/TO/WORKFLOW/COMPONENT
    with:
      test_input: "world"

That way you can easily define a collection of steps once and reuse it everywhere. It would be great if there was a more standard way to modularise pipelines.

7

Hey @benmcp!

We don’t support this out of the box and the way that GHA does it is via their parser; they scan for uses and inject it in to the workflow, or so I believe.

There is a way to do this, which I’ve written about previously and should hopefully help.

Cheers!