🚀 Defining Pipeline Build Steps with YAML


#1

You can now define pipeline steps using YAML directly in Buildkite. The feature is currently in beta, and we’d love to hear what you think.

How to convert your pipeline to YAML

You can switch to defining pipelines with YAML on the Pipeline Settings page. You’ll see a section like this:

Environment Variable Precedence

With the introduction of this new way of defining steps in Buildkite, we’ve tweaked the precedence logic for environment variables.

The current precedence/merging logic looks like this:

pipeline_environment. # defined on the pipeline
  merge(step_environment). # defined on the step
  merge(build_environment). # passed through the API or the "New Build" dialog
  merge(buildkite_environment) # $BUILDKITE_ and friends

The fact that custom build environment overrides custom step environment can be problematic. As an example, given a pipeline like this…

env:
  LEVEL: "pipeline"

steps:
  command: "echo $LEVEL"
  env:
    LEVEL: "step"

…if you create a build using the “New Build” dialog and add an environment variable of “LEVEL=build”, it would override all instances of $LEVEL (even if you defined an override at the step level).

We’ve changed it to this:

pipeline_environment.
  merge(build_environment). # 👇 (swapped)
  merge(step_environment). # ☝️ (swapped)
  merge(buildkite_environment)

So creating a build with a custom environment overrides the pipeline environment, but the step environment always takes precedence.

The other notable change is that we’ve stopped the env block in a pipeline.yml file from taking precedence over all other environment variables. Using the same pipeline.yml file from earlier…

env:
  LEVEL: "pipeline"

steps:
  command: "echo $LEVEL"
  env:
    LEVEL: "step"

…if you were to upload this pipeline currently with buildkite-agent pipeline upload command, the result would be echo pipeline, which is probably not what you’d expect. We consider this a bug, so we’ve fixed it. The result is now echo step as you’d expect. Additionally, any environment variable defined in the top level env block, will be made available to all future buildkite-agent pipeline upload commands.

Interpolation Support

Environment variable interpolation is also available:

env:
  EMOJI: "🦙"

steps:
  - command: >
      echo "$BUILDKITE_ORGANIZATION_SLUG <3 $EMOJI"

You can also use the same “default value” syntax as you use in Bash:

steps:
  - command: >
      echo "$BUILDKITE_ORGANIZATION_SLUG <3 ${EMOJI-🐶}"

In this example, if EMOJI isn’t defined, it will default to “:dog:

You can learn more about how interpolation works here: https://buildkite.com/docs/agent/v3/cli-pipeline#environment-variable-substitution

Only a subset of environment variables are available for interpolation at this level:

  • BUILDKITE_BRANCH
  • BUILDKITE_MESSAGE
  • BUILDKITE_COMMIT
  • BUILDKITE_PIPELINE_SLUG
  • BUILDKITE_ORGANIZATION_SLUG
  • BUILDKITE_PULL_REQUEST
  • BUILDKITE_PULL_REQUEST_BASE_BRANCH
  • BUILDKITE_PULL_REQUEST_REPO

Let us know if you’d like any other ones included. (Some variables like BUILDKITE_BUILD_NUMBER can’t be supported as interpolation happens before the build is created)

We love feedback!

Please get in touch if you have any thoughts/comments on the feature. The goal is to eventually make this default, and remove the current way of defining steps. We don’t have a timeline on when we’ll make the full switch.


#4

Does this mean moving away from shell based commands per web-interface-step in favor of moving these steps directly to yaml?
As long as I’d still have the option to keep all my steps in yaml format in a git repository, and using the pipeline upload command to upload them, this sounds like a welcome improvement!

Does this also mean a source repository as part of a pipeline will become optional?


#5

Gave this a spin on an existing pipeline, and I’ve hit a bit of a problem. Environment variables defined in the pipeline upload step only apply to that step and so are not accessible in any of the steps uploaded from source. The original (pre upgrade) env vars were global to all steps in the pipeline and it was relying on that to work.

Any suggestions to work around?


#6

The lack of pipeline-level environment variable is a problem for us too. Is there a way to set pipeline level environment variables in the new yaml?

Otherwise great change. We wanted the ability to set a concurrency group on the upload step which this can now do.


#7

@thom.koomen Good question! Shell based commands are always going to be a thing, and buildkite-agent pipeline upload is still around. We’ve just changed how you define your initial steps. Instead of using the old step based UI that had limited controls, you can just use the same YAML syntax that you know to define your initial steps.

For example, many of our pipelines look like this:

I’d love to see repository-less pipelines in the future as an option! I think there’s some good ideas here, to support use-cases like one off pipelines, or even recurring jobs you want to run on agents but don’t need a Git repository for.

Thanks for the questions!


#8

@jam13 hrm…that doesn’t sound right at all! Just to clarify, do you mean in this pipeline here:

steps:
  - command: "buildkite-agent pipeline upload"
    env:
      FOO: "bar"

FOO=bar isn’t available as an interpolation in the uploaded pipeline.yml file?


#9

@chrisrbnelson this should work:

env:
  SOMETHING: "test"

steps:
  - command: "buildkite-agent pipeline upload"

Let me know how you go with that!


#10

So this is the first time we’ve used the forum to announce a beta feature and collect feedback on it, and I think having questions/feedback post directly to this topic could get a bit unwieldy quickly! (it also means I can’t use the forums “here is the solution” button to pin answers to the top of posts)

I might lock this post, and ask if you want to ask a question or give some further feedback - do it in a new topic here: https://forum.buildkite.community/c/beta-support-feedback/yaml-pipeline-steps. I really appreciate you taking the time to post here @thom.koomen @jam13 @chrisrbnelson! (also sorry for the annoyance around having to change topics now, I’m still learning the best way to do this :stuck_out_tongue: )


#11

#12