Pass metadata to triggered pipelines with pure YAML

Features Requests
1

I’ll paste the interaction from Slack for posterity:

When using a trigger step (https://buildkite.com/docs/pipelines/trigger-step), I see that we can pass meta-data to the triggered pipeline. I have some data that I generate and set as meta-data within the triggering pipeline that I’d like to pass to the triggered pipeline. Absent dynamically generating the content of the trigger step, is there any way I can set this up using pure YAML? Something like, say:

- trigger: "dependent-pipeline"
  label: "Doing nifty things"
  build:
    meta_data:
    - key_that_i_set_earlier_in_the_triggering_pipeline

And @keithpitt responded:

Hey @cm! It’s a good question! The only way to do it at the moment is to have a step that re-generates the trigger step at runtime and pipeline uploads it
(basically kinda what you were thinking)
However, I hadn’t considered the docker-compose model before as a quick shortcut…
Can you shoot this through to support and say “Keith said this could be a good idea?”

Thanks!

2

+1 to this - we have a lot of pipelines which trigger other pipelines with dynamic metadata.

e.g:

steps:
  - command: "create_deployment.sh" # sets deployment-name metadata

  - wait: ~

  # script which reads metadata + uploads trigger step below
  - command: "profile_deployment.sh" 

  - trigger: "deployment-profiling"
    build:
       meta_data:
          # has to be resolved by script, doesnt work in steps.yaml
          deployment-name: $(buildkite-agent meta-data get deployment-name)

It would be ideal if we could remove the profile_deployment.sh script and just directly put that trigger step into the yaml

3

This is a great idea, thanks for taking the time to raise this! I have raised this internally with the team!

4

Hey, I’ve seen this idea floated around in a few places, and I’d like to ask is there any update on this feature? I’d be happy to submit a PR to implement it, but unfortunately I would need some help finding where this behaviour happens on the agent.

Our specific use case for this is that we have build with lots of options, and we have some defaults set for UI triggered builds, but we’d like to add a second “advanced” build that just triggers the first with some metadata customised. It would be really great if we didn’t have to duplicate the metadata parsing logic in both pipelines in a script, and instead could just do:

- trigger: "multi-option-build"
  build:
    meta_data:
        option_a: "meta:option_a"
        option_b: "meta:option_b"
5

Welcome back to the forum, @Donal! Thank you for your suggestion!

Unfortunately, metadata is not handled by the Buildkite agent but by Buildkite itself, so a PR for an agent will not solve your issue. The idea of implementing passing metadata to triggered pipelines with YAML has been floating around for some time on the forum. However, for the time being, the only possible workaround is using dynamic pipelines, trigger step, and a command to get metadata from the trigger pipeline.

You’ll need the triggered pipeline (Pipeline B) to use the command buildkite-agent meta-data get "release-version" in the pipeline steps to get the metadata contained in the trigger pipeline (Pipeline A).

For example:

Pipeline A (trigger)

steps:
  - trigger: pipeline-b-slug
    build: 
      meta_data:
        release-version: "2.0"

Pipeline B (triggered)

steps:
  - command: echo $(buildkite-agent meta-data get "release-version")

I hope this helps.

Cheers!