How to download artifacts back from triggered pipeline

General
1

Hello,

I am trying to implement following workflow working that involves two pipelines, let’s call them A and B. The basic workflow goes:

  • A produces an artifact
  • A triggers B and waits.
  • B download the artifact from A using BUILDKITE_TRIGGERED_FROM_BUILD_ID environment variable.
  • B finishes its pipeline with another artifacts generated
  • A resumes and wants to download the artifact generated by B.

One of the way we thought of was to access Builds API to get the BUILD_ID of the triggered step. But that doesn’t seem clean and we need to give away token to hit the API. I was hoping there’s a way to get either triggered_build_id using buildkite-agent command, such as buildkite-agent step get.

Do you have any suggestion on how to achieve this?

Thanks,

2

:wave: Welcome to the community! :hugs:

It’s possible to achieve what you are looking for by using build meta-data, although it may not be too pretty :sweat_smile: You’ll have for example:

Pipeline A:

steps:
  - command: buildkite-agent artifact upload "test.jpg"
  - trigger: pipeline-b
    build:
      env:
        TRIGGER_JOB_ID: $BUILDKITE_JOB_ID
  - wait: 
  - command: buildkite-agent artifact download "*.*" .  --build "$(buildkite-agent meta-data get "triggered_build_id")"

Pipeline B:

steps:
  - command: buildkite-agent artifact download "*.jpg" . --build $BUILDKITE_TRIGGERED_FROM_BUILD_ID
  - command: buildkite-agent artifact upload "another.txt"
  - command: buildkite-agent meta-data set "triggered_build_id" "$$BUILDKITE_BUILD_ID" --job $TRIGGER_JOB_ID

A clarification, being unable to interpolate the BUILDKITE_JOB_ID it’s a caveat of the YAML steps editor, so if you get the error “Interpolating “BUILDKITE_JOB_ID” is currently not supported” then uploading a pipeline file would allow it because the variables are interpolated be the agent.

Another option that I believe is nicer is to pass the JOB_ID from the parent to the child (like Pipeline A), and from the child’s job, do the artifact upload with that job_id

3

Hi @paula

I didn’t know one could upload an artifact or create meta data across pipelines. This is very helpful tip that I could use. Thank you :bowing_man:

For the problem that I described earlier, pipeline B is maintained by other team, so I will have to coordinate with them. If we agree to go with suggested workflow, I think metadata seems little more flexible since Pipeline A has control over where to download the artifacts to.

I still think it would be great if triggered_build_id is retrievable using buildkite-agent step get.

Thank you so much for your help.

2 Likes
4

Hi @Germicide5416 i am from the product team. Thanks a lot for the feedback. This is not prioritised for the near term but will be in our backlog to get to. We’ll let you know once we pick it up.

5

Hi @paula
Tried the second approach when the triggered job uploads artifacts to the parent job by its job id.
The sub-job example:

command: buildkite-agent artifact upload my-file  --job $${BUILDKITE_TRIGGERED_FROM_BUILD_ID}

Got an error (replaced job id with <JOB_ID>):

fatal: failed to upload artifacts: uploading artifacts: POST https://agent.buildkite.com/v3/jobs/<JOB_ID>/artifacts: 404 Not Found: Could not find job: <JOB_ID>

I double-checked that BUILDKITE_TRIGGERED_FROM_BUILD_ID(a buildkite managed variable) is correct and matches the parent build id. What am I doing wrong?

6

Hi @pazonec ,

This is Suma from Buildkite support team. Firstly, welcome to Buildkite community and thank you for reaching out to us with your question.

Issue I see here is you are passing build id (BUILDKITE_TRIGGERED_FROM_BUILD_ID) to the --job which is expecting job id.

This is the reason it failed to upload the artifact. So like shown in example from Paula, you would need to pass the job id of the parent pipeline build to the triggered pipeline and then use that for upload

steps:
  - command: buildkite-agent artifact upload "test.jpg"
  - trigger: pipeline-b
    build:
      env:
        TRIGGER_JOB_ID: $BUILDKITE_JOB_ID
  - wait: 
  - command: buildkite-agent artifact download "*.*" .  --build "$(buildkite-agent meta-data get "triggered_build_id")"

Now use the TRIGGER_JOB_ID in the child pipeline artifact upload command.

I hope this answers your question. Please feel free to reach out if you have any further questions.

7

@suma $BUILDKITE_JOB_ID in the env no longer works anno 2026. It says you should replace it with a runtime interpolation ($$BUILDKITE_JOB_ID), but that will replace it will the triggered ID, not the triggering build ID.

Our use case is a bit similar where we have a “build” triggered substep of our “release” pipeline (when not built yet). I want to be able to get our built artifact (saved on the build pipeline) without resorting to the HTTP API, but there’s no way to get the ID of the triggered build. I’ve tried all combinations of meta-data setting, to no avail. Also buildkite-agent step get --format json --step stage in the parent does not include a “triggede_build.id” path, which would have made sense.

Any great ideas?

8

Hey @emielm

Because ${BUILDKITE_BUILD_ID} cannot be interpolated at runtime in a previously uploaded pipeline, the solution is to upload the pipeline dynamically and pass the required environment variables during that upload step.

Parent Pipeline (Release Pipeline)

steps:
  - label: "Parent Upload"
    command: "buildkite-agent pipeline upload .buildkite/release.yml"

.buildkite/release.yml

# Parent pipeline (the triggering “release” pipeline)
steps:
  - label: ":arrow_right: Trigger build pipeline"
    key: "trigger_build"
    trigger: "merge"   # slug of the child pipeline
    build: 
     env:
      PARENT_TRIGGER_JOB_ID: "${BUILDKITE_JOB_ID}"
  - wait: ~
  - command: |
      export MY_VAR="$(buildkite-agent meta-data get child_build_id)"
      buildkite-agent artifact download "dist/release.tgz" . --build "$$MY_VAR"

In this setup, the parent pipeline uploads its pipeline definition so ${BUILDKITE_JOB_ID} can be interpolated at upload time and passed to the triggered pipeline. The child pipeline then uses that value to write the build ID back to the parent via meta-data, which the parent retrieves after the wait step.

Child Pipeline (Build Pipeline)

steps:
  - label: "Child Pipeline Upload"
    command: "buildkite-agent pipeline upload .buildkite/build.yml"

.buildkite/build.yml

steps:
  - label: ":hammer: Build + upload + report build id to parent"
    command: |
      set -euo pipefail

      mkdir -p dist
      echo "hello from child build ${BUILDKITE_BUILD_ID}" > dist/info.txt
      tar -czf dist/release.tgz -C dist info.txt

      buildkite-agent artifact upload "dist/release.tgz"
      buildkite-agent meta-data set child_build_id "$BUILDKITE_BUILD_ID" --job "$PARENT_TRIGGER_JOB_ID"