How to download artifacts back from triggered pipeline

Hello,

I am trying to implement following workflow working that involves two pipelines, let’s call them A and B. The basic workflow goes:

  • A produces an artifact
  • A triggers B and waits.
  • B download the artifact from A using BUILDKITE_TRIGGERED_FROM_BUILD_ID environment variable.
  • B finishes its pipeline with another artifacts generated
  • A resumes and wants to download the artifact generated by B.

One of the way we thought of was to access Builds API to get the BUILD_ID of the triggered step. But that doesn’t seem clean and we need to give away token to hit the API. I was hoping there’s a way to get either triggered_build_id using buildkite-agent command, such as buildkite-agent step get.

Do you have any suggestion on how to achieve this?

Thanks,

:wave: Welcome to the community! :hugs:

It’s possible to achieve what you are looking for by using build meta-data, although it may not be too pretty :sweat_smile: You’ll have for example:

Pipeline A:

steps:
  - command: buildkite-agent artifact upload "test.jpg"
  - trigger: pipeline-b
    build:
      env:
        TRIGGER_JOB_ID: $BUILDKITE_JOB_ID
  - wait: 
  - command: buildkite-agent artifact download "*.*" .  --build "$(buildkite-agent meta-data get "triggered_build_id")"

Pipeline B:

steps:
  - command: buildkite-agent artifact download "*.jpg" . --build $BUILDKITE_TRIGGERED_FROM_BUILD_ID
  - command: buildkite-agent artifact upload "another.txt"
  - command: buildkite-agent meta-data set "triggered_build_id" "$$BUILDKITE_BUILD_ID" --job $TRIGGER_JOB_ID

A clarification, being unable to interpolate the BUILDKITE_JOB_ID it’s a caveat of the YAML steps editor, so if you get the error “Interpolating “BUILDKITE_JOB_ID” is currently not supported” then uploading a pipeline file would allow it because the variables are interpolated be the agent.

Another option that I believe is nicer is to pass the JOB_ID from the parent to the child (like Pipeline A), and from the child’s job, do the artifact upload with that job_id

Hi @paula

I didn’t know one could upload an artifact or create meta data across pipelines. This is very helpful tip that I could use. Thank you :bowing_man:

For the problem that I described earlier, pipeline B is maintained by other team, so I will have to coordinate with them. If we agree to go with suggested workflow, I think metadata seems little more flexible since Pipeline A has control over where to download the artifacts to.

I still think it would be great if triggered_build_id is retrievable using buildkite-agent step get.

Thank you so much for your help.

2 Likes

Hi @Germicide5416 i am from the product team. Thanks a lot for the feedback. This is not prioritised for the near term but will be in our backlog to get to. We’ll let you know once we pick it up.