How to download artifacts back from triggered pipeline

General
1

Hello,

I am trying to implement following workflow working that involves two pipelines, let’s call them A and B. The basic workflow goes:

  • A produces an artifact
  • A triggers B and waits.
  • B download the artifact from A using BUILDKITE_TRIGGERED_FROM_BUILD_ID environment variable.
  • B finishes its pipeline with another artifacts generated
  • A resumes and wants to download the artifact generated by B.

One of the way we thought of was to access Builds API to get the BUILD_ID of the triggered step. But that doesn’t seem clean and we need to give away token to hit the API. I was hoping there’s a way to get either triggered_build_id using buildkite-agent command, such as buildkite-agent step get.

Do you have any suggestion on how to achieve this?

Thanks,

2

:wave: Welcome to the community! :hugs:

It’s possible to achieve what you are looking for by using build meta-data, although it may not be too pretty :sweat_smile: You’ll have for example:

Pipeline A:

steps:
  - command: buildkite-agent artifact upload "test.jpg"
  - trigger: pipeline-b
    build:
      env:
        TRIGGER_JOB_ID: $BUILDKITE_JOB_ID
  - wait: 
  - command: buildkite-agent artifact download "*.*" .  --build "$(buildkite-agent meta-data get "triggered_build_id")"

Pipeline B:

steps:
  - command: buildkite-agent artifact download "*.jpg" . --build $BUILDKITE_TRIGGERED_FROM_BUILD_ID
  - command: buildkite-agent artifact upload "another.txt"
  - command: buildkite-agent meta-data set "triggered_build_id" "$$BUILDKITE_BUILD_ID" --job $TRIGGER_JOB_ID

A clarification, being unable to interpolate the BUILDKITE_JOB_ID it’s a caveat of the YAML steps editor, so if you get the error “Interpolating “BUILDKITE_JOB_ID” is currently not supported” then uploading a pipeline file would allow it because the variables are interpolated be the agent.

Another option that I believe is nicer is to pass the JOB_ID from the parent to the child (like Pipeline A), and from the child’s job, do the artifact upload with that job_id

3

Hi @paula

I didn’t know one could upload an artifact or create meta data across pipelines. This is very helpful tip that I could use. Thank you :bowing_man:

For the problem that I described earlier, pipeline B is maintained by other team, so I will have to coordinate with them. If we agree to go with suggested workflow, I think metadata seems little more flexible since Pipeline A has control over where to download the artifacts to.

I still think it would be great if triggered_build_id is retrievable using buildkite-agent step get.

Thank you so much for your help.

2 Likes
4

Hi @Germicide5416 i am from the product team. Thanks a lot for the feedback. This is not prioritised for the near term but will be in our backlog to get to. We’ll let you know once we pick it up.

5

Hi @paula
Tried the second approach when the triggered job uploads artifacts to the parent job by its job id.
The sub-job example:

command: buildkite-agent artifact upload my-file  --job $${BUILDKITE_TRIGGERED_FROM_BUILD_ID}

Got an error (replaced job id with <JOB_ID>):

fatal: failed to upload artifacts: uploading artifacts: POST https://agent.buildkite.com/v3/jobs/<JOB_ID>/artifacts: 404 Not Found: Could not find job: <JOB_ID>

I double-checked that BUILDKITE_TRIGGERED_FROM_BUILD_ID(a buildkite managed variable) is correct and matches the parent build id. What am I doing wrong?

6

Hi @pazonec ,

This is Suma from Buildkite support team. Firstly, welcome to Buildkite community and thank you for reaching out to us with your question.

Issue I see here is you are passing build id (BUILDKITE_TRIGGERED_FROM_BUILD_ID) to the --job which is expecting job id.

This is the reason it failed to upload the artifact. So like shown in example from Paula, you would need to pass the job id of the parent pipeline build to the triggered pipeline and then use that for upload

steps:
  - command: buildkite-agent artifact upload "test.jpg"
  - trigger: pipeline-b
    build:
      env:
        TRIGGER_JOB_ID: $BUILDKITE_JOB_ID
  - wait: 
  - command: buildkite-agent artifact download "*.*" .  --build "$(buildkite-agent meta-data get "triggered_build_id")"

Now use the TRIGGER_JOB_ID in the child pipeline artifact upload command.

I hope this answers your question. Please feel free to reach out if you have any further questions.