Download Artifacts from parallel builds

General
1

I’ve created a set of parallel builds which all have the same step-name is there a way to download the artifacts from all of the parallel builds. all the files in each step are unique but it seems to only download from the last one.

1 Like
2

Hey Mark!

Yeah, artifacts in parallel builds with the same step name do automatically get generated the same name, and that makes it hard to download them! We have a similar problem in Buildkite’s buildkite pipeline. We run 50 parallel tests steps and generate a simplecov file for each of them, then we need to download all 50 and collate them together to generate our total test coverage.

How we solved it is:

  1. At the end of our bash script that run the parallel test steps, we take the coverage file generated, give it a new name that includes the BUILDKITE_JOB_ID so that it’s unique and move it to a tmp folder:
tar -czvf "tmp/simplecov-resultset-$BUILDKITE_JOB_ID.tar.gz" -C coverage .resultset.json
  1. We tell the agent to upload this file, by adding the artifact path to our pipeline.yml:
  - name: "RSpec"
    command: ".buildkite/steps/knapsack_specs"
    artifact_paths:
      - "tmp/simplecov-resultset-*.tar.gz"
  1. And then in our script that generates the test coverage, we download all the simplecov artifacts
buildkite-agent artifact download "tmp/simplecov-resultset-*.tar.gz" .

Hope that helps! Happy scripting!

3

thank you, I realized that specifying the name of the step was my issue as soon as I removed that optional argument everything started working.

4

This seems like something that could have been added to the docs a while ago.

@Rose Can you take a look at searching/downloading artifacts from parallel steps by nickofthyme · Pull Request #1607 · buildkite/docs · GitHub and Add note about using `--step` with parallelized steps by nickofthyme · Pull Request #1681 · buildkite/agent · GitHub and reword them as needed to get this issue more exposure?

Thanks!