Download Artifacts from parallel builds

I’ve created a set of parallel builds which all have the same step-name is there a way to download the artifacts from all of the parallel builds. all the files in each step are unique but it seems to only download from the last one.

Hey Mark!

Yeah, artifacts in parallel builds with the same step name do automatically get generated the same name, and that makes it hard to download them! We have a similar problem in Buildkite’s buildkite pipeline. We run 50 parallel tests steps and generate a simplecov file for each of them, then we need to download all 50 and collate them together to generate our total test coverage.

How we solved it is:

  1. At the end of our bash script that run the parallel test steps, we take the coverage file generated, give it a new name that includes the BUILDKITE_JOB_ID so that it’s unique and move it to a tmp folder:
tar -czvf "tmp/simplecov-resultset-$BUILDKITE_JOB_ID.tar.gz" -C coverage .resultset.json
  1. We tell the agent to upload this file, by adding the artifact path to our pipeline.yml:
  - name: "RSpec"
    command: ".buildkite/steps/knapsack_specs"
    artifact_paths:
      - "tmp/simplecov-resultset-*.tar.gz"
  1. And then in our script that generates the test coverage, we download all the simplecov artifacts
buildkite-agent artifact download "tmp/simplecov-resultset-*.tar.gz" .

Hope that helps! Happy scripting!

thank you, I realized that specifying the name of the step was my issue as soon as I removed that optional argument everything started working.