Dynamic pipeline never starts if step fails

We want to spawn a dynamic pipeline from a failing build step, but things doesn’t seem to work as expected. Roughly what we have is this:

# do some things and capture $EXIT_CODE

if [ "$EXIT_CODE" -eq 3 ]; then

  # known exit code, we want to trigger an async pipeline here to process an artifact, that should not block the main pipeline

  buildkite-agent artifact upload "$some_artifact"
  buildkite-agent pipeline upload <<EOF
steps:
  - label: "foo"
    trigger: some-async-pipeline
    async: true
    build:
      commit: "$BUILDKITE_COMMIT"
      branch: "$BUILDKITE_BRANCH"
      env:
        TRIGGERED_FROM_JOB_ID: "$BUILDKITE_JOB_ID"
        TRIGGERED_FROM_STEP_KEY: "$BUILDKITE_STEP_KEY"
        SOME_ARTIFACT: "$some_artifact"
EOF
fi

exit $EXIT_CODE

What seems to be happening is that the async pipeline can never start at all, unless we change the final line to exit 0. This is not desirable as we want the build to fail. We have tested with sleep commands and it seems like the uploaded pipeline has an implicit dependency on the step that uploads it, and will wait until that step has finished, and only start if it is successful. Our expectation would be that a pipeline uploaded using buildkite-agent pipeline upload starts immediately without regard for the status of the current step being executed.

Is there a different way to achieve what we want? I.e. if the step is going to fail, spawn an asynchronous dynamic pipeline that can process our artifact on a different agent, which does not block the main pipeline.

Hi @glindstedt_embark :wave: and welcome to Buildkite!

I’ve tested your approach using the script you shared, and everything seems to work as expected—the trigger step gets dynamically uploaded when the exit code is 3 , it correctly triggers the downstream build asynchronously without blocking, and the original build still fails as intended. If you’re still running into issues, it might help to see your full script since there’s a chance the exit code isn’t being captured properly on your end. if it includes sensitive information, you can reach out to support@buildkite.com.