We want to spawn a dynamic pipeline from a failing build step, but things doesn’t seem to work as expected. Roughly what we have is this:
# do some things and capture $EXIT_CODE
if [ "$EXIT_CODE" -eq 3 ]; then
# known exit code, we want to trigger an async pipeline here to process an artifact, that should not block the main pipeline
buildkite-agent artifact upload "$some_artifact"
buildkite-agent pipeline upload <<EOF
steps:
- label: "foo"
trigger: some-async-pipeline
async: true
build:
commit: "$BUILDKITE_COMMIT"
branch: "$BUILDKITE_BRANCH"
env:
TRIGGERED_FROM_JOB_ID: "$BUILDKITE_JOB_ID"
TRIGGERED_FROM_STEP_KEY: "$BUILDKITE_STEP_KEY"
SOME_ARTIFACT: "$some_artifact"
EOF
fi
exit $EXIT_CODE
What seems to be happening is that the async pipeline can never start at all, unless we change the final line to exit 0
. This is not desirable as we want the build to fail. We have tested with sleep
commands and it seems like the uploaded pipeline has an implicit dependency on the step that uploads it, and will wait until that step has finished, and only start if it is successful. Our expectation would be that a pipeline uploaded using buildkite-agent pipeline upload
starts immediately without regard for the status of the current step being executed.
Is there a different way to achieve what we want? I.e. if the step is going to fail, spawn an asynchronous dynamic pipeline that can process our artifact on a different agent, which does not block the main pipeline.