I am trying to pass some arguments from one pipeline to another using a trigger step but I am unable to do so.
Suppose the following (abridged) trigger step:
- trigger: "some-other-pipeline"
label: "My step label"
build:
env:
MY_ARG: "$(buildkite-agent meta-data get my_arg)"
The intention here is to set MY_ARG with whatever value was set to my_arg with the buildkite-agent meta-data get set command in the same pipeline and pass that through to some-other-pipeline.
That however doesn’t work and MY_ARG always ends up with an empty value in some-other-pipeline because I believe shell commands aren’t getting executed inside the env section?
One solution that works is doing something similar to:
Your understanding is correct. Shell commands aren’t evaluated in a static trigger step’s env block it’s treated as pipeline config, not shell input, so $(buildkite-agent meta-data get "my_arg") won’t run there, leaving MY_ARG empty.
Your dynamic pipeline upload approach is the best and recommended approach for runtime values , so the resolved value (e.g. MY_ARG: "1.1") gets passed through as seen in my example below.
I’ve noticed that static arguments declared either in an env block at the top of the pipeline file or even in a select block are also visible within the trigger block using the {metaenv?.MY_VAR} syntax but I guess this isn’t shell but rather buildkite level resolution?
@savvas that’s correct; we’re essentially using a YAML parser via Ruby to determine what the values are on the pipeline where input and env are being set, then interpolating those values in the pipeline YAML.
As I was reading this thread, I was wondering if an alternate solution for the OP couldn’t be to read the my_arg metadata of the calling build… directly from the triggered build?
i.e. calling MY_ARG=$(buildkite-agent meta-data get --build "$BUILDKITE_TRIGGERED_FROM_BUILD_ID" my_arg) directly within the script/command of the some-other-pipeline build that they triggered and that needs to use that MY_ARG value? Would that work too?
steps:
- label: ":mag: Get Parent Build Meta-data"
command: |
MY_ARG="$(buildkite-agent meta-data get --build "$BUILDKITE_TRIGGERED_FROM_BUILD_ID" my_arg)"
echo "The ARG is $$MY_ARG"
Small gotcha to watch out for: you need $$MY_ARG (double $) when referencing it later, otherwise Buildkite’s own variable interpolation strips it out before the shell even runs (it doesn’t recognize MY_ARG since it’s not a real Buildkite variable).
One downside compared to the build.env option though MY_ARG here is just a shell variable scoped to that one step. It won’t carry over to other steps in the pipeline, and it won’t show up as an actual env var in the UI. If you want it available across the whole triggered build (and visible as a proper env var), setting it via build.env on the trigger step itself is the more convenient route.
So, if that needed to be propagated further to a downstream pipeline then I guess one would have to buildkite-agent meta-data set it on this step and then grab it again using the $BUILDKITE_TRIGGERED_FROM_BUILD_ID from there?
Re the $$ I must admit that has tripped us up a few times so far.. So, my understanding is that upon the first pass Buildkite will stip the leading $ and attempt to match the remaining against it’s own environment variables therefore converting a declared variable of say $MY_VAR to merely MY_VAR for the shell to resolve, which it naturally can’t find, is that correct?
During buildkite-agent pipeline upload, i.e. when Buildkite parses your pipeline.yml file and uploads it to generate the actual jobs for your build based on its configuration, it pre-processes that pipeline.yml file first to substitute any $VARIABLE in it with the corresponding value
At the time this buildkite-agent pipeline upload runs, the environment will already contain all the BUILDKITE_* variables defined in this doc, as well as all environment variables declared upstream (e.g. passed by an env: attribute in the parent job in which you had your trigger: for example. Or any environment variable you might have set when triggering a CI build manually via the “New Build” green button from the Buildkite UI then opening the “Options > Environment variable” box before validating.
So at that stage, any instance of e.g. $BUILDKITE_BRANCH in your pipeline.yml will be interpolated by buildkite-agent pipeline upload during that preprocessing and replaced by its value. And same for instance of $FOO if the FOO env var is defined at the time of this pipeline-upload stage (e.g. because it was passed by env: from the triggering build or provided manually when you used the “New Build” button).
And later, way after your steps have been uploaded and only at the time your job will later be picked up by an available agent, that agent will run that command in a shell… which might trigger a second stage of shell interpolation (but this time evaluated at the time your job runs)
So there’s one interpolation at “pipeline upload” time by buildkite, then the usual interpolation by your shell when the command of your job is actually run.
And using $$ (or \$, both work) in your pipeline.yml file is a way to escape that $ character so that pipeline-upload replaces it with a verbatim $ instead of interpolating the variable.
If that helps that’s similar to if like if you were to use echo "echo \$FOO" >myscript.sh in a terminal to echo a verbatim echo $FOO in your script that would then only interpolate the value of the FOO variable when the myscript.sh is finally run,
By the way, another way to avoid needing to escape the $ in your pipeline.yml via $$ or \$ —so that the $ is kept verbatim until the command runs and is interpolated by the running shell in your agent—is to not have your command containing that echo $MY_VAR or whatever directly in your pipeline.yml file, but instead put it in a separate script, e.g. instead of:
# pipeline.yml
steps:
- label: arg-demo
command: |
MY_ARG="$(buildkite-agent meta-data get --build "$BUILDKITE_TRIGGERED_FROM_BUILD_ID" my_arg)"
# Be sure to escape the dollar sign by doubleing it or using a backslash
# So that the preprocessing of this pipeline.yml file during upload
# just replaces it with a literal single dollar sign (so that it can then
# only be interpolated at the time the shell runs this command when
# your job runs) instead of interpolating the variable too soon.
echo "The ARG is $$MY_ARG"
You can instead move the code from your command in a dedicated script file (let’s call it arg-demo.sh) and call it in your YAML’s command: attribute:
#!/bin/env bash
# arg-demo.sh
MY_ARG="$(buildkite-agent meta-data get --build "$BUILDKITE_TRIGGERED_FROM_BUILD_ID" my_arg)"
# no deed for escaping the `$` anymore in the code below,
# thanks to the indirection we used of putting this in a separate script, and
# not directly into the `pipeline.yml` that gets processed by pipeline upload
echo "The ARG is $MY_ARG"
That way there’s no $ —that risks being interpolated too soon by the pipeline upload step in the pipeline.yml —anymore.
PS: Note that a subtle difference between the first approach (code directly in your pipeline.yml ) vs my example above (code in separate script) is that in the latter the $BUILDKITE_TRIGGERED_FROM_BUILD_ID env var is evaluated at the time your job runs (and that arg-demo.sh script is run), while on the former it was evaluated as soon as your pipeline.yml is processed during pipeline upload
In that particular case that shouldn’t matter, because that BUILDKITE_TRIGGERED_FROM_BUILD_ID env var is visible both during the pipeline upload… and at the time the job executes the command.
That is not the same for the MY_ARG (and why you need to escape it with $$MY_ARG or \$MY_ARG if you put it in the pipeline.yml), which is not defined at the time your pipeline.yml is pre-processed by the pipeline upload (hence why just keeping $MY_ARG without escape in the pipeline.yml would be interpolated to the value of MY_VAR at that time… which is… nothing) and is only defined by the shell code when that code is finally executed when your job runs.
I see, gotcha So, first process all these sources on a pre-upload stage and then run the declared command steps on some agent which naturally has no connection/context to the triggering/uploading environment as it runs on it’s own vm/host, I assume.
Right, I see..that’s actually pretty cool as it can also help improve readability significantly on pipeline.yaml, especially if that’s rather complex! (naturally, at the expense of having a few more shell scripts to manage but I think with some careful naming conventions that shouldn’t be too much of an issue).
!
So, my understanding is that upon the first pass Buildkite will stip the leading