As a long term user of Buildkite and, specifically, the AWS Elastic Stack to orchestrate infrastructure as code pipelines, I have often made use of the manual “New Build” feature to manually initiate builds. Often I will use the Options»Environment Variables key value pairs to inject and override environment variables in my pipelines.
However, recently I have observed that specifically the “merging” / overriding of environment variables as defined in the “New Build” modal are no longer taking precedence over the environment as specified in the pipeline.yml. e.g.
- name: "Generating :buildkite: Pipeline"
env:
ENVIRONMENTS: "buildkite github dev"
command:
- make pipeline
agents:
queue: terraform
branches:
main
Environment variables | Buildkite Documentation does a good job outlining environment variable precedence but it does not really cover the case with Options»Environment Variables key value pairs.
After verifying to the best of my knowledge that my setup has not changed in anyway (pipeline + infra) since the last successful manual build, I believe at least empirically the Options»Environment Variables key value pairs took precedence over those defined as per the above pipeline snippet.
Has something changed here on the Buildkite end recently or have I missed something on my end of the setup?
Hope you are well, thanks for the question and welcome to the Buildkite Community Forum
It looks like you are using YAML Steps for your pipeline (please let me know otherwise) - but in the case above - any key/value pair that is defined in the Options»Environment Variables menu will take precedence over any pipeline OR step level environment variables with the same key.
Should you use the older Legacy Steps - there is noted compatibility changes across variables defined at a pipeline and step level (basis is that step level variables take precedence if defined with the move). In the linked doc you shared - the Pipeline Settings page referred to is the Environment Variables input as part of pipelines utilising Legacy Steps (not applicable if using YAML steps as above):
So I think I kinda understand what has happened here. Originally (just over a month ago) I setup the pipeline by hand via the UI (the way I have many times in the past selecting “Read steps from repository”) and added the pipeline.yml to the repo.
At a later point I started managing the pipeline in question via the Buildkite Terraform provider which when I imported the pipeline into the Terraform workflow flipped me over to using the YAML Steps editor for the pipeline (in contrast to whatever the default is for the UI pathway).
It would seem the process of setting the uploading of the pipeline steps via Terraform somehow changed the precedence rules
The thing I do not quite understand (if I am reading your response correctly) is, “New Build” Options»Environment Variables with YAML Steps for the pipeline should take precedence over any pipeline OR step level environment variables with the same key. But this not what I am observing since importing & managing this pipeline via the Buildkite Terraform provider.
I guess if I want the original expected behaviour I would need to recreate the pipeline the good old fashion way and leave it out of Terraform?
Looks like on that former picture - you’ve found the default way that the UI sets new pipelines to be created in (its Legacy Steps by default, but you can change the default behaviour for new pipelines to use the YAML editor in your organizations settings page).
Me and my immediate team are actually maintainers of the Terraform Provider - happy that you’re using such a tool for managing your Buildkite infrastructure (in this case pipelines). The underlying pipeline creation/update calls do use YAML steps as you’ve found out (the actual input for said mutations expects YAML step input).
As for the original ask - given that you’ve set step level environment variable (ENVIRONMENTS) - that should be the variable that is set through to your builds: as the provider will just set the actual config and leave it there. Correcting what I mentioned before - the deeper level the variable is set in, the higher precedent it is placed. If you have both Options»Environment Variables (calling this the Build popup) and set in pipeline YAML, the former will take precedence. If the variable was defined in all 3 layers (Build popup and on both pipeline/step level in YAML) - it is the step variable that will override the former two.
I guess the moral to the story is when using Legacy Steps (default UI way) one can rely on using Build popup to override all environment variables (even if they are defined at the pipeline step level), but when using the YAML steps editor (either explicitly via UI or implicitly via GraphQL API) pipeline step level environment variables trump Build popup environment variables.
No worries - and you’ve summed it up quite nicely there!
Think of the popup as a chance to override any variable set on a pipeline level - but of course any variable set at step level will take precedence over it/pipeline level variables
(its Legacy Steps by default, but you can change the default behaviour for new pipelines to use the YAML editor in your organizations 