Buildkite env vars set in the pipeline is broken

Pipelines
1

Hi Team,

We upgraded buildkite version to the latest ( v3.51.0), after upgrading the elastic-ci stack to latest version Buildkite stack v6.3.0.

Now when we the builds are triggered from webhook set in bitbucket, the ENV vars set in buildkite pipeline settings in UI, do not get set.
But when triggered via restapi, and ENV vars set in the api, the variables are set in the build.

I have gone through the release notes of buildkite and env variable precedence order, could not get why they are getting set when webhook is triggered.

It was working fine till yesterday before the upgrade. I did see an outage in morning today for buildkite, i am not sure, if anything related to it.

We are still waiting for another bitbucket merge trigger to see if it was just a one of issue.

The pipeline.yml looks like this .

env:
  APPNAME:
  ENVIRONMENT:
  DESTINATION:
  ENABLE_BROWSER_CONSOLE:
  PREDESTINATION:
  SKIP_BLOCK_STEP:

steps:
  - command: "./scripts/notify.sh"
    label: "notify"
    agents:
      - "queue=frontend"

Any help help/suggestions would be appreciated.

Thanks
Regards
Suraj

2

Hello @surajthakur!

Thanks for the message - I hope you are well :wave:

From the above - are those build level envs being set as per the pipeline’s YAML? Have tested a similar setup (with defining build level env vars/using them inside a script) and it’s as normal from agents on v3.51.0 that have been spun up by scaling on one of my v6.3.0 CI stacks.

Has that aforementioned BB merge trigger come in to see if this was a once-off occurrence? If you are still seeing similar behaviour - please send us through a link to the pipeline/details on the script (without anything sensitive/tokens etc) to support@buildkite.com so we can take a look at this individually.

Cheers :+1:

James

3

Hi @james.s

The pipeline failed again due to ENV var are not being set. They are not picked that are set in pipeline settings configuration.

I have a quick question
What does setting these in top of the pipeline.yml means ? I understand it will set up ENV for all steps, but leave them blank what does that mean?
The piplelines have been working for same config and pipeline.yml from a long time, but only started failing today.

  • I am suspecting since, ENV set in pipeline.yml is given more preference over ENV set in pipeline settings and it is getting blanked due to the below.
env:
  APPNAME:
  ENVIRONMENT:
  DESTINATION:

I will send the pipeline build details through email.

Thanks
Regards,
Suraj

4

No worries - thanks for the confirmation @surajthakur!

Shall wait for the case to be raised, but on your question - if you define env variables without their value from within the pipeline’s YAML as above, they’ll just be empty strings unless they’ve been set from the upper pipeline settings/new branch build configuration. If the build level env has values defined in its corresponding pipeline YAML, but you ‘override’ those value from the new build configuration, that will take the place of what is defined.

Hope that makes sense - and cheers also!

James

5

Thanks for prompt responses. Really appreciate.

Thats makes sense, I think that was happening till yesterday.
Two scenarios.
Case1:
If I am creating a build by clicking new build and specifying the ENV vars, the vars get set, it overrides the pipeline.yml

image
image946×1034 56.1 KB

Case2:
When the same pipeline is triggered by bitbucket using webhook (like when merging a branch)
The three ENV var which is in the pipeline settings already there, does not get set or override the pipeline.yml config as case 1.
And this is just started happening today. I have sent example of previous builds in the email.

APPNAME=test
PREDESTINATION=test-bucket
DESTINATION=test-bucket

Regards
Suraj

6

This case is solved, by removing top level ENV vars which were initially set for defining variables.
But after removing them, the problem got solved.
Thanks for the support over the email and here @james.s and support team.

7

No worries at all Suraj - happy to hear!

Please let us know if anything else arises here or via a email as before.

Hope the rest of the week goes well and cheers :+1:

James