Environment variables set in S3 not accessible from build step

[ldev654]

Hello,

I have setup the AWS CI stack and have got it all working (as well as adding an “environment” file in the secrets S3 bucket). However, although I can see in the build logs “Fixing permissions …” it says the var was added, example “MY_KEY added”, it is not accessible from a build step which is running a script.

I have managed to do a workaround which adds an empty env var to my pipeline.yaml and this works as in it gets overrided with the value defined in the s3 file, but I feel like I shouldn’t need to do this and that all secrets/vars added in the environment file should be accessible.

env:
  MY_KEY: ""

Is there something I am doing wrong, or extra I need to do? Here’s the pipeline.yaml

common:
  docker-login: &dockerLogin
    docker-login#v3.0.0:
      propagate-environment: true
      mount-buildkite-agent: true
      username: "n/a"
      password-env: ACCESS_TOKEN
      server: registry-link
      volumes:
        - "/var/run/docker.sock:/var/run/docker.sock"

env:
  IMAGE_TAG: "${BUILDKITE_BRANCH}_${BUILDKITE_BUILD_NUMBER}"
  MY_KEY_EXAMPLE: "" 

steps:
  - label: ":docker: Build & Push to Registry"
    key: build_push
    command: .buildkite/scripts/build-push.sh
    plugins:
      - *dockerLogin
  - label: ":k8s: Deploy to Kubernetes"
    depends_on: build_push
    # Added /bin/bash as a workaround for now, since normally it fails (perm denied).
    command: /bin/bash .buildkite/scripts/deploy-helm.sh
    plugins:
      - *dockerLogin
      - docker#v3.0.0:
          image: my-registry/helm-helpers:0.1.0
          propagate-environment: true
          always-pull: true

[james.s]

Hey @ldev654!

Hope you’re well; great question too

Since you’re using the Elastic Stack - first premise was whether you were using its’ managed secret S3 bucket; which you are using here. Since the exported variables are placed in each job of your pipeline - you’ll want to evaluate these added variables at runtime with $$ in particular (i.e in your case; $$MY_KEY_EXAMPLE.

You wouldn’t need to set a top level env either with the above too!

[ldev654]

Hi @james.s

Thank you for your reply. I have tried doing this using $$ before the env variable, however it does not evaluate to anything. When I try echo $$DO_ACCESS_TOKEN from my sh script, it just outputs “7DO_ACCESS_TOKEN” & then of course this fails in below commands that try to use this auth token variable since it is not resolving to the value.

(DO_ACCESS_TOKEN is the env variable)

Is this approach able to work from a sh script?

[stephanie.atte]

Hello @ldev654,

Appreciate the update.

Could you please forward a link to the build to support@buildkite.com? This will help us examine it further.

Cheers!