# Meta as environment variables

**URL:** https://forum.buildkite.community/t/meta-as-environment-variables/1927
**Category:** Features Requests
**Created:** [February 25, 2022, 5:58pm UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927 "2022-02-25T17:58:36Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![kgillette](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/kgillette/32/1043_2.png) [@kgillette](https://forum.buildkite.community/u/kgillette)
#### Post date: [February 25, 2022, 5:58pm UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/1 "2022-02-25T17:58:36Z")

</div>

We have some pipelines that depend heavily on build metadata, and have steps that look like:

```nohighlight
- command: |
    export SOMEVAR1=$(buildkite-agent meta-data get some-var-1)
    export SOMEVAR2=$(buildkite-agent meta-data get some-var-2)
    ./script

```

Given that metadata has close conceptual relationship between environment variables and metadata (they’re both data that come from “the environment,” be that parent processes or other buildkite jobs), it would be valuable to be able to formally initialize environment variables based on meta-data, like so:

```nohighlight
- env:
    SOMEVAR1: meta-data:some-var-1
    SOMEVAR2: meta-data:some-var-2
  command: ./script

```

---

<div class="post-metadata">

### Author: ![jeremy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/jeremy/32/1040_2.png) [@jeremy](https://forum.buildkite.community/u/jeremy)
#### Post date: [February 26, 2022, 1:32am UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/2 "2022-02-26T01:32:59Z")

</div>

Hi @kgillette great suggestion! I’ve raised this as a feature request with our Product team and it will be reviewed 🙂

---

<div class="post-metadata">

### Author: ![saurabh](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@saurabh](https://forum.buildkite.community/u/saurabh)
#### Post date: [July 6, 2022, 12:17pm UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/3 "2022-07-06T12:17:07Z")

</div>

Any update on this feature? Any workarounds for now?

---

<div class="post-metadata">

### Author: ![suma](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/suma/32/1306_2.png) [@suma](https://forum.buildkite.community/u/suma)
#### Post date: [July 6, 2022, 6:15pm UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/4 "2022-07-06T18:15:23Z")

</div>

@saurabh

First of all welcome to Buildkite community and thank you for reaching out with your questions

We do not have any updates yet on this feature request. Regarding workaround, as mentioned in the initial post the current option is to use `buildkite-agent meta-data get var` in order to fetch the meta data value and assign it to an environment variable.

Please let us know if you are observing any issues with that approach at the moment.

---

<div class="post-metadata">

### Author: ![saurabh](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@saurabh](https://forum.buildkite.community/u/saurabh)
#### Post date: [July 7, 2022, 6:32am UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/5 "2022-07-07T06:32:28Z")

</div>

But the workaround mentioned will only export the variable for the single step from what i understand, I want to export it across the whole pipeline and I can’t export it in the same step since I am using the docker-compose plugin for one of my steps

---

<div class="post-metadata">

### Author: ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)
#### Post date: [July 7, 2022, 7:06am UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/6 "2022-07-07T07:06:45Z")

</div>

Thats right @saurabh 👍

You will need to execute that command in each step where you need the environment variable.  
And the docker-compose plugin allows you to mount the `buildkite-agent` into the container so that you can do this at runtime.

If you need it before then, you can use [dynamic pipelines](https://buildkite.com/docs/pipelines/defining-steps#dynamic-pipelines) to upload the steps whilst defining the variable so that it is available for the command.

---

<div class="post-metadata">

### Author: ![sj26](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/sj26/32/17_2.png) [@sj26](https://forum.buildkite.community/u/sj26)
#### Post date: [July 7, 2022, 8:00am UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/7 "2022-07-07T08:00:39Z")

</div>

Hi folks! Thanks for the feedback, it’s an interesting idea. But we don’t have any immediate plans to do this. It’s a bit complicated to think about with the lifecycles involved. Env is defined when a job is uploaded, while meta-data changes over the course of a build. It’s also possible to achieve the result today with the commands already mentioned.

If you want to use env across multiple steps at once I’d advise using the top-level env key:

> **[Defining Your Pipeline Steps](https://buildkite.com/docs/pipelines/defining-steps#step-defaults)**
>
> Pipeline steps are defined in YAML and are either stored in Buildkite or in your repository using a pipeline.yml file.

Longer term we’re considering doing more powerful interpolation of values including meta-data into dynamic pipeline uploads and commands.

---

<div class="post-metadata">

### Author: ![saurabh](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@saurabh](https://forum.buildkite.community/u/saurabh)
#### Post date: [July 7, 2022, 8:58am UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/8 "2022-07-07T08:58:55Z")

</div>

I’m trying to export for the whole pipeline using the top level env, but I need to run a command to get my environment values.  
env:  
CURRENT\_UID : $(id -u):$(id -g)  
But, I noticed that fro environment variables the value is not “executed”, it gets assigned the commnd string instead. Am i doing it wrong here?

---

<div class="post-metadata">

### Author: ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)
#### Post date: [July 7, 2022, 1:09pm UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/9 "2022-07-07T13:09:11Z")

</div>

That is working as expected @saurabh. Env variables are plain string values. To achieve what you want, you’ll need to dynamically generate the pipeline yaml using a command step. Then upload the generated yaml with `buildkite-agent pipeline upload`

---

<div class="post-metadata">

### Author: ![sj26](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/sj26/32/17_2.png) [@sj26](https://forum.buildkite.community/u/sj26)
#### Post date: [July 8, 2022, 10:05pm UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/10 "2022-07-08T22:05:32Z")

</div>

Yeah, sorry @saurabh, we don’t support command interpolation into pipeline yaml files directly. But you can do the interpolation with bash, first, in a [dynamic pipeline](https://buildkite.com/docs/pipelines/defining-steps#dynamic-pipelines) like Jarryd suggests:

```auto
#!/usr/bin/env bash
# Bash will interpolate the command results before the yaml is passed to the upload command:
buildkite-agent pipeline upload <<YAML
env:
  CURRENT_UID: $(id -u):$(id -g)
steps:
- ...
YAML

```

---

<div class="post-metadata">

### Author: ![kgillette](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/kgillette/32/1043_2.png) [@kgillette](https://forum.buildkite.community/u/kgillette)
#### Post date: [September 6, 2022, 3:10pm UTC](https://forum.buildkite.community/t/meta-as-environment-variables/1927/11 "2022-09-06T15:10:08Z")

</div>

> Longer term we’re considering doing more powerful interpolation of values including meta-data into dynamic pipeline uploads and commands.

I’ve been thinking about this as well. At least in the pipelines I write, I’m usually not trying to get metadata shared between completely unrelated steps: there’s typically a wait step or chain-of-dependency from the step getting the metadata back to the step setting the metadata, and as such, as an author, I can be confident that the metadata will by set by the time it’s read later.

The issue is that metadata is handled in an imperative way (any step can set metadata at any time, and any step can get metadata at any time), rather than handling it in a declarative way, such as described in the original post. If steps declared that they are using certain meta-data, and other steps declare that they may be setting meta-data variables, then static analysis of the pipeline becomes possible, and the buildkite server/orchestration can validate that metadata reads and writes are non-concurrent (that there’s either a wait/block step or a dependency separating each phase of reads and writes).
