Best practices for handling redacted vars in Buildkite pipelines

Pipelines
1

Hello,

I would like to understand the recommended best practices for working with redacted variables in our pipelines.

With the Buildkite agent update 3.95.0, the buildkite-agent set metadata command started enforcing argument variable redaction [1].

Since then, we have encountered errors[2] when setting simple values, such as:

buildkite-agent meta-data set attempts_count "1"

I understand the principle behind redacting variables that match patterns of sensitive values (e.g., *_TOKEN variables). However, when someone defines a TOKEN, it is typically a long alphanumeric string, covering most letters and numbers. It is highly expected that all numbers will be included in the redacted set.

Another example: I created a temporary directory using mkdtemp, and the generated random string contained a sequence of characters that matched part of our TOKEN string defined in /etc/buildkite-agent/hooks/environment.
(we have a shared volume and I need to pass the name of directory amongst pipeline’s steps)

What is the recommended approach to handling this?
Do we need to explicitly add --redacted-vars='' in most places where buildkite-agent meta-data set is used?

I appreciate any insights or recommendations.

Thanks in advance!

[1]

[2]

2025-03-21 09:05:56 WARN   Meta-data value for key "attempts_count" contained one or more secrets from environment variables that have been redacted. If this is deliberate, pass --redacted-vars='' or a list of patterns that does not match the variable containing the secret
2025-03-21 09:05:56 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 1/10 Retrying in 2s)
2025-03-21 09:05:58 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 2/10 Retrying in 3.216s)
2025-03-21 09:06:01 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 3/10 Retrying in 5.172s)
2025-03-21 09:06:06 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 4/10 Retrying in 8.317s)
2025-03-21 09:06:15 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 5/10 Retrying in 13.374s)
2025-03-21 09:06:28 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 6/10 Retrying in 21.508s)
2025-03-21 09:06:50 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 7/10 Retrying in 34.587s)
2025-03-21 09:07:25 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 8/10 Retrying in 55.619s)
2025-03-21 09:08:20 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 9/10 Retrying in 1m29.442s)
2025-03-21 09:09:50 WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 10/10)
buildkite-agent: fatal: failed to set meta-data: POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing
2

Hi @chalda!

Welcome to the Buildkite community! I really appreciate the thoughtful post with references! Very cool. :smiling_face_with_sunglasses:

I hear what you are saying. To me it would feel a little capricious for the agent to redact substrings of environment variable values.

Really out of my own curiosity, I tried to reproduce what you are describing, but I was unable to get the agent to complain at me in the same way you are showing.

Could you possibly provide a little reproducible example?

3

Hello.

Thank you for the kind response. This is for example a test script I was experimenting with.

agents:
  queue: "snapshots"

env:
  TEST_TEST: "abcabc"

steps:
  - label: "Test"
    commands:
    - echo "--> Setting attempts_count 1 no redacted"
    - buildkite-agent meta-data set --redacted-vars='' attempts_count 1 || echo 'failure'
    - echo "--> Setting attempts_count 1 default redacted"
    - buildkite-agent meta-data set attempts_count 1 || echo 'failure'
    - echo "--> Setting attempts_count 1 random redacted"
    - buildkite-agent meta-data set --redacted-vars='TEST' attempts_count "1" || echo 'failure'
    - echo "--> Setting attempts_count abc TEST redacted"
    - buildkite-agent meta-data set --redacted-vars='*_TEST' attempts_count "abc" || echo 'failure'

The output of this pipeline in the step log was

$ echo "--> Setting attempts_count 1 no redacted"
buildkite-agent meta-data set --redacted-vars='' attempts_count 1 || echo 'failure'
echo "--> Setting attempts_count 1 default redacted"
buildkite-agent meta-data set attempts_count 1 || echo 'failure'
echo "--> Setting attempts_count 1 random redacted"
buildkite-agent meta-data set --redacted-vars='TEST' attempts_count "1" || echo 'failure'
echo "--> Setting attempts_count abc TEST redacted"
buildkite-agent meta-data set --redacted-vars='*_TEST' attempts_count "abc" || echo 'failure'
--> Setting attempts_count 1 no redacted
--> Setting attempts_count 1 default redacted
2025 ... WARN   Meta-data value for key "attempts_count" contained one or more secrets from environment variables that have been redacted. If this is deliberate, pass --redacted-vars='' or a list of patterns that does not match the variable containing the secret
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 1/10 Retrying in 2s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 2/10 Retrying in 3.216s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 3/10 Retrying in 5.172s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 4/10 Retrying in 8.317s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 5/10 Retrying in 13.374s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 6/10 Retrying in 21.508s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 7/10 Retrying in 34.587s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 8/10 Retrying in 55.619s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 9/10 Retrying in 1m29.442s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 10/10)
buildkite-agent: fatal: failed to set meta-data: POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing
failure
--> Setting attempts_count 1 random redacted
--> Setting attempts_count abc TEST redacted
2025 ... WARN   Meta-data value for key "attempts_count" contained one or more secrets from environment variables that have been redacted. If this is deliberate, pass --redacted-vars='' or a list of patterns that does not match the variable containing the secret
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 1/10 Retrying in 2s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 2/10 Retrying in 3.216s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 3/10 Retrying in 5.172s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 4/10 Retrying in 8.317s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 5/10 Retrying in 13.374s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 6/10 Retrying in 21.508s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 7/10 Retrying in 34.587s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 8/10 Retrying in 55.619s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 9/10 Retrying in 1m29.442s)
2025 ... WARN   POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing (Attempt 10/10)
buildkite-agent: fatal: failed to set meta-data: POST https://agent.buildkite.com/v3/jobs/.../data/set: 400 Bad Request: Value is missing
failure

For the call buildkite-agent meta-data set attempts_count 1 we can consider we have a ENV variable GITHUB_TOKEN that is a variable with content consisting of character 1 (it could be like "FD77-2792-91A9-6CE3").

4

@jerrid, maybe a suggestion just came to light?

5

Hi @chalda

Amna jumping in for Jerrid here! :wave:

Thanks for sharing the script — it really illustrates the issue clearly.

You’re right to use --redacted-vars='' in cases like this. Given the current agent behavior, that’s the recommended way to bypass redaction for values you know are safe. Just keep in mind that disabling redaction also disables protection for sensitive values — so it’s best reserved for clearly non-sensitive contexts like this. Since the pattern-based matching can be a bit aggressive, this is currently the cleanest path forward when you’re dealing with safe values that happen to overlap with redacted content.