Which token is failing?

I’m trying to do some local debugging of my build script, and at the end, the script uploads artifacts to buildkite.

buildkite-agent artifact upload "python/dist/*" "s3://$BUCKET_NAME/$BUILDKITE_JOB_ID" \
                                                --job "$BUILDKITE_JOB_ID" \
                                                --agent-access-token "$BUILDKITE_AGENT_TEST_TOKEN"

I’ve checked the BUILDKITE_AGENT_TEST_TOKEN several times, and it has the right value, and pulled directly from this page: https://buildkite.com/organizations/expanso/clusters/8e34d15e-2106-4a12-a1a5-937fa995ab73/tokens

But now i’m getting this error:

+ buildkite-agent artifact upload 'python/dist/*' s3://bacalhau-build-pipeline-artifacts/test-FFEEACAA --job test-FFEEACAA --agent-access-token LyFZovKVEXAMPLECsbjQyr
2024-06-25 11:56:39 INFO   Found 2 files that match "python/dist/*"
2024-06-25 11:56:40 INFO   Uploading to Amazon S3 ("s3://bacalhau-build-pipeline-artifacts/test-FFEEACAA"), using your agent configuration
2024-06-25 11:56:40 INFO   Creating (0-2)/2 artifacts
2024-06-25 11:56:41 WARN   POST https://agent.buildkite.com/v3/jobs/test-FFEEACAA/artifacts: 401 Unauthorized: Invalid access token (Attempt 1/10 Retrying in 500ms)
fatal: failed to upload artifacts: uploading artifacts: POST https://agent.buildkite.com/v3/jobs/test-FFEEACAA/artifacts: 401 Unauthorized: Invalid access token

(the token was directly copied and pasted from that page)

Is there anything I can do to debug this? Do different tokens have the ability to upload?

Hey @aronchick

Welcome to Buildkite community.

The agent-acess-token flag returns a 401 even if their pass a valid token, because, the are two kinds of agent tokens: “agent registration tokens”, and “agent session tokens”.

The agent access token is set when the agent exchanges its registration token for a session token, which is stored in $BUILDKITE_AGENT_ACCESS_TOKEN. So if you are not providing the access token via the flag on the “buildkite-agent artifact upload” command, it will default to using the value set in $BUILDKITE_AGENT_ACCESS_TOKEN when the agent first registers itself with Buildkite. So in your case I would suggest you to remove the --agent-access-token flag and it should work as expected (assuming that you successfully authenticated with AWS S3, using any of the included credential providers) Here is some examples: buildkite-agent artifact v3 | Buildkite Documentation

If you’re still running into trouble after removing the --agent-access-token flag, please do let us know support@buildkite.com.


Oh I see! Yeah this was super confusing.

Honestly, I was trying to debug using a local script - how do I generate this token without an agent? Or is there a pattern for running locally?

Hey @aronchick! :wave:

You will need to set up an agent to use buildkite-agent artifact upload because the agent needs to register with Buildkite.

You can spin-up an agent locally using buildkite-agent start (buildkite-agent start v3 | Buildkite Documentation).

oh i see - this makes some sense.

is there a doc on how to do local development/debugging of Buildkite?

I saw this - GitHub - buildkite/cli: A command line interface for Buildkite. - but it appears to be unupdated for 2 years.

I’m just looking to do local debugging of my pipelines.

Hi @aronchick ,

You can use the bk cli’s bk local run for running the pipelines locally, however not all pipeline steps features are supported. Development has also discontinued on that branch. We are now developing a new version of bk cli (branch 3.x) that provides more interaction with your buildkite org’s pipelines and builds.

For debugging, we usually start with the job logs that you can see from the UI, but is also available for download via API Jobs API | Buildkite Documentation.

For agent logs, for self-hosted agents running locally, you should be able to access them in your own environments. But for running agents via the elastic stack, we have a guide available for troubleshooting and collecting the logs here Troubleshooting the Elastic CI Stack for AWS | Buildkite Documentation.

We also have integrations available for tracing the buildkite agent - Tracing in the Buildkite Agent v3 | Buildkite Documentation.

Hope the above help to get you started and for any other questions, don’t hesitate to reach out!