Secure pipeline secrets

Our github repo is public and we run Buildkite build for every PR, including external contributions. Also our Buildkite build needs secrets. I followed recommendations from Managing pipeline secrets | Buildkite Documentation and placed them into env file in our private S3 bucket. Logs are also private (hidden) but nothing prevents some malicious external “contributor” to send a PR with a simple test which does POST request to external endpoint and send content of os.Environ() to it.

What are the best practices here?

Hi @alexshtin!

That’s a good question, and we’re still figuring out what the best answers are!

Most of the users, included ourselves, enable read-only public access for a pipeline, and not building third-party forks.
On a pipeline with a GitHub repository, in the pipeline settings, you can tell it not to build pull requests from third parties (forks). Then you can manually trigger builds if a change doesn’t look malicious.
f you do want to build forks but want to have a manual review step, all branch names will be prefixed by the fork username, so instead of BUILDKITE_BRANCH=“master” it might be BUILDKITE_BRANCH=“user:master”. You can detect these on your agents using a pre-command hook, or something similar which can’t be modified by the committer and pipeline upload a block step so that a trusted user can vet the code before allowing the build.

You can also use the agent’s environment hook to add strict checks for what repositories, commands, and plugins are allowed to run. You can learn more about that here.

Secrets are tricky. Any build running on your infrastructure will have access to any secrets within the build environment. So it’s best not to run untrusted code without reviewing it first.

There was also a good suggestion from another user in the forum, it’s a workaround but that could probably help you.

Hopefully that gives you some options!

Cheers,

1 Like

Hi Paula.

I finally got back to that problem. I cleared our secretes but I still want to implement manual vet review before running build from 3rd party folks. I understand that I can get github user name from branch but this will require me to whitelist all allowed users somewhere (hard code in agent script?). I would better use BUILDKITE_BUILD_CREATOR_TEAMS env and check if user belongs to some hard coded team (let’s say “AllowedBuilders”) and will manage members of this team using web UI. Am I a right that this env has a list of teams from my org only? I mean, if random user creates same “AllowedBuilders” team on his org, it won’t be passed to BUILDKITE_BUILD_CREATOR_TEAMS env, right?

Also it is not quite clear for me how to block the build? Is it simple exit 1 from agent pre-command hook? But how does allowed reviewer unblock the build after the vet step is passed?

Thanks,
Alex.

Hi Paula I think I figured out all my questions in the post you linked above.

We are using AWS Elastic stack for agents. What is the easiest way to add agent hook to it?

Hi @alexshtin !

Yup, you are correct. You can check an example in our docs of programmatically manage your teams from here.

If you are using our AWS Elastic Stack, you can take a look on our Build Secret section.
Another alternative could be to create your own AMI by modifying this part of the packer provisioner in order to add the options you need: elastic-ci-stack-for-aws/bk-install-elastic-stack.sh at master · buildkite/elastic-ci-stack-for-aws · GitHub

Cheers!

Thanks, @paula.

I am trying to avoid building custom AMI. I would love to see something similar to env file in secret bucket: hooks directory in S3 bucket (apparently another one) that got copied into the agent before starting it up. But what I can tell from quick scan of shell scripts, it is not supported right now and I don’t have any other option beside building my own AMI. Can you confirm it?

Hey!

Yup, that’s correct. I’ll try to find alternatives for you, but at first glance, there is no other option at the moment, sorry!