Buildkite currently supports a “build on PR” trigger, with a single limitation for particular branch names. In addition to this limitation, it would be nice to have the ability to limit automatic builds with some form of allow list. For example:
- A GitHub team or role (such as “owners”)
- Users listed in an OWNERS file associated with the diff
- A basic text field of listed users
The motivation for this is that in a larger setting—for example an open source project—we may want some builds to start automatically, but some to be manually triggered, i.e. PRs created by not-yet-trusted contributors.
Forgive me if this feature exists in some way… I did not se anything in the help section.
Hey!
This is a tricky needle to thread, because as you suggest, there’s a lot of different control mechanisms one might want/need to apply.
In general, our recommendation is really that you get the best control by handling this in code yourself, in your first build step (which is configured in the UI, and thus not open to modification by the contributor).
Unfortunately, the trade-off of that flexibility is that it makes for a pretty non-trivial setup process 
As one example, for rails/rails, this is the YAML configured inside the pipeline settings in the UI. (It’s copied into the repository for reference purposes only.)
It implements a behavior rather along the lines you describe: in this case, it knows that as long as the .buildkite/ directory hasn’t been modified from a known-safe version, all the contributor-controlled execution will run safely tucked inside docker containers, and the build is free to proceed… whereas if there has been a change to the pipeline configuration, it inserts a block step to require manual review.
Note that that first step runs inside a checkout of the candidate commit, but is careful not to be influenced by those surrounding files until it’s satisfied they’re trusted.
Thanks for this response. Will get back to you in a day or two.
Hi Matt. Thanks for the detailed response. I wouldn’t mind doing this, however I just remembered that we looked at moving pipeline.yml out of the repo when you guys announced the feature to move it to the UI. However, in speaking with your support it seems that we cannot access the environment to get $BUILDKITE_BUILD_NUMBER, which we rely on for a variety of things.
Do you have any ideas here?
Hey @clambertops – You’re correct there, unfortunately $BUILDKITE_BUILD_NUMBER is not immediately available in the YAML because it’ll try and interpolate it before it exists. But since this env var is available in the job’s environment I thought of a workaround using printenv while we look at improving this:
steps:
- command: 'buildkite-agent meta-data set "example" "$(printenv BUILDKITE_BUILD_NUMBER)"'
key: 'setup'
- command: 'echo "build number is: $(buildkite-agent meta-data get "example")"'
depends_on: 'setup'
It might still be best to keep a pipeline.yml in your repo so you don’t have to worry about this limitation, and keep the pipeline settings yaml as a place to authorize your build before you run the pipeline upload step.
