Playing around with how to set this up. We currently run a ton of tests in a pipeline on PR open.
General question for people who are using GH Merge Queues: do you run the same pipeline twice in your setup? Once when the PR is opened and a second time when the PR lands in the merge queue?
Or do you only run your tests pipeline when the PR lands in the merge queue?
More questions:
If I have two pipelines:
gh-readonly-queue/main/*What are the correct settings for required status checks to pass here?
If I include both like this:
I can’t actually put the PR in the merge queue because the buildkite/merge-queue-tests-merge-queues status check doesn’t trigger on general PR create.
And if exclude buildkite/merge-queue-tests-merge-queues as a required status check, the PR seems to skip the merge queue (or it goes through the queue so fast) and never triggers the buildkite/merge-queue-tests-merge-queues pipeline.
Here’s my PR, unable to land in the merge queue because of the required status checks:
Hey @ian-persona!
As you can see, there isn’t an optional check that you can implement, so I feel as though there may not be an easy way around this, however I’m going to take a look and see what we can find. 
Cheers!
@benmc thanks. How are people using Buildkite with Merge Queues in Github generally? Are you not listing the merge queue test pipeline as a required check?
Or do you set up a ruleset in Github that matches the Merge Queue branch pattern and put the required check on there? Edit: Never mind, that can’t work as rulesets are for target branches, not the branches the changes are coming from.
The pattern that we tend to see is that folks don’t necessarily run all of their tests on PRs and merges, with the assumption being that if the PR has passed tests then those tests would also pass in main, else the tests maybe don’t provide an accurate representation of a production environment.
We also advise that folks use the Skip builds with existing commits option to avoid double builds of the same commit.
If you’d like, you’re welcome to email us at support@buildkite.com so that we can dive a bit deeper and provide some additional input that’s more relevant to your use case.
Cheers!
In our experience here you often don’t know that a test is inadequate until multiple commits are merged together.
The very situation we’re trying to cover with merge queues is: branch A passes test 1, branch B passes test 1, but branch A+B fail test 1.
Right now, you only discover this when you merge to main and break main. This breaks main’s deployability and it breaks tests for everyone who branches off main after this breaking point.
It’s advantageous, in a busy monorepo, for us to catch as much of this type of breakage before commits land in main.
I do think there’s confusing documentation from Buildkite on the canonical way to do this setup with Github. Two blog posts and at least one post on the Discourse server don’t align well to make for a true solution right now.
Would be good, IMHO, to have this hashed out in public and stand as an artifact for those to who come after.
But if you think support email is better, I can take this there.
Thanks!
Hey @ian-persona!
Thanks for the added context there! More than happy to continue discussion here, we just tend to use email if we’re heading down a road where you may need to share links with us that you don’t want on a public forum, such as build URLs, but if that does become the case then you could just include a link to this thread.
I’ll see if I can find a source of truth for the optimal way to approach what you’re attempting to do!
Cheers!
Just to follow up on this; I’ve asked for some input from the product engineers here who have been working on the merge queue functionality. They’ll be following up on this thread soon.
Cheers!
I think I’ve got this mostly figured out now! The crux of the issue is this:
Any Buildkite pipeline that you want to have gate your merge queue needs to be in the required status checks for the ruleset you’re using to protect your
mainbranch. And any check that’s a required status check needs to pass on a pull request to be able to get the PR in to the merge queue.
So here are the two scenarios I’ve tested and my testing results.
Can you check them against what you know and use on your side @benmc?
Scenario 1: You want to run the same pipeline against PRs and merge queue branches
If you’re running the same Buildkite pipeline on PR open and on entry in to the merge queue the setup is pretty straightforward.
You do not want to put a branch limiting pattern on your Buildkite pipeline in this case. You just want to make sure that the following options are checked for the pipeline:
You can use whatever you want for the rest of the options, but those are the essential ones to have turned on.
On the Github side you’ll want to make sure that the buildkite/my-pipeline pipeline is listed as a required status check in either your ruleset or your branch protection rules where you’re enabling the merge queue.
You should make sure your pipeline code works in both the scenario where you’re running against a pull request and running against a branch update. The big difference between the two cases is branch runs will have BUILDKITE_PULL_REQUEST_BASE_BRANCH set to an empty string on branch builds – so if you’re relying on that to get a list of changed files or whatnot, you’ll need to anticipate it being "" and adjust your pipeline code accordingly.
Scenario 2: You want to run different pipelines against PRs and merge queue branches
This is the harder case.
Both of these pipelines need to be in the branch protection rules on Github as required status checks. That means they’ll both run during pull requests and during merge queue branch creation.
In our case, we use buildkite-builder to build our pipelines. So the first pipeline step is always “generate the actual pipeline”. This makes it easy to check the branch name and, if it doesn’t match a pattern (or does match a pattern), short circuit the pipeline.
In the pipeline we only want to apply to pull requests, we have the Ruby code check to see if the branch name starts with gh-readonly-queue – if it does we don’t generate any steps and the pipeline stops there and is green. If it doesn’t, we generate all the steps we’d like to run during a pull request.
In the pipeline we only want to apply to merge queue branches, we do the same check for the branch name, but if we get a match we generate steps. Otherwise we stop there.
This ensures both pipelines run on PR and branch creation. But they do work in the scenario we want them to do work.
You could, of course, use the same check to make a single pipeline do different things in the PR and merge queue branch case, but we have long and complex pipeline files and liked the physical file separation we got doing it this way. It also made the UI nicer as engineers seeing a failure in the merge queue see it as “the merge queue pipeline failed” not “the PR pipeline failed” so it’s easier to know what to debug and where to look.
Thanks for that detail! As you mentioned, scenario 1 should be straight forward and generally how merge queues work out of the box; a single ruleset that checks on the status of external checks and acts accordingly.
Scenario 2 is more on the advanced end of merge queue usage, from reading our own docs/blog post and the docs in GH. The automation you have in place with buildkite-builder seems great and super handy. I’d imagine it’s something we’d be keen on publicising a little more too for folks who have more advanced requirements of merge-queues.
FWIW, the blog post is the current advice we have on using merge queues, but I think folks here would be keen on adding to that. Does your Ruby code just check on the presence/value of the BUILDKITE_PULL_REQUEST_BASE_BRANCH and perform accordingly?
