We have many developers submitting PRs, and when merged, they generate many more jobs than are run pre-merge. For example, a proposed changeset to our Android app directory will run about common jobs and about 5 Android test jobs, for a total of 10 jobs, but then post-merge runs the entire pipeline of 70 jobs. The post-merge jobs should ideally be scheduled at a lower priority, since developers are not waiting on these results to continue development, whereas pre-merge jobs should return results as quickly as possible.
What we would ideally like to do is dynamically assign step priorities based on whether the build is a pull request, or branch build, by modifying the pipeline yaml that is uploaded to Buildkite.
Feature Spec
Add a numeric priority field to the step block of an uploaded pipeline. When Buildkite schedules jobs against agents, it will first assign higher priority jobs in order of creation, then submit lower priority jobs in order of creation, and so on.
For example, if 100 jobs with priority 4 are waiting for agents, and a PR is submitted with 10 jobs that have priority 8, then those 10 jobs would effectively jump the queue and be assigned before the remaining 100 jobs would be assigned.