I run Buildkite in a kubernetes cluster. I put together different pods of buildkite-agent with different “resource_class” tags, and agent priorities.
In the way I have set it up, our scheduled builds uses a smaller pool of agents and our rebuilar builds uses a larger pool of agents. With the agent priority, builds are scheduled into our standard pods, but can overflow into our “long-running” pods as needed.
Since I am using K8S, I configured the resource requests to “burst” and then oversubscribe the pods.
I have not thought about branch priorities, but I could probably set something up like that too. We had not needed it though, because there is enough capacity to build both master and feature branches together, and the lead time for deploying from master is reasonably fast, even when there are feature branches being built.