Hybrid Agent Setup

Hello,

We use the Elastic CI stack for AWS, but also have some fixed agents in the same queue. The goal would be that the elastis stack only scales when all the fixed agents are busy.

Is it possible that the AutoscalingFunction works on modified measurements for the Buildkite Queue (e.g. if we have 10 fixed agents, we want ScheduledJobsCount to be max(0, x-10)? I have no insight into this part, so I’d be grateful if someone could give me a quick hint before I start digging.

This small adjustment should be sufficient to avoid unnecessary scaling with additional fixed agents in the queue.

Thanks!

Hello, @bunert and welcome to the Buildkite Community!
Could you please tell us if you’re using clusters?

Best!
Karen

Hi Karen,

Yes, we switched to clusters around the same time we introduced the self-hosted Buildkite agents.

The improved UI in the cluster queue overview is great, but it also quickly showed that scaling does not work the way we wanted it to.

Thanks for confirming about using the clusters, @bunert!

We discussed the issue with the team and the short answer would be “it is not possible in the exact way you’re thinking it would work”, however it can be accomplished in a different way.

A work-around for this would be separating your jobs into tasks of different priority and using different queues. For instance, with a “priority” queue for your fixed agents and an “elastic” queue for the EC2 agents. But it is not possible to only trigger the AWS agents once all of your fixed ones are busy without some clever scripting.

The idea here would be to use two queues in the cluster, one primary/fixed queue and one EC2 queue, and use scripting to flip the queue based on a conditional (you’d want to experiment here). But a direct approach like you’re thinking isn’t possible/not supported at this time.

Best!
Karen

Hi Karen

I took a look into the buildkite-agent-scaler and doing some modifications in the scaling_calculator.go seems to work for my use case.

The elastic CI stack for AWS already supports providing a custom BuildkiteAgentScalerServerlessARN, so that worked quite smooth :slight_smile:

Thanks for the quick reply!

Hello, @bunert! Glad to see that you found a workaround that is working for you. If you need help with anything else - we’re always here, happy to help.

Best!
Karen