In our current setup, we run both cloud agents and local agents today to optimize for cost. Ideally the local agent take most of the load and then cloud agents scale up as needed to handle the backlog. To enable graceful load balancing between the two pools, we do not add any specific agent tags.
We observe the current behavior scales up the cloud agent immediately and most of the time local agents end up picking the job before the cloud agent even has time to get brought up, thus wasting money running nothing. Is there some setting I can enable to optimize the current behavior so we only scale up cloud agents after some time has passed?
Thanks for your message - and interesting question all the same!
Given that from your description, the pipeline is targeting a queue that has both local and cloud based agents, and your mention of “scaling”, you are utilising our CI Stack (correct me if this isnt the case!).
The current scaling mechanisms supported (as of writing) are here - there isnt a direct parameter in the Cloudformation Stack to set scaling after a set time has passed, but the closest parameter I believe that you could configure to achieve what you require would be the ScaleOutFactor to be less than the default 1.0 (which then would slow down scaling out of cloud based agents, which is handled by the Autoscaling Lambda).
