# Experimental Lambda-based Scaler 🦑 **URL:** \ **Category:** Elastic CI Stack for AWS\ **Created:** [April 10, 2019, 4:15am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425 "2019-04-10T04:15:40Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![anon18197598](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@anon18197598](https://forum.buildkite.community/u/anon18197598)\ **Post date:** [April 10, 2019, 4:15am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/1 "2019-04-10T04:15:40Z")
The latest version of the stack, [v4.3.1](https://github.com/buildkite/elastic-ci-stack-for-aws/releases/tag/v4.3.1), introduces a `EnableExperimentalLambdaBasedAutoscaling` parameter, which when set to `true` will disable the default Amazon AutoScaling powered scaling behaviour in favour of a [Lambda](https://github.com/buildkite/buildkite-agent-scaler) that handles the scale-out. Due to a combination of avoiding the intrinsic wait-times for native autoscaling and a much faster polling rate, the stack scales much, much faster up from zero to whatever capacity you need. We are seeing wait time reduction on builds with cold stacks of up to 50%. Scale down is handled with the new `--disconnect-after-idle-timeout` flag that was added to the agent in [v3.10.0](https://github.com/buildkite/agent/releases/tag/v3.10.0). After the agent has been idle for a while (configured with `ScaleDownPeriod`), it disconnects and then terminates the instance and [decrements the autoscaling group desired count atomically](http://docs.aws.amazon.com/cli/latest/reference/autoscaling/terminate-instance-in-auto-scaling-group.html). The result is a much, much faster scale out and a much simpler scale-in process that no longer requires [lifecycled](https://github.com/buildkite/lifecycled). We’d love to hear how it works for your stacks and what issues you encounter. The plan is to make this the default for v5.0.0. ## FAQ ### Why not handle scale-in with the lambda too? We tried! For some unknown reason, ASG’s don’t fire lifecycle hooks when scaling in via directly setting `DesiredCount`. They do however fire them when terminating an instance with [`TerminateInstanceInAutoScalingGroup`](https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_TerminateInstanceInAutoScalingGroup.html). We were perplexed too. ### What about the other scaling configuration options? With the new autoscaler enabled, the following options are respected: - MinSize - MaxSize - ScaleDownPeriod - InstanceCreationTimeout Conversely, the following options are completely ignored: - ScaleUpAdjustment - ScaleDownAdjustment - ScaleCooldownPeriod We might consider implementing `ScaleUpAdjustment` if there is interest, it could provide a min-bound for scale up. ### How about all the metrics the stack used to publish? With the new scaling enabled, we disable the old [buildkite-agent-metrics](https://github.com/buildkite/buildkite-agent-metrics) lambda in favour of a smaller, nimbler all-in-one lambda that collects metrics and does the scaling. Whilst we do also publish `ScheduledJobsCount` and `RunningJobsCount`, that’s all. You can still run [buildkite-agent-metrics](https://github.com/buildkite/buildkite-agent-metrics) directly, it works nicely without a queue so you can have one lambda power a whole organization of metrics. Let us know if there are any metrics you really miss and we’ll consider adding them back. ### Is anything broken? I think I might have broken `BuildkiteTerminateInstanceAfterJob` 🤔 ### Help! My instances occasionally are very slow! If you are using an instance type that has burstable CPU credits, you might be running into this: [https://serverfault.com/questions/740498/why-do-ec2-t2-instances-sometimes-start-with-zero-cpu-credits](https://serverfault.com/questions/740498/why-do-ec2-t2-instances-sometimes-start-with-zero-cpu-credits) You can avoid cycling instances so often by using a much longer `ScaleDownPeriod`. ### Why won’t my ASG’s provision more than 10 instances at a time? Turns out there is a hidden setting on ASG’s that limit increases to batches of 10. If you email AWS support they will change this for you. ---
**Author:** ![pagameba](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/pagameba/32/362_2.png) [@pagameba](https://forum.buildkite.community/u/pagameba)\ **Post date:** [April 26, 2019, 2:00pm UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/2 "2019-04-26T14:00:21Z")
I’ve been using this scaler for a couple of weeks now and I really like the decrease in allocation time for build instances, we’ve seen a decrease from about 3.5 mins to 1.5 mins (qualitative assessment, I didn’t actually go back and measure too carefully). However, our build times are not decreasing because it doesn’t respect the ScaleUpAdjustment. Our builds generally follow a fan out/in approach where we build a docker image then run multiple tests on it simultaneously. Previously we’d wait 3.5+ minutes for the first set of images to become available but then tests would immediately have an agent available for running as soon as the build finished. Now, only a single agent is scheduled for the build phase but we also have to wait on agents in the test phase as well, so the net effect is about the same overall. It would be really useful for us if you implemented ScaleUpAdjustment. Thanks for all the great work! ---
**Author:** ![anon18197598](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@anon18197598](https://forum.buildkite.community/u/anon18197598)\ **Post date:** [April 28, 2019, 6:35am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/3 "2019-04-28T06:35:52Z")
Good idea, I’ve got a PR up at [https://github.com/buildkite/buildkite-agent-scaler/pull/12](https://github.com/buildkite/buildkite-agent-scaler/pull/12). Feedback welcome. ---
**Author:** ![joffotron](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/joffotron/32/580_2.png) [@joffotron](https://forum.buildkite.community/u/joffotron)\ **Post date:** [June 20, 2019, 6:54am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/4 "2019-06-20T06:54:05Z")
We’ve seen some behaviour where `MinSize` doesn’t look like it’s being respected. Had it set to ‘8’ in the stack, yet all our build agent instances were terminated. Suggestions as to how to debug? ---
**Author:** ![anon18197598](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@anon18197598](https://forum.buildkite.community/u/anon18197598)\ **Post date:** [June 20, 2019, 7:09am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/5 "2019-06-20T07:09:00Z")
Which stack version or autoscaler version? ---
**Author:** ![joffotron](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/joffotron/32/580_2.png) [@joffotron](https://forum.buildkite.community/u/joffotron)\ **Post date:** [June 20, 2019, 7:19am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/6 "2019-06-20T07:19:31Z")
Hah, sorry! I’ve just had a conversation with our contractor-ops person and turns out they were trying something out and had terminated all the instances manually! No bugs to see here! Thanks :-) ---
**Author:** ![anon18197598](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@anon18197598](https://forum.buildkite.community/u/anon18197598)\ **Post date:** [June 20, 2019, 7:30am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/7 "2019-06-20T07:30:37Z")
No problems! Glad it’s all working! ---
**Author:** ![pme](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/pme/32/711_2.png) [@pme](https://forum.buildkite.community/u/pme)\ **Post date:** [January 28, 2020, 3:43pm UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/8 "2020-01-28T15:43:45Z")
FYI: it is possible to invoke the scaler lambda more frequently than once per minute which is the fastest you can do with a CloudWatch trigger, or by keeping the lambda running, by utilizing a step function. ![flowchart](https://us1.discourse-cdn.com/flex016/uploads/buildkite1/original/1X/f014ef696e71d3ed3181771a3efc6c044b56a2d1.png) I’ve implemented this for our scaler and it works like a charm: > **[rockset/high-frequency-lambda](https://github.com/rockset/high-frequency-lambda)** > > Contribute to rockset/high-frequency-lambda development by creating an account on GitHub. ---
**Author:** ![anon18197598](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@anon18197598](https://forum.buildkite.community/u/anon18197598)\ **Post date:** [February 2, 2020, 9:23pm UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/9 "2020-02-02T21:23:02Z")
Yeah, I opted for running the function for longer in the new autoscaler. It seemed simpler than step functions. ---
**Author:** ![phan.le](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/phan.le/32/820_2.png) [@phan.le](https://forum.buildkite.community/u/phan.le)\ **Post date:** [June 18, 2020, 5:50am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/10 "2020-06-18T05:50:00Z")
Have you seen any behavior where: 1. A MinSize is set (i.e. 1) 2. An idling agent exits and its host is removed from ASG causing the desired size (0) \< min size (1) 3. A new machine is created to fill the gap. So machines keep being shut down (because of idle agents) and spinning up. ---
**Author:** ![sj26](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/sj26/32/17_2.png) [@sj26](https://forum.buildkite.community/u/sj26)\ **Post date:** [June 19, 2020, 12:15am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/11 "2020-06-19T00:15:23Z")
@phan.le sorry, we haven’t — the agent shouldn’t exit by itself when idle, only if you ask it to (or it gets scaled in by AWS). Want to send us a message through to [support@buildkite.com](mailto:support@buildkite.com) with the specifics of your case? Edit: unless you’re using `--disconnect-after-idle-timeout`? In which case the idle timeout and the min size might be fighting! ---
**Author:** ![anon18197598](https://avatars.discourse-cdn.com/v4/letter/a/9d8465/32.png) [@anon18197598](https://forum.buildkite.community/u/anon18197598)\ **Post date:** [June 19, 2020, 1:09am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/12 "2020-06-19T01:09:53Z")
Yeah, apologies @phan.le, this is unfortunately something that is very hard to do with the lambda scaler. Maintaining a minimum set of agents is really hard as they terminate when idle and can’t co-ordinate with each other to keep at least N running. I’d recommend that you use a much longer disconnect after idle timeout for now! ---
**Author:** ![phan.le](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/phan.le/32/820_2.png) [@phan.le](https://forum.buildkite.community/u/phan.le)\ **Post date:** [June 19, 2020, 1:48am UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/13 "2020-06-19T01:48:53Z")
Got it, thanks for responding! The reason we want to keep a small minimum of agents is to reduce latency when a commit is pushed. ---
**Author:** ![max](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/max/32/1523_2.png) [@max](https://forum.buildkite.community/u/max)\ **Post date:** [September 25, 2024, 4:45pm UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/14 "2024-09-25T16:45:24Z")
In case others run into anything similar in the future: For some reason, setting `disconnect-after-idle-timeout` did not fully work for me (at least, as I interpreted its functionality from this forum post). It would stop the `buildkite-agent` service on the underlying server/instance, but it would not ‘terminate the instance and decrement the autoscaling group desired count atomically’ as stated above. I’m not sure why this is the case, but regardless, a workaround is as follows: In your buildkite agent’s cloud-init (or similar) script after you start the buildkite agent, run a background script that does something like: ```auto #!/bin/bash sleep 300 while service buildkite-agent status | grep -q 'Running'; do sleep 60 done INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) AWS_REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) aws ec2 terminate-instances --instance-ids "$INSTANCE_ID" --region "$AWS_REGION" ``` Depending on your ASG config, you may need additional commands before the terminate to decrement the ASG desired-counts accordingly. Feels pretty janky but it works. ---
**Author:** ![ivanna](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/ivanna/32/1410_2.png) [@ivanna](https://forum.buildkite.community/u/ivanna)\ **Post date:** [September 25, 2024, 11:06pm UTC](https://forum.buildkite.community/t/experimental-lambda-based-scaler/425/15 "2024-09-25T23:06:43Z")
Hey @max! Thank you for sharing your workaround with the Buildkite community! Can you confirm that you are using the Elastic Stack to manage your Buildkite agents? If so, what version? As the `--disconnect-after-idle-timeout` flag is designed to only stop the Buildkite agent when it’s idle, but it doesn’t handle terminating the instance or adjusting the autoscaling group’s desired count on its own. These actions are automatically managed when you’re using the Elastic Stack. To dig into can you share your custom configuration including the bootstrap and hook to help identify underlying issues. You can sent those details to **[support@buildkite.com](mailto:support@buildkite.com)**. Thanks,