# Autoscaling disconnects active agents **URL:** \ **Category:** Elastic CI Stack for AWS\ **Created:** [June 27, 2023, 9:13pm UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983 "2023-06-27T21:13:03Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![samuel](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@samuel](https://forum.buildkite.community/u/samuel)\ **Post date:** [June 27, 2023, 9:13pm UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/1 "2023-06-27T21:13:03Z")
Hi team, Currently running into an issue where auto-scaling is bringing down active agents occasionally when there are idle agents. Are there any settings I can change to prioritize bringing down idle agents? ---
**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\ **Post date:** [June 27, 2023, 9:45pm UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/2 "2023-06-27T21:45:02Z")
Hi Sam! 👋 You can use the `ScaleInIdlePeriod` property that corresponds to the `disconnect-after-idle-timeout` from the [agent configuration](https://buildkite.com/docs/agent/v3/configuration#disconnect-after-idle-timeout) : [elastic-ci-stack-for-aws/packer/linux/conf/bin/bk-install-elastic-stack.sh at master · buildkite/elastic-ci-stack-for-aws · GitHub](https://github.com/buildkite/elastic-ci-stack-for-aws/blob/master/packer/linux/conf/bin/bk-install-elastic-stack.sh#L186) (number of _seconds_ the agent must be idle before terminating). `ScaleInIdlePeriod` isn’t an ASG parameter; it terminates an agent, but the instance can still be up. Hope this helps! Best, ---
**Author:** ![jkburges](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/jkburges/32/986_2.png) [@jkburges](https://forum.buildkite.community/u/jkburges)\ **Post date:** [July 31, 2023, 11:07pm UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/3 "2023-07-31T23:07:06Z")
I have the same issue I think. I can’t find docs about agent status, but I would assume that if there’s a running job, then it’s not idle, and therefore shouldn’t be shut down? But alas, this is what I saw. Here’s the timeline from the relevant job: ![image](https://us1.discourse-cdn.com/flex016/uploads/buildkite1/original/2X/f/fb370cd56c187d84e1fe51137e7ad40bfe33ae07.jpeg) and the corresponding activity history from the ASG: ![image](https://us1.discourse-cdn.com/flex016/uploads/buildkite1/original/2X/a/a512f23f8d1551bb4c8d5591b5e71bdb0a093528.jpeg) Perhaps the scaling logic only tells the ASG to reduce size, then the ASG just picks randomly one of the instances? But it would be good if that took into consideration whether or not there was a job in progress. ---
**Author:** ![james.s](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/james.s/32/1305_2.png) [@james.s](https://forum.buildkite.community/u/james.s)\ **Post date:** [July 31, 2023, 11:36pm UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/4 "2023-07-31T23:36:03Z")
Hey @jkburges! Thanks for the message 👍 For your stack in question above (thanks for the pictures also) - what was the `ScaleInIdlePeriod`? As @paula mentioned above, the parameter isn’t one set on the ASGs themselves, but at an agent instance level ([UserData](https://github.com/buildkite/elastic-ci-stack-for-aws/blob/34cc18a3abd5b144cd3d67d43af4617a442587a3/templates/aws-stack.yml#L1146) specifically). The scaler (by default) doesn’t scale in: its more based on the termination script [added](https://github.com/buildkite/elastic-ci-stack-for-aws/blob/34cc18a3abd5b144cd3d67d43af4617a442587a3/packer/linux/scripts/install-buildkite-agent.sh#L68) which then sets the ASG’s desired count. Cheers! ---
**Author:** ![jkburges](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/jkburges/32/986_2.png) [@jkburges](https://forum.buildkite.community/u/jkburges)\ **Post date:** [July 31, 2023, 11:41pm UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/5 "2023-07-31T23:41:21Z")
> [@james.s](#): > > what was the `ScaleInIdlePeriod` ? 600 which I guess is the default, pretty sure we’re not setting it. ---
**Author:** ![james.s](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/james.s/32/1305_2.png) [@james.s](https://forum.buildkite.community/u/james.s)\ **Post date:** [August 1, 2023, 12:19am UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/6 "2023-08-01T00:19:17Z")
Thanks for confirming @jkburges - 600 seconds/10 minutes being the default. Happy for this to be sent to a dedicated item to `support@buildkite.com` but it would be worth seeing what the actual activity on the job for the last 10 minutes was for the agent to consider itself being idle/shutting down after the allotted time. Cheers đź‘Ť ---
**Author:** ![jkburges](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/jkburges/32/986_2.png) [@jkburges](https://forum.buildkite.community/u/jkburges)\ **Post date:** [August 1, 2023, 12:25am UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/7 "2023-08-01T00:25:08Z")
> [@james.s](#): > > actual activity on the job for the last 10 minutes was for the agent to consider itself being idle/shutting down after the allotted time It is deploying to AWS Elastic Beanstalk, using the `eb deploy` CLI tool. The EB side takes a while (over 10 minutes) so it’s basically sitting there doing nothing, waiting for the status to change. I should say that I haven’t noticed this problem before, and we do a lot of deploys. ![image](https://us1.discourse-cdn.com/flex016/uploads/buildkite1/original/2X/a/ad12e07ece7112bddb94f5cf9efdf19f72abde30.jpeg) ---
**Author:** ![james.s](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/james.s/32/1305_2.png) [@james.s](https://forum.buildkite.community/u/james.s)\ **Post date:** [August 1, 2023, 12:41am UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/8 "2023-08-01T00:41:58Z")
Cheers - could very well be a contributing factor there - though of course, it depends on how _long_ the `eb deploy` CLI takes to wrap up. It might be worth considering bumping the timeout period to accomodate - though I concur with the actual shutdown/termination scripts to account for `active` agents. Another thing - has this CI stack been configured with Spot instances with `InstanceType`? Not ruling it our per-se, but reclamation might be at play if so ---
**Author:** ![jkburges](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/jkburges/32/986_2.png) [@jkburges](https://forum.buildkite.community/u/jkburges)\ **Post date:** [August 1, 2023, 1:11am UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/9 "2023-08-01T01:11:25Z")
Yes, we use spot instances. But I think the ASG message above in the screenshot “taken out of service in response to a user request” suggests that it’s not this? Could be wrong 🙂 ---
**Author:** ![james.s](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/james.s/32/1305_2.png) [@james.s](https://forum.buildkite.community/u/james.s)\ **Post date:** [August 1, 2023, 1:40am UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/10 "2023-08-01T01:40:38Z")
Suspected the case 🙂 You are right that its more a _User_ (script) level request from first glance, though noting that from the logs , the `Deployment succeeded` entry indicates the finishing of the EB deployment, then for the last ~7 minutes, the agent could then be sitting idle. Depending on how long the deployment normally takes to run before that command in itself wraps up, the idle period then would be whats triggering its shutdown. Also worth potentially trialling this with a on demand instance - primarily for the case of reclaims (but also taken into consideration the cost differential). Cheers! ---
**Author:** ![jkburges](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/jkburges/32/986_2.png) [@jkburges](https://forum.buildkite.community/u/jkburges)\ **Post date:** [August 1, 2023, 2:18am UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/11 "2023-08-01T02:18:31Z")
> [@james.s](#): > > Depending on how long the deployment normally takes to run before that command in itself wraps up, the idle period then would be whats triggering its shutdown. That screenshot was a bit misleading due to the clipping, the full log is: > 2023-07-31 19:14:59 INFO Deployment succeeded. Terminating old instances and temporary Auto Scaling group. i.e. the `eb deploy` command is still running as the deploy hasn’t finished due to still having some cleanup to do. > the agent could then be sitting idle This bit I don’t understand: if `eb deploy` is still running, within the context of a job (also running), then the agent should not be considered idle, right? Or is idleness based on something else? ---
**Author:** ![james.s](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/james.s/32/1305_2.png) [@james.s](https://forum.buildkite.community/u/james.s)\ **Post date:** [August 1, 2023, 2:45am UTC](https://forum.buildkite.community/t/autoscaling-disconnects-active-agents/2983/12 "2023-08-01T02:45:29Z")
Ah - turns out truncation occurred: no worries, so the actual command then was still running. The agent processes have a fixed lifetime based on the parameters as above, though they should be extended if they are running a job - so crossing off all options with the Spot instance ask before. Its all part of the lifecycle of the agent (handled via [lifecycled](https://github.com/buildkite/lifecycled)) I believe what is happening is the `eb deploy` command is still running, but under the hood the lifecycle is potentially considering the agent to be idle, even though its actively running the deploy command. It would be worth for a deeper investigation and potential feedback on our side too: happy for you to raise this to `support@buildkite.com` so we can then action this individually 🙂