# Is buildkite-agent intended to be used on preemptible instances? **URL:** \ **Category:** General\ **Created:** [October 22, 2020, 10:42pm UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297 "2020-10-22T22:42:10Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![onsails](https://avatars.discourse-cdn.com/v4/letter/o/b9bd4f/32.png) [@onsails](https://forum.buildkite.community/u/onsails)\ **Post date:** [October 22, 2020, 10:42pm UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297/1 "2020-10-22T22:42:10Z")
I am running buildkite-agent inside a google kubernetes cluster on preemptible node pool. When node goes down (or when I simulate this situation by just killing pod) agent stays in buildkite agents lists for a very long time. I know that there is a `health-check-addr` option but it’s not an option for agents running inside private clusters. Is there any way to set some kind of timeout which sould be tracked on buildkite API side? ---
**Author:** ![onsails](https://avatars.discourse-cdn.com/v4/letter/o/b9bd4f/32.png) [@onsails](https://forum.buildkite.community/u/onsails)\ **Post date:** [October 23, 2020, 11:24am UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297/2 "2020-10-23T11:24:11Z")
I suppose perfect solution would be an option like `eviction-id` which tells API that if there is a new agent spawned, the old one with the same `eviction-id` should be stopped by buildkite API. ---
**Author:** ![juanitofatas](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/juanitofatas/32/957_2.png) [@juanitofatas](https://forum.buildkite.community/u/juanitofatas)\ **Post date:** [October 27, 2020, 2:09am UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297/3 "2020-10-27T02:09:42Z")
Hi Andrey, I am not familiar with this but I think it is ok to use on Preemptible instances. There is similar work that adds scripting around the life cycle of Agent. Example of such scripting work from our Elastic CI Stack for AWS: [https://github.com/buildkite/elastic-ci-stack-for-aws/pull/737](https://github.com/buildkite/elastic-ci-stack-for-aws/pull/737/files) I found Google has an article on terminating with grace ​https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-terminating-with-grace But I cannot see a way to send double SIGTERM to the agent, but maybe could implement a preStop hook that sends the first SIGTERM then sleeps for 20 seconds. Once that is done, Kubernetes will send the second SIGTERM which will force the job to stop and the agent will be able to deregister. I hope this helps! Cheers, Juanito​​​​​​ ---
**Author:** ![megan](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/megan/32/882_2.png) [@megan](https://forum.buildkite.community/u/megan)\ **Post date:** [October 30, 2020, 8:31am UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297/4 "2020-10-30T08:31:30Z")
Is this a suggestion or this option exists to fix this issue ? ---
**Author:** ![juanitofatas](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/juanitofatas/32/957_2.png) [@juanitofatas](https://forum.buildkite.community/u/juanitofatas)\ **Post date:** [November 2, 2020, 3:25am UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297/5 "2020-11-02T03:25:22Z")
Sorry would need to implement scripting similar to [https://github.com/buildkite/elastic-ci-stack-for-aws/pull/737](https://github.com/buildkite/elastic-ci-stack-for-aws/pull/737/files) for now on GCP. ---
**Author:** ![harisekhon](https://avatars.discourse-cdn.com/v4/letter/h/6de8d8/32.png) [@harisekhon](https://forum.buildkite.community/u/harisekhon)\ **Post date:** [December 15, 2020, 6:31pm UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297/6 "2020-12-15T18:31:43Z")
Buildkite is not well suited to preemptible instances because even with a spot termination script to shut down the agent, you usually only have 30 seconds to do that and since most builds take longer than 30 seconds you won’t be able to wait and will incur false negative build failures. See this ticket I’ve just raised which would make BuildKite more suitable for use with pre-emptible instances: > [@Reschedule builds on other agents rather than Fail builds when agents time out or are killed (machine shut down or put to sleep)](https://forum.buildkite.community/t/reschedule-builds-on-other-agents-rather-than-fail-builds-when-agents-time-out-or-are-killed-machine-shut-down-or-put-to-sleep/1388): > > Please can you implement build retries on other agents to handle when the BuildKite agent goes away due to the agent or machine being shut down or machine put to sleep. In distributed processing systems it is common to retry a task 4 times on 4 different hosts before declaring a task as actually failed, because a lot of the time those failures are due to temporary issues or machines dying or in my case being shut down or put to sleep because I run BuildKite agents on my laptop. This will also … ---
**Author:** ![harisekhon](https://avatars.discourse-cdn.com/v4/letter/h/6de8d8/32.png) [@harisekhon](https://forum.buildkite.community/u/harisekhon)\ **Post date:** [December 17, 2020, 1:36pm UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297/7 "2020-12-17T13:36:09Z")
Update: you will need to automatic retries for when the agent is lost or killed, in which case it should work, see the thread I linked which has the details. ---
**Author:** ![onsails](https://avatars.discourse-cdn.com/v4/letter/o/b9bd4f/32.png) [@onsails](https://forum.buildkite.community/u/onsails)\ **Post date:** [December 25, 2020, 5:15pm UTC](https://forum.buildkite.community/t/is-buildkite-agent-intended-to-be-used-on-preemptible-instances/1297/8 "2020-12-25T17:15:02Z")
I’ve found an easy way for agents running on kubernetes not to hang on redeployment. Perhaps it will work for preemption event too, didn’t have a chance to test yet. 1. Create pre-stop.sh and add it to a container: ```auto #!/usr/bin/env bash set -euo pipefail # https://buildkite.com/docs/agent/v3#signal-handling` echo "Sending SIGTERM to buildkite-agent" > /dev/termination-log pkill -TERM buildkite-agent # TODO: send SIGQUIT after sleep 25? ``` 1. Add preStop hook to deployment: ```auto apiVersion: apps/v1 kind: Deployment metadata: ... spec: ... template: spec: containers: - name: buildkite-agent image: ... lifecycle: preStop: exec: command: ["/usr/bin/env", "bash", "/pre-stop.sh"] ``` On the next deployment or a pod deletion buildkite-agent will notify api that it’s stopped. _Waring_: for some reason it doesn’t wait for an ongoing step to finish, step fails with `Terminating bootstrap after cancellation with terminated`.