Is buildkite-agent intended to be used on preemptible instances?

[harisekhon]

Buildkite is not well suited to preemptible instances because even with a spot termination script to shut down the agent, you usually only have 30 seconds to do that and since most builds take longer than 30 seconds you won’t be able to wait and will incur false negative build failures.

See this ticket I’ve just raised which would make BuildKite more suitable for use with pre-emptible instances:

Reschedule builds on other agents rather than Fail builds when agents time out or are killed (machine shut down or put to sleep)

Please can you implement build retries on other agents to handle when the BuildKite agent goes away due to the agent or machine being shut down or machine put to sleep. In distributed processing systems it is common to retry a task 4 times on 4 different hosts before declaring a task as actually failed, because a lot of the time those failures are due to temporary issues or machines dying or in my case being shut down or put to sleep because I run BuildKite agents on my laptop. This will also …