# Newer versions of the Buildkite agent don't respect the cancel grace period **URL:** \ **Category:** General\ **Created:** [April 30, 2024, 4:35pm UTC](https://forum.buildkite.community/t/newer-versions-of-the-buildkite-agent-dont-respect-the-cancel-grace-period/3684 "2024-04-30T16:35:31Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![etoddstrongdm](https://avatars.discourse-cdn.com/v4/letter/e/977dab/32.png) [@etoddstrongdm](https://forum.buildkite.community/u/etoddstrongdm)\ **Post date:** [April 30, 2024, 4:35pm UTC](https://forum.buildkite.community/t/newer-versions-of-the-buildkite-agent-dont-respect-the-cancel-grace-period/3684/1 "2024-04-30T16:35:31Z")
Posting here as Github issues don’t seem to be monitored… Somewhere in the last few releases, the Buildkite agent stopped respecting `BUILDKITE_CANCEL_GRACE_PERIOD`, which we currently set to `600`. Instead it kills the job after the default 10 second grace period. I rolled back our agent to 3.65 and that version seems to respect it properly. Also reported here: [Newer versions don't respect `BUILDKITE_CANCEL_GRACE_PERIOD` · Issue #2748 · buildkite/agent · GitHub](https://github.com/buildkite/agent/issues/2748) ---
**Author:** ![athreya](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@athreya](https://forum.buildkite.community/u/athreya)\ **Post date:** [April 30, 2024, 7:04pm UTC](https://forum.buildkite.community/t/newer-versions-of-the-buildkite-agent-dont-respect-the-cancel-grace-period/3684/2 "2024-04-30T19:04:31Z")
Hello Evan. Welcome to the Buildkite community! We appreciate you bringing this to our attention. Starting from agent version [3.66.0](https://github.com/buildkite/agent/releases/tag/v3.66.0), we have expanded graceful cancellation to all job phases. Essentially, this means that if a cancellation signal is sent while an agent is running a job, the agent will complete the current job and then shut down gracefully by default. This feature essentially replicates the functionality of `BUILDKITE_CANCEL_GRACE_PERIOD`, which ensures that the agent will shut down once the current job is finished or once the time specified in the `BUILDKITE_CANCEL_GRACE_PERIOD` environment variable is reached, whichever comes first. I conducted tests using a code snippet that traps “`SIGTERM`” and executes a small piece of code to guarantee a graceful termination. After multiple tests, I found that the job only ended within a ~10 second timeframe in the following scenarios: - When I forced the agent to cancel by issuing another cancel signal after the initial SIGTERM. - When I cancelled the job through the UI. However, during these scenarios,I noticed that the `BUILDKITE_CANCEL_GRACE_PERIOD` was not being honored, which is expected when we forcefully stop the agent. I’m curious to know if you experienced the same behavior or if it was different. Did you observe that cancelling a job while using the later versions of the agent (3.66.0 or above) still immediately cancels the job after the initial cancel signal, without waiting for the current running job to complete.? Cheers! 🚀 Athreya. ---
**Author:** ![etoddstrongdm](https://avatars.discourse-cdn.com/v4/letter/e/977dab/32.png) [@etoddstrongdm](https://forum.buildkite.community/u/etoddstrongdm)\ **Post date:** [April 30, 2024, 8:09pm UTC](https://forum.buildkite.community/t/newer-versions-of-the-buildkite-agent-dont-respect-the-cancel-grace-period/3684/3 "2024-04-30T20:09:49Z")
Yes my specific repro steps were: - Run buildkite agent with `BUILDKITE_CANCEL_GRACE_PERIOD=600` in the environment - Start a job - Cancel the job via Buildkite UI - Job handles the SIGTERM and starts a graceful shutdown - 10 seconds later, the job is forcefully killed From reading the pull requests, I thought the new `signal-grace-period` setting was intended to apply an _additional_ wait period in addition to the normal `cancel-grace-period`. If that’s the case, then I think this is a bug. On the other hand if the default 9 second `signal-grace-period` is meant to _override_ the `cancel-grace-period`, then I would have appreciated some documentation making it clear that this is a breaking change! ---
**Author:** ![ivanna](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/ivanna/32/1410_2.png) [@ivanna](https://forum.buildkite.community/u/ivanna)\ **Post date:** [April 30, 2024, 10:35pm UTC](https://forum.buildkite.community/t/newer-versions-of-the-buildkite-agent-dont-respect-the-cancel-grace-period/3684/4 "2024-04-30T22:35:01Z")
Hello @etoddstrongdm, Ivanna here stepping in for Athreya due to the timezones. Thank for providing more details on this issue. We have tested using Buildkite agent version v3.66.0 and v3.71.0, and confirmed that neither version respects the `BUILDKITE_CANCEL_GRACE_PERIOD`. I will create the escalation for our internal pipeline team. Your initiative in creating a GitHub issue will certainly help in tracking and resolving this matter effectively. I will include the reference to the escalation ticket, and we will keep you updated! Thank you once again for your contribution to the Buildkite community! ---
**Author:** ![sarah.j](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@sarah.j](https://forum.buildkite.community/u/sarah.j)\ **Post date:** [May 1, 2024, 12:11am UTC](https://forum.buildkite.community/t/newer-versions-of-the-buildkite-agent-dont-respect-the-cancel-grace-period/3684/5 "2024-05-01T00:11:47Z")
Hello Evan, Thank you so much for raising this. My name is Sarah and I am engineer on the Pipelines team. I am posting here to acknowledge that our escalation team have received this issue. We will post again here when we begin our investigation or have other updates 🙂 We really appreciate you providing the extra details on the issue. We hope to get back to you soon. Cheers, Sarah