I’m setting up Buildkite to deploy a relatively complex service that runs in several dozen locations around the world. To minimize disruptions, each region gets deployed in the middle of the night, in the region’s local timezone.
We already have a map of our regions and the corresponding desired deployment start times in UTC, and with Buildkite’s dynamic pipelines we can elaborate the action graph and rollout plan from that source of truth (yay!). However, what we currently can’t do is make buildkite wait for the appropriate time before executing a subgraph.
Our current plan is to hack around this by conditioning each region’s deployment on an approval step, and then building additional logic outside buildkite to trawl through the BK API and click the buttons on a schedule. However, this is obviously quite clunky, and the whole point of buildkite is to save us from having to build our own deployment orchestration machinery 
We also considered having command steps that sleep() for the appropriate time, but for any given deploy part of the graph is going to wait close to 24 hours, and that would cause egregious amounts of billed minutes. Buildkite can implement such a step on the control plane side at minimal steady state cost, similar to approval steps. We can’t replicate that outside of the control plane.
Would it be possible to add a “time-based wait” step type, that waits on the control plane side (i.e. without occupying a build agent) until a particular timestamp before completing? Ideally with an option for a human to overrule and have it complete immediately (similar to current approve steps). IOW, I’m picturing a step something like this in yaml:
- sleep: "Wait for Europe/Amsterdam deploy window"
until: "2024-12-13T02:00:00Z"
I picked RFC 3339 as the timestamp format just because all programming languages are likely to have an RFC3339 parser and formatter available, but I’ll settle for any format I can explain to a computer - unix seconds, seconds relative to pipeline upload, RFC822, … Relative time to when the step was uploaded would make such a step useful for people doing more static yaml pipelines, although for our use case where we’re generating a complex dynamic pipeline on every build, absolute timestamps would be preferable.
Buildkite is reasonably good at orchestrating complex deployments in general, this is the single big showstopper we’ve encountered so far. As I say, it’s possible for us to write a pile of code to work around it outside buildkite, but if we start down the road of building parts of the control plane ourselves, it’s harder for me to argue the value of using buildkite for this rather than something else. I would love to not have to have those conversations :)
,