This alludes to spot instances, but to add one clear use case that’s impacting us is that we use Google Cloud and Preemptible instances.
These instances are cycled in 24 hours or less, so the symptom of losing hosts is rather prolific across all of our pipelines, and all of our steps. Having an ability to set a standard retry either on the pipeline level or across all pipelines in a project would be helpful. As is, we’ll need to go through and ad this to every step in all of our pipelines.