We have a few builds with flaky elements (eg external APIs, spot instances being terminated, etc) where we specifically don’t use Buildkite’s per-step retry functionality as we don’t want the flakiness being masked / ignored.
What I’d love is a “This build passed, but we had to retry some steps” notification/state. Build would appear as green but with a caveat, and would be reflected in slack/other notifications about the build.