Ability to iterate mid-pipeline

Features
#1

I have a pipeline that’s composed of many ordered steps in the DAG. Steps publish and consume metadata and artifacts to avoid repeating work.

If a step fails towards the right-hand end, to be able to iterate on fixing that, my current option is to make a fix then wait a while to see what happens. For game builds, this could be several hours.

I would like to be able to restart a build that has failed, from the point (or points, I suppose) of failure, with a newly chosen revision (probably defaulting to HEAD in the same branch), so that I can avoid needing to wait for all the previous steps that were green to be green again.

I’d like the restarted build to have access to the same build-state (artifacts and metadata and env) as it had at the point of failure.

I’d be totally fine with not being able to restart a restarted pipeline, if the first restart cleared the first failure and exposed a second; I’d still be sped up, here, and I appreciate that the semantics could get complex.

I would not want a restarted build to change how commit-statuses got published onward to things like SCM providers, because I would absolutely still want only an end-to-end green build to be canonical.

I have thought about ways to make the artifacts possible to come from previously-successful builds of the pipeline - but this is both fiddly, and not necessarily suitable if the reason for the failure is relating to how the artifacts are dealt with.

Does anyone have any different ideas about how this set of problems could be solved today?

#2

Hi @petemounce! :wave:

The best I’ve managed in this sort of vein is to grab a copy of the job’s environment variables and the run the scripts with matching env until it works. If you’re highly integrated with meta-data and artefacts etc it does get tricky though. A sneaky way is to grab an agent access token from one of your running agents somehow and use that, but it’s a bit dicey.

This is great feedback, and something we’ve done some pie-in-sky thinking about. We’d love to make debugging complex build pipelines easier, with debugger-style attachment points, or being able to replicate builds in certain states, etc. While I don’t have anything to share at the moment, we’ll take this feedback into that thinking.

Cheers,
Sam

#3

Grabbing an agent token is definitely sneaky, but also totally not an option from our end due to our need to constrain access into the build farm to maintain guarantees about its predictability.

Thanks, I appreciate your response, and look forward to seeing what you come up with. Is there an issue tracker or similar that I might watch, or, how will you communicate updates back? If the answer is “this thread” that’s fine; I’m just trying to figure out what to expect.

#4

Thanks @petemounce, this forum is where we’re tracking feature requests like this generally, and we try to update them as things are released. So this is probably the right spot, and should receive an update when there’s something new. :bowing_man:t2:‍♂

1 Like