Hi there!
We have a lot of pipelines that are dynamically generated based on dependency (and “what changed”) information from elsewhere. Here’s a high-level one of such pipelines, which mostly just triggers other pipelines and waits for their results:
The actual dependencies here look something like…
…but we have to flatten them into a straight line, because Buildkite doesn’t (yet? ) support declaring dependencies between jobs in any other way. This requires “shaking out” the dependency graph into groups of jobs that can be run together — something like tsort. Easy. But the result will always leave something to be desired, and sometimes that something is really significant.
Consider in the above example if “build web client” takes 15 minutes longer than any other job in its group, and “test backend” takes 15 minutes longer than any other job in its group. From that alone, the overall build will take 15 minutes longer than necessary. We have a lot of such scenarios in different parts of our pipelines, and they’re really starting to add up to a lot of unnecessary build latency.
Now, I can imagine some ways to work around this, e.g., a pipeline that manages its own graph of jobs internally and keeps adding a new “poll until ready to spawn new job” job to the end each time it realises another dependency has been satisfied. But then I’d effectively be reimplementing what I think of as being some of the core functionality of Buildkite — i.e. job queueing and scheduling — at which point I’d end up asking myself some deeper questions that I’d rather not have to think about.
So what would work really well for me?
I can understand that you’d be hesitant to add any kind of explicit DAG support to Buildkite unless it was something you believed you could support with a great user experience through the UI. I’d like to suggest something that I believe will really help power users in the short-term, and doesn’t preclude or interfere with a grander/friendlier design further down the line.
My problem would be solved if, in addition to the existing wait steps, I could declare “tags” and “wait for tags” on jobs, like so:
{
label: "Build web-client",
command: "...",
tags: ["build-web-client"],
},
{
label: "Test web-client",
waitForTags: ["build-web-client"]
}
A job with “wait for tags” would not start until for each of those tags there is either (1) no job that provides that tag, or (2) all jobs with that tag have finished.
As far as the UI is concerned, I’d personally be happy enough just having jobs float around mysteriously if they have any unsatisfied dependencies. But I guess that could be a bit confusing, so how about marking jobs with unsatisfied dependencies with some tiny tag, and then allow hovering over them to reveal what they’re waiting on:
…
(Maybe with a tool-tip as well, explaining exactly what’s going on?)
This might be a bit of extra clutter in the UI, but it would only ever show up for power-users who have explicitly opted in to this complexity by declaring dependencies in their pipelines. For extra cleverness, you could make it automatically infer “wait steps” where jobs can be precisely linearised, just to tidy up the display of it where possible.
What do you reckon? I’d be overjoyed to have access to a feature like this, even without any UI attached to it; dependencies/DAGs are often the natural way (and always the most flexible way) to express dependencies between tasks, and we’re going to keep suffering a lot of wasted time in our pipelines without some kind of support for them.
If you have anything similar to this in the works, or some other idea that solves the same problem, any chance of getting alpha/beta access? Or if it’s not yet that far along, would you be willing to share some ideas about what the design is likely to look like, so that we can have a think about how it might fit into our world?
Thanks!
P.S.
Sorry, new users can only put one image in a post.
Sorry, new users can only put 2 links in a post.
Are you able to make me not a “new user” so that I can re-upload those other images to display inline?
UPDATE: Hurrah! I’ve replaced the imgur links with inline images. Muchas gracias!