Steps starting at the same time shouldn't imply dependency groups

[gerry]

I have a pipeline

- command: building large docker image
- command: running small steps that don't need that large image
- command: build and run rubocop step

- wait

- command: lots of test steps running in parallel that depend on the large image

The rubocop step sometimes takes longer than the build step (which I might be able to address, but not right now), however all the testing steps after the wait are only dependant on the build of the docker image.

I could separate the rubocop step into it’s own pipeline, but I could see this being the case for a lot of our linters/checkers, which would mean devs are possibly having to open up a lot of different tabs to find errors in their builds. I suspect even one more will be annoying. I also like the idea that a fail on the rubocop step could then cause an immediate failure on everything after that point. This way we are encouraging doing basic checks locally before pushing it up to CI.

Has anything like this been discussed before? I couldn’t find anything when searching. A command option such as this would likely be ideal… blocking_type: [wait|no_wait|no_wait_but_fail] where the wait step would then only wait for any steps with a blocking_type of wait (the default).

Maybe I’m thinking about this all wrong?

edit: Oh actually it does look like this has been raised before: Feature Request: More granular control over pipelines · Issue #305 · buildkite/feedback · GitHub

[anon46008679]

Hi @gerry! This sounds a little similar to the following?

Declare dependencies between jobs

Hi there! We have a lot of pipelines that are dynamically generated based on dependency (and “what changed”) information from elsewhere. Here’s a high-level one of such pipelines, which mostly just triggers other pipelines and waits for their results: [42%20pm] The actual dependencies here look something like… [55%20pm] …but we have to flatten them into a straight line, because Buildkite doesn’t (yet? ) support declaring dependencies between jobs in any othe…

[gerry]

Thanks @anon46008679 yeah something where I could specify the dependencies of each job would indeed be ideal, my idea was more so that you don’t have to make too many changes to the current way things work. You would be modifying the wait step to have dependencies instead of each job declaring them. But individual job dependencies would certainly be more flexible.