Update on DAG implementation: wait steps are back!

Over the last couple of months since it was first announced we’ve been making improvements to the DAG behaviour, and to ease the migration path.

You still need to enable it with dag: true in your YAML/JSON step definition, either in the UI or during a step upload.

We also still don’t have any UI changes – though this now is being worked on. We think it might look something like this:

The big change from the previous post is that wait steps now work as they do in existing non-DAG builds, creating automatic dependencies based on the order steps are defined in. This should make it much easier to experiment with, and transition to, a partially DAG-based pipeline: you can continue using your current pipeline definition as-is, and then insert additional dependencies via depends_on where needed.

As an example, given an existing pipeline that looks like this:

steps:
  - command: "bin/build-docker"
  - wait
  - command: "bundle exec rubocop"
  - command: "bundle exec rspec"
  - wait
  - command: "bin/coverage"

You can now replace only the later wait with a narrower depends_on, while leaving the rest of the pipeline unchanged:

dag: true
steps:
  - command: "bin/build-docker"
  - wait
  - command: "bundle exec rubocop"
  - key: rspec
    command: "bundle exec rspec"
  - command: "bin/coverage"
    depends_on: "rspec"

With that change, the coverage step will no longer wait for rubocop to finish, and can instead start as soon as rspec is completed.

2 Likes

Hi,

In a support case I was recommended to look into DAG to be able to add block steps which after unlock could allow subsequent steps to run immediately without waiting for previous steps to complete.

In my testing however, block steps appear to work just the way they did without DAG, ie. similar to a wait step.

Is it actually possible to achieve block steps that don’t “wait” after unlocking?

Our use case is using block steps for deployment, where the block step provides a UI to select a server environment and subsequently triggers a deploy pipeline and we need the developers to be able to deploy an “in progress” or even failed build in certain situations.

Also on another note your example here uses “key” to define the id of a step while the original post (Initial implementation of the DAG) used “id”.

Which one should we use?

Hey @dbackeus,

We’ve actually just released input steps which will do what you’re looking for! https://buildkite.com/docs/pipelines/input-step

And key is the one to go for - we swapped from id during the beta :blush:

Let me know if you’ve got any other questions! There’s also some new docs out about dependencies, you can find them here: https://buildkite.com/docs/pipelines/dependencies

I’m again not sure this solves our issue of wanting to have a block step providing a UI which after being unlocked does not wait for previous steps to execute the following steps? Please provide an example of how to achieve that if possible.