Initial implementation of the DAG

:wave: We’ve recently shipped an initial implementation of DAG support within the pipeline.yml file.

This post will serve as interim documentation until we make it generally available. We may change how some of the internals of the DAG works as we receive feedback, but we’ll try and not make any backwards incompatible or breaking changes. If we do, we’ll reach out to you individually.

The get started with the DAG, you’ll need to enable it in your pipeline.yml definition:

Once you’ve set dag: true, you’ll notice a few things:

  1. Nothing changes in the UI. DAGs don’t have a visualisation currently, and it’s not something we’ve got on the cards to work on just yet. We hope a future iteration of this feature will include something like this.
  2. wait steps no longer work. This is by design, but something we’re considering adding support back for.

In your pipeline.yml file, you’ll need to change how to setup dependencies between steps as wait steps are no longer available.

Here’s a basic example of a pipeline with dependencies:

steps:
  - command: "a.sh"
    id: "a"
  
  - command: "b.sh"
    id: "b"
    depends_on: "a"

You need to ensure each step as a unique id attribute. This is what you’ll use in your depends_on setup.

Even though we haven’t built a visualisation for the DAG yet, you can dependencies listed in the “Timeline” tab for a step:

image

Multiple dependencies can be defined using an array:

steps:
  - command: "a.sh"
    id: "a"
  
  - command: "b.sh"
    id: "b"

  - command: "c.sh"
    id: "c"
    depends_on:
     - "a"
     - "b"

continue_on_failure has been replaced with allow_dependency_failure

  - command: "c.sh"
    id: "c"
    allow_dependency_failure: true
    depends_on:
     - "a"
     - "b"

And it can be configured on a per-dependency basis using allow_failure

  - command: "c.sh"
    id: "c"
    depends_on:
     - step: "a"
       allow_failure: true
     - step: "b"
     - step: "c"

We’ve already got some ideas of things we want to improve, but we’d love to hear what you think!