Initial implementation of the DAG

:wave: We’ve recently shipped an initial implementation of DAG support within the pipeline.yml file.

This post will serve as interim documentation until we make it generally available. We may change how some of the internals of the DAG works as we receive feedback, but we’ll try and not make any backwards incompatible or breaking changes. If we do, we’ll reach out to you individually.

The get started with the DAG, you’ll need to enable it in your pipeline.yml definition:

Once you’ve set dag: true, you’ll notice a few things:

  1. Nothing changes in the UI. DAGs don’t have a visualisation currently, and it’s not something we’ve got on the cards to work on just yet. We hope a future iteration of this feature will include something like this.
  2. wait steps no longer work. This is by design, but something we’re considering adding support back for.

In your pipeline.yml file, you’ll need to change how to setup dependencies between steps as wait steps are no longer available.

Here’s a basic example of a pipeline with dependencies:

steps:
  - command: "a.sh"
    id: "a"
  
  - command: "b.sh"
    id: "b"
    depends_on: "a"

You need to ensure each step as a unique id attribute. This is what you’ll use in your depends_on setup.

Even though we haven’t built a visualisation for the DAG yet, you can dependencies listed in the “Timeline” tab for a step:

image

Multiple dependencies can be defined using an array:

steps:
  - command: "a.sh"
    id: "a"
  
  - command: "b.sh"
    id: "b"

  - command: "c.sh"
    id: "c"
    depends_on:
     - "a"
     - "b"

continue_on_failure has been replaced with allow_dependency_failure

  - command: "c.sh"
    id: "c"
    allow_dependency_failure: true
    depends_on:
     - "a"
     - "b"

And it can be configured on a per-dependency basis using allow_failure

  - command: "c.sh"
    id: "c"
    depends_on:
     - step: "a"
       allow_failure: true
     - step: "b"
     - step: "c"

You don’t need to change your agent version to use the DAG. Everything is handled on our side.

We’ve already got some ideas of things we want to improve, but we’d love to hear what you think!

5 Likes

This is working great for us thanks! Has resolved our concurrency issues in our more complex pipelines.

within the pipeline.yml file

Can’t we use it in the JSON format? I am dynamically generating the pipeline definition by logging the JSON object to console in a nodeJS file.

node myPipeline.js

// myPipeline.js
let pipeline = [
 {label, command, agents},
  ...
]
console.log(pipeline)

@harryi3t Yep, you can also use it in the JSON format. I’m sorry for our confusing words here… we were trying to say you can’t use it via our old build steps web interface, vs the new way of defining steps using YAML/JSON.

Question: What if I have a step (A) that only runs on a certain branch (branches: “master”). While I’m on a feature branch, that step does not run. How do the steps (ids: B, C, D) that depend on the no-op step work? I’m hoping that the build would consider that step A “done” and run steps B, C, D? But I can see others use cases where if step A doesn’t run, then B, C, and D does not either.

If I need the former, how is that achieved?

Found a small bug.

My pipeline still had a block step in it, but all steps after the block had depends_on set, to the block was ignored (as expected).

So all steps completed, but the block step was never unblocked. This resulted the status in github to still be reported as: Pending — Build #1013 passed and blocked.

I then went and unblocked the pipeilne (despite all steps having completed), but the github status never updated.

If I manage to unblock the block step before all commands completed, then it correctly reported the success status to github.

Probably just a small bug that not many should hit, because I know that I shouldn’t be using block steps anymore if I am exclusively using depends_on.