How can I depends_on the possible missing steps?

Pipelines
1

Hi folks, I am now using monorepo-diff plugins to trigger both frontend and backend changes at the same time. But I have another step which depends on some steps in frontend and backend, which may lead to build failure because sometimes only frontend or backend are triggered, and the dependent step does not exist at all. It is like the following example:

steps:
  - label: "Triggering pipelines"
    plugins:
      - chronotc/monorepo-diff#v2.3.0:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "frontend/"
              config:
                command: "buildkite-agent pipeline upload frontend.yml"
            - path: "backend/"
              config:
                command: "buildkite-agent pipeline upload backend.yml"
 - label: "some step"
    depends_on:
    - some_key_in_frontend_yml
    - some_key_in_backend_yml
   command: "echo hello"

I am not using wait: true here, because I don’t want to wait for all the steps in backend and frontend to finish.
What I want is when some steps to depend on are missing, the dependent step does not fail.

I have read the docs and I know that for such a case, the result is failure

Any suggestion is appreciated~ Maybe I am in the wrong direction for this task.

Thanks

1 Like
2

I struggled with this myself and the only way I could get it to work was way more hacky than I would have liked & expected.

I resorted to manually modifying the pipeline.yml file before upload

# .buildkite/hooks/post-checkout
export JS_FILES_CHANGED_COUNT=$(git --no-pager diff master... --name-only --diff-filter=d -- ./\*\*\*.js ./\*\*\*.jsx ./\*\*\*.ts ./\*\*\*.tsx ./\*\*\*.json ./\*\*\*.graphql ./\*\*\*.snap | sort | uniq | wc -l | awk '{print $1}')

sed_option="-i"
if [[ "$OSTYPE" == "darwin"* ]]; then
  sed_option="-i ''"
fi

if [ "$JS_FILES_CHANGED_COUNT" == "0" ]; then
  sed $sed_option "s/\(- step: .* # if JS_FILES_CHANGED_COUNT\)/# \1/g" ".buildkite/pipeline.yml"
fi

Then in pipeline.yml, annotate the depends_on with the “magic comment”

    depends_on:
      - step: jest # if JS_FILES_CHANGED_COUNT

Really annoying to have to do all this, but I hope it helps.

Hopefully in the future buildkite will add an option for this instead.

3

FTR my jest step looks like this:

  - label: "Jest"
    key: jest
    if: |
      (
        (build.message !~ /skip.?(jest)/ && "${JS_FILES_CHANGED_COUNT}" != "0") ||
        build.branch =~ /^(master|deploy\/production|deploy\/staging|deploy\/sandbox)\$/
      )
4

Thank you for your hacky solution.

Inspired by you, I think maybe another way is to fork or contribute monorepo-diff, and add a when-not-match command to execute some command when there is no diff. So we can add a dummy step with id we want to satisfy the depends_on. Still ugly…

5

Yeah this whole issue seems very solvable natively, because buildkite already has soft-fail and depends_on configuration, such as allow_failure: true. To me it seems like it should be easy to add allow_skipped: true or something similar.

1 Like