Soft fail status in github checks

Features Requests
1

Hello,

We are using the soft_fail attribute on a command step for one of our development team’s CI process. We have the ‘Create a status for each job’ option enabled in the GitHub integration settings on the pipeline.

We’ve noticed the check status on a PR in GitHub shows as successful when the soft fail step has failed. Our developers primarily view their pipeline results for PRs via the GitHub checks UI, so it is not obvious to them that they have caused this step to fail. While we don’t want this step’s failure to fail the build, we would like developers to easily know when they have made the step fail. Is there any way to allow the soft_failure attribute to pass the failed status to the GitHub checks?

Thank you,
Allie

2

Hey @allievt ,

Thanks for reaching out!

The use of the soft_fail attribute is to allow specified non-zero exit statuses not to fail the build and doesn’t have direct correlation to the commit statuses.

We don’t have an inbuilt feature for this, but I think using the Outcome of the soft_fail step and step level commit status customization can help you have a custom commit status if the job soft fails.

steps:
  - command: "exit -1"
    soft_fail: true
    key: 'one'

  - label: 'Check soft fail'
    command: |
      if [ $(buildkite-agent step get "outcome" --step "one") == "soft_failed" ]; then
         cat <<- YAML | buildkite-agent pipeline upload
         steps:
             notify:
                - github_commit_status:
                    context: "my-status-for-soft-fail"
      YAML
      fi

Hope this helps!
Cheers!

3

Thank you for the quick reply and the suggestion for an alternative approach. I copy/pasted your code sample and tried it exactly as is; the Github job level commit statuses all show as green however.

I would like to point out that the job commit status in Github does already show “Soft failed” in the message (see image below). What I would like is for the job’s commit status to show as a failed check instead of a successful check. I still want the overall build status to be successful, so I believe soft_fail is the correct attribute for my use case.

For example in the image below, the commit status for the job would show the red x, and instead of “All checks have passed - 20 successful checks” it would read " Some checks were not successful - 1 failing". This would more clearly reflect the same information that the pipeline shows in Buildkite. In Buildkite it is very easy to see the soft fail step has failed; but the same information is not as clearly conveyed in the job level commit statuses in the Github checks.

image
image1896×504 50.6 KB

4

Hey @allievt

Thanks for the follow up!

Using the soft fail feature ensures it doesn’t fail. Unfortunately you can’t fail the status check if the build doesn’t fail.

Cheers!

5

This is an example of a job level status check, not a build level status check - meaning the jobs that fail (not soft_fail) do show a failed status in Github.