Build status override by human

Pipelines
1

Is there a good recipe for visual regression testing workflow on Buildkite?

If one of the setup steps fails, it is immediate failure.

If the “Unit tests” step fails, it is immediate failure.

If the “Screenshot tests” step fails, it should create a human confirmation dialog and wait for 4 hours and then mark the build as failed.

If a human reviews the screenshot testing results before the timeout, the pipeline should add an extra step (e.g. to accept the new screenshots baseline), and then proceed to the rest of the steps defined in the pipeline.

Obviously, if a human accepts the screenshots, the build has to be marked as successful (unless some post-test steps fail). If the visual differences are rejected, the build has to finish with a failure. Also, it’s ideal to record the username of the person who took the action.

Well, I don’t believe that this use case is too rare. Perhaps, I didn’t look thoroughly enough, but I’d appreciate a qualified help, if possible.

2

Hello Noomorph,

Welcome to the community! Let me help you with your queries. The default behaviour of our pipeline steps is to cancel the whole build if a step fails. However, this can be overridden by explicitly mentioning it to ignore failures. If you anticipate some corner cases that might cause failures, then I would suggest using soft_fail attribute of our command step . On the contrary , if you want the steps to continue on any kind of failures, then you can use the continue_on_failure attribute and set the value to true .

In terms of adding an extra step based on outcome of the screenshot test, you can leverage the use of our dynamic pipeline feature combined with our Block step feature this will allow you to upload a pipeline with anticipated logic if your step fails , and also let you know who unblocked the step.

You can use our depends_on feature to create dependencies between steps , which allows you to proceed to the next step only if the dependent step completes.

I’ve Posted a sample snippet that would probably give you an idea of the how to incorporate these features in your code logic.

- label: "Screenshot tests"
  command: echo "hi"
  key: "Screenshot-tests"
  soft_fail: true

- label: "outcome of screenshot tests"
  key: "keytwo"
  command: |
    if [ $$(buildkite-agent step get "outcome" --step "Screenshot-tests") != "passed" ]; then
        cat <<- YAML | buildkite-agent pipeline upload
        steps:
          - block: ":rocket:  review the failure and unblock this step please"
    YAML
    fi
  timeout_in_minutes: 240
  depends_on: "Screenshot-tests"


- label: ":rocket: Build"
  command: "echo 'job id is hi'"
  depends_on: "keytwo"

Hope this information helps. Cheers :slightly_smiling_face:

3

Thank you, @athreya. The references are immensely useful, and I plan to try this suggestion within a week and come back with what I managed to make, so it can help the community as well.