Custom Variable to decide next step execution

General
1

Hello Team,

Looking to have Custom variable should have flag ( true / false) to decide next step if condition. Can someone please share the recommended way to achieve this?

For example

steps:

  • label: ‘Get Delta files and deploy artifacts’
    commands:
    • chmod +x .buildkite/scripts/delta.sh
    • .buildkite/scripts/delta.sh $$BUILDKITE_COMMIT $$BUILDKITE_BRANCH
    • pwsh .buildkite/scripts/module-yaml.ps1
    • pwsh .buildkite/scripts/module-sailpointutils.ps1
    • “export CUSTOM_VARIABLE=$$(pwsh .buildkite/scripts/deployartifact.ps1)”
    • "echo $$CUSTOM_VARIABLE
  • wait
  • label: “SP Config Export Objects from Sandbox”
    commands:
    • pwsh .buildkite/scripts/module-sailpointutils.ps1
    • buildkite-agent artifact upload ./exportedfile.json
      key: “Export”
      if: “build.branch == ‘sv046x_vault_check && CUSTOM_VARIABLE == true’”
2

Hey @sukurams :wave:

You won’t be able to evaluate $CUSTOM_VARIABLE in your next step, as exporting the variable in the command step will result in the value only being available in the same step, as each step runs in it’s own shell.

The recommendation would be to dynamically generate the SP Config Export Objects from Sandbox, evaluating CUSTOM_VARIABLE in a script that will upload the subsequent steps to the pipeline based on its value. We have some documentation around using conditionals in steps as well that gives some more examples!

3

@jeremy - Don’t we have a way to set custom environment variable from previous steps to refer upcoming steps ?.

4

Hey @sukurams!

No, it’s not possible. If you need to share data between steps you need to use meta-data (`buildkite-agent meta-data` v3 | Buildkite Documentation), but it’s not possible with env vars.

Best,

5

@paula

Thank you. I was trying to do following way to check and limit the steps execution. I see error when I execute following pipeline.

Can you please guide me about this error and check if we can use as I trying below.

022-11-17 11:36:43 WARN POST https://xxxx.xxxx.com/v3/jobs/018d-325797978918/pipelines: 422 One of the steps you provided was invalid: Error parsing if expression: Unexpected -: expected (, comparison operator, &&, ||, ?, or EOF (line 2, column 11) (Attempt 0/60 Retrying in -2562047h47m11.854775807s)
2022-11-17 11:36:43 ERROR Unrecoverable error, skipping retries
2022-11-17 11:36:43 FATAL Failed to upload and process pipeline: POST https://agent.buildkite.com/v3/jobs/01848671-8597-475f-8f99-325797978918/pipelines: 422 One of the steps you provided was invalid: Error parsing if expression: Unexpected -: expected (, comparison operator, &&, ||, ?, or EOF (line 2, column 11)
:rotating_light: Error: The command exited with status 1

steps:
  - label: 'Get Delta files and deploy artifacts'
    commands:
      - pwsh .buildkite/scripts/validatespconfig.ps1 | buildkite-agent meta-data set "SPCONFIG"
      - echo "$$(buildkite-agent meta-data get "SPCONFIG")"
  - wait
  - label: "SP Config Export Objects from Sandbox"
    commands:
      - pwsh .buildkite/scripts/module-sailpointutils.ps1
    key: "Export"
    if: |
        build.branch == 'sv046x_vault_check' && 
         buildkite-agent meta-data get "SPCONFIG" == "true"
  - wait
  - label: "SP Config Import Objects into Production"
    commands:
      - pwsh .buildkite/scripts/module-sailpointutils.ps1
    key: "Import"
    if: |
        build.branch == 'sv046x_vault_check' && 
         buildkite-agent meta-data get "SPCONFIG" == "true"
    depends_on: "Export"
6

Hi @sukurams :wave:

The error is coming from your if statement in the YAML you shared:

Unfortunately, It’s not possible to evaluate the results of the buildkite-agent meta-data command as part of a conditional in your pipeline steps. What you would need to do is run a command step that gets the meta-data and checks it, then uploads the conditional step as part of the buildkite-agent pipeline upload command, like this:

- command: |
      if [[ $$(buildkite-agent meta-data get "SPCONFIG") == "true" && "$BUILDKITE_BRANCH_ID" == "sv046x_vault_check" ]]; then
        cat <<- YAML | buildkite-agent pipeline upload
        steps:
          - label: "SP Config Export Objects from Sandbox"
            commands:
              - pwsh .buildkite/scripts/module-sailpointutils.ps1
      YAML
      fi

Cheers,
Jeremy

7

@jeremy - Thank you. Do you know how to display the meta value after created. validatespconfig.ps1 will return true / false. I wanted to display the value to confirm about value.

- pwsh .buildkite/scripts/validatespconfig.ps1 | buildkite-agent meta-data set "SPCONFIG"
- echo "$$(buildkite-agent meta-data get "SPCONFIG")"
8

Hi @sukurams :wave:

We would want to set values to meta-data to make it available for the other steps, so the best way to confirm and display the value is to confirm it in another step.

Something like this:

steps:
 - label: "Set SPCONFIG"
   commands: pwsh .buildkite/scripts/validatespconfig.ps1 | buildkite-agent meta-data set "SPCONFIG" 
 - label: "Get SPCONFIG" 
   commands: echo "$$(buildkite-agent meta-data get "SPCONFIG")"

Hope this helps!

Cheers,
Lizette