Meta-data in conditional clause

renanberto · August 18, 2020, 5:30pm

Example:

  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
  - wait
  - trigger: "stable"
    label: ":rocket: Release a stable version"
    if: $(buildkite-agent meta-data get release-type) == "stable"

today i try to do a conditional based on a block but it is not possible

sj26 · August 19, 2020, 2:01am

Hi @renanberto!

I saw this comment over in our agent repository, too:

I think this is a great idea so I’ve raised the issue internally and we’ll see what we can do.

renanberto · August 19, 2020, 2:50pm

Cool! tomorrow I’ll spend a few hours to propose the change here: https://github.com/buildkite/conditional! I send it to you on the issue and here to see if I can help you with something!

it is also interesting to say that we did a workaround with hook. At the time of decision we look for metadata and transform it into environment, but it is definitely not the best approach, considering that metadata will not always be present in the step.

Thanks for prioritizing!

sj26 · August 19, 2020, 11:40pm

Thanks for being eager, but that repository is for the language, not how it is hooked up to builds. :-)

sj26 · August 21, 2020, 7:01am

Hi folks! After a deep discussion we’ve decided not to support meta-data in conditionals just yet, sorry! We want to do it well, and have an answer for what happens when meta-data changes over the lifecycle of a build, and we haven’t quite figured that out yet.

For now, you can do this with the buildkite-agent and a bit of scripting — and it’s not super different, like:

# buildkite.yml
steps:
  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
            
  - command: |
      if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
        buildkite-agent pipeline upload buildkite-release-stable.yml
      fi

   # or even

  - command: buildkite-agent pipeline upload "buildkite-release-$$(buildkite-agent meta-data get release-type).yml"

# buildkite-release-stable.yml
steps:
  - trigger: stable
    label: ":rocket: Release a stable version"

We’ll make some noise when conditional support for meta-data changes.

renanberto · August 21, 2020, 2:18pm

thanks for the feedback! We put a case to determine the steps within a command:

  - command: >
      case $(buildkite-agent meta-data get "release-type") in
        "stable") buildkite-agent pipeline upload .buildkite/stable.yml ;;
        "beta") buildkite-agent pipeline upload .buildkite/beta.yml ;;
        *) echo "Unknown release type" && exit 1
      esac
    label: ":rocket: - Release"
    key: "choose-release"
    agents:
      queue: "android"
    if: build.pull_request.id != null

I’ll get that thread dirty to ask you something…We have artifact dependency between jobs, in this case, before I call stable.yml I generate an artifact. How do I get this artifact inside another step that was uploaded by the agent?

E.G:

  - command:
      - "gradle build'"
      - "mv apk/production.apk /tmp/production.apk"
    artifact_paths: 
      - "/tmp/production.apk"
    label: "Build"
    key: "production-release"
    agents:
      queue: "android"

in this case, stable is selected

  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
    key: "request-release"

  - command: >
      case $(buildkite-agent meta-data get "release-type") in
        "stable") buildkite-agent pipeline upload .buildkite/stable.yml ;;
        "beta") buildkite-agent pipeline upload .buildkite/beta.yml ;;
        *) echo "Unknown release type" && exit 1
      esac
    label: ":rocket: - Release"
    key: "choose-release"
    agents:
      queue: "android"

After uploaded I have new step, and I need to the get artifact:

steps:
  - command:
      - buildkite-agent artifact download /tmp/production.apk /tmp --step production-release

Can you help me with this? Again I understand that this is not the place, but I’ve been breaking my head for some time.

Jason · August 23, 2020, 10:24pm

Hey @renanberto, can you send this through to us at support@buildkite.com and we can get someone to go deep on this issue for you.

jerry · September 24, 2020, 5:44am

I’d love for the if: clause mechanism to be made more powerful – at my company we use top-level git diffing to do a lot of downstream decisions w.r.t. which parts of a build hierarchy to run (complex mono-repo!)

And I’d love to be able to build a (more flexible) mechanism to use diff output in if: clauses, which is complicated by the limitations of which data sources can be fed into this mechanism.

If there were some more general mechanism to feed data (files) reliably and without formatting changes into inspectable variables for downstream pipelines and steps that’d open up a large swathe of improvements and simplifications w.r.t. ease-of-use for our developers.

jeffhappily · December 17, 2020, 10:06am

Hey, I’m actually trying to do something similar, just wondering what’s the difference between using buildkite-agent to upload a pipeline that triggers the target pipeline, and directly using buildkite-agent to upload the target pipeline?

And if I want to use the proposed method, how do I pass arguments when doing the upload?
For example

# pipeline.yml
- command: 
  if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
    buildkite-agent pipeline upload trigger.yml
  fi

# trigger.yml
steps:
  - trigger: target
    build:
      env:
        something: ${something}

I want to pass something from pipeline.yml to trigger.yml so that it can pass it into target.yml, how can I do it?

Jason · December 18, 2020, 3:16am

Hey @jeffhappily welcome!

I don’t think there is much difference in the target and trigger pipeline, it’s just an example in this case where the release pipeline most likely already exists and we don’t want to do a pipeline upload for it and a trigger makes more sense.

If I was going to create trigger.yml like that, I would probably do it as a dynamic pipeline which would make passing those values a little easier.

# pipeline.yml
- command: 
  if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
    .buildkite/trigger.sh | buildkite-agent pipeline upload
  fi

jeffhappily · December 21, 2020, 8:00am

Thanks for the reply! I have another similar question, since metadata doesn’t work for the if clause in trigger, will it work in the build.env or build.meta_data? Like is it possible for me to pass in a metadata in the current pipeline to another pipeline with the trigger?

jeffhappily · December 21, 2020, 9:30am

And how does pipeline upload actually work or what’s the expected behavior of pipeline upload?

If I have a pipeline with 4 steps, and if I do a pipeline upload at step 2, what will happen? Will it overwrite step 3 & 4 and run the new steps uploaded, or will it put the uploaded pipeline in between step 2 & 3, or will it run the uploaded pipeline somewhere else?

I tried running

buildkite-agent pipeline upload mypipeline.yml

and the step ran successfully but nothing really happens. I don’t see anything from the mypipeline.yml running

Jason · December 21, 2020, 8:57pm

Hey @jeffhappily The upload command will insert the steps from the script into the build immediately after the upload step.

If you send us through a message to support@buildkite.com with a link to your build and the issues you are having we can help you with your pipeline in a bit more detail.

Dave · May 11, 2021, 2:12am

On the README here GitHub - buildkite/conditional: A small language for conditionals there is a syntax example:
// complex expressions
((build.tag =~ ^v) || (meta-data(“foo”) == “bar”))

I don’t see meta-data() anywhere else in that repo so I guess the example was wishful thinking?

anon18884907 · May 11, 2021, 4:20am

Hi @Dave, welcome to the forum! – yeah, I think that might be a leftover fragment of a design decision we made that’s largely to do with conditionals being evaluated at the time of pipeline upload vs. how meta-data might change over the course of a build. I’ll cross-link a post from @sj26 about this on GitHub.

Hi folks! After a deep discussion we’ve decided not to support meta-data in conditionals just yet, sorry! We want to do it well, and have an answer for what happens when meta-data changes over the lifecycle of a build, and we haven’t quite figured that out yet.

There’s a description of a workaround using scripting in the previously linked issue:

github.com/buildkite/agent

Use metadata in conditionals

opened 11:40PM - 28 Jan 20 UTC

closed 01:41AM - 25 Aug 20 UTC

evandam

Hi there, I'm looking to add a step in my pipeline to block and ask if tests …should be run. Here's the general pattern: ```yaml steps: - block: "Prompt for tests" fields: - select: "Run tests?" key: "run-tests" options: - label: "Sure" value: "yes" - label: "No thanks" value: "no" if: build.source == "ui" # Ideal approach using an "if" (feature request?) - label: "Run tests" key: test command: echo "test me!" if: build.metadata("run-tests", "false") == "true" plugins: - docker-compose#v3.1.0: run: app # Workaround, but doesn't work inside docker image without buildkite-agent - label: "Run tests take 2" key: test command: > if [ "$(buildkite-agent meta-data get run-tests --default false)" = "true" ]; then echo "test me!" else echo "tests skipped" fi plugins: - docker-compose#v3.1.0: run: app ``` I suppose another approach could be to have a hook that tries to read the metadata and export that as an environment variable that can be used, but it feels like a lot of work to do something that should be a native solution. Any thoughts on this?

Dave · May 11, 2021, 4:48am

Thanks for the reply @anon18884907 , makes sense and I appreciate the workaround.

The weird use case I have is an initial step that checks a database and a ready/not ready value to determine whether or not to run the rest of the pipeline.

I thought perhaps something like 3 pipeline files
pipeline.yml contains the initial step and a publish pipeline based on that
noop.yml just echoes that there is nothing to do
dostuff.yml has the full rest of the pipeline

Is that sensible? what would pseudocode for that look like, or is there something cleaner and simpler?

tbh I think this is probably nicer than using metadata in conditionals - I can keep the dostuff.yml pipeline clean

Jason · May 12, 2021, 5:03am

Hey @Dave

Hmm - kinda sounds like you could achieve that with a dynamic pipeline upload on the one pipeline?

If you need any further help with this you could also send us a message at support@buildkite.com with your pipeline(s) and we can look at them a little deeper for you?

Fisher · December 13, 2021, 1:37pm

This conditional feature will be helpful for us, as developers want to manually skip UAT step when committing specific branches, and expect a block step with the selection of "deply in ***” （YES or NO), then the next step (like trigger another pipeline in DEV or UAT) can be skipped if the selection is “NO”.

a_Sam_a · April 6, 2022, 10:11pm

I was able to use meta-data in conditionals by loading them into environment variables, like this:

steps:
  - block: ":rocket: PROD Release!"
    prompt: You are about to create a new release. This cannot be undone.
    fields:
      - select: "Release type"
        key: "release_type"
        multiple: false
        options:
          - label: "Patch"
            value: "Patch"
          - label: "Minor"
            value: "Minor"
          - label: "Major"
            value: "Major"
          - label: "None (What are you doing here then?)"
            value: "None"
  - wait: ~

  - label: "Load release type into env variable"
    command: |
      export release_type=$(buildkite-agent meta-data get release_type)

  - wait: ~

  - label: "Triggering primarily pipeline to build and release"
    if: "build.env('release_type') != 'None'"
    command: |
      echo Building release of "$release_type"

  - label: "Cancelled release"
    if: "build.env('release_type') == 'None'"
    command: echo Cancelled release

paula · April 6, 2022, 10:23pm

Thanks for sharing @a_Sam_a! and welcome to the community!

Topic		Replies	Views
How to make buildkite-agent meta-data get "key" optional General	6	1366	June 10, 2022
Query meta-data by build_id Features Requests	2	310	November 5, 2022
Forward meta-data to triggered build Features Requests	0	960	October 31, 2019
Build step based on input condition Pipelines	1	1033	August 29, 2023
Meta as environment variables Features Requests	10	2271	September 6, 2022

Meta-data in conditional clause

in this case, stable is selected

After uploaded I have new step, and I need to the get artifact:

Related topics