Meta-data in conditional clause

Example:

  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
  - wait
  - trigger: "stable"
    label: ":rocket: Release a stable version"
    if: $(buildkite-agent meta-data get release-type) == "stable"

today i try to do a conditional based on a block but it is not possible

Hi @renanberto!

I saw this comment over in our agent repository, too:

I think this is a great idea so I’ve raised the issue internally and we’ll see what we can do. :man_dancing:

2 Likes

Cool! tomorrow I’ll spend a few hours to propose the change here: https://github.com/buildkite/conditional! I send it to you on the issue and here to see if I can help you with something!

it is also interesting to say that we did a workaround with hook. At the time of decision we look for metadata and transform it into environment, but it is definitely not the best approach, considering that metadata will not always be present in the step.

Thanks for prioritizing!

1 Like

Thanks for being eager, but that repository is for the language, not how it is hooked up to builds. :-)

1 Like

Hi folks! After a deep discussion we’ve decided not to support meta-data in conditionals just yet, sorry! We want to do it well, and have an answer for what happens when meta-data changes over the lifecycle of a build, and we haven’t quite figured that out yet.

For now, you can do this with the buildkite-agent and a bit of scripting — and it’s not super different, like:

# buildkite.yml
steps:
  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
            
  - command: |
      if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
        buildkite-agent pipeline upload buildkite-release-stable.yml
      fi

   # or even

  - command: buildkite-agent pipeline upload "buildkite-release-$$(buildkite-agent meta-data get release-type).yml"
# buildkite-release-stable.yml
steps:
  - trigger: stable
    label: ":rocket: Release a stable version"

We’ll make some noise when conditional support for meta-data changes.

1 Like

thanks for the feedback! We put a case to determine the steps within a command:

  - command: >
      case $(buildkite-agent meta-data get "release-type") in
        "stable") buildkite-agent pipeline upload .buildkite/stable.yml ;;
        "beta") buildkite-agent pipeline upload .buildkite/beta.yml ;;
        *) echo "Unknown release type" && exit 1
      esac
    label: ":rocket: - Release"
    key: "choose-release"
    agents:
      queue: "android"
    if: build.pull_request.id != null

I’ll get that thread dirty to ask you something…We have artifact dependency between jobs, in this case, before I call stable.yml I generate an artifact. How do I get this artifact inside another step that was uploaded by the agent?

E.G:

  - command:
      - "gradle build'"
      - "mv apk/production.apk /tmp/production.apk"
    artifact_paths: 
      - "/tmp/production.apk"
    label: "Build"
    key: "production-release"
    agents:
      queue: "android"
  • in this case, stable is selected

  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
    key: "request-release"

  - command: >
      case $(buildkite-agent meta-data get "release-type") in
        "stable") buildkite-agent pipeline upload .buildkite/stable.yml ;;
        "beta") buildkite-agent pipeline upload .buildkite/beta.yml ;;
        *) echo "Unknown release type" && exit 1
      esac
    label: ":rocket: - Release"
    key: "choose-release"
    agents:
      queue: "android"
  • After uploaded I have new step, and I need to the get artifact:

steps:
  - command:
      - buildkite-agent artifact download /tmp/production.apk /tmp --step production-release

Can you help me with this? Again I understand that this is not the place, but I’ve been breaking my head for some time.

1 Like

Hey @renanberto, can you send this through to us at support@buildkite.com and we can get someone to go deep on this issue for you.

3 Likes

I’d love for the if: clause mechanism to be made more powerful – at my company we use top-level git diffing to do a lot of downstream decisions w.r.t. which parts of a build hierarchy to run (complex mono-repo!)

And I’d love to be able to build a (more flexible) mechanism to use diff output in if: clauses, which is complicated by the limitations of which data sources can be fed into this mechanism.

If there were some more general mechanism to feed data (files) reliably and without formatting changes into inspectable variables for downstream pipelines and steps that’d open up a large swathe of improvements and simplifications w.r.t. ease-of-use for our developers.

1 Like

Hey, I’m actually trying to do something similar, just wondering what’s the difference between using buildkite-agent to upload a pipeline that triggers the target pipeline, and directly using buildkite-agent to upload the target pipeline?

And if I want to use the proposed method, how do I pass arguments when doing the upload?
For example

# pipeline.yml
- command: 
  if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
    buildkite-agent pipeline upload trigger.yml
  fi
# trigger.yml
steps:
  - trigger: target
    build:
      env:
        something: ${something}

I want to pass something from pipeline.yml to trigger.yml so that it can pass it into target.yml, how can I do it?

1 Like

Hey @jeffhappily welcome! :wave:

I don’t think there is much difference in the target and trigger pipeline, it’s just an example in this case where the release pipeline most likely already exists and we don’t want to do a pipeline upload for it and a trigger makes more sense.

If I was going to create trigger.yml like that, I would probably do it as a dynamic pipeline which would make passing those values a little easier.

# pipeline.yml
- command: 
  if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
    .buildkite/trigger.sh | buildkite-agent pipeline upload
  fi
1 Like

Thanks for the reply! I have another similar question, since metadata doesn’t work for the if clause in trigger, will it work in the build.env or build.meta_data? Like is it possible for me to pass in a metadata in the current pipeline to another pipeline with the trigger?

1 Like

And how does pipeline upload actually work or what’s the expected behavior of pipeline upload?

If I have a pipeline with 4 steps, and if I do a pipeline upload at step 2, what will happen? Will it overwrite step 3 & 4 and run the new steps uploaded, or will it put the uploaded pipeline in between step 2 & 3, or will it run the uploaded pipeline somewhere else?

I tried running

buildkite-agent pipeline upload mypipeline.yml

and the step ran successfully but nothing really happens. I don’t see anything from the mypipeline.yml running

1 Like

Hey @jeffhappily The upload command will insert the steps from the script into the build immediately after the upload step.

If you send us through a message to support@buildkite.com with a link to your build and the issues you are having we can help you with your pipeline in a bit more detail.

On the README here GitHub - buildkite/conditional: A small language for conditionals there is a syntax example:
// complex expressions
((build.tag =~ ^v) || (meta-data(“foo”) == “bar”))

I don’t see meta-data() anywhere else in that repo so I guess the example was wishful thinking?

Hi @Dave, welcome to the forum! – yeah, I think that might be a leftover fragment of a design decision we made that’s largely to do with conditionals being evaluated at the time of pipeline upload vs. how meta-data might change over the course of a build. I’ll cross-link a post from @sj26 about this on GitHub.

Hi folks! After a deep discussion we’ve decided not to support meta-data in conditionals just yet, sorry! We want to do it well, and have an answer for what happens when meta-data changes over the lifecycle of a build, and we haven’t quite figured that out yet.

There’s a description of a workaround using scripting in the previously linked issue:

Thanks for the reply @Chloe , makes sense and I appreciate the workaround.

The weird use case I have is an initial step that checks a database and a ready/not ready value to determine whether or not to run the rest of the pipeline.

I thought perhaps something like 3 pipeline files
pipeline.yml contains the initial step and a publish pipeline based on that
noop.yml just echoes that there is nothing to do
dostuff.yml has the full rest of the pipeline

Is that sensible? what would pseudocode for that look like, or is there something cleaner and simpler?

tbh I think this is probably nicer than using metadata in conditionals - I can keep the dostuff.yml pipeline clean

Hey @Dave :wave:

Hmm - kinda sounds like you could achieve that with a dynamic pipeline upload on the one pipeline?

If you need any further help with this you could also send us a message at support@buildkite.com with your pipeline(s) and we can look at them a little deeper for you?