Meta-data in conditional clause

Features
#1

Example:

  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
  - wait
  - trigger: "stable"
    label: ":rocket: Release a stable version"
    if: $(buildkite-agent meta-data get release-type) == "stable"

today i try to do a conditional based on a block but it is not possible

#2

Hi @renanberto!

I saw this comment over in our agent repository, too:

I think this is a great idea so I’ve raised the issue internally and we’ll see what we can do. :man_dancing:

2 Likes
#4

Cool! tomorrow I’ll spend a few hours to propose the change here: https://github.com/buildkite/conditional! I send it to you on the issue and here to see if I can help you with something!

it is also interesting to say that we did a workaround with hook. At the time of decision we look for metadata and transform it into environment, but it is definitely not the best approach, considering that metadata will not always be present in the step.

Thanks for prioritizing!

1 Like
#5

Thanks for being eager, but that repository is for the language, not how it is hooked up to builds. :-)

1 Like
#6

Hi folks! After a deep discussion we’ve decided not to support meta-data in conditionals just yet, sorry! We want to do it well, and have an answer for what happens when meta-data changes over the lifecycle of a build, and we haven’t quite figured that out yet.

For now, you can do this with the buildkite-agent and a bit of scripting — and it’s not super different, like:

# buildkite.yml
steps:
  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
            
  - command: |
      if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
        buildkite-agent pipeline upload buildkite-release-stable.yml
      fi

   # or even

  - command: buildkite-agent pipeline upload "buildkite-release-$$(buildkite-agent meta-data get release-type).yml"

# buildkite-release-stable.yml
steps:
  - trigger: stable
    label: ":rocket: Release a stable version"

We’ll make some noise when conditional support for meta-data changes.

1 Like
#7

thanks for the feedback! We put a case to determine the steps within a command:

  - command: >
      case $(buildkite-agent meta-data get "release-type") in
        "stable") buildkite-agent pipeline upload .buildkite/stable.yml ;;
        "beta") buildkite-agent pipeline upload .buildkite/beta.yml ;;
        *) echo "Unknown release type" && exit 1
      esac
    label: ":rocket: - Release"
    key: "choose-release"
    agents:
      queue: "android"
    if: build.pull_request.id != null

I’ll get that thread dirty to ask you something…We have artifact dependency between jobs, in this case, before I call stable.yml I generate an artifact. How do I get this artifact inside another step that was uploaded by the agent?

E.G:

  - command:
      - "gradle build'"
      - "mv apk/production.apk /tmp/production.apk"
    artifact_paths: 
      - "/tmp/production.apk"
    label: "Build"
    key: "production-release"
    agents:
      queue: "android"

  • in this case, stable is selected

  - block: "Request release"
    prompt: "Fill out the details for release"
    fields:
      - text: "Version"
        key: "version"
        hint: "Include the version according to the standard: 2.x.x.x"
        required: true
      - select: "Type"
        key: "release-type"
        default: "stable"
        options:
          - label: "Stable"
            value: "stable"
          - label: "Beta"
            value: "beta"
          - label: "Debug"
            value: "debug"
    key: "request-release"

  - command: >
      case $(buildkite-agent meta-data get "release-type") in
        "stable") buildkite-agent pipeline upload .buildkite/stable.yml ;;
        "beta") buildkite-agent pipeline upload .buildkite/beta.yml ;;
        *) echo "Unknown release type" && exit 1
      esac
    label: ":rocket: - Release"
    key: "choose-release"
    agents:
      queue: "android"

  • After uploaded I have new step, and I need to the get artifact:

steps:
  - command:
      - buildkite-agent artifact download /tmp/production.apk /tmp --step production-release

Can you help me with this? Again I understand that this is not the place, but I’ve been breaking my head for some time.

1 Like
#8

Hey @renanberto, can you send this through to us at support@buildkite.com and we can get someone to go deep on this issue for you.

3 Likes
#9

I’d love for the if: clause mechanism to be made more powerful – at my company we use top-level git diffing to do a lot of downstream decisions w.r.t. which parts of a build hierarchy to run (complex mono-repo!)

And I’d love to be able to build a (more flexible) mechanism to use diff output in if: clauses, which is complicated by the limitations of which data sources can be fed into this mechanism.

If there were some more general mechanism to feed data (files) reliably and without formatting changes into inspectable variables for downstream pipelines and steps that’d open up a large swathe of improvements and simplifications w.r.t. ease-of-use for our developers.

1 Like
#10

Hey, I’m actually trying to do something similar, just wondering what’s the difference between using buildkite-agent to upload a pipeline that triggers the target pipeline, and directly using buildkite-agent to upload the target pipeline?

And if I want to use the proposed method, how do I pass arguments when doing the upload?
For example

# pipeline.yml
- command: 
  if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
    buildkite-agent pipeline upload trigger.yml
  fi

# trigger.yml
steps:
  - trigger: target
    build:
      env:
        something: ${something}

I want to pass something from pipeline.yml to trigger.yml so that it can pass it into target.yml, how can I do it?

1 Like
#11

Hey @jeffhappily welcome! :wave:

I don’t think there is much difference in the target and trigger pipeline, it’s just an example in this case where the release pipeline most likely already exists and we don’t want to do a pipeline upload for it and a trigger makes more sense.

If I was going to create trigger.yml like that, I would probably do it as a dynamic pipeline which would make passing those values a little easier.

# pipeline.yml
- command: 
  if [[ "$$(buildkite-agent meta-data get release-type)" == "stable" ]]; then
    .buildkite/trigger.sh | buildkite-agent pipeline upload
  fi
1 Like
#12

Thanks for the reply! I have another similar question, since metadata doesn’t work for the if clause in trigger, will it work in the build.env or build.meta_data? Like is it possible for me to pass in a metadata in the current pipeline to another pipeline with the trigger?

1 Like
#13

And how does pipeline upload actually work or what’s the expected behavior of pipeline upload?

If I have a pipeline with 4 steps, and if I do a pipeline upload at step 2, what will happen? Will it overwrite step 3 & 4 and run the new steps uploaded, or will it put the uploaded pipeline in between step 2 & 3, or will it run the uploaded pipeline somewhere else?

I tried running

buildkite-agent pipeline upload mypipeline.yml

and the step ran successfully but nothing really happens. I don’t see anything from the mypipeline.yml running

1 Like
#14

Hey @jeffhappily The upload command will insert the steps from the script into the build immediately after the upload step.

If you send us through a message to support@buildkite.com with a link to your build and the issues you are having we can help you with your pipeline in a bit more detail.