Buildkite with input steps and schedules

Pipelines
1

Hello,

If I have a pipeline which has multiple input select steps, can I have schedules for that particular pipeline and just passing as environment variables which I would get via input steps if running the pipeline manually?

Also, can I have input steps with conditional? As in, if I give in a specific value to an input step(eg Prod) the next input step to show only specific values and if I add staging to have other ones.

Thank you

2

Hey @UnknownTester404 :wave:

Welcome to the Buildkite community! :smile:

It sounds like you want to have two things happening in your pipeline here:

  1. If a build is running on a schedule, rather than a manual build, you want steps that would normally be input steps have some pre-defined inputs and run automatically
  2. You would like for input steps to appear conditionally based on previous inputs

Both of these are possible, and are a good example of why dynamic pipelines are helpful!

We have a great blog post on dynamic pipelines in Buildkite over here, which I would suggest giving a read through.

The tl;dr is, you would create a script that generates your pipeline YAML based on the inputs - you can check BUILDKITE_SOURCE to see if the build is a scheduled build, or was triggered via some other means, and you could then decide whether to generate the steps you need programmatically - as inputs or already pre-determined.

Hopefully that helps, but reach out if you have any questions or need further assistance!

3

So in both cases I should create a script to generate a pipeline file? I’ve read about the dynamic pipelines but it is unclear in which point I should set the script to run since I am declaring the path of the yml file in the steps via UI and have the running pipeline logic in the yml file.

Thanks

4

Hey @UnknownTester404 :wave:

With dynamic pipelines, you could start pointing your pipeline upload step to a script which generate the pipeline based on your custom logic.

For example: adding below in your step upload in the UI will kick of the script to generate pipeline
command: ".buildkite/dynamic_pipeline.sh | buildkite-agent pipeline upload"

Refer example here for script to pipeline generation. This should help you to move your pipeline logic to script using env variable to determine if the pipeline was scheduled or run manually like Jeremy mentioned above.

Hope this helps! Let us know if any questions.

Cheers,
Priya

5

So pretty much in the sh script I am writing from scratch the yml file? Even if there are 200 lines of code?

6

Hello UnknownTester404,

Thanks for reaching out. With dynamic pipeline, you are replacing your pipeline.yml file with a shell scripts. Lets say you have 200 lines in your pipeline.yml file, essentially, you are just adding a bit of code and carrying over the same logic.

For example:

 command : |

      if: build.source == "scheduled"

      cat <<- YAML | buildkite-agent pipeline upload 

        Contents of pipeline.yml file

      YAML

Just to make sure that we undertsand it right, are you intending to have your steps unblocked with a default value, without manual intervention ?

If that is the use case with scheduled build, you can set the default value as mentioned in Input step | Buildkite Documentation , however, this would still require manual intervention to unblock the step as by design, input steps are intended to block your build from completing, but do not automatically block other steps from running unless they specifically depend upon it.

7

Yeah, but in the sh script I would add the entire 200 lines of code for example right?

8

Hey @UnknownTester404!

Not at all, you could just run something like:

steps:
  - label: "Upload another pipeline"
    command: buildkite-agent pipeline upload the_other_pipeline.yml
    if: build.source == "scheduled"
9

So I’ve tried having a general.yml file which has


steps:
  - label: "Upload X pipeline"
    command: buildkite-agent pipeline upload .buildkite/X-pipeline.yaml
    if: "build.env('VARIABLE') == 'X'"
  - label: "Upload Y pipeline"
    command: buildkite-agent pipeline upload .buildkite/Y-pipeline.yaml
    if: "build.env('VARIABLE') != 'X'"

And then I have two yml files X and Y
However I have an error

fatal: Failed to upload and process pipeline: Pipeline upload rejected: Step is missing a type `{"key":"secret:load","pipe-gen":{"type":"step/pipeline-generator"},"depends_on":"secret:configure"}`
``` but I have in both yml files this

steps:

  • label: ‘:vault: Configure Secret’
    key: secret:configure
    commands: .buildkite/scripts/secrets.sh
    plugins:

    • plugin 1

  • pipe-gen:
    type: step/pipeline-generator
    key: secret:load
    depends_on: secret:configure

Any pointers will help. Thanks
10

Fixed it. Thanks for the info

1 Like