Bootstrap process to create a pipeline


#1

Hi! Currently, we have a setup for our repositories where we use a custom command to first generate the pipeline for a service based on environment variables defined for the pipeline. Will the new feature and when it is rolled out to all do away with the ability to run a custom command to generate a pipeline?

To give a concrete example, we have APPLICATION_CATEGORY environment variable we define for a pipeline which is set to say, DOTNET_CORE or JAVASCRIPT and then based on those, the pipeline is generated by a bootstrap command line tool.

This has allowed us to do away with each repository having it’s own pipeline definition and seems to really make it easy to setup new pipelines.


#2

Thanks for the question @amitsaha!

You know…I hadn’t considered that but… I think we’d be pretty close to being able to offer that!

Just to clarify your setup, your initial pipeline looks like this:

steps:
  - command: "generate_pipeline.sh | buildkite-agent pipeline upload"

Which uploads a pipeline like this:

steps:
  # Only include this step if APPLICATION_CATEGORY is DOTNET_CORE
  - command: "run_dotnet_core.sh"

  # Only include this step if APPLICATION_CATEGORY is JAVASCRIPT
  - command: "run_javascript.sh"

Which depending on the value of APPLICATION_CATEGORY it either include the first step, or the second step. Does this kinda match your setup?


#3

Hi @keithpitt - thanks for the reply.

Yes, you are right. The key is of course the generate_pipeline.sh step which uses the various environment variables to generate the pipeline which can be quite different based on those environment variable values.

To give a more concerete idea, here’s how roughly the generate_pipeline.sh script looks like:

#!/usr/bin/env bash

set -e

echo "steps:"

if [ -z "$APPLICATION_CATEGORY" ]; then
  echo "APPLICATION_CATEGORY not defined"
  exit 1
fi

if [ "$APPLICATION_CATEGORY" == "DOTNET_CORE" ]; then
  echo "  - label: \"Build, Test and Publish\""
  echo "    command: \"/usr/local/bin/ci/dotnet_core/entrypoint.bash\""
  echo "    agents:"
  echo "      queue: \"linux\""

  echo "  - wait"

  echo "  - block: \":rocket: Deploy to Test\""

  echo "  - label: \"Deploy to Test\""
  echo "    command: \"/usr/local/bin/ci/dotnet_core/deploy_test.bash\""
  echo "    agents:"
  echo "      queue: \"linux\""

  echo "  - wait"

  echo "  - block: \":rocket: Deploy to Production\""


  echo "  - label: \"Deploy to Production\""
  echo "    command: \"/usr/local/bin/ci/dotnet_core/deploy_prod.bash\""
  echo "    agents:"
  echo "      queue: \"linux\""

elif [ "$APPLICATION_CATEGORY" == "JAVASCRIPT_FRONTEND" ]; then

  echo "  - label: \"Build, Test and Publish to UAT\""
  echo "    command: \"/usr/local/bin/ci/javascript_frontend/deploy_uat.bash\""
  echo "    agents:"
  echo "      queue: \"linux\""
  
  echo "  - wait"

  echo "  - block: \":rocket: Deploy to Production\""


  echo "  - label: \"Deploy to Production\""
  echo "    command: \"/usr/local/bin/ci/javascript_frontend/deploy_prod.bash\""
  echo "    agents:"
  echo "      queue: \"linux\""

else
  echo "APPLICATION_CATEGORY: $APPLICATION_CATEGORY not recognized"
fi

#4

Thanks for the example script! Although this doesn’t solve that use case 100%, it moves us close to a solution! We’ve got something extra in the works that should help you out here :slight_smile:


#5

Great, looking forward to it.