Trigger same pipeline multiple times (different env var)

General
1

I’m using the monorepo plugin, and I’m trying to make it run a single pipeline (pr-build) multiple times, with a different env-var for each service.

However, I get an error from the buildkite-agent when I try :( Any ideas?

Generated Pipeline:
steps:
- trigger: tend-platform
  build:
    message: try again
    branch: move-pr-to-single-pipeline
    commit: a3bbd7b6dd3a2df1301173c23aed99d096c09938
- trigger: pr-build
  build:
    message: try again
    branch: move-pr-to-single-pipeline
    commit: a3bbd7b6dd3a2df1301173c23aed99d096c09938
  env:
    LAMBDA_NAME: db-migrations
- trigger: pr-build
  build:
    message: try again
    branch: move-pr-to-single-pipeline
    commit: a3bbd7b6dd3a2df1301173c23aed99d096c09938
  env:
    LAMBDA_NAME: cognito-triggers
(more of them....)
- trigger: pr-build
  build:
    message: try again
    branch: move-pr-to-single-pipeline
    commit: a3bbd7b6dd3a2df1301173c23aed99d096c09938
  env:
    LAMBDA_NAME: patient-service
 
ERRO[2022-05-09T23:00:25Z] +++ failed to upload pipeline: command `buildkite-agent` failed: exit status 1

Any idea how I can get more info out of it?

2

Hey @nicwise!

Welcome to the community! :blush:

:thinking: This seems like an issue with the pipeline upload, how buildkite-agent is being called rather than with the monorepo plugin. You can try to check the generated yml and upload it directly so you can confirm there is not an issue with that yml file.

Best!

3

I think the problem is…

my main pipeline is thus (with a lot more of this repeated)

steps:
  - label: ":shrug: Pipeline dispatcher"
    env:
      DEBUG: true
    plugins:
      - chronotc/monorepo-diff#v2.2.0:
          diff: .buildkite/diff.sh
          watch:
            - path:
              - "db-migrations/"
              - "packages/"
              - ".buildkite/"
              - "yarn.lock"
              - "package.json"
              config:
                trigger: "pr-build"
                async: true
                env:
                  - LAMBDA_NAME=db-migrations

this results in the following pipeline to be generated from the monorepo plugin

steps:
- trigger: pr-build
  build:
    message: async it all up?
    branch: move-pr-to-single-pipeline
    commit: 0ff1a8e324a986ef8612befffad7d9176d9bbb33
  env:
    LAMBDA_NAME: db-migrations
  async: true

that, however, isn’t a valid pipeline according to the schema (so the json schema in VSCode tells me). it’d have to be:

steps:
- trigger: pr-build
  build:
    message: async it all up?
    branch: move-pr-to-single-pipeline
    commit: 0ff1a8e324a986ef8612befffad7d9176d9bbb33
    env:
      LAMBDA_NAME: db-migrations
  async: true

just not sure how to make the monorepo thing generate that.

4

looks like I had the input a bit wrong

I’ve changed the source pipeline to this:

steps:
  - label: ":shrug: Pipeline dispatcher"
    env:
      DEBUG: true
    plugins:
      - chronotc/monorepo-diff#v2.2.0:
          diff: .buildkite/diff.sh
          watch:
            - path:
              - "db-migrations/"
              - "packages/"
              - ".buildkite/"
              - "yarn.lock"
              - "package.json"
              config:
                trigger: "pr-build"
                build:
                  env:
                    - LAMBDA_NAME=db-migrations
                async: true
            - path:
              - "cognito-triggers/"
              - "packages/"
              - ".buildkite/"
              config:
                trigger: "pr-build"
                async: true
                build:
                  env:
                    - LAMBDA_NAME=cognito-triggers    
            - path:
              - "practitioner-api/"
              - "packages/"
              - ".buildkite/"
              - "yarn.lock"
              - "package.json"
              config:
                trigger: "pr-build"
                async: true
                build:
                  env:
                    - LAMBDA_NAME=practitioner-api
            (etc)

so env is under build. this works, tho if I miss one, I get that error 1, which isn’t overly useful :frowning:

Odds are good it’s YML, which is, at best, a foot gun waiting to happen :slightly_frowning_face: