Using multiple plugins with a template

General
1

Hi,
We have a build pipeline that uses docker images, with multiple steps requiring ECR authentication. In order to avoid repetition of the same code we were advised to put the plugin definition in a template.

The steps that do not have other plugins defined work correctly, but those that do don’t work as desired, somehow the merge of the plugins doesn’t work. Our pipeline definition is:

_templates:
  - &ecr-auth
    plugins:
      - ecr#v2.1.0:
          login: true
          account_ids: "896069866492"
          no-include-email: true
          region: "eu-west-1"

steps:

  - <<: *ecr-auth
    label: ":docker: Build"
    command: |
      make publish

  - label: ":bomb: Tests"
    command: |
      make test

  - block: ":rocket: Deploy to staging! :chicken:"
    fields:
      - select: "I'm deploying to STAGING!"
        key: "unleash-the-chicken"
        default: "yes"
        options:
          - label: ":chicken: yes, release the chicken! :rooster:"
            value: "yes"

  - label: ":helm: :arrow_right: :k8s: staging"
    command: |
      make deploy
    plugins:
      - ssh://git@github.com/reevoo/konfiguration-buildkite-plugin.git#v0.0.2: ~
      - ssh://git@github.com/reevoo/git-buildkite-plugin.git#v0.0.3:
          repo: git@github.com:reevoo/charts.git
          branch: v5.0.0
          dest: charts
      - docker#v2.0.0:
          image: "896069866492.dkr.ecr.eu-west-1.amazonaws.com/kube-release:eks-helm3"
          workdir: "/app"
          volumes:
            - ".:/app"
          environment:
            - "K8S_CLUSTER"
            - "K8S_NAMESPACE"
            - "BUILDKITE_COMMIT"
    env:
      K8S_CLUSTER: hades
      K8S_NAMESPACE: staging
    <<: *ecr-auth

When I run a command like “buildkite-agent pipeline upload --dry-run pipeline5.yaml” I see the following output:

{
  "_templates": [
    {
      "plugins": [
        {
          "ecr#v2.1.0": null,
          "login": true,
          "account_ids": "896069866492",
          "no-include-email": true,
          "region": "eu-west-1"
        }
      ]
    }
  ],
  "steps": [
    {
      "plugins": [
        {
          "ecr#v2.1.0": null,
          "login": true,
          "account_ids": "896069866492",
          "no-include-email": true,
          "region": "eu-west-1"
        }
      ],
      "label": ":docker: Build",
      "command": "make publish\n"
    },
    {
      "label": ":bomb: Tests",
      "command": "make test\n"
    },
    {
      "block": ":rocket: Deploy to staging! :chicken:",
      "fields": [
        {
          "select": "I'm deploying to STAGING!",
          "key": "unleash-the-chicken",
          "default": "yes",
          "options": [
            {
              "label": ":chicken: yes, release the chicken! :rooster:",
              "value": "yes"
            }
          ]
        }
      ]
    },
    {
      "label": ":helm: :arrow_right: :k8s: staging",
      "command": "make deploy\n",
      "plugins": [
        {
          "ecr#v2.1.0": null,
          "login": true,
          "account_ids": "896069866492",
          "no-include-email": true,
          "region": "eu-west-1"
        }
      ],
      "env": {
        "K8S_CLUSTER": "hades",
        "K8S_NAMESPACE": "staging"
      }
    },

I’m aware that the order of the plugins is important as per other topics such as No basic auth credentials, yet using ECR plugin but I can’t get the “merge” to work properly - any help would be appreciated, Thanks

2

Hi Dan,

The plugins list for each step is an array/list, and I believe the YAML spec has no support for merging lists. There’s some relevant discussion on this github issue: https://github.com/yaml/yaml/issues/35.

To work around this limitation in YAML (and others), some of our customers use a scripting language (like python or ruby) to dynamically render their pipeline.yml before it’s uploaded. That’s a super flexible option, but it doesn’t suit all teams.

Other than that, repeating the plugin might unfortunately be your best option.

3

Yes, unfortunately YAML doesn’t deeply merge duplicates. But you can anchor and alias from any point in YAML, so you could define your plugin once and then reference from slightly deeper in your later steps, like:

_templates:
  - &ecr-auth
    ecr#v2.1.0:
      login: true
      account_ids: "896069866492"
      no-include-email: true
      region: "eu-west-1"

steps:
  - label: ":docker: Build"
    command: |
      make publish
    plugins:
      - *ecr-auth

  - label: ":bomb: Tests"
    command: |
      make test

  - block: ":rocket: Deploy to staging! :chicken:"
    fields:
      - select: "I'm deploying to STAGING!"
        key: "unleash-the-chicken"
        default: "yes"
        options:
          - label: ":chicken: yes, release the chicken! :rooster:"
            value: "yes"

  - label: ":helm: :arrow_right: :k8s: staging"
    command: |
      make deploy
    plugins:
      - *ecr-auth
      - ssh://git@github.com/reevoo/konfiguration-buildkite-plugin.git#v0.0.2: ~
      - ssh://git@github.com/reevoo/git-buildkite-plugin.git#v0.0.3:
          repo: git@github.com:reevoo/charts.git
          branch: v5.0.0
          dest: charts
      - docker#v2.0.0:
          image: "896069866492.dkr.ecr.eu-west-1.amazonaws.com/kube-release:eks-helm3"
          workdir: "/app"
          volumes:
            - ".:/app"
          environment:
            - "K8S_CLUSTER"
            - "K8S_NAMESPACE"
            - "BUILDKITE_COMMIT"
    env:
      K8S_CLUSTER: hades
      K8S_NAMESPACE: staging
1 Like
4

Thank you for your responses, we have it working now as per @sj26 suggestion above.