No basic auth credentials, yet using ECR plugin

General
1 
plugins:
  ecr#v1.1.4:
    login: true
    account_ids: "436617320022"
    no-include-email: true
  docker#v2.0.0:
    image: "436617320022.dkr.ecr.us-west-2.amazonaws.com/ci:centos"
    workdir: /data/job
    timeout: 120

Running plugin GitHub - buildkite-plugins/ecr-buildkite-plugin: 🔐 Login to an AWS ECR registry pre-command hook 0s
Authenticating with AWS ECR to 436617320022 5s
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
# AWS_DEFAULT_REGION changed
Running plugin GitHub - buildkite-plugins/docker-buildkite-plugin: 🐳📦 Run any build step in a Docker container command hook 0s
docker Running /bin/sh -e -c 'echo “+++ :hammer: Building” && \ 1s
echo 1 ./build.sh &&
echo “— Compressing build directory :compression:” &&
tar -pczf build.tar.gz build/
’ in 436617320022.dkr.ecr.us-west-2.amazonaws.com/ci:centos
Unable to find image ‘436617320022.dkr.ecr.us-west-2.amazonaws.com/ci:centos’ locally
docker: Error response from daemon: Get https://436617320022.dkr.ecr.us-west-2.amazonaws.com/v2/ci/manifests/centos: no basic auth credentials.
See ‘docker run --help’.
:rotating_light: Error: The command exited with status 125

I’m curious if someone knows why a successful login with the ECR plugin still causes the docker command to fail?

2

Tried 2.1.0, still no difference.

Running plugin GitHub - buildkite-plugins/docker-buildkite-plugin: 🐳📦 Run any build step in a Docker container command hook 0s
$ /var/lib/buildkite-agent/plugins/github-com-buildkite-plugins-docker-buildkite-plugin-v2-1-0/hooks/command
:hammer: Enabling debug mode 0s
docker Running /bin/sh -e -c 'echo “+++ :hammer: Building” && \ 1s
echo 1 ./build.sh &&
echo “— Compressing build directory :compression:” &&
tar -pczf build.tar.gz build/
’ in 436617320022.dkr.ecr.us-west-2.amazonaws.com/ci:centos
$ docker run -it --rm --volume /var/lib/buildkite-agent/builds/automation-builder-fleet/EOSIO/eosio-dot-cdt-dot-testing:/data/job --workdir /data/job --env BUILDKITE_JOB_ID --env BUILDKITE_BUILD_ID --env BUILDKITE_AGENT_ACCESS_TOKEN --volume /usr/bin/buildkite-agent:/usr/bin/buildkite-agent 436617320022.dkr.ecr.us-west-2.amazonaws.com/ci:centos /bin/sh -e -c echo “+++ :hammer: Building” &&
echo 1 ./build.sh &&
echo “— Compressing build directory :compression:” &&
tar -pczf build.tar.gz build/
Unable to find image ‘436617320022.dkr.ecr.us-west-2.amazonaws.com/ci:centos’ locally
docker: Error response from daemon: Get https://436617320022.dkr.ecr.us-west-2.amazonaws.com/v2/ci/manifests/centos: no basic auth credentials.
See ‘docker run --help’.
:rotating_light: Error: The command exited with status 1

3

That’s a tricky one! There can be a few causes. Let’s see if we can narrow it down!

First up, when you have plugins that depend on ordering, it’s a good idea to use a list for plugins vs a map. It’s rare, but there can be ordering issues with maps as yaml doesn’t guarantee enumeration order. Beyond that, the other possibility is that the ECR login is happening in a region other than us-west-2, which would explain the issue. Do you know what region your agent instance is in?

Perhaps you could try:

plugins:
  - ecr#v1.1.4:
      login: true
      account_ids: "436617320022"
      no-include-email: true
      region: us-west-2
  - docker#v2.0.0:
      image: "436617320022.dkr.ecr.us-west-2.amazonaws.com/ci:centos"
      workdir: /data/job
      timeout: 120
1 Like
4

Turns out it was the region that was the issue. However, the - before the plugins seems to have broken it, throwing:Error: Failed to parse plugin definition: Configuration for "github.com/buildkite-plugins/login-buildkite-plugin" is not a hash

All set, thanks for the help!

5

That’s odd, I wonder if perhaps the indenting didn’t come through correctly? The dashes are very important for ordering, otherwise you might login after your docker command!

6

I’m seeing 100% of runs so far order things properly, without dashes. I’ll keep an eye out on the runs and look for it mixing them up.

7

Yup, it’s deterministically unordered! Up to you, but I promise it’s a convention that will eventually lead to very confusing ordering issues.

8

Hey Lox, wanted to follow up on this. So far all of the pipelines are choosing the proper order for plugins using:

plugins:
  ecr#v1.1.4:
    login: true
    account_ids: "123"
    no-include-email: true
    region: "us-west-2"
  docker#v2.1.0:
    debug: false
    image: "123.dkr.ecr.us-west-2.amazonaws.com/ci:centos7"
    workdir: /data/job

Is this a problem with the version of buildkite we’re using? We’re using v3.7.0. Wonder if you have any ideas why I can’t use a list ( - before docker and ecr lines) without it failing. Thanks!

9

When I use this pipeline yaml:

steps:
  - label: "test"
    command: echo hello world
    plugins:
    - ecr#v1.1.4:
        login: true
        account_ids: "123"
        no-include-email: true
        region: "us-west-2"
    - docker#v2.1.0:
        debug: false
        image: "123.dkr.ecr.us-west-2.amazonaws.com/ci:centos7"
        workdir: /data/job

And run it with buildkite-agent pipeline upload --dry-run test.yml, I get the following parse output:

{
  "steps": [
    {
      "label": "test",
      "command": "echo hello world",
      "plugins": [
        {
          "ecr#v1.1.4": {
            "login": true,
            "account_ids": "123",
            "no-include-email": true,
            "region": "us-west-2"
          }
        },
        {
          "docker#v2.1.0": {
            "debug": false,
            "image": "123.dkr.ecr.us-west-2.amazonaws.com/ci:centos7",
            "workdir": "/data/job"
          }
        }
      ]
    }
  ]
}

That all looks right to me! Are you able to replicate those results?

10

buildkite-agent version 3.7.0, build 2659

Odd, I still can’t get the plugins in a yaml list to work. I originally thought it might be that I’ve got - command: and no label. But adding a `- label: “test” and then command on the next line doesn’t change it.

11

Are you able to post the output from buildkite-agent pipeline upload --dry-run @nathan.pierce?

12

Hmm, here is the output of your test and mine in a dry-run:

{
  "steps": [
    {
      "label": "test",
      "command": "echo hello world",
      "plugins": [
        {
          "ecr#v1.1.4": {
            "login": true,
            "account_ids": "123",
            "no-include-email": true,
            "region": "us-west-2"
          }
        },
        {
          "docker#v2.1.0": {
            "debug": false,
            "image": "123.dkr.ecr.us-west-2.amazonaws.com/ci:centos7",
            "workdir": "/data/job"
          }
        }
      ]
    },
    {
      "command": "echo \"+++ :hammer: Building\" \u0026\u0026 \\\necho 1 | ./eosio_build.sh \u0026\u0026 \\\necho \"--- :compression: Compressing build directory\" \u0026\u0026 \\\ntar -pczf build.tar.gz build/\n",
      "label": ":centos: 7 Build",
      "agents": {
        "queue": "automation-large-builder-fleet"
      },
      "artifact_paths": "build.tar.gz",
      "plugins": {
        "ecr#v1.1.4": {
          "login": true,
          "account_ids": "123",
          "no-include-email": true,
          "region": "us-west-2"
        },
        "docker#v2.1.0": {
          "debug": true,
          "image": "123.dkr.ecr.us-west-2.amazonaws.com/ci:centos7",
          "workdir": "/data/job"
        }
      },
      "timeout": 60
    }
  ]
}

I’m going to keep playing around with this… One thing I notice is that “plugins” is not an array like yours is. That could be the reason I’m not getting a mix up of the plugin order.

13

Yup, that is a difference! See my quote from earlier: