Multi-architecture builds using Docker Compose

Docker
1

I am trying to build and push amd64 and arm64 images using Docker Compose. My docker-compose.yml looks like this:

version: '3.8'
services:
  ruby-3.1:
    build:
      context: .
      dockerfile: 3.1/Dockerfile
      platforms:
        - "linux/amd64"
        - "linux/arm64"

My pipeline.yml looks like this:

steps:
  - label: ":docker: Build and Push to ECR"
    plugins:
      - docker-compose#v4.16.0:
          push:
            - ruby-3.1:xxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/ruby:3.1.4
          cli-version: 2

I am using the latest version of the Elastic CI Stack for AWS.

When I try to run this, I get an error about Docker not support multi-architecture builds. I got this same error locally on my Mac until I enabled using containerd for storing and pull images - once I did that, it worked locally.

Is there a way to enable containerd on the stack instances? I think this would enable building multi-architecture images because the instances themselves seems to contain all the QEMU stuff needed to actually do the build.

Thanks!

2

It’s also worth noting that the Docker documentation seems to suggest containerd can be enabled by modifying the daemon configuration.

3

Hey @wgrrrr

Welcome to the Buildkite Community Forums!

Thanks for reaching out, it is a good question and there are a few potential ways to enable containerd on the Elastic CI Stack Instances.

If you created your own AMI instances using the Elastic CI Stack repository, you could update the packer scripts to update the Docker daemon config e.g., in install-docker.sh

Alternatively, as mentioned in a similar issue with the repository, there is the BootstrapScriptUrl parameter in the CloudFormation Stack, which can be used to download a script to update the Docker daemon config on booting the EC2 Instances - saves time building AMIs

Let us know if either those options work for you, and if not, I’d recommend raising it as an Issue on the repository for increased visibility:

Cheers,

Tom

4

Thought I’d close the loop on this topic… I wrote the following instance bootstrap script which configures Docker to use the containerd snapshotter:

#!/usr/bin/env bash

echo "Running Buildkite instance boostrap..."

if [ ! -f /etc/buildkite_run_once ]; then
    echo "This is the first run of the Buildkite instance: running one-time setup..."

    # Backup the original Docker configuration
    cp /etc/docker/daemon.json /etc/docker/daemon.json.orig

    # Customise the Docker configuration to use the containerd snapshotter
    jq '. += {"features": {"containerd-snapshotter": true}} | del(."storage-driver")' /etc/docker/daemon.json.orig > /etc/docker/daemon.json

    # Restart Docker to load our new configuration
    systemctl restart docker

    # Create a file to indicate that the one-time setup has been run
    touch /etc/buildkite_run_once
    echo "Finished running Buildkite one-time setup."
fi

echo "Finished running Buildkite instance boostrap."

This script is injected into the Elastic CI Stack for AWS using the BootstrapScriptUrl parameter. It was also necessary to disable namespacing via the EnableDockerUserNamespaceRemap parameter for some reason. I think this is the bleeding edge of Docker at the moment, so I suspect we may be able to make this work without disabling namespacing in the future (I hope so).

Hope this helps someone!

2 Likes
5

Hey @wgrrrr

Thanks so much for sharing, and letting us know you were able to use the BootstrapScriptUrl parameter.

Interesting to hear that you had to disabled EnableDockerUserNamespaceRemap, so wondering if that affected any of your Builds? As I aware it enabed for the buildkite-agent user to run Docker itself.

But any issues, let us know!

Cheers,

Tom

6

None of the builds were working correctly with EnableDockerUserNamespaceRemap enabled. I received errors when running apt operations inside the containers, and they were “permission denied” type errors (as though I was not running apt as root). I had a feeling it was related to the namespace remapping, and disabled it to test this theory. Indeed, that fixed the issue. Note that there were no other changes OTHER than disabling EnableDockerUserNamespaceRemap, so I am pretty confident in that.

I wonder if there is some conflict between namespace remapping and the experimental nature of the containerd snapshotter? Not sure.

7

Thanks for the extra info!
It is possible that your scenario was further complicated by namespace remapping being enabled in your AWS Elastic Stack (Template parameters in the Elastic CI Stack for AWS | Buildkite Documentation). That would further change how IDs are mapped inside containers you are running.

We haven’t heard about this issue, but we’ll keep an eye on if we see this issue again.
Thanks again!