Security Recommendations on agents with docker

Help
#1

I have a VM that I want to use to run buildkite on an open source project, both for the master (post-review) and PRs (pre-review), and I would like to offer the dockerization experience to the users via the docker plugin for buildkite, as the project heavily relies on it.

However, I obviously do not want users to escalate privileges in the VM through buildkite builds.

Specifically, I was planning to add a user, buildkite, to the machine specifically to run buildkite agent daemon. However, doing so and allowing docker requires

sudo usermod -aG docker buildkite

which opens up the Docker daemon attack surface. This would not be a problem if the builds were reviewed prior to run, but PRs are by definition not reviewed, and thus this would open up a large security hole.

Do you have any specific recommendations on this topic?

#2

Hi Jorge!

Here’s how I approached this for Rails: buildkite-config/initial.yml at master · rails/buildkite-config · GitHub

Essentially, pre-review PR builds are permitted iff they haven’t changed the .buildkite/ config itself, including the docker configuration. Thus, anything the PR might have changed is constrained to run inside docker instances which, while still configured in the same repository, have a known-trusted configuration. In the rare case that a PR needs to modify that gear, its build gets a block step so a trusted user can review those changes before the build proceeds.

Of course this strategy only works if the vast majority of your incoming PRs do not need to modify that set of files which are providing your security boundary.

Does that sound like an option that might be viable for your needs? I’m happy to go deeper with you, either here or via support@, to help explore whether it might adapt to your project.

Cheers,

Matthew

#3

Thanks a lot for your answer.

That is a good option. Thanks a lot for sharing that.

What I (successfully) tried so far was to deploy a rootless version of docker running on a non-privileged user. It disables some docker capabilities, but it is an easy fix for most use-cases. The installation is a bit more cumbersome, but it goes along the lines of (on an ubuntu)

set -e

USERNAME=buildkite-agent
EXTERNAL=external

# prepare a mount point for external
mkdir /media/$EXTERNAL
# partition external disk, `parted`, `mkpart primary ext4 0% 100%`
# change /etc/fstab to add auto-mount of the external disk

sudo chown -R $USERNAME:$USERNAME /media/$EXTERNAL

# place everything docker-related into the external volume via symlink
ln -s /media/$EXTERNAL/docker ~/.local/share/docker

sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

sudo adduser $USERNAME
sudo su -- $USERNAME
mkdir /home/$USERNAME/.ssh/
touch /home/$USERNAME/.ssh/authorized_keys
# add public key to authorized_keys
# ssh to $USERNAME@hostname

curl -fsSL https://get.docker.com/rootless | sh

systemctl --user enable docker
systemctl --user start docker

export DOCKER_HOST=unix:///run/user/1001/docker.sock
docker run hello-world

# Docker ready: let's move to the buildkite
# https://buildkite.com/docs/agent/v3/linux

TOKEN="INSERT-YOUR-AGENT-TOKEN-HERE" bash -c "`curl -sL https://raw.githubusercontent.com/buildkite/agent/master/install.sh`"

# configure auto start of buildkite and stuff on the user.