Builds directory for an agent should have just one checkout per repository

As you can see we have a number of machines with 6 checkouts of the same directory. And since they don’t run git gc automatically, they take up 83.5GB (since checkout would be 8.5GB).

Thanks for posting @stevenhepting! This is an excellent idea, and something we’ve been wanting to address for quite some time.

Currently each agent does a git clone of the repository. If you’ve got 5 different agents on the one machine, then you’ve got 5 full copies of the repository. If the name of your agent changes each time (maybe you’re using %n) then this can get even worse, since depending on when the agent starts/stops, you could have an endless amount of agent directories all with the same copy of the code.

@lox and I’s current thinking is to maintain 1 copy of the repository in /usr/local/var/buildkite-agent/repositories and do a bare clone into the agent directory. We could also allow some options around “shallow cloning” so we only git clone the last 100 commits say, to speed up that process.

Since we’ve got a long running process on the machine, and single checkouts of the repo, this allows us to do a few other cool things like:

  • Periodically run git fetch to prepare the repo for work (we could probably run this every 10 seconds or so safely, and make the timing a config option)
  • Periodically run git gc once a week perhaps? (the timing of this could be configurable as well)

Another thing we’d do is make the repositories paths configurable, so if you’ve got say 5 virtual machines on the same hardware, you could choose a mounted path that they all share, giving you truly 1 version of the repo.

(sorry for the wall of text) how does something like that sound?

1 Like

@keithpitt We are actually only running 1 Buildkite agent on each machine, so the multi-agent situation doesn’t impact us. What we really care about is the “multi-pipeline” situation where 6 pipelines all checkout the same repository. It would be great to have these all use the same repo.

I imagine our case should be even simpler than the multi-agent case since you don’t need to worry about contention, just the directory name for where to perform the build.

Oh right, thanks for clarifying! I think our proposed solution would still work a treat :slight_smile:

The path we git clone the repository into is:

/usr/local/var/buildkite-agent/builds/[agent-name]/[org-name]/[pipeline-name]

If you use the same Git repository in 6 different pipelines, they’d all have their own full copy of the repository.

/usr/local/var/buildkite-agent/builds/[agent-name]/[org-name]/[pipeline-name1]
/usr/local/var/buildkite-agent/builds/[agent-name]/[org-name]/[pipeline-name2]
/usr/local/var/buildkite-agent/builds/[agent-name]/[org-name]/[pipeline-name3]
...

If we took the “only 1 checkout per machine” approach, then we’d only make 1 full git clone into:

/usr/local/var/buildkite-agent/repositories/[some-id-here]

And then do bare clones into each [agent]/[org-name]/[pipeline-name] path.

That’d still mean we’re only managing 1 checkout (saving you all the gigabytes!)

1 Like

@keithpitt Are there any updates on this topic?

I just got back to work from parental leave, and am looking into a number of machines that are filling up their disks with various checkouts.

I’m wondering if I should write a scheduled job to delete some of these checkouts that get run less than 1/day or if this functionality might be on the roadmap.

Hey @stevenhepting, hope your parent leave went well!

Sounds like we at a minimum need a mechanism for garbage collecting old agent checkouts.

I’ve started work on the single repository model in the agent (which will use either shallow clones or workdirs). I’ll actually be at the Airbnb office next week, perhaps we could collaborate or at least discuss? :grinning:

1 Like

This will be available in the next Buildkite Agent release:

And in the next Elastic CI Stack for AWS release:

1 Like

Git Mirrors are not a complete solution to this problem. For some repositories, a bulk of the space is consumed via LFS files, which will not be shared between mirrored repositories.

We’ve actually managed to solve this problem thanks to the configurability of the BUILDKITE_BUILD_CHECKOUT_PATH variable in the global environment hook. This is what we use at Butterfly:

export REPO_SLUG=$(echo "$BUILDKITE_REPO" | cut -d: -f2 | sed 's/\.git$//')
export BUILDKITE_BUILD_CHECKOUT_PATH="$BUILDKITE_BUILD_PATH/$REPO_SLUG"

It works pretty well, but ideally we’d like to have a simple configuration flag for this.

Feature Spec

Add two configuration flags to control checkout path:

checkout_path_includes_pipeline: true by default, settable to false. When true, the pipeline name will be included as part of the checkout path.

checkout_path_includes_hostname: true by default, settable to false. When true, the hostname will be included as part of the checkout path.