How does Caching work between agents?

General
1

Goal: Be able to use docker build caches across agents

Context: I use a buildkite pipeline to run parallel tests on 5-10 agents for some rspec tests. I use docker-compose to bring up my main app and it’s dependencies. Not sure what the right terminology is but we use temporary agents (they get removed when not used, and we get new ones when demand increases). I seem to get cache hits on the existing agents but the new ones that gets created on demand doesnt seem to have those caches to be used.

Questions 1: How can i make sure any agent (new or existing) can use these layer caches? Any examples you can point me at?

Question 2: Should I be thinking of a different strategy building/using docker images when building my pipelines?

2

Hi @benmllr86 ,

Are you able to send us a link to your pipeline or a build url where you were seeing some cache hits with your agents to support@buildkite.com? This would give us a better understanding of your setup like how docker images are loaded to run your tests, how your agents are scaled up/down and what is your infrastructure setup . With a much clearer picture, we’d be able to give you a better advice on how to improve your existing build strategy.

Cheers.