Caching among steps

Hi Buildkite team,

I have an issue when running a pipeline. The pipeline takes too long time to run, initialize steps. Every step need to be initialized with the same work such as cloned repository, install docker container, install node_modules,… The next steps don’t inherit the state from the previous steps.

I would like to know if there is a way to cache its state among steps and don’t need to initialize before running a new step.

Thank you.

Hi @son.propte,

There are many ways to address this. If the issue is just taking too long to clone a repo, you can use git-mirrors feature , or if you want the jobs steps to run on the same agent that performed the pipeline upload, you can make use of the node-affinity feature.


I was going to ask the same question and just tried the “node-affinity” feature to get the benefit of storing the npm dependencies in the same directory across buildkite steps.

I did as this doc says:

And the pipeline has 3 steps that contains dependency installation:

  command: .buildkite/ # contains `pnpm install`

Also configured a root dir for pnpm in the agent-startup hook:

pnpm config set store-dir ~/.pnpm-store

however, the steps still taking 25 seconds and im able to see they are just installation of dependencies.

Hello, @muratgozel! Could you please send a link to that pipeline to for us to take a closer look?


1 Like