Caching among steps

Pipelines
1

Hi Buildkite team,

I have an issue when running a pipeline. The pipeline takes too long time to run, initialize steps. Every step need to be initialized with the same work such as cloned repository, install docker container, install node_modules,… The next steps don’t inherit the state from the previous steps.

image
image807×874 91.4 KB

I would like to know if there is a way to cache its state among steps and don’t need to initialize before running a new step.

Thank you.

2

Hi @son.propte,

There are many ways to address this. If the issue is just taking too long to clone a repo, you can use git-mirrors feature , or if you want the jobs steps to run on the same agent that performed the pipeline upload, you can make use of the node-affinity feature.

Cheers!

3

I was going to ask the same question and just tried the “node-affinity” feature to get the benefit of storing the npm dependencies in the same directory across buildkite steps.

I did as this doc says:

And the pipeline has 3 steps that contains dependency installation:

  ...
  command: .buildkite/unit_tests.sh # contains `pnpm install`
  agents:
      hostname: "$BUILDKITE_AGENT_META_DATA_HOSTNAME"
  ...

Also configured a root dir for pnpm in the agent-startup hook:

pnpm config set store-dir ~/.pnpm-store

however, the steps still taking 25 seconds and im able to see they are just installation of dependencies.

4

Hello, @muratgozel! Could you please send a link to that pipeline to support@buildkite.com for us to take a closer look?

Best!
Karen

1 Like