Plugin branch fresh clone not working

smoreno-allurion · April 28, 2023, 5:46pm

Hi, I’m developing a plugin for Buildkite and event when I set the BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH env to true, the plugin doesn’t get updated.

The log shows:

Preparing plugins	0s
# Plugin "github.com/org/pipeliner#bug/sc-29304/detecting-changes" already checked out (4e7c351)
Preparing working directory

This is the YAML file of the pipeline using the plugin:

steps:
  - label: ":pipeline: Generating pipeline"
    env:
      BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH: "true"
    plugins:
      - ssh://github.com/org/pipeliner#bug/sc-29304/detecting-changes:
          "create-deploy-dependencies": false
          "mode": "build"
          "workspaces-path":
          - "apps"
          - "services"

Am I missing something here? It used to work just fine, but not anymore.

Thanks!

suma · April 28, 2023, 6:53pm

Hi @smoreno-allurion

This is Suma from Buildkite support team. Thank you for reaching out to us with your question.

In order to achieve this you should actually set the flag plugins-always-clone-fresh to true as part of your agent configuration or during agent start up by passing flag --plugins-always-clone-fresh=true which will let the agent know that it needs to perform clone every time and not use local copy of the plugin.

When you set that flag that environment variable BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH will get set to true so instead of defining this environment variable in pipeline, you should either set the flag during agent startup or set it in agent configuration file.

Please let us know if you have any further questions on this.

Thanks,
Suma

smoreno-allurion · April 28, 2023, 7:15pm

Hi Suma,
Thanks for your answer. I’m aware of the flag you mention, but I was under the impression that the same could be achieved by setting that env variable on a step.

The documentation mentions this when it says:
" One way to try this is to add the following step to the Buildkite pipeline where you’re testing your plugin. Configuring BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH on only one step means that other plugins, which are unlikely to be changing in the meantime, won’t get unnecessarily cloned on every step invocation." (source)

Did I misinterpret it? Or may be this feature is not available anymore? I’m quite sure this was working with that env var on the step before.

Thanks!

suma · April 28, 2023, 9:54pm

Hi @smoreno-allurion
Yes, you are correct that one of the option is to able to set this environment variable at step level so it does not apply for every plugin in the pipeline.

I did a test and can see following where it sees that env is set and removes previous checkout of the plugin

|Preparing plugins|1s|
| --- | --- |
||# BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH is true; removing previous checkout of plugin github.com/buildkite-plugins/library-example-buildkite-plugin#v1.0.1|
||# Plugin "github.com/buildkite-plugins/library-example-buildkite-plugin" will be checked out to "/opt/homebrew/var/buildkite-agent/plugins/github-com-buildkite-plugins-library-example-buildkite-plugin-v1-0-1"|

Please can you send your build url to support@buildkite.com where you saw it did not work so we can check further on this.

Thanks,
Suma

suma · April 28, 2023, 11:02pm

Hi @smoreno-allurion

Please can you also confirm agent version is 3.37.0 or above because this feature was introduced in v3.37.0
Here is the release notes

Here is PR in which this change was introduced

github.com/buildkite/agent

feature: Allow forcing clean checkout of plugins

buildkite:main ← toothbrush:paul/force_pull_plugins

opened 01:15AM - 13 May 22 UTC

toothbrush

+414 -57

## Description While developing a Buildkite plugin, it can be useful to just …work on a branch (e.g., `dev`) and refer to it in some test pipeline, e.g.: ```yaml steps: - command: test.sh plugins: - my-org/herd-the-llamas#dev: alpacas: mostly ``` However, [the current implementation of `bootstrap.go(checkoutPlugin)`](https://github.com/buildkite/agent/blob/de27049657ac876ba08c3e7830c5425d9bb0659d/bootstrap/bootstrap.go#L878) only checks that a checkout matching repo name and ref _exists_, not whether it's up-to-date. This PR introduces a CLI flag `--plugins-always-clone-fresh` and an environment option `BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH=true` (thanks @moskyb for the improved names 🙌) which will cause the Buildkite agent to always make a fresh clone of all plugins. For example, in a test pipeline: ```yaml steps: - command: test.sh env: BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH: "true" plugins: - my-org/herd-the-llamas#dev: alpacas: rarely ``` The `BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH=true` setting will only apply to that one step, which is likely enough for the use case i have in mind. Of course there are risks that a user of Buildkite might set this option globally, causing their build agents to repeatedly perform clones of plugins on every job step, but.. surely nobody would do that 😅 Folks have been talking about this for quite a while: https://github.com/buildkite/agent/issues/422! I'm not sure if this PR adequately addresses the concerns in that issue (e.g., be smart and pull if the plugin version is a branch ref, but not if it's a tag ref), but i've found this to be a sufficient workaround for my purposes. ### What other approaches did i consider, and why did i discard them? #### Add a configuration flag per-plugin Initially i thought it'd be swell to do something like this: ```yaml steps: - command: test.sh plugins: - my-org/herd-the-llamas#dev: force_pull: true ``` However, that configuration block is passed directly to the plugin, and we wouldn't want to block a plugin author from having a `force_pull` flag. 🚫 #### Add a step-level config flag Next, i though the "right" place to configure this might be on a step level: ```yaml steps: - command: test.sh force_pull_plugins: true plugins: - my-org/herd-the-llamas#dev: alpacas: are_not_llamas ``` But to my surprise, the `buildkite-agent` tool doesn't have much machinery for interpreting pipeline YAML – i guess that happens on Buildkite-server-side. Fair enough. 🚫 Then it hit me – the idiomatic way to do this is probably via an environment variable. I went digging, and discovered inspiration in https://github.com/buildkite/agent/pull/1242. Crucially, it showed that setting the environment config per-pipeline-step was possible (as opposed to e.g., having to set it before `buildkite-agent` startup, which would be super awkward). That's why i chose this approach. ### Other random weird comments * I did quite a bit of thinking and experimenting around making my force-pull feature smarter, e.g., if a checkout exists, do a `git fetch --force --tags` and try to update worktree state to the latest version of the branch or tag. However, making that robust and general turned out to be surprisingly tricky. In the end i just chose the blunt instrument and made it `rm -rf` the checkout if the user wants a fresh one. There's a discussion to be had around that tradeoff, but this seemed most robust and improved the situation today, even if it's not the most ideologically pure way of doing things. * I was bitten by a very surprising thing (and i might have misunderstood, feel free to correct me): initially i only added the configuration option with environment struct-tag to [`bootstrap/config.go(Config)`](https://github.com/buildkite/agent/blob/de27049657ac876ba08c3e7830c5425d9bb0659d/bootstrap/config.go#L19), but the option only became available to me (as an env var!) once i'd added it with a `cli:"<foo>"` struct tag in [`clicommand/bootstrap.go(BootstrapConfig)`](https://github.com/buildkite/agent/blob/fa992ef4988b6d24820fd17b171c1fceb60ae59c/clicommand/bootstrap.go#L45). To help future travellers i added a signpost comment in https://github.com/buildkite/agent/pull/1636/commits/5ad83dd8a17daf9f90e9d59e09e60d47ef9a3ce3. * I wasn't sure how to test this change apart from literally running a build locally with `buildkite-agent bootstrap` and when i had more certainty, spinning up a Buildkite EC2 box with my tricked-out `buildkite-agent`. So, i haven't got integration tests (yet?) around this feature. I imagine that `bootstrap/integration/plugin_integration_test.go` would be the place to add something, but i haven't thought about that much yet. * If folks think this approach has merit, i'd be happy to update the docs with the flag as well as the plugin-development section to suggest using it.

So if your agent version is lower than 3.37.0 version then that could be the reason you see it is not working. If so please upgrade the agent to 3.37.0 or above.

smoreno-allurion · April 29, 2023, 2:10am

Yes, I can confirm my nodes are running with version v3.45.0.

I’m sending the email to support now.

Thanks Suma!

Topic		Replies	Views
How to make plugin always-pull in every build? General	2	687	October 8, 2021
Shallow clones? Features Requests	4	3133	February 22, 2022
Buildkite env vars set in the pipeline is broken Pipelines	7	810	August 29, 2023
Ability to consistently point Plugins at the HEAD commit on a given Branch Features Requests	3	470	April 18, 2023
Plugin environment variables question General	1	981	October 26, 2020

Plugin branch fresh clone not working

Related topics