Native support for Golang plugins

Features Requests
1

When you’re working in an environment where you’re managing custom plugins that need to work on multiple agent platforms, it can be annoying/difficult to manage hooks. You have to worry about the nuances of how Bash works on the various platforms, or you’re managing duplicate functionality across Bash and Batch files. Also, Bash is a great scripting language, but more and more plugins are used for more complex purposes. Thus, I am proposing first class support for golang hooks.

Now, I 100% acknowledge that I can do what I want today by simply writing a Bash script to call my golang binary, but I’m proposing a more native solution where the existing structure of a plugin is replaced with a single go binary.

  1. All the hooks in a single build. The exact structure of this can vary, but I can imagine numerous ways to incorporate all the hooks into a single place.
  2. Automatic detection that determines which build of the hook we should call. If we establish that you need to build your binaries in a certain way, name them following a certain convention, and put them in a certain place, the buildkite agent could automatically call the correct build based on your platform.
  3. A golang library that makes it easy to write these hooks. Similar to how something like cobra is used to write CLI utilities, we could manage a library around hooks.

Now, if there is interest in this, I’m willing to offer my (company’s) services to help build/test this out. We are thinking of shifting towards “buildkite plugin driven development” where a lot of our automation is done inside Buildkite pipelines, and the “how” for that automation is stored in plugins.

What I’m looking for in terms of a POC is some feature flagged support for this functionality in the native buildkite-agent and ancillary services. What exactly this entails I think can be up for discussion based on the feedback this topic gets.

So, to those reading, if you think it’d be useful to be able to write Buildkite Plugins as Go binaries, please vote!

2

I love this idea!

3

Also love this idea, has there been any first steps on this?

4

I don’t believe there’s been any first steps, but there is still interest in it.

5

I just stumbled upon this and was starting to play with trying to put together some interesting helper libraries to run native Golang plugins with what we got today.

There is one hurdle I came across and I wonder if there’s a chance we could get a small change into Buildkite.

Today, the BUILDKITE_PLUGINS env var holds the full JSON string of the plugins struct. The “currently relevant” plugin parameters are available through BUILDKITE_PLUGIN_<name>_ prefixed variables.

Would it be feasible to have the current plugin configuration exported as JSON too the same way the parameters are?

In other words: Extend ConfigurationToEnvironment with something as simple as:

configJson, err := json.Marshal(p.Configuration)
// handle the error
envSlice = append(envSlice, fmt.Sprintf("BUILDKITE_PLUGIN_CONFIGURATION_%s=%s", p.Name(), string(configJson))
envSlice = append(envSlice, fmt.Sprintf("BUILDKITE_CURRENT_PLUGIN_NAME=%s", p.Name())

Reference: agent/plugin.go at fe4464768534f3cd24c4270c3ab2d28fb23f115b · buildkite/agent · GitHub

Having this would allow a plugin author to easily just:

  1. Know what the internal name of the plugin is (this is useful even for bash plugins)
  2. Parse the JSON themselves, potentially even into a type stricter than map[string]interface{}.

I know this wouldn’t give us native Golang plugins the way OP asked for, but I think it would allow us to write plugins in Go super easily.

Bootnote:
The reason we need the current plugin’s configuration in BUILDKITE_PLUGIN_CONFIGURATION_<name> and cannot just rely on parsing the full BUILDKITE_PLUGINS json is that within the context of the plugin, I don’t actually know which entry in the plugin list is the appropriate one to look at, especially if the same plugin is called more than once in the same step (which is technically possible today I think).

6

I hacked something together here: Export current plugin name and Configuration by moensch · Pull Request #1382 · buildkite/agent · GitHub

7

In lieu of native support for this, I’ve spiked a plug-in that publishes its binary to a small docker image and bootstraps with a short bash script. It’s currently Alipine based, but could be scratch instead. The overhead of musl isn’t high though.

There are other ways to do this, with different strengths and weaknesses. Downloading a release binary directly from Github could work too. We don’t have Go on the agents, so go run is currently out.

In order to discover the plug-in version in use to be able to download it I had to parse the plug-in configuration with jq, and accept that if the plug-in is referenced multiple times that only the first version would be used. I’ll have to check whether the current agent version we’re using supports the env variables added from moensch’s PR.

It was an enjoyable experience though: it was easier to test, I spent less time sorting through Bash issues, and implementing the annotation rendering template in Go was extremely simple, giving support for proper HTML escaping without any extra effort.

Using Go was a lot more verbose than Bash, but being able to use data structures was handy, and the result is pretty simple code (IMO).

There are improvements that could be made, but it was a worthwhile spike and we’ll run with it.

If there was native support, making the agent more usable as a library instead of shelling out to it would be useful.

1 Like
8

That is quite a cool concept @james.telfer! :+1:

I’ll be sure to let the team know of this approach for if/when we come around to rearchitecting the hooks system

9

Has there been any update or progress in this area? We’re mostly interested in step 1 and 2 from the initial suggestion here.

We’re deploying bash scripts to wrap all of our binary plugins with a common naming convention, e.g. hooks/checkout-win-amd64 or hooks/checkout-darwin-arm64, and deploying the whole thing. Would you be interested in a PR that standardised some naming here and provided a way to handle:

  • hooks/checkout/checkout-win-amd64.exe
  • hooks/checkout/checkout-darwin-arm64
    while gracefully preserving the existing behaviour of executing /hooks/checkout.bat and friends?
10

Hi @donal

First of all welcome to Buildkite community and thank you for reaching out to us with your questions.

While it is an area of interest but it is not currently looked at due to other priority items on our roadmap so the situation is still the same on step 1 and 2.

Now regarding PR that standardize naming convention, we definitely welcome your contributions and our team would be happy to review the PR.