Verifying the integrity of third party plugins


#1

We use several plugins provided by buildkite in our builds and pin them to versions. We also have started using third-party plugins provided by other organisations, like CultureAmp’s aws-assume-role plugin.

Pinning versions is not entirely sufficient to ensure that we’re always running the same plugin code between builds: the pin is based on a git tag or branch, which can easily be reset by the repository maintainer. Worse, if a bad actor gains access to the third party repository, they could add their own malicious code and reset the tags for existing versions.

We’ve worked around this by forking plugins to our own repository, but is there a better way to ensure the plugins we run don’t change over time and remain secure?


#2

Would pinning the plugin to the full commit SHA be sufficient? You can put any commitish reference where you’d normally place the tag/branch.


#3

The other option we’ve been discussing is allowing plugins to be vendored. For instance, we could support .buildkite/plugins, which if it existed would be the only place the agent would load plugins from for that build.

The bk tool could be used to read a pipeline.yml and download linked plugins:

bk plugins vendor
> Downloading cultureamp/aws-assume-role into ~/.buildkite/plugins/cultureamp/aws-assume-role

The only issues I can foresee with this is that it would preclude hooks that fire before checkout.


#4

With sha’s it’s not obvious which "version’ you’re using, and whether it’s the latest, although it does solve my immediate problem.

With plugins there’s a lot of copy/paste already, and I’m wondering whether some of that can/should be moved to the top of the pipeline file, etc:

plugins:
  docker-compose#1.1.1#ab01df
    # verifies that 1.1.1 has this sha - so you get both version / lock
    config-file: docker-compose.buildkite.yml
steps:
  plugins:
    docker-compose:
      # inherits version/sha lock, and plugin defaults, from above
      run: ruby

#5

@lox I would love to be able to vendor plugins, but for a different reason: I want to work on customising / making small changes to plugins without having to do so in a different git repository to the build I’m testing the plugin in.


#6

Yup, I hear ya! I’ve often wanted that too. I’d also like https://github.com/buildkite/agent/issues/422 for that too. Maybe I will build that now.


#7

I wrote a very-similar looking gist on the top-level plugin block a while back: https://gist.github.com/lox/1dd425819b4e3874ba394346cb56a74c

I think the difficulty it runs into is that inheritance is really hard to reason about, especially when you are trying to merge complex yaml structures. I think docker-compose’s extends support is a really good example of how badly this can work.

In the meantime, you could use yaml anchors for exactly that functionality if you wanted it! They are also diabolically complicated :sweat_smile:


#8

Nice! We’ve thrown around some ideas like this before, but I don’t think we thought of pinning both the version and sha. :ok_hand:t3:

Now we have the agents block, this mirrors it quite nicely:

(@lox I think the same overriding rules would apply there, where a specific step just wholesale overrides the plugin config defined at the top?)


#9

(We’d probably use arrays instead of a map now though, as it’s the new recommended default:
https://buildkite.com/docs/pipelines/plugins)


#10

@lox yep, agree re YAML/inheritance weirdness being confusing.

How about something a bit more lightweight?

plugin-versions:
  - docker-compose#1.1.1#ab01df
  - ecr#1.1.0#3301df
steps:
  plugins:
    - ecr:
       ...
    - docker-compose:
       ...

Bonus round: if you specify plugins without versions, or with versions but without a commit sha, Buildkite prints a security warning and generations the plugin-versions section for you for you to copy/paste to your pipeline.


#11

I think I’m probably leaning towards vendored plugins as a first port of call for this stuff. It doesn’t solve the problem of de-duplicating config, but it’s fast, efficient and has the best security profile out of the options we’ve looked at.

Makes for a great way to prototype plugins too!


#12

Yep, reasonable.

(blah blah blah 20 character minimum)


#13

We will try and remove the 20 char minimum!

The first major obstacle to vendored plugins is that we’d have to drop support for checkout related hooks: pre-checkout, checkout and also environment, or at least change the semantics of the environment to run after the bootstrap checkout phase.


#14

(just made it 10!)


#15

The top-level plugin block brings with it a whole lot of questions, so for the moment let’s leave it aside and focus on the two things that bring integrity and also development easy. This is what I ended up with

steps:
  - name: ":hammer: :linux:"
    command: ".buildkite/steps/tests.sh"
    plugins:
      - "docker-compose#v2.5.1@51cc6ca5c1b1cc01ccd1d628efa6132f540119de":
          config: .buildkite/docker-compose.yml
          run: agent
      - "./.buildkite/plugins/my-vendored-plugin":
          blah: true

#16

Agree that vendored plugins would solve this for us quite nicely. Also means it’s a lot easier to use “private” plugins without having to push them to a repo (e.g. for testing).

Do you see that plugins via submodules would also be supported? And would the plugin.yml drive things like the plugin name for variables? e.g. BUILDKITE_PLUGIN_MY_VENDORED_PLUGIN?


#17

I think they should mostly just work since you define a vendored plugin via a path, you should just be able to point the path at a submodule:

steps:
  - name: ":hammer: :linux:"
    command: ".buildkite/steps/tests.sh"
    plugins:
      - "./.buildkite/submodules/my-plugin":
          blah: true

The code checkout/submodule phase would have already happened by the time we start running vendored plugins.

Hrmm…just to confirm what you mean here…are you thinking that a vendored plugin my have a different config name prefix to one that’s pulled in globally? It’s not something we discussed, but we’re keen to hear what problems you think it’d solve!


#18

From what I understood the plugin derived its name from the reference? Which is no longer controlled by the plugin author if the plugin is vendored (e.g. the folder could be renamed). Not major, I’m just trying to get my head around if this would be derived from plugin.yml or from the plugin reference?


#19

@zsims we use the plugin name for that, which is basically the basename of the plugin directory.


#20

I’ve got a PR ready for vendored plugins: https://github.com/buildkite/agent/pull/878