Single-step Dry-run from a desktop

Features Requests
1

I’d like the ability to run a single step in a build pipeline using local changes that are not yet pushed to a remote repository.

We use buildkite to run Hardware-In-the-Loop testing on microcontrollers situated in various labs though-out the world. It would be great to allow developers to use buildkite to test their changes on just one of these test rigs before pushing and submitting a PR.

2

Thanks for the suggestion and feedback! I think the Buildkite CLI might do the trick for you?

3

I think this might be close but there are some things missing:

  1. Documentation? I can get some of the commands to work but doing bk build create just ends with exit status 128 and --debug doesn’t provide any meaningful help.
  2. My request was to be able to use a given agent/queue from the cli. It would look something like: bk agent run --step=on-target-s32k-test --artifacts=./build/ where the CLI uploads the binaries under ./build to allow the build step to download them.

This would be helpful for pipelines where building locally was easy but running some of the more complex tests in the pipeline was difficult. Developers could get most of their work done using a local toolchain, getting code written and unit tests running, and then use this feature to validate that they hadn’t regressed any integration tests before they add the noise of a push and PR. Additionally this would reduce the cycle time for debugging these tests since the binaries delivered to the integration step would be built locally and the developer can short-circuit on just one set of tests without waiting for all the previous steps to complete.

Let me give you a made-up example: imagine a company that is building satellites. They might have subsystems broken out into test racks, say reaction wheels, that were used by a build pipeline to verify firmware changes for the vehicle. These rigs might cost upwards of $100K (USD) a piece and may even pose safety risks to be near them. As such there would be only a few and they would not be located near a developer’s desk. The feature I’m requesting would allow a developer in that company to build her binaries at her desk in Zurich and then initiate a test run on this reaction wheel rig in Brussels. Because she has skipped all other steps in the multiple pipelines the company has she’s able to get the test scheduled almost immediately and finds that her change has broken some required functionality within minutes. As we all know, the closer to the source that a software defect is found the less it costs to fix it.

4

Sorry about the documentation! bk cli is still young, and we could definitely improve things there. @lox is working on improving it.

We’re totally onboard with everything you described, that’s the idea of the bk local run command.

The bk build create command actually creates a build in Buildkite, but the bk local run runs a local agent just like you’ve described. Want to give it a go?

You should be able to run:

bk local run --filter=on-target-s32k-test

I think it uses step labels, and not step id's (https://buildkite.com/docs/pipelines/command-step#command-step-attributes)

5

I think I’m asking for precisely the opposite of this functionality?

bk build create --filter=on-target-s32k-test

but even if this was supported it wouldn’t work because my on-target tests pull artifacts built in the previous step. What I’m looking for is a way to run the previous step locally:

bk local run --filter=on-target-s32k-test

which totally works (:partyparrot:) but to then use the build artifacts from that local run in a special build in the pipeline. Something that might look like:

bk build create --filter=on-target-s32k-test --upload-artifacts="./build/*.hex"

This would create a special build (i.e. not grouped by the branch name in the list of builds like how PR builds work) that would be given the artifacts specified on the command-line where they match the artifacts property in the step yaml.

It’d also be really cool to find out why bk build create doesn’t work for me but I guess I should open a topic under help for that.

P.S. --filter does use the label which is weird. id would be more appropriate for this.

6

FYI: opened https://github.com/buildkite/cli/issues/63 to track this. I’m still not sure if this is supposed to work or not because the basic build create functionality is horked for me.

7

Okay, now that I got past issue #63 on github I see that the bk build create does not support --filter. So my feature request stands: I’d like bk build create to support --filter and I’d like special access tokens that allow select people to build without giving them full graphql and organization access. This would be similar to pull-requests in github but the changes to build would come from a developer’s local workstation.

Even more I’d like to be able to give out expiring tokens for this through Buildkite so that I didn’t have to email them to people. Here’s the user story:

Chloe in Shenzhen fixes a bug in libuavcan that she verifies on an ST Micro microcontroller she uses for her drone. She wants to submit the change upstream but she isn’t sure it will work on other microcontrollers. Chloe contacts Scott in Seattle and asks if he can test her changes in his device lab in Seattle. Scott enables this by creating a temporary token in Buildkite with access to a special pipeline setup for this purpose and then gives Chloe access through her buildkite account. Chloe copies the token from her account on the Buildkite website, updates her bk cli, and is able to run bk build create --pipeline=seattle-hil-lab to test local changes in minutes. After getting it to run in this pipeline she pushes the fix to her github fork and submits a PR.

This same workflow would be relevant for software simulation pipelines where a developer wants to verify a fix in a regression simulation suite before finalizing their work and submitting a full PR. The higher-level story is enabling faster iteration on hard-to-configure test infrastructure by enabling ad-hoc builds with narrow permissions.

Thanks for listening, sorry for the long ramble.