Maybe there already is a way to achieve this, but as far as I can tell, the buildkite-agent pipeline upload command only takes a single file as argument.
I’d like to be able to upload multiple files at the same time so they get added simultaneously.
Because uploading them one at a time inserts them in reverse order (per documentation), I am forced to upload the last pipeline first, which means steps might start running that I really want to only execute after an earlier pipeline file completes (with wait commands at the end of those earlier pipelines to ensure this).
Is there a non-obvious solution to this?
Or, how hard would it be to add the ability to upload multiple files in a single command and have all the steps get added at the same time in a deterministic order?
Secondly, when uploading more steps, they inherit the created timestamp of the uploading job which ties into how they are scheduled. If you add a wait step before/after the uploaded steps, they will wait for the step doing the upload to completely finish before proceeding. You’ll still need to upload in reverse order, but no new steps will be started until all the uploading has completed and that step has finished.
And lastly, are you able to combine these separate files into one file before you upload them? So that you can just upload the one file?
I am aware of that plugin (although I don’t recognize the org/user of this specific version), and we are actively using this already for triggering pipelines based on independent portions of our mono-repo. We certainly do not run a single pipeline; there are over 100.
However, within a single independent system in the mono-repo, there are usually multiple resources and services defined that do not all change with every PR. We have implemented some custom magic to allow multiple pipeline.yml files to be detected within a single system and uploading only the ones that are at the top of a code-tree that has diffs in it. Effectively a custom monorepo-diff derivative that operates within a single pipeline to complement the top-level fan-out across our many pipelines corresponding to independent systems.
The problem with “adding a wait step at the top” is that it needlessly serializes all of our compound build. If a System contains three Services that need to be built, I’d like all three to build at the same time, since they are still nominally independent, even though they might collectively require integration tests once they are all built.
I did consider combining all the files into a single file, but just concatenating the steps: seemed dangerously subject to potential indentation or formatting issues I cannot fully oversee. I guess I could use a YAML parser, parse all the files, and then combine the steps in the parsed object and re-serialize them… but that seems excessively complicated for a use-case where I basically just want to “add the steps from these N buildkite files in a single upload, please”. Since the agent/server already implement all the parsing and combining logic for steps, it seems safer to leverage the existing smarts than to try and jerry-rig my own.
I still think this is doable and hopefully fairly trivial. I think the hard part is going to be trying to explain it here!
The first and most simplest solution I think might still be to combine the files prior to upload. You have reservations about that but I don’t think it would be that difficult or dangerous. You could use yq to help you here. I’ve found this doc explaining how to merge yaml files and done a test locally and it just worked https://mikefarah.gitbook.io/yq/operators/multiply-merge?q=merge
And back to the other option to upload multiple files; I think there are still a few options:
First, It sort of sounds like you could do with another layer of pipelines that get triggered, 1 for each of your “Services” you mention. So instead of trying to upload N files with M steps each that are working with a particular Service, you would instead upload N files, each with a single trigger step in them. Those trigger another pipeline that houses the M steps. I think that would help with the problem of serialisation/parallelisation here. So overall, you’d have the first pipeline that uses the monorepo-diff plugin to trigger another pipeline, this pipeline then further triggers others to increase the fan out. You might need to take this concept and apply it a little more to your exact use-case though but see what you think.
Second, we have 2 architectures of pipelines. The oldest one is using purely wait steps and is very sequential and simple. But we have a DAG mode version now where the build can fork and merge and build very complex pipelines with. I think this second mode would help you. When a build is in DAG mode, steps that are uploaded implicitly depend on the uploading step - so they don’t start until that has finished. This goes back to my first recommendation about wait steps, but won’t introduce the serialisation between each “Service” because all steps will be dependent on the uploading step.
Builds can start out in non-dag mode and enter DAG mode as soon as a step is uploaded with depends_on. You can also force DAG mode by adding dag: true at the top-level of a yaml upload.
You can run further with this by adding explicit dependencies to all the steps as well. But one thing to keep in mind is that dependencies need to exist at upload time which might not work if you have to upload the last steps first that might need to depend on early steps which are not yet uploaded. The good news is that, with DAG mode, and with a fully defined graph of dependencies, the order of upload doesnt really matter.
From here, its possible this is going to get really complex. So the best thing to do would be to have a play around with it and then come back with any questions if thats the approach you want to take. I don’t think I’d be able to provide all information you’d need without you having some experience (assuming you dont already) with this sort of setup.
Lastly, I’ll share that if you append /dag to your build URL you’ll see a visualisation of the build graph. It’s has a few issues still around some steps types but should go pretty far to help you here. It’s just not quite production ready so we don’t have a specific button in the UI to take you there. Basically, use with caution.
Thanks, yq looks interesting; I’ve used jq extensively. It’ll take some effort to update tooling so yq is available in the context where it’ll be needed, so that might not be the most obvious solution for us.
We were wanting to avoid triggering further pipelines; we already have 100s of pipelines, and the latency/overhead every additional trigger introduces is not ideal.
However, the DAG mode sounds interesting – is this behaviour documented anywhere? I didn’t realize once it’s in this mode completion of current step is guaranteed.
The DAG page is currently not yet available for public release since it still needs some product design details. But if you have more questions around it as you explore, please don’t hesitate to reach out here.
