steps:
- group: ":gear: test"
steps:
- command: echo "1"
- group: ":gear: test"
steps:
- command: echo "2"
I would expect this yaml to merge the two groups since they have the same group name with two commands:
instead i get
two groups with 1 command each and seemingly the same group name:
You also cannot add identical keys under the identical group names to group them, it fails with not allowing the same key twice
Hi @chanukov!
Thanks for checking with us! Group steps with the same name within a pipeline will not be grouped together. So what you are seeing is the expected behavior.
Group merging happens only when you upload a pipeline that has a group with the same name as the group that uploaded this pipeline.
Example:
Parent Pipeline:
steps:
- group: "Setup"
steps:
- commands:
- "buildkite-agent pipeline upload" //this uploads the below child pipeline
- echo "start"
Child Pipeline:
steps:
- group: "Setup"
steps:
- command: "docker build"
In this case, the above 2 groups will be merged as the group name in the child pipeline is the same as the group name that uploaded this pipeline.
Here is the doc reference of group merging. Hope this clarifies!
Best,
Nithya
Thank you,
Is there a plan to possibly add merging groups within a pipeline.
I see no logical situation where i would want two identically named groups.
I currently have a python script which takes my pipelines and generates a merged one and uploads it, but I really would love to see this working out of the box.
As a side note the docs imply that Group merging does work the way i thought, you may want to call it out directly that groups in one pipeline with the same name will not merge.
Thanks for checking back with us @chanukov!
I would like to know more about the scenario where you would need to have your pipeline with multiple group-step of the same names but merge them on the BK UI. I wonder why not group them in the pipeline.yml itself! Kindly clarify.
As a side note the docs imply that Group merging does work the way I thought,
Just submitted a PR buildkite/docs: Pull Request 1336 to the doc this morning :)
you may want to call it out directly that groups in one pipeline with the same name will not merge.
Good point! I will update the doc again.
Best,
Nithya
We have a tool which takes multiple yaml files, and merges them before uploading one yaml to BK based on changed bazel targets with multiple steps, to allow us to dynamically decide which tests should run dynamically.
I want to add some of these steps into groups, and have them display nicer and grouped, I have to instead of appending them as I would a step to the file add them inside the group. As stated, I am able to merge the groups myself, but just think it would be simpler to have it work within a pipeline.
I’d also like to request this feature.
In our usecase, we have a rather complicated buildkite pipeline setup for the Julia Language project where jobs which are logically grouped together don’t always share a similar “pipeline lineage”.
As an example, in our CI pipeline we will take a single version of Julia, build it on N platforms, test it in M scenarios (each platform will have at least one test scenario, but some may have more than one, such as alternative debug modes on linux x86_64), and then finally we’ll have N upload jobs to send the artifacts up to S3 if they pass all their tests.
All of these pipelines are being dynamically templated/generated, and so the “lineage” of the pipeline is complex. For instance, it’s not so simple for us to have the “packaging” steps be emitted by the same step that emits the “uploading” steps, so if we wanted to have groups such as “Linux”, “macOS”, “Windows”, “FreeBSD”, etc…, it would force us to split up our “meta” YAML files into many more files than we otherwise would prefer. (E.g. create sub-trees for each “group”, rather than having a single templating “package launch” job that launches all of the packaging jobs, and a single templating “tester launch” job that launches all of the testing jobs, etc…).
Additionally, we wanted to use group steps to provide some flexibility in depends_on, but the lineage constraint makes that difficult as well. To give a concrete example:
We build one x86_64-linux-gnu tarball, then send this off to two testers (one multithreaded, one singlethreaded, to test both code paths). We also spawn an upload job that pushes any build that passes all of its tests up to S3. Because all of these jobs are templated across many different OS’s and architectures, it would be very nice if the upload job could use something like depends_on: "test_group_x86_64-linux-gnu". However, because the testers are being templated, this means that we would need to have the testers be templated from a job that is dedicated to launching only the x86_64-linux-gnu testers (So that the launching job belongs to the same group).
In my usecases, I can’t imagine ever not wanting groups with the same name to be merged.
Hi @staticfloat
Welcome to Buildkite community and sharing your question with us
Let me reiterate the question to make sure I understand your usecase
You have a pipeline lets say pipelineA which runs builds of Julia language where each step in pipeline triggers build on N platforms where it runs M test scenarios per platform.
Now you want group merging such that for each pipeline upload for each platform it merges into a group in UI which is not working for you.
Is this understanding correct ? If so for security reasons please can you send details of the build link to support@buildkite.com so that we can look into this further.
Once again welcome to the community. Thank you
Regards,
Suma
Just to add to this i have had multiple developers ask me why group merging is not working how they imagine it should.
I am not sure I can envision any scenario where I or anyone would want to have tow groups with identical label names in UI. I can convert this to a Feature Req if that makes it easier because what i would like and i think what staticfloat wants is for it to merge any identical group names into one group. The fact that I can have two groups with the same name in UI actually makes it less useful than simply not having any groups as you literally have no idea which thing is in which group.
In my initial example I do not know which test has echo 1 or which is echo 2. so to find the test with echo 2 I might have to open both groups. This is true for any scenario where you have X identical name groups, instead of grouping things together it adds a layer of obfuscation and you end up not knowing where the thing you are looking for is.
i think what staticfloat wants is for it to merge any identical group names into one group.
That is correct.
Now you want group merging such that for each pipeline upload for each platform it merges into a group in UI which is not working for you.
In particular, I want builds with the same group name, launched from any pipeline lineage, to be merged together. This is because for complicated pipeline setups, it may be difficult to have the parent job share the same group as the child job.
The example I was trying to give above, (where we have pipeline templating being used to launch all build jobs, the another templating job launching all test jobs, then another templating job launching all upload jobs) is a good showcase where the parent-child relationship of jobs does not match the desired grouping.
To make this extremely concrete, imagine I have the following pipelines that launch the following child pipelines. I’m omitting any dependencies between steps here to make things simpler:
build_linux-x86_64build_linux-i686build_linux-aarch64build_windows-x86_64build_windows-i686build_macos-x86_64build_macos-aarch64test_linux-x86_64test_linux-i686test_linux-aarch64test_windows-x86_64test_windows-i686test_macos-x86_64test_macos-aarch64upload_linux-x86_64upload_linux-i686upload_linux-aarch64upload_windows-x86_64upload_windows-i686upload_macos-x86_64upload_macos-aarch64The parent-child requirement that the group step currently imposes means that I cannot have build_linux-*, test_linux-* and upload_linux-* all belong to the same group. Really, what I want is to simply have the build_linux-x86_64 job declare itself as part of the linux group, and then have the test_linux-x86_64 job also declare itself as part of the linux group, and have them be merged together. There is no situation that I can think of where I would want these two linux groups to be shown as separate in the buildkite GUI.
Also, if you think this is a lot of jobs, note that this is not our full list of supported architectures, and it doesn’t even get into the complications of multiple testing scenarios for different platforms, and it doesn’t include our miscellaneous jobs such as whitespace checks, etc… So tools to efficiently group jobs and clean up the GUI from having an explosion of jobs all over the place are much desired!
Hey @staticfloat
Thank you for sharing more details on this use case. While I am checking further on this, I wanted to see if you had a chance to check build matrix which lets you simplify complex build configurations by expanding a step template and array of matrix elements into multiple jobs.
Please let us know if you had a chance to look at this and if so what is your thoughts on this with respect to your usecase.
Hey, build matrices look really nice! In simpler cases, we could use this to great effect, thank you for pointing it out.
In this case, we have multiple dimensions of data but they shouldn’t be combinatorially expanded. As an example, in this file the highlighted line defines multiple pieces of information for a single build, and the only way I know of to use this in a matrix attribute is to have essentially an empty matrix and a bunch of adjustment/with blocks below it. But since we will have more than 10 entries in this matrix eventually, we’ll cross the matrix adjustment limit.
I don’t want to side-track this topic however; even with a matrix, we wouldn’t be able to, for instance, have all Linux builds launched from one matrix then be grouped with all the Linux tests launched from another matrix. That is the core feature that is under discussion here, and since we’ve already built a bash-based templating system for these .arches files in our repository, we’re happy to use that and not use build matrices in this case.
Hi,
As you saw Build Matrix feature is still not publicly available so I quickly wanted to check if this is something that can help with your use case and if the matrix adjustment limits will be a blocker for your use case.
Thank you for confirming that this would not help with your use case. I will go ahead and open a feature request with our product team and what will be helpful is names of your pipeline or current builds so I can share those as reference for our product team.
Sorry for the late response. You can see our pipeline source here, with builds here. Note that this repository is our “testing” repository where we iterate on changes to the buildkite configuration, but that this configuration is used by the main julia repository and those builds show up here, here and here.
Hi @staticfloat
Thank you for sharing the details. I will open a feature request with our product along with reference pipelines and builds. I just want to let you know that we will not have an ETA on the feature request as it is currently not part of the roadmap but our team will look into it.
We also would love to have this feature. We won’t ever know in advance the groups we will have in a pipeline, so having groups uploaded separately merge together is key for us.
FWIW, this is also how I would expect and want this to work.
Hi there, is there any chance this can make it on the roadmap? This is still a suboptimal user experience.
Hi @john.soo,
Thanks for the message!
At the moment, this feature isn’t on our roadmap. However, I’ll bring it up with the team and get their thoughts.
Cheers!