Hi, I’m using API directly to upload test result data as JSON, however branch dropdown is never filled in with actual branch names (has only “all branches” selection), but in the UI under “Recent runs” I can see that the branch name is there. This makes it impossible to select specific branch. See following screenshot to get the idea
What could be a problem there?
Hey @afds
Thanks for reaching out!
The reason you are not able to see a branch in the switch branch drop down is because you need to set a default branch to filter by in the test-suite settings
https://buildkite.com/organizations/{org-name}/analytics/suites/{test-suite-name}/edit
You can also use https://buildkite.com/organizations/{org-name}/analytics/suites/{test-suite-name}/runs?period=7days&branch=main to filter the report by a specific branch or period
Hope this helps!
Cheers!
Stephanie
Hey @stephanie.atte,
Thanks for quick reply. I tried to set default branch to filter by in the test-suite settings, then I can see it in the dropdown, but the problem is that I can’t see other branches, only the one that I set in the settings.
Direct url works fine to show runs for other branches, though I would expect this drop down to be pre-filled with available branches so that I don’t have to manually enter branch name. Another problem is that this does not work for “Least reliable” tests section i.e. https://buildkite.com/organizations/{org-name}/analytics/suites/{test-suite-name}/tests?branch={any branch name that is not in the settings}&filter=reliability&period=7days returns empty
Hey @afds!
Thanks for the reply and additional info!
Are you able to provide any information around how these tests get uploaded? You mentioned the API, is that directly via curl, or is curl running in a Docker container, for example? Have you included the branch in the curl request as documented here?
I cannot be certain, but I have a feeling that if the branch isn’t passed then it may be inheriting the default branch for the suite.
Cheers!
It is internal Maven plugin, so just a java code that performs the upload, and yes it specifies run_env[branch] field in the request (and all other relevant fields, though run_env[CI] might be different one and has jenkins as upload happens from Jenkins).
I can see different branch names under “Recent runs”, the problem is that branch selection dropdown is not populated and other sections like “Least reliable test” does not show/group by other branch names.
If this helps, here is the example project
Interesting, I can see the same behaviour with Test Collector plugin as well, branches drop down never gets updated with new branches.
Hey @afds,
I reproduced this on my end as well. I was only able to switch between the default branch and “all branches”.
If no default is set then only “all branches” shows. This seems to be the intended behaviour with the assumption that most of the tests are probably gonna be on a specific branch.
Hope this helps!
Cheers!
Hey,
To me, this limits product usefulness. Documentation also vaguely mentions that it should be possible to compare results across all branches, but currently this is not possible.
Could you please escalate this question to Product/Engineering team?
Thank you.
Hey @afds!
Currently, the functionality allows you to compare your default branch to all branches, by this it means compare the branch you want to be the default, or “highest value” vs every other branch. I have raised a request with the Test Analytics team to enable the selection of individual branches from the dropdown.
Cheers
Hi @afds !
Michelle here, product manager for Buildkite Test Analytics. Thanks for reaching out with your question!
Right now, Test Analytics only officially supports selecting one branch (ie, a default branch) as a comparison against all branches. This then appears in the drop down and is an option to filter across all information in the product, as you describe.
The typing an existing branch into the URL does work as a work-around, but like you say, this doesn’t result in a page filter in the same way that the drop-down does. The type-in in the URL is a bit of a cheeky extra that we didn’t block because we felt it might be useful to folks, even if it doesn’t fully filter. It’s not a fully supported feature, however.
We are thinking about how we might extend filtering options in the future, though. Can you tell me more about what you’re trying to use the ability to filter on any branch for? Ideally, what are you after?
Cheers
Michelle
