Hey BuildKite Team, I’m using the Test Engine Runs API to get failed executions - Runs API | Buildkite Documentation. I can see that these executions match what I see in the buildkite UI when I look at the same run ID.
However, I really want to know the failed test count. There doesn’t seem to be an API to get the failed test count. And I can’t seem to get the same number if I dedupe test execution by location or test ID.
Could you please let me know what the easiest way to fetch failed test count would be?
At the moment there isn’t a single API field that returns “failed test count” for a run. The usual approach is to compute it from executions. To match the UI-style numbers, you generally want to ignore skipped and count unique failing tests, which you can do as count(distinct test_id) across the failed executions for that run.
If you are seeing a mismatch, one common cause is the failed-executions endpoint result cap/pagination, so make sure you are retrieving all failed executions before deduping.
Can you tell me more about what you have in mind re: failed tests? Do you mean unique tests that failed at least once (count distinct test_id), or total failed executions including retries (count failed executions)?
Got it. I’m counting unique failed tests and it looks like I’m getting more failed test count than what’s shown in the UI. Which is strange. I want to know the number of unique tests that failed at least once. I don’t care about retries.
Seems like the only way right now is to manipulate the failed executions result. Thanks!
I see. Can you reach out to support@buildkite.com with more details. That way, you can share build link and some other sensitive details with us and we can help you further on this.
,