Support testsets and tests with no duration

Test Analytics
1

The julia Test.jl stdlib framework has the following type of common structure

@testset "Package" begin
    # some setup code to produce x. Any errors are reported as errors in the testset
    @test x == 1 # test some variable value
    # more setup code to produce y
    @test y == 2
    @testset "module" begin
        # more setup code to produce z
        @test z == 3
    end
end

With this model, testsets have meaningful duration, but tests do not, as they’re often just checking boolean comparisons.

There’s also so many of them that it would be inefficient to record their duration, but that’s a lesser point.

It would be good for the buildkite test engine to:

  1. Support grouping results into testsets and have those testsets have duration info
  2. Support results with no duration. You can set the duration to 0.0 but it reports in the UI as 0 us and contributes to headers like total test suite duration, which is not representative.
2

Hi @IanButterworth ,

Thanks for the above feedback. I’ll raise them to our product team!

Cheers!

3

Thanks. For now we have worked around this by making all results 0s, but adding a single “unknown” result for each top-level group, with the real duration. We opted to not have them marked as pass/fail to avoid them showing up as specific failures on the jobs, as it’d just be noise ontop of the actual failing results.

4

Appreciate you sharing the approach; it gives us a good reference point as we think about better support for test grouping and duration reporting.