Test failures showing up in Test Digest but not on jobs

Test Analytics
1

I have test failures that are showing up on “Test Digest”, with the correct “View job” links, but the tests don’t show up on the job “Tests” tab.

For instance here where the “View job” links will take you to the right place, like this and note the “Tests” tab exists but is empty.

They also show up correctly as failures here.

2

Perhaps one important detail to add. The results.json files are uploaded by another job, but we set the job id to the test job.

I wondered if it’s that the check for results on that job is being run before they’re uploaded from the upload job, but I think a job having the “Tests” tab at all means it knows there were results for that job, so I doubt it’s that.

3

Hi @IanButterworth :waving_hand:

Thanks for bringing this out to our attention. This seems to be odd. If the tests showed up correctly in the Test Suites’ failed tests, then there could be some broken links somewhere. I’m raising this with our engineering to have a further check and we will get back to you when we have more information.

Cheers!

1 Like
4

Thanks. In case it’s relevant here is how we upload the results files

5

Hi @IanButterworth

Thank you for providing this additional information. I have shared it with the relevant team for review. Please feel free to reach out if you have any further details to add or any questions.

6

Hi @IanButterworth, I’m an engineer with Test Engine and have been investigating this ticket today.

Thanks for raising this and for giving clarity around your upload process, we’ve found the issue and deployed a fix this afternoon, so you can see your failed tests by job now.

Please let us know if there are any other issues!

7

Hi Ian,

It turns out the issue did result to the results being uploaded from a different job to the one they came from. Specifically, that query to load the Tests tab was scoped to the start/end time of job, and could not see the test executions that were uploaded after the job’s end timestamp.

Would you mind sharing the motivation for splitting the uploading to Test Engine out to separate steps? I’m curious because there’s some improvements we could investigate relating to how we handle test execution timestamps. It’s likely some other customer pipelines will do a similar separation of steps/jobs, and it would be helpful for us to understand the use-case.

Cheers,
Paul

8

Thanks. Our upload jobs are signed and our test jobs aren’t. Because we’re OSS we don’t give the test runners access to private tokens to prevent misuse.

1 Like
9

To give a little bit more context, we use a buildkite plugin called cryptic to separate our build jobs into two categories; signed and unsigned. The signed jobs will treehash their inputs (e.g. buildkite pipeline .yaml files, shell scripts that will be run, etc…) and if they do not match a committed signature file, the agent will not decrypt certain secrets (environment variables, files, etc…).

Because we restrict access to the signing key, and because it’s onerous to sign things all the time, we only sign “sensitive” jobs, such as things that have access tokens (uploading to S3, uploading test results to buildkite, signing executables on macOS, etc…).

This is why we have our tests run in an unsigned job, but then the results are collated and uploaded in a signed job.

1 Like
10

Thanks for the detail @staticfloat :ok_hand:t3: Makes perfect sense.

It’s always fascinating to hear about the different ways people use Buildkite.