Support for test results

samuelkdavis_mcf · November 7, 2019, 8:24pm

TeamCity has really good test result support. It would be nice if buildkite could offer similar, as we’ll likely need a separate tool to track our test results and history.

The features I would like is:

A list of individual test results. Support for potential grouping - we use BDD, so feature/scenario/scenario with examples grouping support would be good. We want to see a visualization at a high level of what’s working and what’s not.

Also a history over time per test, and potentially grouping histories. Being able to tell if a feature is breaking often, or if there’s a race condition in the test itself that only occurs in 10% of runs is really handy to keeping the test suite from rotting.

We also use Extent to generate a html report for our tests. These UI’s could potentially be supported natively

moensch · November 7, 2019, 10:36pm

Do any of these tools support just XUnit input? We built a small buildkite plugin that runs as a post-command hook which simply expected a XUnit XML file on disk and uploads it to a given location for further processing.

We ended up building our own reporting frontend because we had very simple needs and no existing one - but I bet TeamCity has some feature that accepts XUnit as input and will just render that for you (via some API).

samuelkdavis_mcf · November 8, 2019, 1:00am

I believe TeamCity supported multiple formats, but last time I touched it was 2 years ago, can’t be too sure.

I’m running dotnetcore, so have the option of a few types I believe.

github.com

Microsoft/vstest-docs/blob/master/docs/report.md#available-test-loggers

# Reporting test results
Test discovery, execution results in a test run can be controlled with test
loggers. This document will cover details on installation, usage and authoring
of test loggers.


## Test loggers
A test logger is a test platform extension to control reporting of test results.
It can perform tasks when a test run message, individual test
results or the test run completion events are reported by the test platform.

You can author a test logger to print messages on the console, generate result
files of a specific reporting format, or even report results to various CI/CD
services. Default inputs to a test logger can be provided in the command line.

 Please refer to [this section](https://github.com/Microsoft/vstest-docs/blob/master/docs/report.md#create-a-test-logger) for instructions on creating a test logger and [todo]()
if you're interested in the architecture of a test logger.

### Available test loggers
| Scenario | Nuget Package | Source Repository |

This file has been truncated. show original

dotnet test --test-adapter-path:. --logger:“nunit;LogFilePath=test-result.xml”