# Notebook for investigating build failures, especially flaky tests

**URL:** https://forum.buildkite.community/t/notebook-for-investigating-build-failures-especially-flaky-tests/1110
**Category:** General
**Created:** [June 15, 2020, 12:02am UTC](https://forum.buildkite.community/t/notebook-for-investigating-build-failures-especially-flaky-tests/1110 "2020-06-15T00:02:04Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![huon](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/huon/32/455_2.png) [@huon](https://forum.buildkite.community/u/huon)
#### Post date: [June 15, 2020, 12:02am UTC](https://forum.buildkite.community/t/notebook-for-investigating-build-failures-especially-flaky-tests/1110/1 "2020-06-15T00:02:04Z")

</div>

I’ve been investigating some flaky tests on one of the projects I worked on. I wanted to make sure I’d got them all, and what better way than getting a computer to check. We upload our tests results for every builds step as (JUnit) XML artifacts, so it wasn’t too hard to use the Buildkite API and [pybuildkite](https://github.com/pyasi/pybuildkite) to write a Jupyter notebook that examines each of them to find flaky tests in two ways:

- failures within a build that passed, meaning that a human retried some of the steps and it eventually worked, without the code changing
- multiple builds of a single commit, where some builds and some failed (this works best if the configuration between builds isn’t too different)

I’ve uploaded the notebook at: [https://gist.github.com/huonw/5b15172499251ce88ac42a6a926e6162](https://gist.github.com/huonw/5b15172499251ce88ac42a6a926e6162) including example output from [https://github.com/stellargraph/stellargraph](https://github.com/stellargraph/stellargraph). (It may need edits to work with other projects or variants of JUnit XML.)

To run on another project, it can be:

- downloaded via “Raw”: [https://gist.githubusercontent.com/huonw/5b15172499251ce88ac42a6a926e6162/raw/9b335f7a8b5f04e8a16f15bceaa63697065d41fa/flaky%20tests.ipynb](https://gist.githubusercontent.com/huonw/5b15172499251ce88ac42a6a926e6162/raw/9b335f7a8b5f04e8a16f15bceaa63697065d41fa/flaky%2520tests.ipynb)
- run online in Google Colab: [https://colab.research.google.com/gist/huonw/5b15172499251ce88ac42a6a926e6162/flaky-tests.ipynb](https://colab.research.google.com/gist/huonw/5b15172499251ce88ac42a6a926e6162/flaky-tests.ipynb)

Hope this helps someone!