I work at Cruise.
We have quite a sophisticated test annotation machinery that utilizes many html elements to hide details, format lists, format code, color the output etc.
I tracked this down to the rendering logic of buildkite.
There is something very peculiar going on with interactions between html that we upload and buildkite markdown parser.
For example it breaks our lists when there are markdown “header” strings like
====================
---------------------------
which we have quite a lot in the tests from pytest and bazel runner.
It would be very convenient and simplify our life if there would be a mode where we can turn-off all markdown logic and just provide the html we want.
This is the annotation we are trying to upload.
I did my best to escape all markdown special symbols, but it still renders incorrectly.
Note that this is just one example. It breaks very often in different ways, so I’d really prefer if there is a way (like a flag for the buildkite-agent annotate) to disable markdown altogether and provide plain html.
Otherwise we will chase down all this corner-cases and spend a lot of your and our time.
<p class="h2">:bazel: Opt-full Soft-Failure Summary</p>
<p class="h4">These tests failed in the build. This data will be used to determine which tests to quarantine during merges to develop.</p>
<p class="h3 pt2"><a href="/cruise/cruise/builds/99#008c0537-a223-4593-a7cb-7e9ea89a3c38">Test failure</a></p>
<details >
<summary>
<code>2 failures</code>
</summary>
<dl>
<dt>Reproduction</dt>
<dd>
<p>Run the command locally to reproduce the errors in this job</p>
<pre class="term"><code>bazel test --compilation_mode=opt -- test_target_1 test_target_2</code></pre>
</dd>
</dl>
<dl>
<dt>Failed Targets</dt>
<ul>
<li>
<details>
<summary>test_target_1</summary>
<dl>
<dt>Reproduction</dt>
<dd>
<p>Run the command locally to reproduce the errors in this target</p>
<pre class="term"><code>bazel test --compilation_mode=opt -- test_target_1</code></pre>
</dd>
</dl>
<dl>
<dt>Logs</dt>
<dd>
<a href="artifact://artifacts/opt-full/67d68413-a85b-4c32-8cc9-34c3c1c51a8a/build/testlogs/test_target_1/test.log" target="_blank">artifacts/opt-full/67d68413-a85b-4c32-8cc9-34c3c1c51a8a/build/testlogs/test_target_1/test.log</a>
</dd>
</dl>
<dl>
<dt>Details</dt>
<dd>
<pre class="term"><code>exec ${PAGER:-/usr/bin/less} "$0" || exit 1
<span class="term-fg33">Executing tests from test</span>
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
==================================================
Running command: "valgrind --exit-on-first-error=yes --gen-suppressions=all --num-callers=30 --child-silent-after-fork=yes --fullpath-after=/proc/self/cwd/ --suppressions=external/valgrind/default.supp --suppressions=tools/dynamic_analysis/valgrind_runtime_suppression.txt --memcheck:error-exitcode=40 --memcheck:leak-check=full --helgrind:error-exitcode=41 --run-libc-freeres=no --tool=memcheck --memcheck:track-origins=no --read-inline-info=no test"
==================================================
==21== Memcheck, a memory error detector
==21== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==21== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==21== Command: test
==21==
==21== Warning: set address range perms: large range [0x59cad000, 0x99088000) (defined)
==21== Warning: set address range perms: large range [0x99088000, 0xc5658000) (defined)
==21== Warning: set address range perms: large range [0xe39ab000, 0xfba61000) (defined)
{
<insert_a_suppression_name_here>
Memcheck:Cond
fun:_ZN2at4_ops10as_strided4callERKNS_6TensorEN3c108ArrayRefINS5_6SymIntEEES8_NS5_8optionalIS7_EE
}
==21==
==21== Exit program on first error (--exit-on-first-error=yes)
[CRUISE]: The Valgrind command failed.
To get more actionable logs, rerun the test using "--config=valgrind-verbose" instead.
This configuration will be slower, but it will improve the logs by using the options "--memcheck:track-origins=yes --read-inline-info=yes" which will add information about:
- where the uninitialized memory used was allocated (if any),
- calls to inline functions.
If the test is timing-out, please update its timeout attribute in the build file.
Remote action 6e9f1adcbd5782d5ce208faaacd58b75d203647880f7579b9932a77cf27925be/401 failed with action exit code 40 and gRPC status 0</code></pre>
</dd>
</dl>
</details>
</li>
<li>
<details>
<summary>test_target_2</summary>
<dl>
<dt>Reproduction</dt>
<dd>
<p>Run the command locally to reproduce the errors in this target</p>
<pre class="term"><code>bazel test --compilation_mode=opt -- test_target_2</code></pre>
</dd>
</dl>
<dl>
<dt>Logs</dt>
<dd>
<a href="artifact://artifacts/opt-full/67d68413-a85b-4c32-8cc9-34c3c1c51a8a/build/testlogs/test_target_2/test.log" target="_blank">artifacts/opt-full/67d68413-a85b-4c32-8cc9-34c3c1c51a8a/build/testlogs/test_target_2/test.log</a>
</dd>
</dl>
<dl>
<dt>Details</dt>
<dd>
<pre class="term"><code>exec ${PAGER:-/usr/bin/less} "$0" || exit 1
<span class="term-fg33">Executing tests from test</span>
–––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
==================================================
Running command: "valgrind --exit-on-first-error=yes --gen-suppressions=all --num-callers=30 --child-silent-after-fork=yes --fullpath-after=/proc/self/cwd/ --suppressions=external/valgrind/default.supp --suppressions=tools/dynamic_analysis/valgrind_runtime_suppression.txt --memcheck:error-exitcode=40 --memcheck:leak-check=full --helgrind:error-exitcode=41 --run-libc-freeres=no --tool=memcheck --memcheck:track-origins=no --read-inline-info=no test"
==================================================
==21== Memcheck, a memory error detector
==21== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==21== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==21== Command: test
==21==
==21== Warning: set address range perms: large range [0x59cad000, 0x99088000) (defined)
==21== Warning: set address range perms: large range [0x99088000, 0xc5658000) (defined)
==21== Warning: set address range perms: large range [0xe39ab000, 0xfba61000) (defined)
{
<insert_a_suppression_name_here>
Memcheck:Cond
fun:_ZN2at4_ops10as_strided4callERKNS_6TensorEN3c108ArrayRefINS5_6SymIntEEES8_NS5_8optionalIS7_EE
}
==21==
==21== Exit program on first error (--exit-on-first-error=yes)
[CRUISE]: The Valgrind command failed.
To get more actionable logs, rerun the test using "--config=valgrind-verbose" instead.
This configuration will be slower, but it will improve the logs by using the options "--memcheck:track-origins=yes --read-inline-info=yes" which will add information about:
- where the uninitialized memory used was allocated (if any),
- calls to inline functions.
If the test is timing-out, please update its timeout attribute in the build file.
Remote action 6e9f1adcbd5782d5ce208faaacd58b75d203647880f7579b9932a77cf27925be/401 failed with action exit code 40 and gRPC status 0</code></pre>
</dd>
</dl>
</details>
</li>
</ul>
</dl>
</details>
We keep hitting such problems and it’s just a whac-a-mole!
Here is a recent one: markdown breaks down and the <span> tags that suppose to be html show up as a text…
Hi! The product team was on leave, so they didn’t have the chance to take a look at this feedback, but we raised it again.
I’m not sure if this is something we can work on in the near future, but we’ll keep you posted as soon as we have news.
Hey @sergei.vorobev! I’m just checking out your request. This is great feedback for me to go through. We don’t have any work on annotations scheduled in our upcoming cycles, but I’ll let you know when we begin discovery and prioritisation. Stay tuned, I’ll likely reach out for some feedback from you and to understand your usage of annotaitons!
Thank you @eleanor ! Happy to provide any additional context / feedback.
I just had to workaround another issue today (when parts of the output were unintentionally converting to code blocks, because 4 spaces indent).
Separately, we are hitting the 1Mb limit for annotations fairly regularly, would be cool to be able to configure it or maybe just 10x it, unless there is some major reason not to do it.
