Test-collector plugin: Test Analytics returning error code

Test Analytics
1

This is what i’m running with:

steps:
  - label: "Test"
    command:
      - apt-get update
      - apt-get -y install ant
      - ant clean test
    plugins:
      - test-collector#v1.10.0:
          files: "**/test-reports/*.xml"
          format: "junit"

The tests run fine, but when it gets to the final section:

Running plugin test-collector pre-exit hook
$ /buildkite/plugins/github-com-buildkite-plugins-test-collector-buildkite-plugin-v1-10-0/hooks/pre-exit
Uploading '2' files matching '**/test-reports/*.xml'
Uploading './build/test-reports/TEST-HelloTest.xml'...
{
  "message": "Validation failed: Value must be a valid SHA-1 hash"
}
/buildkite/plugins/github-com-buildkite-plugins-test-collector-buildkite-plugin-v1-10-0/hooks/pre-exit: line 181: kill: (3061) - No such process
Uploading './build/test-reports/TEST-TestSortingAlgorithm.xml'...
{
  "message": "Validation failed: Value must be a valid SHA-1 hash"
}

Any ideas why it isn’t accepting this - am I missing an environment variable, or configuration on the test-analytics? I have already configured with the correct API Token, and that section of the ui shows ‘Waiting for test data’

2

Hello again @Rafe - just saw this and was going to mention the use of the test collector plugin, which it looks like you’ve come across and started to use, which is fantastic :+1:

That does look like its relative to the commit SHA not being set correctly and for what the TA API expects. Are you still running these tests using the Buildkite agent image? If its another image entirely, we’d expect something similar to this (passed in environment variables for the test collector).

Cheers :slightly_smiling_face:

3

I’m still using the Buildkite image, this is how I am running it on my local machine:

docker run \
  -e "BUILDKITE_ANALYTICS_TOKEN=#################" \
  -t \
  --name buildkite-agent-ubuntu \
  buildkite/agent:3-ubuntu start --token "#########################"

Should I be telling it which environment variable to look at for the SHA, or passing it in explicitly in the aforementioned yaml?

4

I can see references to setting run_env[commit_sha] and others - where should I add that to my yaml?

5

No worries - you shouldn’t need to have to pass the variables through to the container like that example: as where the builds are running is where the test collector plugin is also running (specifically, its pre-exit hook) which obtains the build’s environment variables - see specifically 113 where the commit SHA is being obtained from the build context - which you’d not need to explicitly set).

I’ve done a local test on my side (Mac host OS) and all works fine - I pass through an Agent token/Analytics token from my env - pre-set them:

docker run -it --rm --name buildkite-agent -e "BUILDKITE_ANALYTICS_TOKEN"=$BUILDKITE_ANALYTICS_TOKEN buildkite/agent:3-ubuntu start --token $BUILDKITE_AGENT_TOKEN --tags queue=docker

Just wondering - what VCS are you using? Was a case earlier with BitBucket and short SHAs that I wonder if its propagating through here.

6

Currently I’m pulling it down from a test repo that I have on github

7

That was the issue:

image

It was using HEAD and not converting it over to the actual SHA
When i replaced the word HEAD with an actual SHA value, it works.

That sounds suspiciously like a bug in the tool, as I can’t see it practical to always make that change in order to get it it work?

8

Glad to hear it was the SHA after all!

I’d point you in the direction of using the resolve-commit-after-checkout experiment when starting up the container. TheBUILDKITE_COMMIT initially is sent to the agent as an unresolved reference, in which it sets a meta data key with the resolved COMMIT_SHA after checkout - and used as BUILDKITE_COMMIT. Enabling that experiment will update said variable to the actual SHA after the code is checked out in the first step - and thus to the test collector plugin when its hook is invoked :+1:

9

Definitely making progress, I’ve got the agent running with that feature on. I tried a run this morning, with HEAD as the commit, but now the test collector isn’t accepting the results (with a new error):

test:
    [mkdir] Created dir: /buildkite/builds/1d88a736ad03-1/ETC/TESTTHINGHERE/build/test-reports
    [junit] Running HelloTest
    [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.127 sec
    [junit] Running TestSortingAlgorithm
    [junit] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec
BUILD FAILED
/buildkite/builds/1d88a736ad03-1/ETC/TESTTHINGHERE/build.xml:31: Tests failed
Total time: 1 second
🚨 Error: The command exited with status 1
user command error: exit status 1
Running plugin test-collector pre-exit hook
$ /buildkite/plugins/github-com-buildkite-plugins-test-collector-buildkite-plugin-v1-10-0/hooks/pre-exit
Uploading '2' files matching '**/test-reports/*.xml'
Uploading './build/test-reports/TEST-HelloTest.xml'...
{"id":"73174528-96ac-408b-aa52-d3d937fc07a6","run_id":"##########################","queued":3,"skipped":0,"errors":[],"run_url":"https://buildkite.com/organizations/ETC/analytics/suites/TESTTHINGHERE/runs/##########################"}/buildkite/plugins/github-com-buildkite-plugins-test-collector-buildkite-plugin-v1-10-0/hooks/pre-exit: line 181: kill: (2159) - No such process
Uploading './build/test-reports/TEST-TestSortingAlgorithm.xml'...
{"id":"##########################","run_id":"##########################","queued":2,"skipped":0,"errors":[],"run_url":"https://buildkite.com/organizations/ETC/analytics/suites/TESTTHINGHERE/runs/##########################"}
10

Hello again @Rafe!

Great to hear that - this seems to be relative to the first test report thats been uploaded (in particular, /TEST-HelloTest.xml. Does those resultant tests in that testcase contain the expected attributes (<classname> and <name> )?. It looks like there wasn’t an error on the second test report though - and seemingly has gone through and processed in your testsuite.

I think it might be worth looking at in more details - feel free to send us a email over at support@buildkite.com, and in particular with any sharable test reports that you can (those two from above would be ideal) and the build links to trace the run_ids to see what is occurring there.

Cheers :slightly_smiling_face:

11

Thanks - I have sent an email off and will see how that goes.