Hello,
I would like to filter test cases run by bktec using Playwright test tags, but I noticed when I set
BUILDKITE_TEST_ENGINE_TEST_CMD=“pnpm playwright test --grep ‘@sanity’”
I get multiple agents that run 0 tests and one agent that is running all of the tests. Is there a way to filter tests run by BKTEC using test tags such that test splitting works properly?
Hi @Benjamin ,
Are you able to share the exact step when running your test case with the env var BUILDKITE_TEST_ENGINE_TEST_CMD ? In our docs, we have noted that it is necessary to include --json --testLocationInResults --outputFile {{resultPath}} in the command. You can read more about it here.
Hope this helps.
Cheers!
I set the results path by adding a json reporter in Playwright config and setting the results path to a shared json file location in BUILDKITE_TEST_ENGINE_RESULT_PATH.
When I set my pipeline to use the BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN instead of test tags, I can see in Test Analytics that test splitting has analytics so it seems like this part is working correctly?
What I think is happening is I set the test command to something like
BUILDKITE_TEST_ENGINE_TEST_CMD="pnpm playwright test --grep '@sanity'"
and then when I run the bktec command, bktec populates {{testExamples}} with tests that do not match the test tag. This makes it so that when the test command runs on the test examples the flag --grep "@sanity" returns 0 tests for that partition.
Does this sound correct? If not then I may be doing something wrong. Otherwise I would ideally like to find a way to filter {{testExamples}} by Playwright tags so that this does not happen.
Hey @Benjamin 
It seems like the issue lies in how the tests are being filtered and split. When you set BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN to identify and process only the matching test files, test splitting works correctly, and you can see analytics for the splits. Could you confirm if this understanding is correct?
Now, instead of using BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN, it looks like you’re attempting to filter tests dynamically based on a tag or marker (e.g., @sanity) by including --grep '@sanity' in the BUILDKITE_TEST_ENGINE_TEST_CMD. which results in zero tests.
I can suggest adding a —list tag, It won’t directly solve this issue but can help debug by showing which tests match your filter (@sanity).
Also, if you can send the url of the build to support@buildkite.com, we can look into this further.
Hey @benmc,
That is exactly the limitation I am running into. My current solution is to use the grep command to get a test file pattern to provide to bktec and use the --grep flag a second time within the test command provided to bktec to do more granular filtering of which tests within a file get run.
This avoids the issue of agents with 0 tests being run but I am wondering if this will affect test splitting since execution data per file may be inconsistent and I am not necessarily executing every test in a file on each run. I will like my test run to the buildkite support email.
No worries, @Benjamin! We’ll keep an eye out for that email