Pytest-split + github actions behavior

I was trying to split my pytest suite (currently triggered via pipeline + buildkite-agent-k8s) in 2 parts.

Previously, I’ve used pytest-split in GitHub actions to accomplish this:

jobs:
  build:
    strategy:
      matrix:
        split: [1,2,3,4]
    steps:
    - uses: actions/checkout@v3
      with:
        fetch-depth: 0  # OR "2" -> To retrieve the preceding commit.
    - name: Run 1/4 of the Tests
      run: |
	pip install pytest-split
        python -m pytest --splits 4 --group ${{ matrix.split }} --splitting-algorithm least_duration --store-durations tests/

What is the correct way to do it in build kite?

Hi @sshleifer ,

Here’s what the step would roughly be like in Buildkite:

steps:
  - name: "python test splitting"   
    commands:  
    - pip install pytest
    - pip install pytest-split
    - pytest -q projects/simplepytest/test_compare.py  --splits 3 --group {{matrix}} --splitting-algorithm least_duration
    matrix:
    - 1
    - 2
    - 3
    plugins:
      - docker#v5.12.0:
          image: "python:latest"          

With the above steps, I used our docker plugin to setup the test environment with pytest and pytest-split. I also used the build matrix configuration to expand the groups based on your example.

However, this does not have to be the case. You can read more about our Test Suites product and explore parallelising builds when configuring your step with parallel jobs.

In addition, when you wish to explore more about migrating more Github Actions to Buildkite Pipelines, you can explore our migration tool. Hope this helps.

Cheers!