Concurency and parallelism

General
1

I have a step like this:

  • name: ‘E2E’
    concurrency: 1
    concurrency_group: “e2e”

So there is only 1 e2e tests run at a time for all the pipeline.

Recently, I implemented the parallelism to make the tests run faster, then it became:

  • name: ‘E2E’
    parallelism: 4
    concurrency: 1
    concurrency_group: “e2e”

This step will create 4 sub threads (which is great!!) to run parallelly, but still, only 1 thread is run at a time. So I update it a bit:

  • name: ‘E2E’
    parallelism: 4
    concurrency: 4
    concurrency_group: “e2e”

Now the 4 sub threads were created and ran parallelly, but it broke my first criteria: only 1 e2e tests run at a time for all the pipeline.

For example:

  • pipeline 1: 4 sub-threads, running 2, finished 2
  • pipeline 2: 4 sub-threads, running 2, queuing 2 (because concurrency is 4)

I think concurrency should be restricted to the main step, not the parallel thread.
Do you have any advise for this case?

Thank you.

2

Hi there,

That’s a really interesting use case. To understand it better it would be great if we could see it in action. Would you mind sending an email to Support@buildkite.com with a build URL so that we can dig into it?

Nik