We have been running parallel test within a single node using 8/12/16xL instances. However, the performance is not as good as distributed ones.
I would like to try knapsack + knapsack pro with distributed nodes.
We use elastic-ci-stack. The coudformation has 8/12XL nodes configured for autoscaling configurations.
Do I need to setup a new stack with a different configuration? I would like the current single very large node based pipeline to co-exist with the in-development distributed small nodes based builds.
How can I try another pipeline build with regular instances, but distributed?
I believe in this case it would be best to create a second Elastic CI Stack with smaller instances and a distinct queue. Something like queue: distributed, while keeping your existing large-node stack with its own queue e.g queue: parallel-large. In the pipeline configuration, use the queue: distributed for the experimental builds and use queue: parallel-large for the existing ones each targeting the correct agent pool. After this, configure each pipeline’s steps with appropriate parallelism (e.g 10-20 for distributed) and use Knapsack Pro Queue Mode for the distributed pipeline to dynamically balance tests.