Can I assign a CI variable on the parallelism value?

General
1

Let’s say I have this step on the Build steps:

steps:
  - label: 'Run tests'
    commands: 'make run_tests'
    parallelism: 4

Can I make the value (e.g. 4) on parallelism as a variable like ${PARALLEL_RUN}?

so it looks like something like this:

env:
  PARALLEL_RUN: 4

steps:
  - label: 'Run tests'
    commands: 'make run_tests'
    parallelism: ${PARALLEL_RUN}

Ultimately, this pipeline is a triggered pipeline from other repositories so I would like to have more flexibility in terms of running the jobs in parallel depending on the codebase’s pipeline running.

2

Hi @ebanster

No, we cannot use variable interpolation for parallelism and only option here is to use dynamic pipeline where you can use environment variable to define the parallelism for the pipeline definition which will be uploaded using the dynamic pipeline process.

1 Like
3

Thanks @suma can you show me an example on how this dynamic pipeline can be done?

4

Hey @ebanster!

You can check our docs that have a complete explanation of how dynamic pipeline works Defining Your Pipeline Steps | Buildkite Documentation, and you can check a working example here GitHub - buildkite/dynamic-pipeline-example: An example of how to generate dynamic build pipelines in Buildkite

Best!

1 Like
5

@ebanster
you could use YAML anchor/alias instead
here is an example

global_vars:
  - &PARALLEL_RUN
  	4

steps:
  - label: 'Run tests'
    commands: 'make run_tests'
    parallelism: <<: *PARALLEL_RUN
1 Like
6

I transferred the CI steps into the yaml file instead but I still get an error on the yaml file that its expecting an integer.
On my pipeline.yaml file, both these gives me the ‘integer’ error:

steps:
  - label: 'Run tests'
    commands: 'make run_tests'
    parallelism: ${PARALLEL_RUN} # integer value tried to be passed from CI variable

global_vars:
  - &PARALLEL_RUN
    4

steps:
  - label: 'Run tests'
    commands: 'make run_tests'
    parallelism: <<: *PARALLEL_RUN
7

Hi @ebanster,

As mentioned in the earlier reply, we cannot use variable interpolation for parallelism. Therefore, the suggested use of YAML anchor/alias will still give you the same errors.

Posting back here the suggestion earlier.

Cheers,
Lizette

8

Hi @lizette Thanks. Would you know if it only recognises a shell script?
Instead of using a shell script (pipeline.sh), we are using a Makefile. I tried to include it with the following:

run_tests: .env
	echo "  - label: 'Run tests'"
	echo "    command: 'make run_tests'"
	echo "    parallelism: ${BUILDKITE_PARALLELISM}"
.PHONY: run_tests

Then on my pipeline.yml I have:

steps:
  - label: 'Run tests'
    command: 'make run_tests'
9

Hi @ebanster,

The example you provided will just echo the steps into the terminal. The steps won’t be executed by the Buildkite Agent as you would expect. I believe the only way to do this is to create a YAML file, pass it on to the agent and use the buildkite-agent pipeline upload command.

Here’s a sample snippet:

In your pipeline steps:

env:
   PARALLEL_RUN: 3

steps:
- label: ":Run steps:"
  command: pipeline.sh | buildkite-agent pipeline upload

In your pipeline.sh

cat <<YAML
steps:
- command: echo This is a dynamic step
  label: "Test %n"
  parallelism: ${PARALLEL_RUN}
YAML

Hope this helps!

1 Like
10

Thanks @lizette this worked!