BUILDKITE_JOB_ID, parallelism and docker-compose, always the same value

Docker
1

Hi,

When running a command in parallel, I can well see distinct BUILDKITE_JOB_ID value for each parallel job.
But, when using the docker-compose plugin, the given value for BUILDKITE_JOB_ID is always the same, and does not match any existing BUILDKITE_JOB_ID values.

So when using the junit annotate plugin, obviously the link to the linked job is wrong. Am I missing something?

Regards,

- label: ":rspec: Run tests"
    parallelism: 150
    command: |
      bundle exec rake knapsack:rspec["--format progress --format RspecJunitFormatter --out reports/junit/junit-buildkite-job-$BUILDKITE_JOB_ID.xml]
    artifact_paths:
      - reports/**/*
    plugins:
      - docker-compose#v3.9.0:
          config: build/docker-compose.yml
          shell: ["/bin/bash", "-e", "-c"]
          workdir: /usr/src/app
          run: app
          volumes:
            - "./reports:/usr/src/app/reports"
          env:
            - BUILDKITE
            - BUILDKITE_BUILD_NUMBER
            - BUILDKITE_BUILD_URL
            - BUILDKITE_JOB_ID
            - BUILDKITE_BRANCH
            - BUILDKITE_COMMIT
            - BUILDKITE_MESSAGE
            - BUILDKITE_PARALLEL_JOB
            - BUILDKITE_PARALLEL_JOB_COUNT
2

Hi @tetienne

I think the issue here is on line 4 (in the command attribute). You are using $BUILDKITE_JOB_ID which is evaluated on the agent when this step is uploaded to our API, so the job ID you are seeing is that of the job which uploaded this parallel command step.

To get around this, you need to use runtime interpolation: Environment Variables | Buildkite Documentation with $$BUILDKITE_JOB_ID. That will evaluate the job ID within the docker container at runtime.

Give that a shot and see how it goes. If that doesn’t solve the problem, it might be best to email support@buildkite.com with a link to your build so we can take a deeper look, unless you are happy to post the link in public here

3

Hi @jarryd,

Thanks you, your suggest was correct. I already tried before posting, but cancelled my pipeline when I saw in the log $BUILDKITE_JOB_ID. I’ve retried and wait the end of the jobs, and the variable was well replaced.

4

Excellent! :+1: