Dynamically replace label of ~matrix~ parallel step

[ianwremmel]

I have a matrix step that runs a deploy command. The matrix node id determines which of several apps get deployed. Deploys can vary significantly between apps (some are cloud formation, some go to fly.io, etc). It’s kinda annoying trying to remember which node id goes with which app. Is there any way to replace the step label in the buildkite ui programmatically? If not, can this be a feature request?

[lizette]

Hi @ianwremmel ,

You can update a step’s label programmatically using the buildkite-agent command buildkite-agent step. You can find some examples in our docs on how to update a step’s label here - buildkite-agent step v3 | Buildkite Documentation . However, it can be tricky to apply this on a matrix expansion in a command step. However, you can run some tests to experiment if this can suit your need.

Otherwise, if you can share some more details on your scenario or an example. We might be able to assess if this does require a feature request.

Cheers!

[ianwremmel]

ok, so, this almost works. Part of the problem is probably that I asked the wrong question . I’m using parallelism, not matrix steps, in this case. It appears that only the first update works gets applied and it’s applied to all of the parallel jobs.

here’s my step definition:

.parallel_step: &parallel_step
  parallelism: 4

.retry: &retry
  automatic:
    - exit_status: -1 # Agent was lost
      limit: 2
    - exit_status: 143 # No such container
      limit: 2
    - exit_status: 255 # Forced agent shutdown
      limit: 2

.step: &step
  retry: *retry
  timeout_in_minutes: 10

steps
  - label: ':cloudformation: deploy'
    <<: [*step, *parallel_step]
    depends_on:
      - build
    command: .buildkite/scripts/deploy
    key: 'deploy'

and here’s the relevant part of the deploy script:

#!/usr/bin/env bash

set -euo pipefail

select_project () {
  readarray -t apps < <(list_all_apps)
  echo "${apps[$NODE_INDEX]}"
}

project="$(select_project)"

buildkite-agent step update "label" "$action $NODE_INDEX/$NODE_COUNT $project"

./scripts/sam "$action" aws "$project"

[lizette]

Hi @ianwremmel ,

Thanks for sending an example through and I had a further look into it. I think the reason you are seeing this behaviour is because the jobs generated from a parallel (or even matrix step), would have the same step ID. And the command buildkite-agent step update would update labels by the step ID. I can raise this as a feature request to be able to programmatically update a job’s label belonging to a parallel step.

Cheers!

[lizette]

Hi @ianwremmel ,

I just had a thought on this one. An alternative solution for you would be to use Build Annotations and annotate your build with the specific deployments ran on your parallel jobs. Let us know if this approach can work for you.

[ianwremmel]

Annotations don’t quite help because they’re so much farther away in the new UI. When I get to the build page, the first thing I see is the sidebar.

From this view, it’s unclear which node is deploying which app. Eventually, I’ll start get used to working it out from alphabetical order, but for now, it’s just more cognitive overhead.

Even after clicking on one of these (e.g., if one fails), the job screen no longer displays annotations (at least, as far as I’ve noticed), so I don’t think they’ll help in this case.

[Owen]

@ianwremmel I see what you mean! I just wanted to check, did you know that you can set a different tab to be the default view when you go to a build page in the new UI? So if it would help, you could set the Annotations tab to be the default. Just wanted to mention that as not everyone is aware of it.

[ianwremmel]

thanks @Owen, yea, I’m aware of that. I might try something with annotations, but I currently have a lot of them. The useful ones tend to get pushed off the first page, so it still takes a few clicks to get to them.

[lizette]

Hi @ianwremmel ,

Thanks for sharing your thoughts. I’ll note this as a feature request for the product team to assess. At the moment, the workaround that can work for you is by using annotations, and specifying priorities so you can customise which ones you want to be shown first.

Cheers!

[ianwremmel]

ooh, I didn’t know about priorities. that’ll definitely help, thanks!