Pipeline agent tag restrictions bleeding between pipelines

[phos]

Hi BK Team!

I have an issue with targeting specific agents in my pipelines.

There are several pipelines. Which pipelines are uploaded is determined based on the monorepo-diff#1.3.2 plugin. For example:

---
steps:
    - label: "Triggering builds"
      plugins:
          - chronotc/monorepo-diff#1.3.2:
                diff: "git diff --name-only HEAD~1"
                watch:
                    - path:
                          - "app1/"
                      config:
                          command: "buildkite-agent pipeline upload app1/.buildkite/app1.pipeline.yml"
                          agents: 
                              queue: default
                    - path:
                          - "app2/"
                      config:
                          command: "buildkite-agent pipeline upload app2/.buildkite/linter.pipeline.yml"
                          agents: 
                              queue: linter

Example entrypoint to my pipelines is above.

1. I need every step in app1.pipeline.yml to run on the same agent, on the same machine, so it has the following structure:

---
agents:
    name: "${BUILDKITE_AGENT_NAME}"

steps:
    - name: stage1
    - wait
    - name: stage2
    - wait
    - name: stage3
    - wait
    - name: stage4
    - wait

This works. stages 1-4 all run on buildkite-#, for example. From stage1>Timeline in the BK UI:

The issue manifests in the 2nd pipeline, linter.pipeline.yml, the linter queue is properly selected for the linter.pipeline.yml step itself, however, the steps within that pipeline are stuck waiting indefinitely to be picked up because the name=buildkite-7 tag restriction seems to have snuck into this pipeline, even though the agent queue is different (screenshot above shows buildkite-3, but I’ve just mixed those up here. The stuck build and the build that just ran are both targeting an agent with name=buildkite-7 (this 2nd one shouldn’t be doing that):

(the name of my agent serving this queue is buildkite-esxi)

The linter.pipeline.yml itself doesn’t have a name: set at the top level, yet it is being targeted anyway as you can see in the screenshot above. Here is the linter.pipeline.yml, which is only targeting the agent by queue:

---
agents:
    queue: "linter"

steps:
    - group: ":broom: app2 Linter"
      steps:
          - label: "check-js"
            command:
                - cd app2/
                - make check-js
          - label: "check-formatting"
            command:
                - cd app2/
                - make check-formatting
          - label: "check-profiles"
            command:
                - cd app2/
                - make check-profiles
          - label: "check-version"
            command:
                - cd app2/
                - make check-version
          - label: "check-metadata-version"
            command:
                - cd app2/
                - make check-metadata-version

Q: How do I get this name=buildkite-7 out of my linter.pipeline.yml step agent targeting, when I haven’t defined it?

Agents:

• 7 independent agents, each on their own machine. Tags from EC2. (queue=default) - names are buildkite-[1-7]
• 6 spawned agents, on 1 single machine. Tags local from buildkite-agent.cfg file, (queue=linter) - name is buildkite-esxi

This is my first time posting, so please let me know what other details are needed and I can send those! Thanks for your time BK peeps.

[james.s]

Hello @phos!

Welcome to the Buildkite Community and thanks very much for your question!

First/foremost, thank you for the incredibly detailed information - definitely helps with assessing the scene and what you’d like targeted.

I’ve just tested a similar setup as to your second configuration - though I’m using the absolute latest of the monorepo-diff pipeline (which as of typing is v2.5.9 - and used as monebag/monorepo-diff#v2.5.9 in a pipeline’s configuration):

agents:
  queue: uploader

steps:
    - label: "Triggering builds"
      plugins:
          - monebag/monorepo-diff#v2.5.9:
                diff: "git diff --name-only HEAD~1"
                watch:
                    - path:
                          - "app/"
                      config:
                          command: "buildkite-agent pipeline upload .buildkite/pipeline.yml"
                          agents: 
                              queue: one
                    - path:
                          - "beta/"
                      config:
                          command: "buildkite-agent pipeline upload/.buildkite/pipeline2.yml"
                          agents: 
                              queue: two

I’ve also configured pipeline.yml to look similar in nature to your linter.pipeline.yml- note the queue:one targeting for the 4 echo statements:

agents:
  queue: one

steps:
    - command: "echo 1"
    - command: "echo 2"
    - command: "echo 3"
    - command: "echo 4"

When a file is then changed within the app/ directory on the repository: the uploader agent runs the main step/plugin:

Agent one then uploads the resultant pipeline.yml1 and **also** sets the four resultant jons with queue:one` as their targets:

And the first echo job (similar for the other three):

It looks like you’ve got everything configured as I expect for linter.pipeline.yml - and it seems like its due to the older version of the plugin that you’re using. Would you be able to give it a go with the latest version?

Cheers

[phos]

Thank you for your quick response, James!

I will try out the new plugin now, and get back here shortly with the results.

Your reproduction was right with one minor exception, modify pipeline.yml to have a name: directive at the top of it under the agent specification, like this:

---
agents:
    name: "${BUILDKITE_AGENT_NAME}"
    queue: one

steps:
    pipeline1_step1
    pipeline1_step2
    pipeline1_step2
    pipeline1_step2

For example purposes let’s say that this name: "${BUILDKITE_AGENT_NAME}" renders to name: "box-1-name"

All those steps run fine,

Then, you should see pipeline2.yml get uploaded and ran by queue=two, but then none of the command steps execute because the Agent Targeting Rules for each of the pipeline2 steps are showing both name: "box-1-name" (from the first pipeline) AND queue=two - as these queues are on different machines and have different names.

brb shortly with the updated monodiff plugin attempt!

[james.s]

No worries!

Given its the second path in your definition adding the name key to agent targeting: focus was on replication there and getting that key out of targeting (which seemingly is version as from my own example, I’d only expect a queue:one target to come through on there.

Keep us posted

[phos]

New monodiff plugin works great, I think I’ll keep that!

However, diff detection doesn’t seem to be my issue. I’ll use your examples here. Using this same uploader pipeline:

agents:
  queue: uploader

steps:
    - label: "Triggering builds"
      plugins:
          - monebag/monorepo-diff#v2.5.9:
                diff: "git diff --name-only HEAD~1"
                watch:
                    - path:
                          - "app/"
                      config:
                          command: "buildkite-agent pipeline upload .buildkite/pipeline.yml"
                          agents: 
                              queue: one
                    - path:
                          - "beta/"
                      config:
                          command: "buildkite-agent pipeline upload/.buildkite/pipeline2.yml"
                          agents: 
                              queue: two

and pipeline.yml of this:

---
agents:
    name: "${BUILDKITE_AGENT_NAME}"
    queue: one

steps:
    - command: "pipeline 1 echo 1"
    - command: "pipeline 1 echo 2"
    - command: "pipeline 1 echo 3"
    - command: "pipeline 1 echo 4"

then a pipeline2.yml of this:

---
agents:
    queue: two

steps:
    - command: "pipeline 2 echo 1"
    - command: "pipeline 2 echo 2"
    - command: "pipeline 2 echo 3"
    - command: "pipeline 2 echo 4"

With agent serving queue=one on a different machine than the agent serving queue=two

I cannot get any of these 4 steps to run from the 2nd pipeline:

steps:
    - command: "pipeline 2 echo 1"
    - command: "pipeline 2 echo 2"
    - command: "pipeline 2 echo 3"
    - command: "pipeline 2 echo 4"

When I look at the agent targeting rules for command: pipeline 2 echo 1 that didn’t run for example, it contains the name=${BUILDKITE_AGENT_NAME} from the pipeline 1 in it, and spins indefinitely without a home because it’s got queue=two listed as well, and the agent serving queue=two doesn’t have name tag set in it like the agent from pipeline 1 does.

[james.s]

Cheers - thanks @phos!

Thats actually quite interesting - as for a similar (almost exact the same setup) on my side yields targeting just to agents with queue: two on a setup like below:

agents:
  queue: uploader

steps:
    - label: "Triggering builds"
      plugins:
          - monebag/monorepo-diff#v2.5.9:
                diff: "git diff --name-only HEAD~1"
                watch:
                    - path:
                          - "app/"
                      config:
                          command: "buildkite-agent pipeline upload .buildkite/pipeline.yml"
                          agents: 
                              queue: one
                    - path:
                          - "beta/"
                      config:
                          command: "buildkite-agent pipeline upload .buildkite/pipeline2.yml"
                          agents: 
                              queue: two

pipeline.yml (With the `BUILDKITE_AGENT_NAME:

agents:
  name: "${BUILDKITE_AGENT_NAME}"
  queue: "one"

steps:
    - command: "echo 1"
    - command: "echo 1"
    - command: "echo 1"
    - command: "echo 1"

pipeline2.yml

agents:
    queue: two

steps:
    - command: echo "pipe2 echo 1"

With a setup like this - changing a file within the beta directory yields a Triggered job with the uploader agent running the plugin (expected), the upload of pipeline2.yml uploading on an agent with queue=two (also expected):

And finally - its uploaded steps only are targeting agents also with queue=two (as we expect):

I think the potentiality of OS and agent versions might be at play - I’m running these through MacOS Ventura/Agent 3.54.0 - what OS/agent version are you running these on? I feel like this might be the way that the plugin itself is constructing the targeting on the host - as it seems your first pipeline’s targeting is somehow getting merged with the second (would raise an issue on GitHub for this!)

Cheers too - and thanks for all the information so far

[phos]

Hey James,

System is Ubuntu 22.04.2 LTS and my buildkite agent version was buildkite-agent version 3.56.0+x.f69df3866b3b34078df053b29bcd83d20e203c7e

It seems that top-level agent: queue targeting rules are not working properly. I’ve got this agent grabbing jobs from the default queue, even though it obviously is tagged with tag="queue=linter"

Something new though – I manually replaced my agent version with buildkite-agent version 3.53.0, build 6992 and the problem went away.

Potentially this feature has regressed in the latest version of the agent?

[jared]

Hey @phos,

Jared here stepping in for James from another timezone.

Caught myself up on this, and I reviewed the changelogs between Agent versions 3.53.0 and 3.56.0 and I don’t see this as being an intentional change.

I would recommend as James had suggested cracking an issue on our Agent repo here: Issues · buildkite/agent · GitHub so the team can take a look into what changed and why.