Remove targeting rules

General
1

I have a pipeline that runs a series of quick tests in dedicated LXD containers;

At the end of this pipeline, the second-to-last step launches a virtual machine with parameters queue=$BUILDKITE_BUILD_NUMBER set via AWS tags. The buildkite agent launches successfully in this ephemeral virtual machine.

The final step dynamically creates a pipeline like this:

    # meant to be piped to buildkite-agent
    echo "
steps:
  - label: \"k3s integration test\"
    commands:
      - ./run-k3s-tests.sh
    agents:
      queue: $BUILDKITE_BUILD_NUMBER

The new steps are added successfully, but the agent targeting from the previous steps is added to, not replaced. So the step hangs.

image

I suppose I could add an AWS tag container_image: ubuntu-2110-ci to the ephemeral VM. But is there a better way?

What I’d like is to continue this pipeline in a different machine, spawned on command. I’ve already pre-baked an AMI with buildkite that accepts tags from EC2 tags.

Is there a better way to accomplish this?

2

Hi @coleman

Firstly welcome to Buildkite community and thank you for reaching out to us with your questions.

Reviewing your pipeline definition, we can see that you have agent targeting rule defined at pipeline level rather than at a step level. Because of this the agent target rule of “container_image: ubuntu-2110-ci” is also getting appended to this dynamic pipeline step as well.

Any reason for defining agent target rule “container_image: ubuntu-2110-ci” at pipeline level in earlier step ? if no specific reason then, please change that to step level like you have for target rule “queue: $BUILDKITE_BUILD_NUMBER” to avoid appending of target rules from previous steps

3

The reason for container_image is to select the appropriate build environment. All our pipelines have this attribute set to one of a few different OS versions.

4

@coleman Yes, but you can define the agent target rule at the step level instead of pipeline level so that way when you do dynamic pipeline with a step that has agent target rule then earlier steps agent target rules wont get appended.

5

This was the answer I was looking for: adding targeting at the step level, which isn’t something we’ve needed to do until now.

image

That changed removed the “global” targeting, and my dynamically generated step doesn’t set global targeting either. Instead, we set a dynamically generated queue attribute.

    # meant to be piped to buildkite-agent
    echo "

steps:
  - label: \"k3s integration test\"
    commands:
      - echo \"HELLO WORLD\"
      - k3s kubectl get nodes
    agents:
      queue: $BUILDKITE_BUILD_NUMBER
"
6

@coleman Yes, that is what I was also referring to when I was saying setting agent target rule at step level instead of pipeline level so next steps will not get target rules appended.

Happy to see that it solved the issue for you. Thank you again for being part of Buildkite community.

1 Like
7

Agent targeting inheritance is still causing me problems. The previous solution has been working for months, but now I’m extending the pattern to another pipeline and no joy.

I run this function

# Launch a K3s server with a waiting buildkite agent, then generate the
# integration test steps for that specific agent.
cmd_generate_integration_tests() {
  _launch_cluster
  cue export \
    -t build_number=$BUILDKITE_BUILD_NUMBER \
    -t pipeline_slug=$BUILDKITE_PIPELINE_SLUG \
    .buildkite/generate_integration_tests.cue | buildkite-agent pipeline upload
}

It uses the cue tool to generate json (steps) to stdout.

It runs in the context of this step

  - label: "Generate integration tests"
    depends_on: build
    commands:
      - ./scripts/cibuild generate_integration_tests
    agents:
      container_image: ubuntu-2204-ci
      queue: default

In our current setup, I must specify container_image, but because I’ve generated additional steps, the resulting agent targeting includes container_image, even though I explicitly set an agents: directive in the generated steps.

It’s the same story, basically. But the cue dependency forces my hand here, I have to specify container_image: ubuntu-2204-ci to run my step in an environment where the cue cli is guaranteed to be there.

It would probably be more convenient if we used queue to specify the build environment, because that’s always implicitly present anyway, but for now I’m going to work around it by tagging my ephemeral ec2 with container_image.

What I’d love to see is a special case for buildkite-agent pipeline upload at the very least, to not inherit their parent step’s tags, at the very least. Maybe via a flag or something.

In the meantime, I’m adding this advice in my CUE file.

#agents: {
    // run `cue injection help` to see how to set data from the command line
    buildkite_build_number:  string @tag(build_number)
    buildkite_pipeline_slug: string @tag(pipeline_slug)

    // Generated steps inherit their parent's agent targeting, unfortunately.
    // We have to explicitly set these extra values here, and ensure that
    // the EC2 we are targeting as them set as tags, as well.
    queue:           "default"
    container_image: "none"
}

#step: {
    depends_on: "build"
    agents:     #agents
    commands: [...string]
}
8

Hey @coleman!

It sounds like you’re looking for a flag like --no-parent-tags to be available to the buildkite-agent command which would clear the tags of a parent?

Without the full pipeline it’s a bit difficult to understand the full context of how the CI image is being used/built. Have you considered using Docker and the docker-compose plugin to achieve the result you are hoping for? That way you could define the services in a docker-compose.yml file such as

services:
  ci:
    context: .
    image: Dockerfile.cue
  otherService
    context: .
    image: Dockerfile.withoutcue

You’d then use the docker-compose plugin to run the services within the pipeline and could continue to use the default queue.

For example:

steps:
  - label: "Generate integration tests"
    key: "tests"
    depends_on: build
    commands:
      - ./scripts/cibuild generate_integration_tests
    agents:
      queue: default
      plugins:
        - docker-compose#v3.10.0:
            run: ci
  - label: "Run other thing"
    depends_on: tests
    command: echo "Hello, world!"
    agents:
      queue: default
      plugins:
        - docker-compose#v3.10.0:
            run: otherService

If you’d prefer, you could email us a link to your pipeline at support@buildkite.com, we’d likely be able to get a bit more context with how your pipeline is running and how you’re using the various images and tags then.

1 Like
9

A --no-parent-tags does sound like that I want.

I have sent an email to support with a link to my pipeline steps. Appreciate that :+1: