# Group steps together with individual steps
**URL:** https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868
**Category:** Pipelines
**Created:** [August 14, 2024, 1:04am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868 "2024-08-14T01:04:27Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 14, 2024, 1:04am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/1 "2024-08-14T01:04:27Z")
Hello,
Is there a possibility to have in a yml file group steps but also steps outside of group?
Pretty much I am trying to have certain steps like setup, docker build as individual steps and then have a group for around 3 steps in regarding with running tests.
---
### Author:  [@karen.sawrey](https://forum.buildkite.community/u/karen.sawrey)
#### Post date: [August 14, 2024, 3:46am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/2 "2024-08-14T03:46:00Z")
Hello, @UnknownTester404!
Sure, you can have both non-group and group steps in your pipeline yaml configuration. For instance, take a look at an [example in our documentation here](https://buildkite.com/docs/pipelines/group-step#using-wait-steps-in-job-groups) where there are two group steps and an individual command step in a yaml config file.
Best!
Karen
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 14, 2024, 6:29am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/3 "2024-08-14T06:29:46Z")
I’ve tried that and it throws that command is not accepted
---
### Author:  [@paula](https://forum.buildkite.community/u/paula)
#### Post date: [August 14, 2024, 6:32am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/4 "2024-08-14T06:32:56Z")
Are you able to share more info about the error for us to help diagnose. If you can share a build link that would be the most helpful 🙏
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 14, 2024, 7:28am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/5 "2024-08-14T07:28:21Z")
The error received:
```auto
fatal: Failed to upload and process pipeline: Pipeline upload rejected: `command` is not a valid property on a step group. Please check the documentation to get a full list of supported properties.
```
And have something like in the yml file
```auto
steps:
- label: "Something outside"
key: otherStepWhichIsNotInGroup
- group: "My Group"
depends_on:
- otherStepWhichIsNotInGroup
steps:
- label: "Something 1"
command: "scripts/script1.sh ${VARIABLE}"
- label: "Something 2"
command: "scripts/script2.sh ${VARIABLE}"
- label: "Something 3"
command: "scripts/script3.sh ${VARIABLE}"
- label: "Something 4"
command: "scripts/script4.sh ${VARIABLE}"
plugins:
```
---
### Author:  [@tomwatt](https://forum.buildkite.community/u/tomwatt)
#### Post date: [August 14, 2024, 9:39am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/6 "2024-08-14T09:39:13Z")
Hey @UnknownTester404
Thanks for sharing that example, really helpful to understand the context of what is being uploaded and figure out what the issue is.
From what you have shared, the reason for the error is the `plugins:` attribute on the _Group Step_, this attribute is only available on _Command_ Steps. But as some Plugins can be executed without specifying the `command:` attribute on Steps (as they run `command` hooks) the error message is interchangable in what it is referring to. Example of a Plugin being run without `command:` attribute:
```auto
steps:
- plugins:
- docker#v5.11.0:
image: "mesosphere/aws-cli"
always-pull: true
command: ["s3", "sync", "s3://my-bucket/dist/", "/app/dist"]
artifact_paths: "dist/**"
```
This is mentioned in the docs: [Command step | Buildkite Documentation](https://buildkite.com/docs/pipelines/command-step#pipelines-without-command-steps)
For the example shared, it will need to be updated with the `plugins:` attribute specificed on each of the _Steps_ within the _Group Step_ e.g.,
```auto
steps:
- label: "Something outside"
key: otherStepWhichIsNotInGroup
command: ...
- group: "My Group"
depends_on:
- otherStepWhichIsNotInGroup
steps:
- label: "Something 1"
command: "scripts/script1.sh ${VARIABLE}"
plugins:
- some-plugin:
...
- label: "Something 2"
command: "scripts/script2.sh ${VARIABLE}"
plugins:
- some-plugin:
...
- label: "Something 3"
command: "scripts/script3.sh ${VARIABLE}"
plugins:
- some-plugin:
...
- label: "Something 4"
command: "scripts/script4.sh ${VARIABLE}"
plugins:
- some-plugin:
...
```
Will mention if the Plugin to be used across all the _Steps_ within the _Group Step_ is the same and using the same configuration, it is possible to use YAML Anchors and Aliases to reduce duplicate code e.g.,
```auto
common:
- docker_plugin: &docker
docker#v3.3.0:
image: something-quiet
steps:
- label: "Read in isolation"
command: echo "I'm reading..."
plugins:
- *docker
- label: "Read something else"
command: echo "On to a new book"
plugins:
- *docker
```
More details on YAML Anchors and Aliases in Pipelines, and how to override them here:
- [Using plugins | Buildkite Documentation](https://buildkite.com/docs/plugins/using#using-yaml-anchors-with-plugins)
- [Using plugins | Buildkite Documentation](https://buildkite.com/docs/plugins/using#using-yaml-anchors-with-plugins-overriding-yaml-anchors)
Hope that helps, but let us know if you have any more questions or issues!
Cheers,
Tom
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 14, 2024, 10:48am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/7 "2024-08-14T10:48:01Z")
Unfortunately it does not work since it cannot reference a metaenv variable which is set in a previous step via command
```auto
- label: "Install"
key: install
command:
- buildkite-agent meta-data set "DK_IMAGE" ${BUILDKITE_PLUGIN_DOCKER_IMAGE}
```
The DK Image is passed is something like:
```auto
- docker#vx.x.x:
image: "{metaenv.DK_IMAGE}"
```
---
### Author:  [@jeremy](https://forum.buildkite.community/u/jeremy)
#### Post date: [August 14, 2024, 6:05pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/8 "2024-08-14T18:05:08Z")
Hey again @UnknownTester404 👋
You’ll need to use `buildkite-agent meta-data get "DK_IMAGE"` in your subsequent build step, either as part of your build script, or via a hook in order to ensure the value is available for those subsequent steps.
An example could be in an `environment` hook you check for the meta-data and then set it as an environment variable:
```sh
#!/bin/bash
if [buildkite-agent meta-data exists "DK_IMAGE"]; then
export DK_IMAGE=$(buildkite-agent meta-data get "DK_IMAGE")
fi
```
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 15, 2024, 9:40am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/9 "2024-08-15T09:40:26Z")
But the value is needed in the plugin part
How should I treat it in the plugin context usage for each step from the group??
```auto
plugins:
- 'plugin 1':
- docker#vx.x.x:
image: "{metaenv.DK_IMAGE}"
propagate-environment: true
mount-buildkite-agent: true
workdir: /app
volumes:
- /app/node_modules
environment:
- ????
```
---
### Author:  [@jeremy](https://forum.buildkite.community/u/jeremy)
#### Post date: [August 15, 2024, 7:02pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/10 "2024-08-15T19:02:24Z")
@UnknownTester404 👋
So in the context of passing the value of `DK_IMAGE` from meta-data into the plugin, you’ll need to ensure that you are getting the meta-data with the `buildkite-agent meta-data get` [command](https://buildkite.com/docs/agent/v3/cli-meta-data#getting-data), at which point you would be able to reference the value of `DK_IMAGE` directly in your pipeline YAML.
In the step that uploads this pipeline yaml to Buildkite with `buildkite-agent pipeline upload`, you would have an `environment` [hook](https://buildkite.com/docs/agent/v3/hooks) (you can use other hooks that run before `command`, but for this example I’m going to use environment) which checks for the meta-data with key `DK_IMAGE`, and then downloads that meta-data and exports it in the job environment to be referenced by the pipeline upload command. I’ll break this down below with some examples:
Let’s assume for this example I’ve set the meta-data for this build already with this command:
```bash
buildkite-agent meta-data set "DK_IMAGE" "foo:1.2.3"
```
and I have the following pipeline yaml in my repo under
`.buildkite/pipeline.yaml`:
```yaml
plugins:
- plugin-1:
- docker#vx.x.x:
image: $DK_IMAGE
propagate-environment: true
mount-buildkite-agent: true
workdir: /app
volumes:
- /app/node_modules
environment:
- ????
```
And I have the following `environment` hook configured on my agent:
```bash
# Environment hook
#!/bin/bash
if [buildkite-agent meta-data exists "DK_IMAGE"]; then
export DK_IMAGE=$(buildkite-agent meta-data get "DK_IMAGE")
fi
```
What ends up happening is when `buildkite-agent pipeline upload` runs, the value of `$DK_IMAGE` is parsed and is set in the pipeline YAML received by Buildkite, which will look like this:
```yaml
plugins:
- plugin-1:
- docker#vx.x.x:
image: foo:1.2.3
propagate-environment: true
mount-buildkite-agent: true
workdir: /app
volumes:
- /app/node_modules
environment:
- ????
```
It’s important to note that you cannot access meta-data directly in your pipeline YAML (via a declaration like`{metaenv.DK_IMAGE}`), but you _can_ access it via the meta-data command mentioned above. Some other valuable reading would be our docs around [environment variables](https://buildkite.com/docs/pipelines/environment-variables#environment-variable-precedence), as there are some nuances that are good to be aware of!
Hopefully the above helps, but feel free to reach out with any further questions 🙂
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 15, 2024, 7:15pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/11 "2024-08-15T19:15:52Z")
To be honest it is unclear where should I add this:
# Environment hook
#!/bin/bash
if [buildkite-agent meta-data exists “DK\_IMAGE”]; then
export DK\_IMAGE=$(buildkite-agent meta-data get “DK\_IMAGE”)
fi
The variable is set in a previous step so even I have that logic in a sh script, should I call it in each follow-up step which uses the plugin with the metaev.DK\_IMAGE?
---
### Author:  [@jeremy](https://forum.buildkite.community/u/jeremy)
#### Post date: [August 15, 2024, 8:12pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/12 "2024-08-15T20:12:54Z")
In the example I shared, the exported value of `DK_IMAGE` is actually configured in the environment hook - a bash script that will run in every job.
Where you would have that hook depends on the [platform](https://buildkite.com/docs/agent/v3/macos#file-locations-homebrew-installation) you have installed your agent on - for example on macOS with an M1 processor, the hooks are saved in `/opt/homebrew/etc/buildkite-agent/hooks`.
You can read more about hooks in our docs here 🙂:
> **[Buildkite Agent hooks v3](https://buildkite.com/docs/agent/v3/hooks)**
>
> An agent goes through different phases in its lifecycle, including starting up, shutting down, and checking out code. Hooks let you extend or override the behavior of agents at different stages of its lifecycle. You "hook into" the agent at a...
Every step that would refer to that image could be referenced as I have in my example above by referencing the value of `DK_IMAGE`.
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 15, 2024, 9:01pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/13 "2024-08-15T21:01:39Z")
The problem is that I don’t have access to that part (agent configuration) since it is configured by the client’s DevOps team and they are not so eager to make changes to the agents.
---
### Author:  [@jeremy](https://forum.buildkite.community/u/jeremy)
#### Post date: [August 15, 2024, 10:19pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/14 "2024-08-15T22:19:13Z")
That certainly makes things more challenging! You do have some options around storing those hooks in the [repository](https://buildkite.com/docs/agent/v3/hooks#hook-locations).
You can also include the same commands as part of a build script that runs before running the pipeline upload, if you want to avoid using hooks entirely. Hope that helps!
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 23, 2024, 2:26pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/15 "2024-08-23T14:26:37Z")
Hello,
I’ve tried the approach with a pre-command hook repository but it seems that it doesn’t work
So it is printed
```auto
.buildkite/hooks/pre-command
Pre-command repo hook execution
Pre-command repo hook executed
DK_IMAGE is: *IMAGE NAME IS PROPERLY PRINTED HERE*
```
The pre-command hook is something like:
```auto
#!/bin/bash
set -euo pipefail
echo "Pre-command repo hook execution"
if buildkite-agent meta-data exists "DK_IMAGE"; then
echo "Pre-command repo hook executed"
DK_IMAGE=$(buildkite-agent meta-data get "DK_IMAGE")
echo "DK_IMAGE is: ${DK_IMAGE}"
export DK_IMAGE
fi
```
The step is:
```auto
- label: "Label Me"
key: label
command: command here
plugins:
- 1
- 2
- docker#vX.X.X:
image: $DK_IMAGE
propagate-environment: true
mount-buildkite-agent: true
workdir: /workdir
environment:
- VARIABLE 1
- VARIABLE 2
```
tried also with image: $$DK\_IMAGE
---
### Author:  [@jeremy](https://forum.buildkite.community/u/jeremy)
#### Post date: [August 23, 2024, 8:36pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/16 "2024-08-23T20:36:30Z")
@UnknownTester404 👋
In this scenario, $DK\_IMAGE is the correct syntax.
You’ll want to ensure this hook is run on the job that performs the `buildkite-agent pipeline upload` in your pipeline - because of the way that environment variable interpolation works in Buildkite, `DK_IMAGE` **must** be set in the job’s environment _before_ the pipeline is uploaded, so that when the pipeline is parsed during upload the value of `DK_IMAGE` is correctly interpolated.
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 23, 2024, 8:50pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/17 "2024-08-23T20:50:39Z")
Sorry, but isn’t pre-comand hook running before each command? Am I missing something?
---
### Author:  [@suma](https://forum.buildkite.community/u/suma)
#### Post date: [August 23, 2024, 11:58pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/18 "2024-08-23T23:58:18Z")
Hey @UnknownTester404 ,
This is Suma from Buildkite support team jumping in on this.
Since the variable interpolation happens at the time of pipeline upload so suggestion is to set the environment variable using the environment hook.
Otherwise when the pipeline gets uploaded the image tag will be empty which I think is what you are observing.
If not, please send the build url to [support@buildkite.com](mailto:support@buildkite.com) so we can take a look at this further and let you know what else could be the issue here.
---
### Author:  [@UnknownTester404](https://forum.buildkite.community/u/UnknownTester404)
#### Post date: [August 24, 2024, 7:41am UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/19 "2024-08-24T07:41:24Z")
As I saw in the documentation, the environment hook can be sent only on agent which I mentioned that I don’t have access to. At the repository level it seems that it isn’t supported. I tried to set it in repo hooks folder but it wasn’t working
---
### Author:  [@paula](https://forum.buildkite.community/u/paula)
#### Post date: [August 24, 2024, 1:05pm UTC](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868/20 "2024-08-24T13:05:00Z")
👋 Unfortunately, as Suma mentioned, because interpolation happens at pipeline upload, you’ll need to use an `environment` hook.
You can send us your build links so we can look into the issue further.
[Next page](https://forum.buildkite.community/t/group-steps-together-with-individual-steps/3868.md?page=2)