# Parsing of dynamically generated pipeline fails

**URL:** <https://forum.buildkite.community/t/parsing-of-dynamically-generated-pipeline-fails/3604>\
**Category:** Pipelines\
**Created:** [March 25, 2024, 6:48am UTC](https://forum.buildkite.community/t/parsing-of-dynamically-generated-pipeline-fails/3604 "2024-03-25T06:48:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![muratgozel](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@muratgozel](https://forum.buildkite.community/u/muratgozel)\
**Post date:** [March 25, 2024, 6:48am UTC](https://forum.buildkite.community/t/parsing-of-dynamically-generated-pipeline-fails/3604/1 "2024-03-25T06:48:23Z")

</div>

The following is generated output of `pipeline.sh` file when pipeline configured as:

```yaml
steps:
  - label: ":pipeline:"
    command: ".buildkite/pipeline.sh"

```

Output:

```log
e_bk;t=1711345498288a~~~ Running script
e_bk;t=1711345498288ae[90m$e[0m ./.buildkite/pipeline.sh
e_bk;t=1711345502332a

e_bk;t=1711345502332a> @org/pkg@v2024.3.9 generate-pipeline /var/lib/buildkite-agent/builds/path

e_bk;t=1711345502332a> pnpm exec generate-pipeline

e_bk;t=1711345502332a

e_bk;t=1711345503401asteps:

e_bk;t=1711345503401a - label: "Deploy Link In Bio To Prod"

e_bk;t=1711345503401a command: |

e_bk;t=1711345503401a pnpm i --filter "@org/pkg" --frozen-lockfile --prod=false --config.dedupe-peer-dependents=false

e_bk;t=1711345503401a pnpm run -r --filter "@org/pkg" setup

e_bk;t=1711345503401a pnpm run -r --filter "@org/pkg" build

e_bk;t=1711345503401a pnpm run -r --filter "@org/pkg" ssg

e_bk;t=1711345503401a rsync -avO --no-perms dist/ /srv/$$APP_HOSTNAME/www/

e_bk;t=1711345503401a env:

e_bk;t=1711345503401a NODE_ENV: production

e_bk;t=1711345503401a INFISICAL_ENV: prod

e_bk;t=1711345503401a APP_HOSTNAME: app.com

e_bk;t=1711345503401a INFISICAL_TOKEN: st426

e_bk;t=1711345503401a concurrency: 1

e_bk;t=1711345503401a concurrency_group: "deploy-$$APP_HOSTNAME"

e_bk;t=1711345503425ae[90m# Successfully shut down Job API servere[0m

```

And when I configure the pipeline to actually work:

```yaml
steps:
  - label: ":pipeline:"
    command: ".buildkite/pipeline.sh | buildkite-agent pipeline upload"

```

It raises a parsing error:

```log
fatal: pipeline parsing of "(stdin)" failed: line 2: did not find expected comment or line break

```

Do you have any idea why it fails to parse the yaml content? It looks pretty valid. I appreciate any guidance.

---

<div class="post-metadata">

**Author:** ![paula](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/paula/32/954_2.png) [@paula](https://forum.buildkite.community/u/paula)\
**Post date:** [March 25, 2024, 7:07am UTC](https://forum.buildkite.community/t/parsing-of-dynamically-generated-pipeline-fails/3604/2 "2024-03-25T07:07:21Z")

</div>

Hey @muratgozel that yaml does look mostly fine. I wonder if it is line endings not being `\n` or something?  
Could you paste just the yaml output, or share a link to the build where you are seeing this for us to investigate further?

There is one issue I see though, which may or may not be related. The `concurrency_group` field does not support runtime interpolation with the `$$`. You’ll need to change that to a fully defined string, eg `deploy-app_com` (replacing the `.` with `_`)

---

<div class="post-metadata">

**Author:** ![muratgozel](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@muratgozel](https://forum.buildkite.community/u/muratgozel)\
**Post date:** [March 25, 2024, 7:06pm UTC](https://forum.buildkite.community/t/parsing-of-dynamically-generated-pipeline-fails/3604/3 "2024-03-25T19:06:23Z")

</div>

the `pnpm run script` command inside `pipeline.sh` was generating the yaml and the empty line it puts to the beginning of the command output was causing the error. I have changed it to `pnpm exec script` and now it works.

Maybe buildkite can skip the empty lines at the beginning/end of the output. I don’t know, this process also hard to test. It cost me 40 triggers and time.

Thank you for `concurrency_group` warning, just fixed it.

---

<div class="post-metadata">

**Author:** ![stephanie.atte](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/stephanie.atte/32/1357_2.png) [@stephanie.atte](https://forum.buildkite.community/u/stephanie.atte)\
**Post date:** [March 25, 2024, 7:29pm UTC](https://forum.buildkite.community/t/parsing-of-dynamically-generated-pipeline-fails/3604/4 "2024-03-25T19:29:18Z")

</div>

Hey @muratgozel

Glad you got this working. Thanks for the feedback