Parsing of dynamically generated pipeline fails

Pipelines
1

The following is generated output of pipeline.sh file when pipeline configured as:

steps:
  - label: ":pipeline:"
    command: ".buildkite/pipeline.sh"

Output:

e_bk;t=1711345498288a~~~ Running script
e_bk;t=1711345498288ae[90m$e[0m ./.buildkite/pipeline.sh
e_bk;t=1711345502332a

e_bk;t=1711345502332a> @org/pkg@v2024.3.9 generate-pipeline /var/lib/buildkite-agent/builds/path

e_bk;t=1711345502332a> pnpm exec generate-pipeline

e_bk;t=1711345502332a

e_bk;t=1711345503401asteps:

e_bk;t=1711345503401a    - label: "Deploy Link In Bio To Prod"

e_bk;t=1711345503401a      command: |

e_bk;t=1711345503401a          pnpm i --filter "@org/pkg" --frozen-lockfile --prod=false --config.dedupe-peer-dependents=false

e_bk;t=1711345503401a          pnpm run -r --filter "@org/pkg" setup

e_bk;t=1711345503401a          pnpm run -r --filter "@org/pkg" build

e_bk;t=1711345503401a          pnpm run -r --filter "@org/pkg" ssg

e_bk;t=1711345503401a          rsync -avO --no-perms dist/ /srv/$$APP_HOSTNAME/www/

e_bk;t=1711345503401a      env:

e_bk;t=1711345503401a          NODE_ENV: production

e_bk;t=1711345503401a          INFISICAL_ENV: prod

e_bk;t=1711345503401a          APP_HOSTNAME: app.com

e_bk;t=1711345503401a          INFISICAL_TOKEN: st426

e_bk;t=1711345503401a      concurrency: 1

e_bk;t=1711345503401a      concurrency_group: "deploy-$$APP_HOSTNAME"

e_bk;t=1711345503425ae[90m# Successfully shut down Job API servere[0m

And when I configure the pipeline to actually work:

steps:
  - label: ":pipeline:"
    command: ".buildkite/pipeline.sh | buildkite-agent pipeline upload"

It raises a parsing error:

fatal: pipeline parsing of "(stdin)" failed: line 2: did not find expected comment or line break

Do you have any idea why it fails to parse the yaml content? It looks pretty valid. I appreciate any guidance.

2

Hey @muratgozel that yaml does look mostly fine. I wonder if it is line endings not being \n or something?
Could you paste just the yaml output, or share a link to the build where you are seeing this for us to investigate further?

There is one issue I see though, which may or may not be related. The concurrency_group field does not support runtime interpolation with the $$. You’ll need to change that to a fully defined string, eg deploy-app_com (replacing the . with _)

1 Like
3

the pnpm run script command inside pipeline.sh was generating the yaml and the empty line it puts to the beginning of the command output was causing the error. I have changed it to pnpm exec script and now it works.

Maybe buildkite can skip the empty lines at the beginning/end of the output. I don’t know, this process also hard to test. It cost me 40 triggers and time.

Thank you for concurrency_group warning, just fixed it.

4

Hey @muratgozel

Glad you got this working. Thanks for the feedback