Agent should chunk uploads rather than user

Features Requests
1

We upload lots of pipelines > 500 steps long. Can the agent do the chunking for us?

Especially when streaming steps to the agent it can be quite a bit of manual effort to pre-chunk steps.

2

Hello, @john.soo! Thank you for your feedback!
Pipeline uploads and parallel builds and jobs are the best ways to deal with long pipelines. Division of steps into logical chunks, however, still needs to be done manually.

Best!

3

Well, my point is that buildkite-agent pipeline upload knows better the chunking limitations than the user. Can the chunking be done in the agent instead?

4

@john.soo that’s a good point and I will share it with the team!
Many thanks!

Cheers!
Karen

1 Like
5

Thank you!

6

Hey @john.soo

Just jumping in for Karen as she is out for the day.

Can you open an issue on our Agent Repo

Cheers!

1 Like
7

Sure! Done! `buildkite-agent pipeline upload` should chunk uploads for the user · Issue #2224 · buildkite/agent · GitHub

Thank you!