# Agent should chunk uploads rather than user

**URL:** <https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065>\
**Category:** Features Requests\
**Created:** [July 19, 2023, 3:54pm UTC](https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065 "2023-07-19T15:54:42Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![john.soo](https://avatars.discourse-cdn.com/v4/letter/j/7ba0ec/32.png) [@john.soo](https://forum.buildkite.community/u/john.soo)\
**Post date:** [July 19, 2023, 3:54pm UTC](https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065/1 "2023-07-19T15:54:42Z")

</div>

We upload lots of pipelines \> 500 steps long. Can the agent do the chunking for us?

Especially when streaming steps to the agent it can be quite a bit of manual effort to pre-chunk steps.

---

<div class="post-metadata">

**Author:** ![karen.sawrey](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/karen.sawrey/32/1188_2.png) [@karen.sawrey](https://forum.buildkite.community/u/karen.sawrey)\
**Post date:** [July 19, 2023, 4:31pm UTC](https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065/2 "2023-07-19T16:31:46Z")

</div>

Hello, @john.soo! Thank you for your feedback!  
Pipeline uploads and [parallel builds and jobs](https://buildkite.com/docs/tutorials/parallel-builds) are the best ways to deal with long pipelines. Division of steps into logical chunks, however, still needs to be done manually.

Best!

---

<div class="post-metadata">

**Author:** ![john.soo](https://avatars.discourse-cdn.com/v4/letter/j/7ba0ec/32.png) [@john.soo](https://forum.buildkite.community/u/john.soo)\
**Post date:** [July 19, 2023, 4:39pm UTC](https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065/3 "2023-07-19T16:39:52Z")

</div>

Well, my point is that `buildkite-agent pipeline upload` knows better the chunking limitations than the user. Can the chunking be done in the agent instead?

---

<div class="post-metadata">

**Author:** ![karen.sawrey](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/karen.sawrey/32/1188_2.png) [@karen.sawrey](https://forum.buildkite.community/u/karen.sawrey)\
**Post date:** [July 19, 2023, 4:51pm UTC](https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065/4 "2023-07-19T16:51:49Z")

</div>

@john.soo that’s a good point and I will share it with the team!  
Many thanks!

Cheers!  
Karen

---

<div class="post-metadata">

**Author:** ![john.soo](https://avatars.discourse-cdn.com/v4/letter/j/7ba0ec/32.png) [@john.soo](https://forum.buildkite.community/u/john.soo)\
**Post date:** [July 19, 2023, 5:02pm UTC](https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065/5 "2023-07-19T17:02:02Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![stephanie.atte](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/stephanie.atte/32/1357_2.png) [@stephanie.atte](https://forum.buildkite.community/u/stephanie.atte)\
**Post date:** [July 19, 2023, 6:02pm UTC](https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065/6 "2023-07-19T18:02:28Z")

</div>

Hey @john.soo

Just jumping in for Karen as she is out for the day.

Can you open an issue on our [Agent Repo](https://github.com/buildkite/agent/issues)

Cheers!

---

<div class="post-metadata">

**Author:** ![john.soo](https://avatars.discourse-cdn.com/v4/letter/j/7ba0ec/32.png) [@john.soo](https://forum.buildkite.community/u/john.soo)\
**Post date:** [July 19, 2023, 6:48pm UTC](https://forum.buildkite.community/t/agent-should-chunk-uploads-rather-than-user/3065/7 "2023-07-19T18:48:31Z")

</div>

Sure! Done! [`buildkite-agent pipeline upload` should chunk uploads for the user · Issue #2224 · buildkite/agent · GitHub](https://github.com/buildkite/agent/issues/2224)

Thank you!