# Collect all logs

**URL:** <https://forum.buildkite.community/t/collect-all-logs/1518>\
**Category:** Features Requests\
**Created:** [April 1, 2021, 8:57pm UTC](https://forum.buildkite.community/t/collect-all-logs/1518 "2021-04-01T20:57:27Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![christian](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/christian/32/942_2.png) [@christian](https://forum.buildkite.community/u/christian)\
**Post date:** [April 1, 2021, 8:57pm UTC](https://forum.buildkite.community/t/collect-all-logs/1518/1 "2021-04-01T20:57:27Z")

</div>

Hey all,

We’re looking into piping all of our build logs into elastic search + kibana. Is there a non-hacky way for us to collect _all_ of the logs produced by every build? The [docs suggest piping into `tee`](https://buildkite.com/docs/pipelines/managing-log-output#storing-the-original-log) but this won’t really scale (in terms of the number of places I’d need to add that pipe).

What I think I want is something like `log-destinations: ['/var/log/my-log-file.log']` inside the agent configuration. From there they can be picked up by something like the CloudWatch agent.

---

<div class="post-metadata">

**Author:** ![jess](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/jess/32/24_2.png) [@jess](https://forum.buildkite.community/u/jess)\
**Post date:** [April 1, 2021, 11:46pm UTC](https://forum.buildkite.community/t/collect-all-logs/1518/2 "2021-04-01T23:46:19Z")

</div>

Hi Christian,

It’s not currently possible directly from the agent, but you could consider writing a [command hook](https://buildkite.com/docs/agent/v3/hooks#main) on your agents which tees all the output automatically.

You could alternately fetch the logs [from the REST API](https://buildkite.com/docs/apis/rest-api/jobs#get-a-jobs-log-output), but that’s probably fiddlier.

---

<div class="post-metadata">

**Author:** ![petemounce](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/petemounce/32/339_2.png) [@petemounce](https://forum.buildkite.community/u/petemounce)\
**Post date:** [April 5, 2021, 1:48pm UTC](https://forum.buildkite.community/t/collect-all-logs/1518/3 "2021-04-05T13:48:34Z")

</div>

We override to add our output capture/redirect in the `bootstrap-script` agent configuration so we get everything, rather than just command logs.

That allows us to query on system failures like failed checkouts too, for example. It will also allow us to centrally filter logs out so buildkite doesn’t see them, which we have a use case for.

---

<div class="post-metadata">

**Author:** ![christian](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/christian/32/942_2.png) [@christian](https://forum.buildkite.community/u/christian)\
**Post date:** [April 5, 2021, 11:48pm UTC](https://forum.buildkite.community/t/collect-all-logs/1518/4 "2021-04-05T23:48:21Z")

</div>

3 suggestions! What a treat. Thank you @jess and @petemounce. Will dig into these options and report back.

---

<div class="post-metadata">

**Author:** ![deacy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/deacy/32/816_2.png) [@deacy](https://forum.buildkite.community/u/deacy)\
**Post date:** [July 28, 2022, 12:53am UTC](https://forum.buildkite.community/t/collect-all-logs/1518/5 "2022-07-28T00:53:29Z")

</div>

@christian not sure if you’re still looking to solve this problem, but a feature to pipe job logs into a temporary file ([buildkite/agent#1564](https://github.com/buildkite/agent/pull/1564)) was recently released (as of agent v3.36.0).

With this feature, the problem is a lot easier to solve, and could be done by accessing the temp file from an agent pre-exit hook, for example

---

<div class="post-metadata">

**Author:** ![deacy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.buildkite.community/deacy/32/816_2.png) [@deacy](https://forum.buildkite.community/u/deacy)\
**Post date:** [August 5, 2022, 10:18pm UTC](https://forum.buildkite.community/t/collect-all-logs/1518/6 "2022-08-05T22:18:17Z")

</div>

You can now find info about the logs file in the docs: [Buildkite Agent Configuration v3 | Buildkite Documentation](https://buildkite.com/docs/agent/v3/configuration#enable-job-log-tmpfile)
