Hey all,
We’re looking into piping all of our build logs into elastic search + kibana. Is there a non-hacky way for us to collect all of the logs produced by every build? The docs suggest piping into tee but this won’t really scale (in terms of the number of places I’d need to add that pipe).
What I think I want is something like log-destinations: ['/var/log/my-log-file.log'] inside the agent configuration. From there they can be picked up by something like the CloudWatch agent.
Hi Christian,
It’s not currently possible directly from the agent, but you could consider writing a command hook on your agents which tees all the output automatically.
You could alternately fetch the logs from the REST API, but that’s probably fiddlier.
We override to add our output capture/redirect in the bootstrap-script agent configuration so we get everything, rather than just command logs.
That allows us to query on system failures like failed checkouts too, for example. It will also allow us to centrally filter logs out so buildkite doesn’t see them, which we have a use case for.
3 suggestions! What a treat. Thank you @jess and @petemounce. Will dig into these options and report back.
@christian not sure if you’re still looking to solve this problem, but a feature to pipe job logs into a temporary file (buildkite/agent#1564) was recently released (as of agent v3.36.0).
With this feature, the problem is a lot easier to solve, and could be done by accessing the temp file from an agent pre-exit hook, for example
You can now find info about the logs file in the docs: Buildkite Agent Configuration v3 | Buildkite Documentation