Hey all,
We’re looking into piping all of our build logs into elastic search + kibana. Is there a non-hacky way for us to collect all of the logs produced by every build? The docs suggest piping into tee but this won’t really scale (in terms of the number of places I’d need to add that pipe).
What I think I want is something like log-destinations: ['/var/log/my-log-file.log'] inside the agent configuration. From there they can be picked up by something like the CloudWatch agent.
Hi Christian,
It’s not currently possible directly from the agent, but you could consider writing a command hook on your agents which tees all the output automatically.
You could alternately fetch the logs from the REST API, but that’s probably fiddlier.
We override to add our output capture/redirect in the bootstrap-script agent configuration so we get everything, rather than just command logs.
That allows us to query on system failures like failed checkouts too, for example. It will also allow us to centrally filter logs out so buildkite doesn’t see them, which we have a use case for.
3 suggestions! What a treat. Thank you @jess and @petemounce. Will dig into these options and report back.