Sending BuildKite build logs to logging solutions such as Splunk, Loki, Datadog

General
1

We run a lot of builds and that produces tons of data, metrics, and situations that require some heavy duty alerting, reporting, dashboards. While the BK UI is decent, it is useful mainly for the pipeline that may be of interest to me or some team. There is no aggregate way of looking at this mountain of data.

This is where I think leveraging our existing investments in Splunk/Loki/Datadog/etc would be very useful.

Question for BK Team

  1. On the buildkit agent, are there any directory/files (build logs + build artifacts) that we can monitor via Splunk/Loki/Datadog agents so that these logs are sent in real-time to our logging systems?
  2. If yes to #1, are all these files have timestamps for each line of event?

Some Use Cases

  • Our logging systems can have sophisticated alerts and can kill builds if we detect something of concern
  • We can aggregate test metrics across pipelines and use these metrics as a way to kick off other pipelines
  • We run a lot of tools that produce a variety of reporting formats. We have small CLIs that can
2

Hi @araza,

We are looking at supporting logging solutions but nothing is planned for immediate roadmap. So at the moment option would be for you to setup log exports to your own logging solutions.

3

Hi Suma,
I don’t think I would need anything from your team. I just need to know if the builds logs and testing artifacts (junit reports, etc) are present somewhere on the buildkite agent. Since we host our own build kite agents, we have complete control over them and can install our logging agents to monitor build kite logs and any build artifacts.

4

Hi @araza

By default, build logs are stored in encrypted form in Buildkite’s Amazon S3 buckets, but you can also store the archived build logs in your private AWS S3 bucket.

This feature is only available to customers on the Buildkite Enterprise plan.

Other option of storing original build log is following the mechanism explained in our docs you can store build log to a file which you can then export to your logging solution.

You can also have the build logs uploaded as artifacts along with any other artifacts from your build and have them sent to your own private S3 bucket as explained here

Please let us know if you have any follow up questions. Thank you.