Hi,
I’m a bit surprised at the low number of tests included in the Pro Tier. 250 tests for free is extremely low and then offering additional tests at $0.10 quickly gets very expensive. What’s even more unfortunate is that this pricing model is antithetical to good software development practices. Ideally, we should have lots of fast-running tests, but were we to adopt Test Analytics, we’d be incentivised to write fewer tests! The feature is useful, but unfortunately we won’t adopt it due to this pricing model.
Hey @ronan-mch,
Welcome to the community!
Thanks for taking the time to share this, it’s a totally fair concern.
One important clarification is that Test Engine is priced on managed tests, not raw executions (which are unlimited). Each unique test is only counted once per day, even if it runs many times, and billing uses the 90th percentile of daily unique tests. To accomplish this, Buildkite measures usage by calculating the number of managed tests that have executed (run) at least once each day, and at month’s end, disregard the top 10%.
By disregarding the highest usage days, you can freely add new tests, experiment with different testing strategies, or temporarily increase test coverage without worrying about billing spikes. The pricing isn’t meant to discourage adding tests, you can run high-value tests without increasing cost unless you’re growing the number of unique managed tests. Test Engine also includes things like historical tracking, test state management, ownership, and flakiness and performance insights to help improve test quality.
If you’d like, please email support@buildkite.com with your organization details and we can take a look at what this would actually look like for your account.
Cheers!