We use the https://github.com/buildkite/elastic-ci-stack-for-aws to run buildkite agents in our own AWS account. We then have a few different stacks with the main breakdown of one stack for on-demand machines and another stack for spot machines. We then use these machines across a multitude of steps and pipelines. This attempts to avoid the cost of waiting for new machines to scale in and out. We have been facing the question of how much does a pipeline cost per day? We can see that we can view the price for the stack with associated tags via the AWS cost explorer but would love to have the same metrics with more insight. We have some analysis in the past where we call the buildkite API to see build and job runtime. However, this feels very heavy-handed and is not presented as nicely as the AWS cost explorer.
We thought about creating another scheduler that tags the ec2 instances with the pipeline that they are running and then removes it when there is no job that an agent is running. However, that becomes more difficult with multiple agents per EC2 instance. I think it would be really great to have some more insight into the cost per pipeline rather than the cost at the agent level!
P.S. Buildkite has been really amazing!