On the other hand, the hosted Linux/arm64 agents “work”, so they must be using some suitable base image.
Does the base image actually have to be hosted-agent-base? Because immediately overriding FROM with a custom (Rocky Linux) image “works”.
When are the agent images built? Using a moderately complex one seems to increase “waiting for agent” from ~5s to ~60s, so maybe they’re built “just in time”?
Generally, the current “agent image” definition process is rather inconvenient, as there’s no feedback on what happens after clicking “Create Agent Image”. It would be much nicer to be able to point to a specific (public) hosted image, and deal with creation separately.
This is Suma from Buildkite support team. First of all welcome to Buildkite community and thank you for reaching out to us with your questions and feedback on hosted agents.
How can we create an agent image to run on hosted Linux/arm64 agents?
Base image for agent images to be used for hosted agents is fixed which is hosted-agent-base . Is that since the hosted-agent-base image is linux/amd64 it is not working for your builds ? Please can you share more on why you want to change the base agent image ?
Does the base image actually have to be hosted-agent-base? Because immediately overriding FROM with a custom (Rocky Linux) image “works”.
If you noticed when you commit the Dockerfile with your own FROM it adds another layer but the base image layer stays there as shown in below screenshot
When are the agent images built? Using a moderately complex one seems to increase “waiting for agent” from ~5s to ~60s, so maybe they’re built “just in time”?
As soon as you create the image the agent image gets build and like shown in above screenshot you should be able to see image as available which you will mention in your hosted queue. Is it always consistent that when using different image wait time is increasing ? If so please can you share with us few example build urls at support@buildkite.com so we can check on it
Generally, the current “agent image” definition process is rather inconvenient, as there’s no feedback on what happens after clicking “Create Agent Image”. It would be much nicer to be able to point to a specific (public) hosted image, and deal with creation separately.
When you click on “Create Agent Image”, like shown in above screenshot it will show you image building and image available once build is done. Are you not seeing them when you do that on your end ?
Please let us know the details so we can check further on this.
Yeah, we want to build arm64 binaries on an arm64 runner, and so we need an arm64 base image to do this.
What I expected to be happening is that the hosted-agent-base actually has both amd64 and arm64 versions, and creating a custom base image actually creates two corresponding images.
Can you tell me what default base image the ARM agents actually run?
We’d prefer to build on RHEL/Rocky, not Ubuntu. This is not a blocker though.
More importantly, this would allow us to move the brunt of building this image outside of the limited current functionality (cf. below); we’d just put FROM <our-image> (with a public <our-image>) in the web form and be done with it. Bonus points if this image could be in a private repository (in our case Google Artifact Repository), and use OIDC authentication similar to buildkite-agent oidc to pull in.
Sure, I see this. But it just either works or fails (and in the ARM case, pretends to work…), and it’s click ops with no logs or anything. Your product is a CI pipeline, I think you can appreciate the difference between “type Dockerfile contents in web interface, click, wait, get a yes/no response” (current situation for custom base images) and flexibly code-driven build and update images.
Yes, consistent. And once I switch back to the default image, agent wait time immediately reduces to 5s again. Is there an out of band way to share example URLs? I’d rather not publish them here.
Thank you for providing additional feedback and inputs. I will follow up on this feedback and get back to you with more details.
Please send the build urls to support@buildkite.com where you see using custom image causes build time to be longer and build url where when you switch to default image it is faster.
,