Trying to pull a docker image during the bootstrap script during initialization. I can in the elastic-stack.txt logs that were are successfully pulling the image and a subsequent docker images confirms the image exists, but after initialization is complete, the image no longer appears to exist. Any ideas?
I noticed there is some cron jobs that prunes and none of their logs seem to indicate they’re removing logs nor can I find any other traces in other misc logs.
That’s an interesting question. It sounds like perhaps some commands might be running on agents that don’t have access to the images? It would be helpful if we are able to take a look at an example build where this is happening. Would you be able to share a build URL with support@buildkite.com, so we can take a closer look and help you out with this?
So, I’m accessing the instances directly before they even get schedule for a job. I have root access and am still unable to detect the images (or any traces of them in /var/lib/docker, /var/log, etc.).
This is happening around agent initialization, so not sure if there is a particular build which is relevant.
@jl-applied Interesting… Would you be able to share your stack parameters at all (with any sensitive information removed)? I wonder if you have something set that might be messing with the image cache on the instance
