If you’re using an HTTP cache, it’s pretty easy in this case - you can use a subdirectory of your http cache per silo key.
EG your example would become:
build --remote_http_cache=https://storage.googleapis.com/my-bucket/1
build --google_default_credentials
This is how we run ours, and it works great. This does not translate to gRPC-based remote caches, but for a GCS/HTTP cache, it’s the simplest way to go by far.