We are using the Elastic CI Stack with a bootstrap file stored in S3. We had an issue recently where a bad file was pushed to S3, which then caused any instance that happened to reboot after it to use the bad bootstrap.
To avoid this in the future, we would like to tie changes to the bootstrap file as part of the cloudformation, so that changes to the bootstrap file would only be rolled out if/when the cloudformation template was deployed.
I believe the easiest way to achieve that would be to enable S3 bucket versioning on the bucket used to store the bootstrap file, and then have the specific bootstrap file version id passed in as a cloudformation parameter. Is that possible? Alternatively, do you have other recommendations on how to more safely rollout changes to the bootstrap script?
Could you use the
BootstrapScriptUrl parameter? It takes an HTTPS URL for a bootstrap script to run at boot. If you passed a pre-signed URL for the S3 object that pointed to a particular version, and gave it a long expiry time, I wonder if that would do the trick.
Our stack doesn’t support passing along a version ID for the bootstrap script as it also supports pulling the script from any public URL.
However, some options I can think of are to push a the script to a new key each time - perhaps with a timestamp or git hash in it and reference that in the stack update so new instances pull the version they are tied to. So if it is bad, only new instances would fail to become healthy.
I also think, with S3 object versioning, you can revert a version in the AWS console quite quickly which should help alleviate this issue in future.
Thanks, I’ll investigate these options!