Buildkite elastic stack MinSuccessfulInstancesPercent

Elastic CI Stack for AWS
1

Hi Team,

I received an error in buildkite elastic stack where I kept MinSize and MaxSize both equal to 1.
Because my requirement is to create a stack for elastic-builders which runs only one instance ondemand all the time, so it can keep the docker warm cache.

I received this error while creating the stack

image
image3168×370 73.3 KB

Could you please suggest what went wrong?

Thanks
Suraj

2

Hello @surajthakur !

We’re taking a look at this for you and will get back to you as soon as possible

Thank you!
Michelle

3

Hello @surajthakur !

I tested these settings on my own stack and it seemed to work. I’m curious why yours didn’t.

Would you be able to provide us with your Cloudwatch logs found here:

/buildkite/elastic-stack/{instance-id}
/buildkite/system/{instance-id}

Thanks! :slight_smile:
Michelle

4

Hi @Michelle

Thanks for replying.

After checking the logs, I understood why it didnt work, but I am not sure what is solution to it.
I am using buildkite token in AWS Secrets manager, and docs say after creation of the stack, update the permissions of secret in aws secret manager with the ARN of IAM role of the lambda and instance policy.

Now in this case, where the Minsize=1 and maxsize=1 , the instance is triggered to create right when the stack is getting created, it fails because it cant fetch the secret from AWS secrets manager.

Here are the logs thats requested

/buildkite/elastic-stack/{instance-id}

image
image3024×436 79.1 KB

/buildkite/system/{instance-id}

image
image2962×524 95.7 KB

I might be missing something, but currently out of clue. I hope you find something.

Thanks
Regards
Suraj

5

I think i got an idea,
I will need to create a stack , with minsize=0 and maxsize=1.
set the secret permissions in aws secret manager.
then update the stack with minsize=1 and maxsize=1, because the ARN of iamroles do not change when we update the stack.

Is that a possible solution to this?

Thanks

6

Hello @surajthakur :slight_smile:

Thanks again for reaching out to us!

As per Michelles’ testing above (also confirmed with my own testing) - should those two roles be made previous - filling the two roles into the Secret Manager secret’s resource permissions should do what you’ve described (after they each have been created the first time round in Stack creation).

Cheers - please let us know how it goes :+1:

James

7

Hi @james.s
Thanks for the reply.

It worked.

I also realised, since Max and Min size are equal, the nested cloudformation stack and the lambda function and its iam role for autoscaling is not created, so essentially it will be only one role for that stack, which you can add in the permission in the first place itself before the creation, the iam role for it is just the name of the stack
For example, the stack for elastic builder is buildkite-agent-elastic-builders-Role

So the IAM role for it is arn:aws:iam::<aws-account-id>:role/buildkite-agent-elastic-builders-Role

So this role can be added to aws secrets permissions before the creation of stack. This way, permission to create fetch the secret will already be in place, when the stack is trying to create instance during stack creation and stack will not rollback.

I am not sure of next step, but some information can be added to docs here

Thanks again for help.

Regards,
Suraj Thakur

8

Hey @surajthakur!

No worries - and thanks for the confirmation also.

As you’ve seen - when you specify both the minSize and maxSize parameters to 1, the Lambda/associated role and Stack isn’t created as part of its spin up. I’ll arrange some docs updates as you’ve suggested to that page for more clarity.

Thanks again and also for all the communication earlier :slightly_smiling_face: