System Administration & Network Administration
amazon-web-services amazon-ec2 autoscaling amazon-kms
Updated Mon, 23 May 2022 22:38:07 GMT

How to auto-scale ec2 instances with an encrypted root volume?


I am trying to configure an autoscaling setup in AWS where the Node Launch Template includes encrypting the root volume (EBS). I have configured a service linked role, and a CMK in Amazon KMS with an IAM policy as per the documentation.

However, I am getting the following error when the ASG looks to create the instances:

Launching a new EC2 instance: i-0123456789xxx. Status Reason: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InternalError: Client error on launch

The troubleshooting documentation simply points at the original documentation and suggests the IAM policy isn't configured properly - I'm struggling to work out what's not right though.

The service linked role is configured on the ASG: SLR on ASG, and the SLR has the correct permissions in the IAM policy for the key that was used to encrypt the volume:

{
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms: Encrypt",
            "kms: Decrypt",
            "kms: ReEncrypt*",
            "kms: GenerateDataKey*",
            "kms: DescribeKey"
        ],
        "Resource": "*"
    },
    {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms: CreateGrant",
            "kms: ListGrants",
            "kms: RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    }

Note that launching the same AMI manually, specifying the root volume is encrypted with the same key, works. This suggests a problem with the SLR perhaps?

Or, do I need to create an AMI where the root volume is already encrypted?

UPDATE 11/05/2020:

Turns out there was a formatting error - there is a space after each colon in the Actions section. Removing that has fixed it and it works as expected now.

{
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
        ],
        "Resource": "*"
    },
    {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    }



Solution

Turns out there was a formatting error - there is a space after each colon in the Actions section. Removing that has fixed it and it works as expected now.

{
        "Sid": "Allow use of the key",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:GenerateDataKey*",
            "kms:DescribeKey"
        ],
        "Resource": "*"
    },
    {
        "Sid": "Allow attachment of persistent resources",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::0123456789:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        },
        "Action": [
            "kms:CreateGrant",
            "kms:ListGrants",
            "kms:RevokeGrant"
        ],
        "Resource": "*",
        "Condition": {
            "Bool": {
                "kms:GrantIsForAWSResource": "true"
            }
        }
    }