The other day I was discussing with a colleague about the implications of changing KMS keys once the data is already encrypted. The consensus was that there is little literature in that regard. For that reason I’m writing this post to give you some insight on how to deal with KMS key rotation.
There’s no doubt that when it comes to data security at rest, AWS Key Management Service (AWS KMS, or just KMS for short) is the best approach.
-
Works with most AWS Services.
-
It’s simple to set-up.
-
Offers a good balance between simplicity of management and key control.
Regarding the balance between easier management and key control, KMS offers three types of keys.
-
AWS owned key: Keys owned and managed by AWS. They are the simplest to use but they offer no control to the customer. Also these keys are shared between customers.
-
AWS managed key: Keys owned by the customer but managed by AWS. Usually associated with a single AWS Service. These keys live only in the customer AWS account and are not shared with other customers.
-
Customer managed key: Keys owned and managed by the customer, a.k.a. us, who are responsible of the lifecycle for the key, where it is used and the key security (configuration of its key policy).
In this post I’m going to talk about this last type of keys, Customer managed keys (CMKs).
Where do KMS keys come from? #
To understand key rotation we must know how keys are made and from what they are made.
When we create a KMS key, we choose the key material origin.
-
KMS: It’s the recommended approach. AWS creates and manages the key material for the KMS key.
-
External (Import Key material): You create and import the key material for the KMS key.
-
AWS CloudHSM key store: AWS KMS creates key material in the AWS CloudHSM cluster of your AWS CloudHSM key store.
-
External key store: The key material for the KMS key is in an external key manager outside of AWS.
Unless we have a very strict security posture, KMS key material is the preferred option for the majority of the cases. AWS CloudHSM key store and External key store are advanced use cases that are not covered by this post.
The main difference between KMS key material and External key material is that in the former AWS is responsible for the creation, custody and rotation of the key material. In the latter, it is our sole responsibility as customers to do all of that. A KMS generated key material never leaves the hardware where it was generated making it very difficult to compromise. On the other hand, for an external customer key material we should take care of securing the generation process as well as storing them in a secure and durable manner.
Knowing these differences is important to know how to effectively rotate KMS keys.
Ways to rotate a KMS key #
There is not a single way of rotating KMS keys. It depends on the type of key material used and the control we would like to have in the rotation process.
Automatic key rotation #
For KMS keys with KMS managed key material it is possible to enable automatic key rotation once a year. This process keeps the same logical resource, our KMS key, but rotates its key material. The old key materials of the key are kept, and the only way to delete them is deleting the KMS key.
From that point forward new encrypt operations will use the new key material of the key, while the decrypt operations will use the key material that was used at the moment of the encryption. This guarantees that any data key encrypted with that KMS key will be able to be decrypted.
Manual rotation #
Manual rotation could be used in either keys with KMS managed key material and keys with imported key material. Also for keys with imported key material it is possible to accomplish it in two different ways:
Generate new key #
For KMS keys with KMS key material and imported key material. This method consist of generating a new KMS key and pointing the KMS alias used in the old key to the new one. That way it is not necessary to make any change in our application code or AWS Services that used that KMS Alias.
When a KMS Alias is pointed to another KMS key, all new encrypt operations will be carried by the new KMS key, but in a similar way that what happens with automatic key rotation, the decryption of data keys encrypted with the old KMS key is guaranteed as long as the old keys exist and were enabled. The decrypt operation uses metadata to know which is the corresponding decryption key.
Import new key material #
This method is only valid for KMS keys with imported key material. It consists in uploading new key material to the KMS key. However, unlike Automatic key rotation in KMS keys with KMS key material, for KMS keys with imported key material the old key materials are not kept when a new one is uploaded. This makes it impossible to decrypt any data key encrypted before the key material rotation.
What to do when a key is compromised #
Although it is very difficult to have a KMS key compromised it’s not impossible. As we see earlier, AWS managed key material never leaves the hardware where it was generated, but, for customer managed key material it’s different.
For KMS keys with external key material the customer must take care of the custody and security of the key material. This includes storing it securely and not disclosing it just like any other secret or password.
Let’s see some examples on how to rotate a KMS key depending on where it is used. These examples include how to prevent access to keys with compromised key material.
Rotate a KMS key used for generating data keys #
We used a KMS key ee740549-6491-47b0-810d-1365b9b52792 with alias my-key to generate data keys that we will use to encrypt our files.
% aws kms list-aliases
{
"Aliases": [
{
"AliasName": "alias/my-key",
"AliasArn": "arn:aws:kms:us-east-1:372922107867:alias/my-key",
"TargetKeyId": "ee740549-6491-47b0-810d-1365b9b52792",
"CreationDate": "2024-01-23T15:06:33.424000+01:00",
"LastUpdatedDate": "2024-01-23T15:06:33.424000+01:00"
}
]
}
% aws kms generate-data-key --key-id alias/my-key --key-spec AES_256
{
"CiphertextBlob": "AQIDAHgzqwxkDivbMS0RKdvlqyaQj/+MMUb4yxnnJYe+A6nwCQF8JkM9izf5rY6uVnY4n/uxAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMUhGbd0AHxyS5TcwGAgEQgDvOECsCoq/uGWnt8N4QlS3lXvdKGuwUTRkdPoKaZuwbTQgkyd6rCZ/ez1xuobFGfnesf0yFnc34AnRYuw==",
"Plaintext": "8RqgnZ3G+c5YzQTK3o9DnZAguHFbpWCoYC2aCNPg0lo=",
"KeyId": "arn:aws:kms:us-east-1:372922107867:key/ee740549-6491-47b0-810d-1365b9b52792"
}
We could decrypt our Ciphertext Blob using the same KMS key.
% aws kms decrypt --ciphertext-blob AQIDAHgzqwxkDivbMS0RKdvlqyaQj/+MMUb4yxnnJYe+A6nwCQF8JkM9izf5rY6uVnY4n/uxAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMUhGbd0AHxyS5TcwGAgEQgDvOECsCoq/uGWnt8N4QlS3lXvdKGuwUTRkdPoKaZuwbTQgkyd6rCZ/ez1xuobFGfnesf0yFnc34AnRYuw==
{
"KeyId": "arn:aws:kms:us-east-1:372922107867:key/ee740549-6491-47b0-810d-1365b9b52792",
"Plaintext": "8RqgnZ3G+c5YzQTK3o9DnZAguHFbpWCoYC2aCNPg0lo=",
"EncryptionAlgorithm": "SYMMETRIC_DEFAULT"
}
We already have data encrypted with our data key when we realized our KMS key (the key material) has been compromised.
If we simply rotate the key with another one, pointing the alias to the new key all the new data keys we generate will be secured, but the current ones and the data protected with them will be at risk.
This scenario is even worse if we replace the key material. We’ll lose access to the data key and the data protected under it.
The right approach then is as follows:
-
Re-encrypt data keys generated with the compromised KMS key. This process doesn’t alter the content of the data key, only its encryption, not being necessary to re-encrypt the data it protects. For this we will use a new KMS key
9772c95e-c9c5-43fb-bfcd-f8c4678f9e49and point the aliasmy-keyto it.
% aws kms update-alias --alias-name alias/my-key --target-key-id 9772c95e-c9c5-43fb-bfcd-f8c4678f9e49 % aws kms list-aliases { "Aliases": [ { "AliasName": "alias/my-key", "AliasArn": "arn:aws:kms:us-east-1:372922107867:alias/my-key", "TargetKeyId": "9772c95e-c9c5-43fb-bfcd-f8c4678f9e49", "CreationDate": "2024-01-23T15:06:33.424000+01:00", "LastUpdatedDate": "2024-01-23T16:54:02.384000+01:00" } ] } % aws kms re-encrypt --ciphertext-blob AQIDAHgzqwxkDivbMS0RKdvlqyaQj/+MMUb4yxnnJYe+A6nwCQF8JkM9izf5rY6uVnY4n/uxAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMUhGbd0AHxyS5TcwGAgEQgDvOECsCoq/uGWnt8N4QlS3lXvdKGuwUTRkdPoKaZuwbTQgkyd6rCZ/ez1xuobFGfnesf0yFnc34AnRYuw== --destination-key-id alias/my-key { "CiphertextBlob": "AQICAHjupf6EVUcdZoJA0fyIbMmGwu8KRy7wa/C4PTQSmct1SgGGvw309CV/AUGYhIa3SWSdAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMDsd/aY/QTctxYJ9uAgEQgDvZIU5u53bQyGHNwt1pKW+ylH6J4KIyGKV1whOA7SCkUBY3eaQgQbXWOSBgYe8Z5Fy/8gOBSl6SElwBUg==", "SourceKeyId": "arn:aws:kms:us-east-1:372922107867:key/ee740549-6491-47b0-810d-1365b9b52792", "KeyId": "arn:aws:kms:us-east-1:372922107867:key/9772c95e-c9c5-43fb-bfcd-f8c4678f9e49", "SourceEncryptionAlgorithm": "SYMMETRIC_DEFAULT", "DestinationEncryptionAlgorithm": "SYMMETRIC_DEFAULT" } -
Disable the compromised KMS key to prevent any decrypt operation.
% aws kms disable-key --key-id ee740549-6491-47b0-810d-1365b9b52792 -
From that moment only will it be possible to decrypt our data key with the new KMS key.
% aws kms decrypt --ciphertext-blob AQIDAHgzqwxkDivbMS0RKdvlqyaQj/+MMUb4yxnnJYe+A6nwCQF8JkM9izf5rY6uVnY4n/uxAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMUhGbd0AHxyS5TcwGAgEQgDvOECsCoq/uGWnt8N4QlS3lXvdKGuwUTRkdPoKaZuwbTQgkyd6rCZ/ez1xuobFGfnesf0yFnc34AnRYuw== An error occurred (DisabledException) when calling the Decrypt operation: arn:aws:kms:us-east-1:372922107867:key/ee740549-6491-47b0-810d-1365b9b52792 is disabled.% aws kms decrypt --ciphertext-blob AQICAHjupf6EVUcdZoJA0fyIbMmGwu8KRy7wa/C4PTQSmct1SgGGvw309CV/AUGYhIa3SWSdAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMDsd/aY/QTctxYJ9uAgEQgDvZIU5u53bQyGHNwt1pKW+ylH6J4KIyGKV1whOA7SCkUBY3eaQgQbXWOSBgYe8Z5Fy/8gOBSl6SElwBUg== { "KeyId": "arn:aws:kms:us-east-1:372922107867:key/9772c95e-c9c5-43fb-bfcd-f8c4678f9e49", "Plaintext": "8RqgnZ3G+c5YzQTK3o9DnZAguHFbpWCoYC2aCNPg0lo=", "EncryptionAlgorithm": "SYMMETRIC_DEFAULT" }
Rotate a KMS key used in S3 #
In this scenario we have an S3 bucket my-bucket-102353703712 with SSE-KMS default encryption key 93237725-c352-41c7-a762-3f001e97c9af
% aws s3api get-bucket-encryption --bucket my-bucket-102353703712
{
"ServerSideEncryptionConfiguration": {
"Rules": [
{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "aws:kms",
"KMSMasterKeyID": "arn:aws:kms:us-east-1:102353703712:key/93237725-c352-41c7-a762-3f001e97c9af"
},
"BucketKeyEnabled": false
}
]
}
}
Any file we put on the bucket is automatically encrypted with this default KMS Key.
% cat my-file.txt
File content
% aws s3 cp my-file.txt s3://my-bucket-102353703712
upload: ./my-file.txt to s3://my-bucket-102353703712/my-file.txt
We could check that our file is encrypted and the key used for that purpose.
% aws s3api get-object --bucket my-bucket-102353703712 --key my-file.txt my-file.txt
{
"AcceptRanges": "bytes",
"LastModified": "2024-01-26T15:48:09+00:00",
"ContentLength": 13,
"ETag": "\"9c2d5a5c8d2d27ca8308ad71310ade58\"",
"ContentType": "text/plain",
"ServerSideEncryption": "aws:kms",
"Metadata": {},
"SSEKMSKeyId": "arn:aws:kms:us-east-1:102353703712:key/93237725-c352-41c7-a762-3f001e97c9af"
}
% cat my-file.txt
File content
At this point, we need to rotate our KMS key just like in the previous scenario. For S3 buckets the best approach is to re-encrypt our bucket objects once we change the default bucket KMS key.
-
Change the bucket default KMS Key. This process doesn’t alter the encryption of the objects that already exist in the bucket. Since we are rotating our KMS Key due to the fact that the key has been compromised, we must take further actions.
% aws s3api put-bucket-encryption \ --bucket my-bucket-102353703712 \ --server-side-encryption-configuration '{"Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "aws:kms","KMSMasterKeyID": "1e51777f-fefd-4bc5-8671-c1851e7d07b3"}}]}' % aws s3api get-bucket-encryption --bucket my-bucket-102353703712 { "ServerSideEncryptionConfiguration": { "Rules": [ { "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "aws:kms", "KMSMasterKeyID": "1e51777f-fefd-4bc5-8671-c1851e7d07b3" }, "BucketKeyEnabled": false } ] } } -
Re-encrypt bucket objects. To do so we could simply perform a copy operation on the whole bucket.
It is worth to mention that there are many ways to accomplish the re-encryption of all objects in a bucket. The one presented here is one of the simplest, but it could have some limitations. For buckets with thousands or millions of objects, it could be advisable to rely on S3 Batch Operations.
% aws s3 cp --recursive s3://my-bucket-102353703712 s3://my-bucket-102353703712 copy: s3://my-bucket-102353703712/my-file.txt to s3://my-bucket-102353703712/my-file.txt % aws s3api get-object --bucket my-bucket-102353703712 --key my-file.txt my-file.txt { "AcceptRanges": "bytes", "LastModified": "2024-01-26T16:13:22+00:00", "ContentLength": 13, "ETag": "\"6b856ce783e96cac811638a2a071fc22\"", "ContentType": "text/plain", "ServerSideEncryption": "aws:kms", "Metadata": {}, "SSEKMSKeyId": "arn:aws:kms:us-east-1:102353703712:key/1e51777f-fefd-4bc5-8671-c1851e7d07b3" } % cat my-file.txt File content -
Finally, we disable the compromised KMS key to prevent further usage.
% aws kms disable-key --key-id 93237725-c352-41c7-a762-3f001e97c9af
Rotate a KMS key used in EBS volumes #
Since EBS volumes cannot be re-encrypted, the rotation of KMS keys must be approached indirectly. The only way of re-encrypting an EBS volume content is generating a snapshot and then creating a new volume from it using a different KMS key.
Our starting point is an EBS volume encrypted with a KMS key d6f8b1ee-032c-4244-9d44-827861e6f9fa.
% aws ec2 describe-volumes --volume-id vol-09be7ec8867c85f21
{
"Volumes": [
{
"Attachments": [],
"AvailabilityZone": "us-east-1a",
"CreateTime": "2024-01-27T17:56:36.159000+00:00",
"Encrypted": true,
"KmsKeyId": "arn:aws:kms:us-east-1:386088430154:key/d6f8b1ee-032c-4244-9d44-827861e6f9fa",
"Size": 10,
"SnapshotId": "",
"State": "available",
"VolumeId": "vol-09be7ec8867c85f21",
"Iops": 3000,
"VolumeType": "gp3",
"MultiAttachEnabled": false,
"Throughput": 125
}
]
}
As in previous examples, we must rotate the KMS key. We proceed as follows:
-
Create a snapshot of the volume.
% aws ec2 create-snapshot --volume-id vol-09be7ec8867c85f21 { "Description": "", "Encrypted": true, "OwnerId": "386088430154", "Progress": "", "SnapshotId": "snap-0a908c9b806fbadc5", "StartTime": "2024-01-27T18:06:13.864000+00:00", "State": "pending", "VolumeId": "vol-09be7ec8867c85f21", "VolumeSize": 10, "Tags": [] } -
Then create a new volume from the snapshot referencing the new KMS key.
% aws ec2 create-volume --availability-zone us-east-1a --encrypted --kms-key-id 3d70f1b8-7a3b-4e9e-a6b9-6db1a6471f5d --snapshot-id snap-0a908c9b806fbadc5 --volume-type gp3 { "AvailabilityZone": "us-east-1a", "CreateTime": "2024-01-27T18:15:47+00:00", "Encrypted": true, "KmsKeyId": "3d70f1b8-7a3b-4e9e-a6b9-6db1a6471f5d", "Size": 10, "SnapshotId": "snap-0a908c9b806fbadc5", "State": "creating", "VolumeId": "vol-017bbdf1b17c1c8b6", "Iops": 3000, "Tags": [], "VolumeType": "gp3", "MultiAttachEnabled": false, "Throughput": 125 } -
Finally, disable the old KMS key and delete the original volume and its snapshot.
% aws kms disable-key --key-id d6f8b1ee-032c-4244-9d44-827861e6f9fa % aws ec2 delete-volume --volume-id vol-09be7ec8867c85f21 % aws ec2 delete-snapshot --snapshot-id snap-0a908c9b806fbadc5
Rotate a KMS key used in other storage services. #
The rotation of KMS keys in other AWS services varies from service to service, but in a way or another could be extrapolated from the examples given in this post. For instance, the process to rotate or change the KMS key for an Aurora RDS Cluster consists in taking a cluster snapshot and restoring it while changing its KMS key in a similar fashion as we did with the EBS volume.
Other services though have a simpler rotation process, for example in DynamoDB is as simple and transparent as just specify a new KMS key in the encryption options for the table and the service takes care of the re-encryption process.
Wrapping up #
As we saw in this post, the chances of getting a KMS key compromised are very low since the AWS generated key material never leaves AWS hardware unencrypted. Customer generated key material should be used only for very specific use cases and the customer is responsible for the custody and durability of the key material.
All in all, it is possible to rotate and block old keys, but it has an operational overhead. The procedure for rotating KMS keys varies from service to service and must be planned and tested carefully.