Secrets Management for Cloud-Native Applications

Psst... Can you keep a secret?

Applications frequently need access to sensitive data, such as database credentials, API keys, passwords and tokens.

Of course, we can't just store these secrets in plain text or hard coded into our applications. Rather, we need to securely protect this sensitive information to ensure that only those with a "need to know" basis can access it.

In this post, I outline various approaches to secrets management and explore some of the most popular solutions out there. We'll finish with some guidance on how you can choose the right solution for your particular application.

Before discussing secrets management, let's frame the problem with some guidelines to follow when dealing with sensitive data.

Secrets management guidelines

Secrets should be encrypted at rest and use multiple encryption keys to limit blast radius.
Secrets should be rotated often.
You should automate the creation of secrets using strong algorithms.
There should be support for various access patterns across container environments such as development, test, and production.
There should be isolated access to secrets on a container/application level rather than at the host level.

There are many approaches to secrets management, ranging from basic roll-your-own techniques to full-blown secrets management solutions. Let's take a closer look at some of the options.

A simple, roll-your-own technique for secrets management

A simple technique for securely storing and accessing sensitive data uses a strategy of encryption at rest coupled with a secure object store to persist the encrypted cipher text.

When using AWS, we can leverage the Key Management Service (KMS) for encryption and decryption and Amazon Simple Storage Service (Amazon S3) for the object store. To implement our secrets handling, we first need to create a KMS customer master key (CMK).

Note that when using this technique, we are using two different resources (KMS CMK and S3 bucket/key), each of which can be locked down using IAM permissions. This gives us very fine-grained access control (and auditing!) of secrets, with very little effort.

Storing a secret

To store a secret:

Use KMS::Encrypt with our CMK to convert the secret to cipher text.
Store the cipher text in S3 using S3API::PutObject.

Example code (bash using AWS CLI)

#!/bin/bash

# Encrypt local file
# NOTE:  When you use the AWS CLI, the output, `CiphertextBlob` is 
#        Base64-encoded. So decode before storing (so subsequent 
#        `decrypt` call will be valid).
aws kms encrypt --key-id $KMS_KEY_ALIAS --plaintext fileb://$PUT_FILE \\
    --output text --query CiphertextBlob | base64 --decode > tmp.encrypt

# Copy encrypted file to S3
aws s3api put-object --region $AWS_REGION --bucket $BUCKET --key $KEY --body tmp.encrypt

Retrieving a secret

To retrieve a secret:

Read the cipher text from S3 using S3API::GetObject.
Decrypt the cipher text by calling KMS::Decrypt

Example code (bash using AWS CLI)

#!/bin/bash
# Retrieve encrypted file from S3
aws s3api get-object --region $AWS_REGION --bucket $BUCKET \\
    --key $KEY tmp.encrypt

# Decrypt
# NOTE:  When you use the AWS CLI, the decrypted output is 
#        Base64-encoded. So we must Base64 decode after 
#        decryption.
aws kms decrypt --ciphertext-blob fileb://tmp.encrypt --query Plaintext \\
    --output text | base64 --decode

Secrets management solutions

While you can implement your own secrets management solution, in general, you probably shouldn't. Instead, you should consider using a purpose-built secrets management solution. When you use a secrets management solution, you can expect a robust set of capabilities, such as:

Securely store and tightly control access to sensitive data
Protect data at rest and in transit
Centralized management capabilities
Secure authorization and authentication mechanisms
Integration with key management and encryption providers
Auditing
Secrets rotation and revocation

Vault

One popular solution is Hashicorp's Vault. Vault is vendor-agnostic and works well for on-premise, hybrid and enterprise environments. It also offers a multi-cloud approach. However, Vault is a self-hosted solution. You'll need to install the software on server(s) you manage and you'll be responsible for scalability and availability. When considering cost, you'll need to account for Vault licensing fees, along with the infrastructure cost to host the software.

AWS options

On the other hand, AWS offers not one, but two, managed services for secrets management.

Both Systems Manager Parameter Store and AWS Secrets Manager are managed services that allow you to securely store key/value pairs. Since they have similar functionality, it can sometimes be confusing to decide which to choose for your secrets management. Let's compare the two services.

Similarities
- Both are managed key/value store services
  - Allow you to store values under a name or key
  - Keys can have prefixes
- Both use KMS for encryption and decryption
- Both are referenceable in CloudFormation and integrated with other AWS services such as ECS
Key differences
- Types of values
  - Parameter Store allows for both encrypted and unencrypted values, whereas Secrets Manager only supports encrypted values.
- Cost
  - Parameter Store is free when using standard tier parameters (less than 4KB). Secrets Manager charges $0.40/secret per month, as well as API usage charges.
- Features
  - Secrets Manager is purpose-built for secrets only, and has several additional key features over Parameter Store, including:
    - Auto rotation of credentials, including seamless integration with RDS database services
    - Password generation (generate random secrets)
    - Secrets can be shared across accounts

Choosing a secrets management solution

Given the availability of robust, mature purpose-built secrets management solutions, it's hard to justify a do-it-yourself approach. Why spend any time or resources on "undifferentiated heavy lifting"?

So, which secrets management solution to choose?

If you want a self-hosted solution or need multi-cloud capabilities, Vault is the right choice. Otherwise, as long as you are using AWS, you should choose either Systems Manager Parameter Store or Secrets Manager or a combination of both.

Personally, because of the flexibility and cost model, I prefer to use Parameter Store for most of the sensitive data used by my applications. But for RDS database instances, I like using Secrets Manager because of its direct support for credential rotation.

Next steps

In a future post, I'll dive into the details of how to implement secrets management using Systems Manager Parameter Store for a containerized application running on Amazon Elastic Container Service (ECS).