Thoughts on building software and startups.

A weekly podcast about cloud native development, AWS, and building distributed systems.


Secrets Management for Cloud-Native Applications

Psst... Can you keep a secret?

Applications frequently need access to sensitive data, such as database credentials, API keys, passwords and tokens.

Of course, we can't just store these secrets in plain text or hard coded into our applications. Rather, we need to securely protect this sensitive information to ensure that only those with a "need to know" basis can access it.

In this post, I outline various approaches to secrets management and explore some of the most popular solutions out there. We'll finish with some guidance on how you can choose the right solution for your particular application.

Before discussing secrets management, let's frame the problem with some guidelines to follow when dealing with sensitive data.

Secrets management guidelines

There are many approaches to secrets management, ranging from basic roll-your-own techniques to full-blown secrets management solutions. Let's take a closer look at some of the options.

A simple, roll-your-own technique for secrets management

A simple technique for securely storing and accessing sensitive data uses a strategy of encryption at rest coupled with a secure object store to persist the encrypted cipher text.

When using AWS, we can leverage the Key Management Service (KMS) for encryption and decryption and Amazon Simple Storage Service (Amazon S3) for the object store. To implement our secrets handling, we first need to create a KMS customer master key (CMK).

Note that when using this technique, we are using two different resources (KMS CMK and S3 bucket/key), each of which can be locked down using IAM permissions. This gives us very fine-grained access control (and auditing!) of secrets, with very little effort.

Storing a secret

To store a secret:

Example code (bash using AWS CLI)


# Encrypt local file
# NOTE:  When you use the AWS CLI, the output, `CiphertextBlob` is 
#        Base64-encoded. So decode before storing (so subsequent 
#        `decrypt` call will be valid).
aws kms encrypt --key-id $KMS_KEY_ALIAS --plaintext fileb://$PUT_FILE \\
    --output text --query CiphertextBlob | base64 --decode > tmp.encrypt

# Copy encrypted file to S3
aws s3api put-object --region $AWS_REGION --bucket $BUCKET --key $KEY --body tmp.encrypt

Retrieving a secret

To retrieve a secret:

Example code (bash using AWS CLI)

# Retrieve encrypted file from S3
aws s3api get-object --region $AWS_REGION --bucket $BUCKET \\
    --key $KEY tmp.encrypt

# Decrypt
# NOTE:  When you use the AWS CLI, the decrypted output is 
#        Base64-encoded. So we must Base64 decode after 
#        decryption.
aws kms decrypt --ciphertext-blob fileb://tmp.encrypt --query Plaintext \\
    --output text | base64 --decode

Secrets management solutions

While you can implement your own secrets management solution, in general, you probably shouldn't. Instead, you should consider using a purpose-built secrets management solution. When you use a secrets management solution, you can expect a robust set of capabilities, such as:


One popular solution is Hashicorp's Vault. Vault is vendor-agnostic and works well for on-premise, hybrid and enterprise environments. It also offers a multi-cloud approach. However, Vault is a self-hosted solution. You'll need to install the software on server(s) you manage and you'll be responsible for scalability and availability. When considering cost, you'll need to account for Vault licensing fees, along with the infrastructure cost to host the software.

AWS options

On the other hand, AWS offers not one, but two, managed services for secrets management.

Both Systems Manager Parameter Store and AWS Secrets Manager are managed services that allow you to securely store key/value pairs. Since they have similar functionality, it can sometimes be confusing to decide which to choose for your secrets management. Let's compare the two services.

Choosing a secrets management solution

Given the availability of robust, mature purpose-built secrets management solutions, it's hard to justify a do-it-yourself approach. Why spend any time or resources on "undifferentiated heavy lifting"?

So, which secrets management solution to choose?

If you want a self-hosted solution or need multi-cloud capabilities, Vault is the right choice. Otherwise, as long as you are using AWS, you should choose either Systems Manager Parameter Store or Secrets Manager or a combination of both.

Personally, because of the flexibility and cost model, I prefer to use Parameter Store for most of the sensitive data used by my applications. But for RDS database instances, I like using Secrets Manager because of its direct support for credential rotation.

Next steps

In a future post, I'll dive into the details of how to implement secrets management using Systems Manager Parameter Store for a containerized application running on Amazon Elastic Container Service (ECS).

Entrepreneur, technologist and startup junkie. Founded two companies, one with $24M in VC, the other bootstrapped with SBA loan.

View Comments