Integrating Secrets Manager with AWS EKS

In one of my earlier blogs written on — “Securing Secrets with Kubernetes”, many people had asked questions around — Why can’t we use native Cloud services like AWS Secrets Manager or AWS Systems Manager Parameter Store to secure secrets and use AWS SDK to retrieve them on the fly?

You can off-course do that. In-fact, that is one of the multiple ways of handling secrets in Kubernetes. IMHO, your choice of approach depends on the constraints you have while Architecting and Designing your application.

Assuming you are constrained with using AWS Secrets Manager because it supports Encryption at Rest, Secret Rotation and has been pre-approved by your Security Team, this blog provides you enough details on how you can use secrets (managed by AWS Secrets Manager) inside AWS EKS pods.

There are multiple ways of using AWS Secrets Manager service in EKS Pods —

1. Write custom code using AWS SDK and load secrets from the Secret Manager during start-up of the Pod
2. Write custom Kubernetes Controller which shall sync secrets from AWS Secrets Manager with Kubernetes Native Secret objects which can then be leveraged by the Pods via Environment variables or Mounted Volumes
3. Write MutatingAdmissionWebhook to inject Sidecar container into the Application Pod to load secrets from the AWS Secrets Manager which can then be used by the application container using Mounted Volumes approach
4. Using Kubernetes Secrets Store CSI Driver — It can load secrets from AWS Secrets Manager and mount them on the Pod using Mounted Volumes

The 1st three custom implementation takes lots of efforts from the individuals in terms of development, maintenance, and operational perspective. 4th approach is a standard specification — which integrates Secrets Stores with Kubernetes via a Container Storage Interface (CSI) volume.

Here is the high-level flow diagram of how it works —

Integrating AWS Secrets Manager with AWS EKS

1) User makes a request to create a Pod

2) EKS API Server receives the request and using Pod Scheduler schedules pod to a Node

3) Kubelet running on the Node gets the request to create Pod. Kubelet scans the Pod manifest file and finds volume to be mounted using CSI Driver

Pod Manifest File with CSI Driver

4) Kubelet invokes Secret Store CSI Driver via RPC call to mount the volume in the Pod

5) Secret Store CSI Driver creates the volume and invokes the AWS Secrets and Config Provider (ASCP) to further retrieve the secrets from the AWS Secrets Manager

6) ASCP retrieves the Pod Identity and assumes Role using AssumeRoleWithWebIdentity construct. It then makes API calls to the AWS Secrets Manager service to load the secrets

7) ASCP uses SecretProviderClass manifest file to find which secrets to be loaded and creates the secrets files in the mounted volume

AWS SecretProviderClass Manifest File

7) Pod gets into Started state and files are read from the mounted volume to get the secrets and perform the business.

Secrets Store CSI Driver also offers additional features of — Syncing Secrets Managers secrets with Kubernetes native Secrets objects and Auto-rotation of mounted contents and synced Kubernetes secrets

Mentioned below are the steps for you to try this —

1) Create EKS Cluster with OIDC Provider provisioned —

eksctl create cluster -f https://github.com/waswani/eks-secrets-manager/blob/main/cluster.yaml

To install eksctl utility, check here

2) Install CSI Secrets Store Driver —

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system --set syncSecret.enabled=true

To install Helm, check here

3) Install AWS Secrets and Config Provider —

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

To install kubectl utility, check here

4) Create Secret inside AWS Secrets Manager —

#Export Region and Cluster Name
REGION=us-west-2
CLUSTERNAME=pet-store
SECRET_ARN=$(aws --query ARN --output text secretsmanager  create-secret --name DBCredentials --secret-string '{"dbusername":"scott", "dbpassword":"tiger"}' --region "$REGION")

5) Create IAM Policy —

POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name pet-store-iam-policy --policy-document '{
    "Version": "2012-10-17",
    "Statement": [ {
        "Effect": "Allow",
        "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
        "Resource": ["$SECRET_ARN"]
    } ]
}')

6) Create Service Account —

Service Account will be associated with Policy we created above. This concept is called IAM Role for Service Account (IRSA). Check this blog for details on how IRSA works.

eksctl create iamserviceaccount --name pet-store-service-account --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

7) Create AWS Secret Provider Class

kubectl apply -f https://github.com/waswani/eks-secrets-manager/blob/main/pet-store-secret-provider.yaml

8) Create Application Deployment

kubectl apply -f https://github.com/waswani/eks-secrets-manager/blob/main/pet-store-deployment.yaml

Once you execute the above steps, you can look at the secrets mounted on the Pod as seen below —

AWS Secrets Mounted on Pod

With the approach demonstrated above, you can securely access secrets managed by AWS Secrets Manager.

Do share this blog with you friends and don’t forget to give claps if this has helped you in any way.

Happy Blogging…Cheers!!!