Securing Secrets in Kubernetes

Naresh Waswani
5 min readMay 1, 2021

More or less every application you develop has sensitive data that it uses to execute some business logic. It could be username and password to connect to a Database, or an application key and secret to connect to a 3rd party service. Using these secrets in the code is straight forward but keeping these secrets secure is a big challenge.

If your workload is containerized and you are using Kubernetes (k8s) as an orchestration engine, then there is some relief. k8s has a native resource called Secret which lets you manage and store sensitive data. It sores secrets as unencrypted base64-encoded string. These secrets can be injected into the containers running inside the Pod as Environment variables or can be mounted as Data volumes.

To keep sensitive data secure, k8s secret objects should be encrypted at rest and should be access controlled using k8s RBAC mechanism. If you are leveraging AWS Public Cloud for hosting the k8s cluster (Self hosted or Managed EKS), AWS Key Management Service (KMS) can be leveraged for encrypting data at rest.

Kubernetes manifest files are generally checked-in into the Code repository for version control. But you may not want to check-in the secrets in plain text or as base64-encoded strings into the Git. We all know why!!! But then where do you keep the sensitive data outside of the Kubernetes cluster to ensure they are safe and secure and still leverage them to create K8s native secret objects.

Well there are multiple options to handle this. Couple of them are listed below —

Option 1 — Encrypt your secret text and check-in the encrypted secret into the Git

  1. Encrypt your plain text sensitive data using symmetric or asymmetric algorithm.
  2. Use k8s Custom Resource Definition (CRD) to create custom secret resource using the encrypted secret content.
  3. Create a custom k8s controller which reads the encrypted secret object and at runtime decrypts it and creates a k8s native secret object.

With this approach, you can check-in the encrypted content into the Git repository. And its risk free because the content is encrypted and can only be decrypted by your private key. But then where do you keep the private key??? Back to square one 😃.

To handle the encryption keys and manage the entire operational aspect, Sealed Secrets from Bitnami can help.

Option 2 — Use external secret management service for storing the secrets

  1. You can store the sensitive data to 3rd part secret management service like AWS Secrets Manager or HashiCorp Vault.
  2. Create custom k8s controllers which can fetch secrets from these services based on some configuration and create k8s native secret object at runtime.

External Secrets is one such project which helps you implement this option out-of-box.

You can also enhance your application logic to read secrets from the 3rd party services on application boot-up but the whole idea here it to decouple the secret management from the application business logic and leverage k8s power to manage the same.

Quick overview of Sealed Secrets —

With Sealed Secret open source project — you can encrypt your Secret into a SealedSecret, which is safe to store — even to a public repository. The SealedSecret can be decrypted only by the controller running in the target cluster and nobody else (not even the original author) is able to obtain the original Secret from the SealedSecret.

Sealed Secrets is composed of two parts:

  • A cluster-side controller / operator
  • A client-side utility: kubeseal

The kubeseal utility uses asymmetric crypto to encrypt secrets that only the controller can decrypt.

These encrypted secrets are encoded in a SealedSecret resource, which you can see as a recipe for creating a secret.

And here is a quick overview of using Sealed Secrets to manage Secrets —

Step 1

Install kubeseal, a client utility tool which helps you to create SealedSecret custom Kubernetes resource from the Secret resource manifest file.

> wget -O kubeseal> sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Step 2

Install Custom Controller and Custom Resource Definition (CRD) for SealedSecrets

> kubectl apply -f created created created
deployment.apps/sealed-secrets-controller created created
service/sealed-secrets-controller created created
serviceaccount/sealed-secrets-controller created created created

Step 3

Verify if the sealed-secret controller pod is running

> kubectl get pods -n kube-system -l name=sealed-secrets-controllerNAME                                         READY   STATUS RESTARTS   AGE
sealed-secrets-controller-7c766b885b-d5r2r 1/1 Running 0 7m39s

If you check the logs of the pod, you should see the secret that this controller has created for its own functionality. This secret contains the public/private key pair which it will use for encrypting/decrypting the secrets.

P.S — To run below command, use the controller pod name that you get from the above command.

> kubectl logs sealed-secrets-controller-7c766b885b-d5r2r -n kube-systemcontroller version: v0.15.0
2021/05/01 20:13:34 Starting sealed-secrets controller version: v0.15.0
2021/05/01 20:13:34 Searching for existing private keys
2021/05/01 20:13:35 New key written to kube-system/sealed-secrets-keymt6dg
2021/05/01 20:13:35 Certificate is
2021/05/01 20:13:35 HTTP server serving on :8080

To see the public/private key data, execute below command —

> kubectl get secret -n kube-system -l -o yaml

Step 4

Create a Secret manifest file with name secrets.yaml. We want to secure the DB_USER and DB_PWD data.

apiVersion: v1
DB_PWD: cGFzc3dvcmQ= //base64 encoded
DB_USER: cm9vdA== //base64 encoded
kind: Secret
name: db-secrets

Let us now use the kubeseal command to create SealedSecret resource manifest file from the secrets.yaml file

> kubeseal --format=yaml < secret.yaml > sealed-secret.yaml> cat sealed-secret.yaml apiVersion:
kind: SealedSecret
creationTimestamp: null
name: db-secrets
namespace: default
DB_PWD: AgDaCRi27RV4/sVI2ok7JlqBSKT5+c7gGJog+...
DB_USER: AgAZG67CrrOBnyKIKha7xhJulr+CQGPaE/PpsjvY8jJR0IDO2...
creationTimestamp: null
name: db-secrets
namespace: default

In the above step, kubeseal gets the public key from the Kubernetes cluster and uses that to encrypt the data.

Step 5

Let’s create resource in k8s using the Sealed Secret manifest file.

> kubectl apply -f sealed-secret.yaml

If you check the logs of the controller again, you should see that the controller on creation of the sealed secret intercepted the request and created k8s native Secret object after decrypting the secret data from the SealedSecret resource.

> kubectl logs sealed-secrets-controller-7c766b885b-d5r2r -n kube-system2021/05/01 20:38:06 Updating default/db-secrets
2021/05/01 20:38:06 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"db-secrets", UID:"fd89a7e7-c81a-4110-9de6-6b65195169d3", APIVersion:"", ResourceVersion:"19365", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

Once the k8s native Secret object is created, it can then be injected into the container either as an environment variable or mounted as data volume.

The Sealed secret manifest file created in Step #4 above can be checked-in into the Git repository. The secrets.yaml file can be thrown away as it is not needed anymore. The secret data in the sealed-secrets.yaml file is safe as it is encrypted and can only be decrypted by the Controller running inside the k8s Cluster.

Hope this blog has given you some idea on how secrets can be secured with Kubernetes.

Do give few claps if this blog has helped you in any way. And don’t forget to share with your friends.




Naresh Waswani

#AWS #CloudArchitect #CloudMigration #Microservices #Mobility #IoT