Securing Secrets in Kubernetes

More or less every application you develop has sensitive data that it uses to execute some business logic. It could be username and password to connect to a Database, or an application key and secret to connect to a 3rd party service. Using these secrets in the code is straight forward but keeping these secrets secure is a big challenge.

If your workload is containerized and you are using Kubernetes (k8s) as an orchestration engine, then there is some relief. k8s has a native resource called Secret which lets you manage and store sensitive data. It sores secrets as unencrypted base64-encoded string. These secrets can be injected into the containers running inside the Pod as Environment variables or can be mounted as Data volumes.

To keep sensitive data secure, k8s secret objects should be encrypted at rest and should be access controlled using k8s RBAC mechanism. If you are leveraging AWS Public Cloud for hosting the k8s cluster (Self hosted or Managed EKS), AWS Key Management Service (KMS) can be leveraged for encrypting data at rest.

Kubernetes manifest files are generally checked-in into the Code repository for version control. But you may not want to check-in the secrets in plain text or as base64-encoded strings into the Git. We all know why!!! But then where do you keep the sensitive data outside of the Kubernetes cluster to ensure they are safe and secure and still leverage them to create K8s native secret objects.

Well there are multiple options to handle this. Couple of them are listed below —

Option 1 — Encrypt your secret text and check-in the encrypted secret into the Git

1. Encrypt your plain text sensitive data using symmetric or asymmetric algorithm.
2. Use k8s Custom Resource Definition (CRD) to create custom secret resource using the encrypted secret content.
3. Create a custom k8s controller which reads the encrypted secret object and at runtime decrypts it and creates a k8s native secret object.

With this approach, you can check-in the encrypted content into the Git repository. And its risk free because the content is encrypted and can only be decrypted by your private key. But then where do you keep the private key??? Back to square one 😃.

To handle the encryption keys and manage the entire operational aspect, Sealed Secrets from Bitnami can help.

Option 2 — Use external secret management service for storing the secrets

1. You can store the sensitive data to 3rd part secret management service like AWS Secrets Manager or HashiCorp Vault.
2. Create custom k8s controllers which can fetch secrets from these services based on some configuration and create k8s native secret object at runtime.

External Secrets is one such project which helps you implement this option out-of-box.

You can also enhance your application logic to read secrets from the 3rd party services on application boot-up but the whole idea here it to decouple the secret management from the application business logic and leverage k8s power to manage the same.

Quick overview of Sealed Secrets —

With Sealed Secret open source project — you can encrypt your Secret into a SealedSecret, which is safe to store — even to a public repository. The SealedSecret can be decrypted only by the controller running in the target cluster and nobody else (not even the original author) is able to obtain the original Secret from the SealedSecret.

Sealed Secrets is composed of two parts:

• A cluster-side controller / operator
• A client-side utility: kubeseal

The kubeseal utility uses asymmetric crypto to encrypt secrets that only the controller can decrypt.

These encrypted secrets are encoded in a SealedSecret resource, which you can see as a recipe for creating a secret.

And here is a quick overview of using Sealed Secrets to manage Secrets —

Step 1

Install kubeseal, a client utility tool which helps you to create SealedSecret custom Kubernetes resource from the Secret resource manifest file.

> wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.15.0/kubeseal-linux-amd64 -O kubeseal
> sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Step 2

Install Custom Controller and Custom Resource Definition (CRD) for SealedSecrets

> kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.15.0/controller.yaml

rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
role.rbac.authorization.k8s.io/sealed-secrets-key-admin created
clusterrole.rbac.authorization.k8s.io/secrets-unsealer created
deployment.apps/sealed-secrets-controller created
customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created
service/sealed-secrets-controller created
clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
serviceaccount/sealed-secrets-controller created
role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created

Step 3

Verify if the sealed-secret controller pod is running

> kubectl get pods -n kube-system -l name=sealed-secrets-controller
NAME                                         READY   STATUS RESTARTS   AGE
sealed-secrets-controller-7c766b885b-d5r2r   1/1     Running   0          7m39s

If you check the logs of the pod, you should see the secret that this controller has created for its own functionality. This secret contains the public/private key pair which it will use for encrypting/decrypting the secrets.

P.S — To run below command, use the controller pod name that you get from the above command.

> kubectl logs sealed-secrets-controller-7c766b885b-d5r2r -n kube-system
controller version: v0.15.0
2021/05/01 20:13:34 Starting sealed-secrets controller version: v0.15.0
2021/05/01 20:13:34 Searching for existing private keys
2021/05/01 20:13:35 New key written to kube-system/sealed-secrets-keymt6dg
2021/05/01 20:13:35 Certificate is 
-----BEGIN CERTIFICATE-----
MIIErjCCApagAwIBAgIRAJqYfaZsali26I8pvBXoFGYwDQYJKoZIhvcNAQELBQAw
ADAeFw0yMTA1MDEyMDEzMzVaFw0zMTA0MjkyMDEzMzVaMAAwggIiMA0GCSqGSIb3
DQEBAQUAA4ICDwAwggIKAoICAQDp/yO5PY8ACHBDuguhtfpOwlbScK9hZorJloyx
ixVCc57j1zMSX0pSVcrk1Yuyf6sYvBQtDi16kM70z6y/ODiz+9g87K/jY7B0UAoi
mpzM/T0tWJiG9ixyNMZhHoNREauokSlbERq3Jl8ZNTfmxHWhLH7DhkJ7MdpQfMpK
a3XHcSZyz1mXFqv+OSCCwllWCRHmHgp/vqudAv8+NYm0gnAxKt2fjlv/ObX8J1RI
CtLnlsCpp/9SyVcSTeYYaqjUsI7fTUZ7tkTE/bdQHwf3xe4DhUty7xLqMF1OPSPw
EetL8fGO0VqoSQFKQ0Bf78+8vhAA2cwkuqB6vQQm9pT3yC5niSCUo+jwFcfyknjr
yx8DINbq6K9B40EXh8X7w4I6zwYpyT0GoNU54wW0ki8pHRm7EnFeBOkUvNspzmKn
t/EZEDVq74Kkl/BRNRvKHYlwudSoJuvvX6JM8DVvRp0lMPnXnG3RLSmCP3gEFQBZ
DhbnkwO+6ADX9Q4vyqelWoHWdVGVULDlMDhSzvEhFFgPcZXzWTShH81vfl8M6lpT
U0ysZkA6i3A29XEJpPj35yWPBDWmKF5fLM3ChMt/NSJEeoJN1RboPDAgVUTxEW59
q+Tq09/zlYD7Ch8PNc3IWNXjFNXmCAAOw9Z1VBbD8p6LrC5JvBtPoWYqufWVXQD9
KDe+6wIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAAEwDwYDVR0TAQH/BAUwAwEB/zAN
BgkqhkiG9w0BAQsFAAOCAgEAvu//VzDREYZPPIW1maTxo9C/nHEEuOP0rQU3zVQr
bBYf8N0b5wpCllESCgi0JDJJXrE8KrjfdtawjoBrBlHOdWHF+fIot2KbrC/i37em
/ulMAgiiJzrKM/ExJuCuH60fsSIx4wrg46tQpU8jHFWq7nGnsaE+UN3QPjuvQ+qo
KKDSBLDxLx+q9vBfaXElblh4okUI8Pr4UEEJrYiPzPM6nA9EPpy53N3si4jyDJJb
2IsCUa2bW6iBhpyZOQQUPn22ziWRQ/sYYNmtP/gX0rwtk+Rr8TTdzPYGZcYfMQ6O
TFq4Zo2/TnpCL/CUr2DiSuF2qdWGGvbQOENYq2FNuDI4zeljElcZHXA8nhpbNSJs
7VNqqz5ZTFCKyL0Gn6SawGT7EdwBT2AD3F33Qd/7bXG/On7KdVw6FKHbZOR2RcoS
YFQv7Xr8g/4atQjxDa7R5+zkxd5unsvpFhYM1UfNJc4cjJ7SmfCCHoPGiwZ0OgqB
6SvUVU64QmMMJ/jYAJkYMOakSHaRITHAvvBjpAMKxSjjb7qZD5FnpXLhRY9lNiY6
MnnQRxJskCw+R6geIAHTMzAofMfc1haIEr+3oMFZfyh1LFFsz3B4hMxXYKrWYDje
+96bhAY9X7L0UfREjmw8HCeZneEuBJjX9z/PyIeMdhViLh9uO/MAL1MBxdBVA55+
LW8=
-----END CERTIFICATE-----
2021/05/01 20:13:35 HTTP server serving on :8080

To see the public/private key data, execute below command —

> kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml

Step 4

Create a Secret manifest file with name secrets.yaml. We want to secure the DB_USER and DB_PWD data.

apiVersion: v1
data:
  DB_PWD: cGFzc3dvcmQ= //base64 encoded
  DB_USER: cm9vdA==    //base64 encoded
kind: Secret
metadata:
  name: db-secrets

Let us now use the kubeseal command to create SealedSecret resource manifest file from the secrets.yaml file

> kubeseal --format=yaml < secret.yaml > sealed-secret.yaml
> cat sealed-secret.yaml 
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: db-secrets
  namespace: default
spec:
  encryptedData:
    DB_PWD: AgDaCRi27RV4/sVI2ok7JlqBSKT5+c7gGJog+...
    DB_USER: AgAZG67CrrOBnyKIKha7xhJulr+CQGPaE/PpsjvY8jJR0IDO2...
  template:
    metadata:
      creationTimestamp: null
      name: db-secrets
      namespace: default

In the above step, kubeseal gets the public key from the Kubernetes cluster and uses that to encrypt the data.

Step 5

Let’s create resource in k8s using the Sealed Secret manifest file.

> kubectl apply -f sealed-secret.yaml

If you check the logs of the controller again, you should see that the controller on creation of the sealed secret intercepted the request and created k8s native Secret object after decrypting the secret data from the SealedSecret resource.

> kubectl logs sealed-secrets-controller-7c766b885b-d5r2r -n kube-system
2021/05/01 20:38:06 Updating default/db-secrets
2021/05/01 20:38:06 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"db-secrets", UID:"fd89a7e7-c81a-4110-9de6-6b65195169d3", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"19365", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

Once the k8s native Secret object is created, it can then be injected into the container either as an environment variable or mounted as data volume.

The Sealed secret manifest file created in Step #4 above can be checked-in into the Git repository. The secrets.yaml file can be thrown away as it is not needed anymore. The secret data in the sealed-secrets.yaml file is safe as it is encrypted and can only be decrypted by the Controller running inside the k8s Cluster.

Hope this blog has given you some idea on how secrets can be secured with Kubernetes.

Do give few claps if this blog has helped you in any way. And don’t forget to share with your friends.

Cheers!!!