AWS EKS and the Least Privilege Principle

When hosting workload with AWS, one of the key security principles we follow is — Least Privilege Access. The idea is to give the minimum set of permissions required for the service to perform the business. But when you work with Kubernetes to host your containerised workload, this principle at times goes for a toss. How??? Let’s see —

Assume Example Corp. has 2 applications which needs different permission set to execute their business logic.

1. Both applications are hosted in a single AWS Elastic Kubernetes Service (EKS) managed cluster.
2. Application A needs access to AWS S3 and Application B needs access to AWS DynamoDB.
3. Cluster is launched by a common DevOps team.
4. To follow the best practices in terms of giving the access, DevOps team creates AWS IAM Role and assigns S3 and DynamoDB access permissions to it.
5. The IAM role is assumed by the EC2 worker nodes.

Now, as and when Kubernetes Pods for Application A and Application B are launched in the EKS cluster, they can access S3 and DynamoDB as the EC2 worker nodes on which they are running can assume the role. But is this not breaking the principle of least privilege??? Application A, if wants, can also talk to DynamoDB and Application B can do the same with S3. I know, you must be thinking, a proper code review can prevent this from happening. But what if one of the applications gets compromised at runtime because of some vulnerability, and the attacker gets access to the pod? The Blast Radius will be high 😧!!!

So how do we ensure that Application A has access to only S3 and Application B has access to only DynamoDB despite they running on the same EC2 worker node??? Well, IRSA is the answer. IRSA stands for IAM Roles for Service Accounts.

IRSA solution leverages quite a few concepts to implement the least privilege principle —

1. Mutating Admission Controller webhook to manipulate the Pod manifest file when API Server gets a request to create new Pod.
2. Kubernetes native Service Accounts (SA) resource to pass secure token to the Application. SA is annotated with IAM role to be assumed.
3. Open ID Connect (OIDC) Federation to assume IAM roles via AWS Security Token Service (STS)
4. AssumeRoleWithWebIdentity construct to get temporary IAM credentials in exchange of a token issued by OpenID Connect provider.

Let us understand what is expected from the end user to make it work as expected —

1. Assuming EKS cluster is already set-up, create an Open ID Connect Provider with the Issuer URL which is linked to your EKS Cluster.

Amazon EKS now hosts a public OIDC discovery endpoint per cluster containing the signing keys for the ProjectedServiceAccountToken JSON Web Tokens so external systems, like IAM, can validate and accept the Kubernetes-issued OIDC tokens.

To get the Issuer URL, execute below command —

> ISSUER_URL=$(aws eks describe-cluster --name cluster_name --query cluster.identity.oidc.issuer --output text)

2. Create an IAM Role with the permissions required by the Application to execute its business logic. This role should have a trust relationship with the OIDC Provider configured in Step #1 and the action allowed should be sts:AssumeRoleWithWebIdentity.

Trust Relationship of the Role -
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::xxxx444:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/xxxxxxxx123"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-2.amazonaws.com/id/xxxxxxxx123:sub": "system:serviceaccount:default:Application-A-serviceaccount",
          "oidc.eks.us-east-2.amazonaws.com/id/xxxxxxxx123:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

3. Create Kubernetes Service Account and annotate it with the Role created in Step #2.

apiVersion: v1 
kind: ServiceAccount 
metadata:   
  annotations:     
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxx444:role/s3-readonly-role   
  name: Application-A-serviceaccount   
  namespace: default
secrets: 
- name: Application-A-serviceaccount-token-xxxxx

4. Pass the service account name in the application pod manifest file.

apiVersion: v1
kind: Pod
metadata:
 name: Application-A
spec:
  containers:
  - image: amazon/aws-cli:latest
    command: ["/bin/sh"]
    args: ["-c", "sleep 1000"]
    name: Application-A
    serviceAccountName: Application-A-serviceaccount

And that’s it. Now, when you create a pod, this is what happens —

1. Mutating Admission Controller running in EKS calls EKS Pod Identity Webhook which injects 2 environment variables — AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE in the Pod manifest file and also mounts a volume with name aws-iam-token.
2. AWS_ROLE_ARN contains the Role ARN which was configured in Step #2 in the earlier section.
3. AWS_WEB_IDENTITY_TOKEN_FILE contains location of the JWT token which was issued by the EKS cluster’s OIDC endpoint.
4. aws-iam-token is the projected volume which contains details of the JWT token like path, expiry time, target audience, etc. For details, see here.

After the pod gets deployed, you can check the mutated manifest file —

apiVersion: v1
kind: Pod
metadata:
  name: Application-A
spec:
  containers:
  - args:
    - -c
    - sleep 1000
    command:
    - /bin/sh
    env:
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::xxxx444:role/s3-readonly-role
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: amazon/aws-cli:latest
    name: Application-A
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: Application-A-serviceaccount-token-xxxxx
      readOnly: true
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  serviceAccountName: Application-A-serviceaccount
  volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token
  - name: Application-A-serviceaccount-token-xxxxx
    secret:
      defaultMode: 420
      secretName: Application-A-serviceaccount-token-xxxxx

And now Application A can use AWS SDK or AWS CLI to call only those AWS APIs it is entitled to as part of the IAM Role.

The AWS SDKs have been updated with a new credential provider that calls sts:AssumeRoleWithWebIdentity

With IRSA, you can create IAM Roles at pod level which helps you implement the Least Privilge Principle at the pod level, reducing the Attack Surface should any of the service gets compromised.

Please do share this blog with you friends and give few claps if this has helped you in any way.

Cheers!!!