Kubernetes Secrets Management at Scale with External Secrets Operator

Managing secrets across a multi-environment Kubernetes platform presented several risks and operational burdens:

• Hardcoded secrets in manifests and Helm values exposed credentials in Git repositories
• Manual secret updates required kubectl access and coordination across teams
• No automatic rotation—expired or compromised credentials required manual intervention
• Lack of audit trail made compliance reporting difficult
• Cross-namespace access created security risks when applications shared secrets inappropriately

Architecture Overview

Deployed External Secrets Operator (ESO) integrated with AWS Secrets Manager, using IAM Roles for Service Accounts (IRSA) for secure, credential-free authentication from Kubernetes pods.

1) External Secrets Operator Deployment

Installed ESO via Helm with CRD support:

• Deployed to dedicated external-secrets namespace
• Custom Resource Definitions for SecretStore and ExternalSecret
• Managed via Pulumi for infrastructure-as-code consistency

2) IRSA Configuration

Created a dedicated IAM role with OIDC trust policy:

• Service accounts annotated with IAM role ARN
• No long-lived AWS credentials stored in the cluster
• Wildcard namespace support for flexible deployments

Trust policy pattern:

{
  "Effect": "Allow",
  "Principal": {
    "Federated": "arn:aws:iam::ACCOUNT:oidc-provider/oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID"
  },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringLike": {
      "oidc.eks....:sub": "system:serviceaccount:*:external-secret-operator"
    }
  }
}

3) Namespace-Isolated SecretStores

Created per-namespace SecretStore resources:

• Each namespace gets its own SecretStore pointing to AWS Secrets Manager
• Service account references ensure proper IRSA binding
• Supports multiple AWS regions if needed

Flow

1. Create IAM role with Secrets Manager read/write permissions and OIDC trust.

2. Deploy External Secrets Operator to the cluster via Helm.

3. Create Kubernetes ServiceAccount annotated with the IAM role ARN in each target namespace.

4. Define SecretStore resources referencing the ServiceAccount for JWT-based AWS authentication.

5. Create ExternalSecret resources specifying the secret path and refresh interval.

6. ESO automatically syncs secrets from AWS Secrets Manager to native Kubernetes Secrets.

7. Applications consume secrets as standard environment variables or volume mounts.

4) Environment-Based Secret Paths

Secrets organized by environment and service using a consistent naming convention:

• {env}/app/{service-name} → e.g., dev/app/backend-api, prod/app/worker

This pattern enables predictable access and easy automation across environments.

5) Configurable Refresh Intervals

Different refresh rates based on sensitivity:

• 10 minutes for frequently rotated credentials (API keys)
• 24 hours for stable secrets (database credentials)
• Force-sync via annotation for immediate updates

• External Secrets Operator (Kubernetes operator)
• AWS Secrets Manager (secret storage)
• AWS IAM + OIDC (IRSA authentication)
• Pulumi (infrastructure automation)
• Helm (operator deployment)
• EKS (Kubernetes platform)

• Zero hardcoded secrets: All credentials externalized to AWS Secrets Manager
• Automatic synchronization: Secrets update without manual intervention
• Namespace isolation: Each application namespace has dedicated access controls
• Audit compliance: AWS CloudTrail logs all secret access
• Simplified onboarding: New services inherit secrets infrastructure automatically

• Secret Sync Latency: <30 seconds
• Hardcoded Secrets Eliminated: 100%
• Secret Rotation Downtime: Zero (automatic refresh)

• Use namespace-scoped SecretStores to enforce least-privilege access patterns
• Implement IRSA over static credentials—eliminates rotation overhead and improves security posture
• Set appropriate refresh intervals: too frequent wastes API calls, too infrequent delays updates
• Use force-sync annotations in CI/CD pipelines for immediate secret availability after deployments
• Organize secrets with consistent path conventions ({env}/app/{service}) for predictable access