External secrets and AWS Secrets Manager with Kubernetes
In this post, I’m going to describe how to sync secrets from AWS Secrets Manager to Kubernetes with example Terraform and Kubernetes manifests. I’ll be using the External Secrets Operator to pull and create secrets and Reloader to restart pods when a secret has changed.
Another option out there is the Kubernetes Secrets Store CSI Driver with the AWS Secrets Store CSI Provider; however, I found limitations with this solution. As of writing this, there is no way to have it automatically update the K8s secret it creates when it’s updated in Secrets Manager. It automatically updates the secret volume mounted on the pod, but if you want to set environment variables, you need to rely on the K8s secret it creates and doesn’t update. Plus I like the cleaner non-volume method from External Secrets Operator.
All the referenced Terraform code can be obtained here.
Providers/Versions
I’m using the EKS module for the endpoint in these providers and you can see how that’s setup in the modules repo referenced earlier.
providers.tf
1locals {
2 env = "sandbox"
3 region = "us-east-1"
4}
5
6provider "aws" {
7 region = local.region
8 default_tags {
9 tags = {
10 env = local.env
11 terraform = true
12 }
13 }
14}
15
16provider "helm" {
17 kubernetes {
18 host = module.eks-cluster.endpoint
19 cluster_ca_certificate = base64decode(module.eks-cluster.certificate)
20 exec {
21 api_version = "client.authentication.k8s.io/v1beta1"
22 # This requires the awscli to be installed locally where Terraform is executed
23 args = ["eks", "get-token", "--cluster-name", module.eks-cluster.name]
24 command = "aws"
25 }
26 }
27}
versions.tf
1terraform {
2 required_providers {
3 aws = {
4 source = "hashicorp/aws"
5 version = "~> 5.0"
6 }
7 helm = {
8 source = "hashicorp/helm"
9 version = "~> 2.11.0"
10 }
11 }
12 required_version = "~> 1.5.7"
13}
Helm
The “count” here is where it can be enabled or not through the EKS module and can be removed. One thing to highlight for reloader is I’m telling it to watch a specific namespace for changes.
1resource "helm_release" "external_secrets_operator" {
2 count = var.addons["external_secrets"]["enable"] ? 1 : 0
3 namespace = "external-secrets"
4 create_namespace = true
5 name = "external-secrets"
6 repository = "https://charts.external-secrets.io"
7 chart = "external-secrets"
8 version = var.addons["external_secrets"]["version"]
9 depends_on = [
10 aws_eks_node_group.core
11 ]
12}
13
14resource "helm_release" "reloader" {
15 namespace = var.env
16 create_namespace = false
17 name = "stakater"
18 repository = "https://stakater.github.io/stakater-charts"
19 chart = "reloader"
20 version = var.addons["reloader"]["version"]
21 set {
22 name = "reloader.namespaceSelector"
23 value = var.env
24 }
25 depends_on = [
26 aws_eks_node_group.core
27 ]
28}
IAM
The service/app that will be consuming these secrets will need access to said secrets. One of the best ways to do this is to assume an IAM role for that app via OIDC. How to do that is described here.
Example IAM role and policy with access to its specific secret:
1locals {
2 irsa_oidc_provider_url = replace(var.irsa_oidc_provider_arn, "/^(.*provider/)/", "")
3 account_id = data.aws_caller_identity.current.account_id
4}
5
6data "aws_iam_policy_document" "assume_role" {
7 statement {
8 effect = "Allow"
9 actions = ["sts:AssumeRoleWithWebIdentity"]
10
11 principals {
12 type = "Federated"
13 identifiers = [var.irsa_oidc_provider_arn]
14 }
15 condition {
16 test = "StringEquals"
17 variable = "${local.irsa_oidc_provider_url}:sub"
18 values = ["system:serviceaccount:${var.env}:${var.name}"]
19 }
20 condition {
21 test = "StringEquals"
22 variable = "${local.irsa_oidc_provider_url}:aud"
23 values = ["sts.amazonaws.com"]
24 }
25 }
26}
27
28resource "aws_iam_role" "role" {
29 name = "${var.name}-${var.env}"
30 assume_role_policy = data.aws_iam_policy_document.assume_role.json
31 managed_policy_arns = [
32 aws_iam_policy.role.arn
33 ]
34 tags = {
35 service = var.name
36 }
37}
38
39resource "aws_iam_policy" "role" {
40 name = "${var.name}-${var.env}"
41 policy = jsonencode({
42 Version : "2012-10-17",
43 Statement : [
44 {
45 Effect : "Allow",
46 Action : [
47 "s3:GetObject",
48 "s3:ListBucket",
49 ],
50 Resource : [
51 "${module.s3_bucket.arn}/*",
52 module.s3_bucket.arn
53 ]
54 },
55 {
56 Effect : "Allow",
57 Action : ["secretsmanager:GetSecretValue"],
58 Resource : [
59 "arn:aws:secretsmanager:${var.region}:${data.aws_caller_identity.current.account_id}:secret:${var.env}/image-app-??????"
60 ]
61 },
62 ]
63 })
64}
Demo
Here we’re creating a SecretStore for this app that uses its own serviceAccount, which is setup for OIDC IAM authentication. Next, we have the ExternalSecret that’s pulling two values out of a json based secret on Secrets Manager called “sandbox/image-app” with a refresh interval of 5 minutes.
SecretStore
1apiVersion: external-secrets.io/v1beta1
2kind: SecretStore
3metadata:
4 name: image-app
5spec:
6 provider:
7 aws:
8 service: SecretsManager
9 region: us-east-1
10 auth:
11 jwt:
12 serviceAccountRef:
13 name: image-app
ExternalSecret
1apiVersion: external-secrets.io/v1beta1
2kind: ExternalSecret
3metadata:
4 name: "image-app"
5spec:
6 refreshInterval: 5m
7 secretStoreRef:
8 name: image-app
9 kind: SecretStore
10 target:
11 name: image-app
12 creationPolicy: Owner
13 data:
14 - secretKey: db_user
15 remoteRef:
16 key: "sandbox/image-app"
17 property: db_user
18 - secretKey: db_password
19 remoteRef:
20 key: "sandbox/image-app"
21 property: db_password
Here’s a stripped down deployment manifest where I’m creating environment variables for the two values in the “image-app” secret. Take a note of the annotations on this deployment. We’re telling Reloader to keep an eye on the “image-app” secret, so when the secret is updated or rotated, Reloader will perform a restart on this app. Hopefully you have your rolling update strategies and pod disruption budgets set to your liking.
deployment
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: image-app
5 namespace: sandbox
6 annotations:
7 secret.reloader.stakater.com/reload: image-app
8spec:
9 selector:
10 matchLabels:
11 app.kubernetes.io/name: image-app
12 app.kubernetes.io/instance: image-app
13 template:
14 metadata:
15 labels:
16 helm.sh/chart: image-app-0.1.0
17 app.kubernetes.io/version: "1.0.0"
18 app.kubernetes.io/managed-by: Helm
19 app.kubernetes.io/name: image-app
20 app.kubernetes.io/instance: image-app
21 spec:
22 serviceAccountName: image-app
23 containers:
24 - name: image-app
25 securityContext:
26 {}
27 image: "image_app:v1.0.0"
28 env:
29 - name: REGION
30 value: "us-east-1"
31 - name: DBUSER
32 valueFrom:
33 secretKeyRef:
34 name: "image-app"
35 key: db_user
36 - name: DBPASSWORD
37 valueFrom:
38 secretKeyRef:
39 name: "image-app"
40 key: db_password