Kubernetes has one of the most simple and flexible methods for storing and retrieving sensitive data in form of secrets. Kubernetes allows creating and storing of secrets such as database passwords, tokens, etc within Kubernetes namespaces. These secrets can be retrieved in form of environment variables or from the files mounted at a certain location within the pod and then can be used in the applications. Even though the native process is seamless and easy, it has certain disadvantages.

1. Base64 Encodings: The default secrets encodings are base64 encoding which doesn't ensure the security of secrets stored within Kubernetes.

2. Missing Single Source of Truth: There is no single source of truth for managing all your Kubernetes secrets in one place. It is not recommended to store secrets in static YAML files in a git repository.

3. Lack of Vault Integrations: Kubernetes secrets don't come with any mechanisms to integrate with cloud vaults such as AWS secret manager, Azure Key Vault, etc to offer enterprise-grade protection.

Thus, KES was Born!

To address the above challenges, Kubernetes External Secret (KES) was born. It was created by Godaddy for their internal usages and later on open-sourced for the community. It comes with the default integration of multiple cloud-based secret stores and solves the all issues discussed above.

However, the GoDaddy team have stopped maintaining the project and is publicly archived on July 26, 2022.

Introducing External Secrets Operator

External Secrets Operator (ESO) is a Kubernetes operator that helps you integrate with third-party external secrets like Hashicorp Vault, AWS Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager and many more within your Kubernetes secrets. The main objective of ESO is to synchronize secrets from external APIs into Kubernetes. It is a collection of custom API resources - SecretStore, ExternalSecrets, and ClusterSecretStore that helps you maintain the lifecycle of secrets from third-party vaults.

Architecture

ESO controller is installed in a Kubernetes cluster and is a cluster-specific object. To communicate with other secret managers, the API resource - SecretStore / ClusterSecretStore is deployed within the namespace/ cluster and is responsible for the authentication with external APIs. There are different ways you can authenticate using such as generic secret (using access key & secret key), IAM roles, ServiceAccount, etc that can be seen here in the documentation.

Once authentication is done, it will create an API resource called ExternalSecret that is responsible for interacting with your secrets stored in vaults and then creating a secret out of the box in respective namespaces.

This is a high-level architecture on how ESO works. Now, let's try some hands-on.

Getting Started with ESO

ESO installation is pretty straightforward. We need to install the ESO controller and then we are ready to get our external secrets into Kubernetes. For controller installation, we will use the helm. It is the Kubernetes package manager. For more information about helm, feel free to read the below blog.

Execute the following commands to install ESO Controller

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
   external-secrets/external-secrets \
   -n external-secrets \
   --create-namespace

Once the controller is deployed, you can see all three pods up and running in external-secrets namespace as shown in the below image.

Yayy!! The controller is up and running. It is all you need to get started with External Secrets. Now let's take the example of AWS Secrets Manager and fetch secrets from AWS and create Kubernetes secrets.

AWS Secret Manager with ESO

Execute the following steps to create a secret in AWS Secret Manager and get it into Kubernetes using ESO.

Step-1: Create a secret using GUI named user-creds with two key-value pairs, username & password. Once created, you can see the dashboard something like this -

Step-2: Let's now create a generic secret with an access key and secret access key that would be used by SecretStore for authentication. Execute the below command to create a secret.

[Note: Make sure KeyId & SecretKey generated should have access to the secrets manager]

echo -n 'KEYID' > ./access-key
echo -n 'SECRETKEY' > ./secret-access-key
kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key -n external-secrets

Step-3: Create SecretStore uses the generic secret for authentication. Execute the following manifest to create SecretStore named cncf-thane-demo

cat <<EOF | kubectl apply -f - 
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: cncf-thane-demo
  namespace: external-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key
EOF

Step-4: Now create a ExternalSecret would fetch the secrets from the AWS secrets manager and bring them to Kubernetes. Execute the following manifest to create ExternalSecret named cncf-thane-demo and provide the necessary details to fetch secrets.

cat <<EOF | kubectl apply -f - 
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: cncf-thane-demo
  namespace: external-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: cncf-thane-demo
    kind: SecretStore
  target:
    name: eso-secret
    creationPolicy: Owner
  data:
  - secretKey: credentials
    remoteRef:
      key: user-creds
      property: username
EOF

After ExternalSecrets being deployed, you can check the secret it created. In our case, it has created a secret named eso-secret that we have specified in ExternalSecret manifest. Let's now check if we get the value of the username key that we have added in the secrets manager.

kubectl get secrets eso-secret -nexternal-secrets -o jsonpath='{.data.credentials}' | base64 -d

Hurray!! We have successfully fetched the value from AWS Secret Manager and imported it into Kubernetes secrets using ESO. We can also use ESO to get multiple secrets or even the plan text secrets into Kubernetes in a similar fashion.

Conclusion

In this blog post, we have seen the need for an External Secrets Operator, KES and how to get started with ESO to fetch secrets from AWS Secrets Manager. Feel free to connect with me on Twitter and LinkedIn and ask your questions if you have any.