Tutorials
12 min read

Managing Kubernetes Secrets with the External Secrets Operator and Doppler

Automated secrets sync from Doppler to Kubernetes.

Oct 25, 2022
Tyler Langlois Avatar
Tyler Langlois
Managing Kubernetes Secrets with the External Secrets Operator and Doppler
Back to the blog
Managing Kubernetes Secrets with the External Secrets Operator and Doppler
Share
Tutorials

API keys, credentials, and other types of sensitive information are the primary way your application calls external services and interacts with the world outside its own runtime. Ensuring that your secrets are secure is one of the most important operational tasks that you need to address, but it's not just rigorous security that's important—providing an ergonomic, audited, and maintainable flow for the management of those secrets is crucial as well. If managing credentials is confusing or difficult, mistakes can undo all of the careful work that goes into keeping secrets secure.

Container orchestrators like Kubernetes offer helpful abstractions to address the need for sensitive values with native secrets. However, a Kubernetes Secret is a somewhat rudimentary object—it lacks encryption by default and normally exists as a plain key/value within etcd. By contrast, services like Vault or Doppler are intentionally designed to provide strong guarantees and well-defined access controls. Can we combine the operational power of Kubernetes with the assurance of a fully managed secret provider?

Yes! The External Secrets Operator is a Kubernetes operator that bridges the gap between Kubernetes' native secret support and external systems that provide a canonical source of truth for secret storage. It does this by leveraging custom resources that define how to retrieve external secrets in order to manage the lifecycle of secret resources in your Kubernetes cluster. In this tutorial, we'll use Doppler as the external secret provider to illustrate using external secrets in a real-world application.

Outcomes

The end goal of this guide is to leverage well-managed secrets in an application deployed in Kubernetes. To achieve this, we'll use:

When finished, you'll understand how to store and manage secrets in Doppler, use those same secrets as native Kubernetes secrets, and ultimately use them in a running application. Let's begin!

Prerequisites

This article assumes that you'll follow along with your own Kubernetes cluster. You may choose to operate in an existing cluster (potentially within a sandboxed namespace), but if you'd like to use a sandbox instead, we suggest using either minikube or kind to create a local Kubernetes installation which will provide a safe environment for testing.

In order to provide a clean and pristine environment, we'll use minikube as the Kubernetes target.

In addition to a functional Kubernetes cluster, ensure that the following command-line tools are installed:

  • helm to install the External Secrets Operator Kubernetes resources. Use the helm quickstart guide to install helm.
  • The Doppler command line utility doppler using the Doppler CLI Guide documentation. Register for a Doppler account to create projects and store external secrets.
  • kubectl to interact with Kubernetes. kubectl is available for a wide range of operating systems, and the Kubernetes documentation provides comprehensive installation guides for kubectl.
  • For a local Kubernetes sandbox, install minikube and follow the instructions to create a local instance of Kubernetes. This can be as simple as minikube start, but consult the documentation if you need additional assistance. minikube bundles a version-compatible installation of kubectl (which saves time) if you choose to use minikube for this exercise

Once installed, confirm that your environment is ready:

The kubectl version command should confirm that kubectl is present and communicating with the Kubernetes API successfully. If you aren't using minikube, simply use the plain kubectl command for the remainder of the tutorial.

1minikube kubectl -- version

The helm and doppler commands should be available:

1helm version
2doppler --version

With the prerequisites installed, let's proceed with building and running our program.

Sample Application

Before we start creating secrets, let's begin by illustrating the external secrets workflow with a simple example service. The following guide will assume the use of minikube to build and load an application container image, so if your Kubernetes environment is different, you may need to adapt the instructions to work with your specific container registry strategy.

First, perform some initial steps to bootstrap a python Flask application:

1mkdir -p ~/tmp/secret-sauce
2cd ~/tmp/secret-sauce
3git init
4echo flask > requirements.txt
5python3 -m venv venv
6./venv/bin/pip install -r requirements.txt

Create a file named app.py that contains the following simple web application:

1from flask import Flask
2from os import environ
3
4app = Flask(__name__)
5
6sauce = environ.get('SECRET_SAUCE')
7
8@app.route("/")
9def index():    
10	if sauce:        
11  	return f"The secret sauce is: {sauce}!"    
12  else:        
13  	return "You'll never find my secret sauce."

Define a small Dockerfile that defines how to build a container image for our application:

1FROM python:3
2
3WORKDIR /usr/src/app
4COPY requirements.txt ./
5RUN pip install --no-cache-dir -r requirements.txt
6COPY . .
7CMD [ "python", "-m" , "flask", "run", "--host=0.0.0.0"]

You can run this application locally with:

1./venv/bin/flask run

Try accessing the running application at http://localhost:5000 to see a response. The root route (/) will render a static string when no secret is present in the environment but will print the secret when the indicated environment variable is defined. Don't print secrets in production! This application is just a demonstration of how to read the value in a real program.

We're ready to load this application into Kubernetes. Assuming that you're following along with minikube, proceed to build the application container in the Kubernetes node under the name "app":

1minikube image build -t app .
2# >> ...lots of output...
3# >> Successfully tagged app:latest

Create a new Kubernetes Deployment file named deployment.yaml:

1apiVersion: apps/v1
2kind: Deployment
3metadata:  
4	name: app
5 spec:  
6 	replicas: 1  
7  selector:    
8  	matchLabels:      
9    	app: flask  
10  template:    
11  	metadata:      
12    	labels:        
13      	app: flask    
14    spec:      
15    	containers:        
16      	- name: webapp          
17        image: app          
18        imagePullPolicy: Never          
19        ports:            
20        	- containerPort: 5000

Take note of a few assumptions in this Deployment:

  • imagePullPolicy has been set to Never because we're running a locally-built image. By default, Kubernetes would attempt to pull the app image, which doesn't exist. As previously mentioned, if you're following this tutorial in an environment other than minikube, you may need to push the container image to a registry and adjust these settings slightly.
  • We've exposed port :5000 which we can access later.

Load this Deployment into your running cluster:

1kubectl apply -f deployment.yaml
2# >> deployment.apps/app created

Finally, forward the port in another terminal window to access the running application in a simple tunnel.

1kubectl port-forward deployment/app 5000:5000
2Forwarding from 127.0.0.1:5000 -> 5000
3Forwarding from [::1]:5000 -> 5000

Send a request to your Kubernetes application to see it in action:

1curl http://localhost:5000
2# >> You'll never find my secret sauce.

Fantastic! Note that we've received the response that indicates no secret value has been injected into the environment. How can we add a secret to the application and see the secret sauce?

Doppler

By using Doppler, we can achieve a great deal of control over application secrets and manage them at each step of our application's lifecycle, from in our local development environment to within a Kubernetes workload.

If you haven't signed up with Doppler, you can do so now. Once you have an account, proceed to use the login command to set up your account locally:

1doppler login

Setup Doppler for the demo application by defining a doppler-template.yaml file in the root of the application directory. This YAML file defines a template that we can import to create a new Doppler project easily from the command line:

1projects:  
2 - name: secret-sauce    
3   description: Kubernetes demo app    
4   environments:      
5   - slug: dev        
6     name: Development        
7     configs:          
8     - slug: dev      
9   - slug: stg        
10     name: Staging        
11     configs:         
12     - slug: stg      
13   - slug: prd        
14     name: Production       
15     configs:          
16     - slug: prd    
17  secrets:      
18   dev:        
19    SECRET_SAUCE: tartar      
20   stg:        
21    SECRET_SAUCE: horseradish      
22   prd:        
23    SECRET_SAUCE: tzatziki

Enter the directory in your shell and run the following doppler command in order to bootstrap your Doppler project. This will create a new project called secret-sauce, set up a few different environments, and load an initial secret value for SECRET_SAUCE:

1doppler import

You'll see output similar to the following:

1┌──────────────┬──────────────┬─────────────────────┬──────────────────────────┐
2ID           │ NAME         │ DESCRIPTION         │ CREATED AT               │
3├──────────────┼──────────────┼─────────────────────┼──────────────────────────┤
4secret-sauce │ secret-sauce │ Kubernetes demo app │ 2022-10-11T22:10:36.567Z │
5└──────────────┴──────────────┴─────────────────────┴──────────────────────────┘

View the secrets for this project with doppler secrets:

1doppler secrets

In addition to some default variables that begin with DOPPLER, you'll also find the secret value we'd like to inject, SECRETSAUCE:

1┌─────────────────────┬──────────────┐
2│ NAME                │ VALUE        │
3├─────────────────────┼──────────────┤
4│ DOPPLER_CONFIG      │ dev5│ DOPPLER_ENVIRONMENT │ dev6│ DOPPLER_PROJECT     │ secret-sauce │
7│ SECRET_SAUCE        │ tartar       │
8└─────────────────────┴──────────────┘

Run a quick test to confirm that our application behaves as expected when a secret is present in its environment variables. To do this, we can use the doppler run command to easily inject the variable into our application.

1doppler run -- ./venv/bin/flask run

Then issue a request to the listening port to see whether the response has changed:

1curl http://localhost:5000
2# >> The secret sauce is: tartar!

Great! Let's continue on to install the External Secrets Operator.

External Secrets

The External Secrets Operator provides the translation layer between Kubernetes' native secrets and external secrets. The operator leverages custom resources in order to model external secrets, which it then retrieves as necessary and translates into native Kubernetes secrets that your workload can easily consume.

The operator is provided as a Helm chart, so first add the upstream external-secrets Helm repository to access the chart:

helm repo add external-secrets https://charts.external-secrets.io
# >> external-secrets has been added to your repositories

Next, install the chart into your Kubernetes cluster. The following command:

  • Instructs helm to use the external-secrets chart,
  • passes -n to install the chart into the external-secrets namespace,
  • creates the namespace as necessary with --create-namespace, and
  • configures the requisite CRDs as well (--set installCRDs=true)
1helm install external-secrets \     
2	external-secrets/external-secrets \     
3  -n external-secrets \     
4  --create-namespace \     
5  --set installCRDs=true

You should see output similar to the following:

1NAME: external-secrets
2LAST DEPLOYED: Tue Oct 11 12:27:18 2022
3NAMESPACE: external-secrets
4STATUS: deployed
5REVISION: 1
6TEST SUITE: None
7NOTES:
8external-secrets has been deployed successfully!

In order to begin using ExternalSecrets, you will need to set up a SecretStore or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).

More information on the different types of SecretStores and how to configure them can be found in our GitHub repository.

With this operator installed, we're now ready to set up Doppler to serve as the source for external secrets.

Integrating Doppler with Kubernetes

The first step is to create a service token that serves as the authentication mechanism for the external secrets operator against Doppler. It should be stored inside of a generic Kubernetes secret that the operator will consume.

You can generate and store the token in a single step with some fancy footwork in the shell to avoid copying and pasting the token around. The following command creates a new Doppler token and then immediately loads it into Kubernetes:

1kubectl create secret generic \    
2	doppler-token-auth-api \    
3  --from-literal dopplerToken=$(doppler configs tokens create --config prd doppler-auth-token --plain)
4# >> secret/doppler-token-auth-api created

Note that we're passing the --config prd flag to doppler to create a token scoped to the prd (production) configuration of our Doppler project. In our application directory, we defaulted to dev, which has its own secrets. With this technique, we're only interacting with development secrets locally while loading production secrets into our container runtime, which keeps the risk of exposure for production secrets minimal.

Next, create a SecretStore CRD that points the operator at your Doppler service token secret. This step sets up Doppler as a source for external secrets that we can call upon later.

1apiVersion: external-secrets.io/v1beta1
2kind: SecretStore
3metadata:  
4	name: doppler-auth-api
5spec:  
6	provider:    
7  	doppler:      
8    	auth:        
9      	secretRef:          
10        	dopplerToken:            
11          	name: doppler-token-auth-api            
12            key: dopplerToken

Create the SecretStore in Kubernetes:

1kubectl apply -f secretstore.yaml
2# >> secretstore.external-secrets.io/doppler-auth-api created

We're ready to inject our secret sauce! A new ExternalSecret is the necessary resource that enables us to synchronize one of our Doppler secrets to a related generic Kubernetes secret. Create a new file called externalsecret.yaml with the following YAML:

1apiVersion: external-secrets.io/v1beta1
2kind: ExternalSecret
3metadata:  
4	name: secret-sauce
5spec:  
6	secretStoreRef:    
7  	kind: SecretStore    
8    name: doppler-auth-api  
9  target:    
10  	name: secret-sauce  
11  data:    
12  	- secretKey: SECRET_SAUCE      
13    	remoteRef:        
14      	key: SECRET_SAUCE

Load it into Kubernetes:

1kubectl apply -f externalsecret.yaml
2# >> externalsecret.external-secrets.io/secret-sauce created

You can confirm that the secret was loaded by the operator by listing it with kubectl:

1kubectl get secret secret-sauce
2# >> NAME TYPE DATA AGE secret-sauce Opaque 1 3s

Success! Note that if you do not see the new secret, you can also inspect the External Secret Operator's logs to debug any issues:

1kubectl -n external-secrets logs -lapp.kubernetes.io/name=external-secrets

Generic secrets can be seen in the Kubernetes dashboard as well. If you're using minikube, you can quickly install and view the Kubernetes dashboard with the following command:

1minikube dashboard

Your browser will open to the dashboard landing page. You may use the left sidebar to navigate to the "Secrets" link to view your new secret:

The only task left to do is to consume the secret from our application. Here's an updated Deployment that references the newly-created Secret. Note the new env property, which injects the variable SECRET_SAUCE drawn from a secret named secret-sauce under the key SECRET_SAUCE:

1apiVersion: apps/v1
2kind: Deployment
3metadata:  
4	name: app
5spec:  
6	replicas: 1  
7  selector:    
8  	matchLabels:      
9    	app: flask  
10  template:    
11  	metadata:      
12    	labels:        
13      	app: flask    
14    spec:      
15    	containers:        
16      	- name: webapp          
17        	image: app          
18          imagePullPolicy: Never          
19          env:            
20          	- name: SECRET_SAUCE              
21            	valueFrom:                
22              	secretKeyRef:                  
23                	name: secret-sauce                  
24                  key: SECRET_SAUCE          
25          ports:            
26          	- containerPort: 5000

Apply the updated Deployment:

1kubectl apply -f deployment.yaml deployment.apps/app configured

The changed Deployment manifest will recreate any necessary pods, so invoke a new forwarded port once the pods have restarted to forward requests to the new containers:

1kubectl port-forward deployment/app 5000:5000

Finally, Issue an HTTP request to the running container to see the injected Doppler secret in action:

1curl http://localhost:5000
2# >> The secret sauce is: tzatziki!

Note that the content of this secret differs from the one we saw in our local development environment. The --config prd flag to the doppler command loaded a token with rights to the prd configuration in Doppler, which injects the correct secret for the configuration that the Doppler token has been scoped to.

Congratulations! You've successfully:

  • Installed the External Secrets Operator into your Kubernetes cluster
  • Managed an application secret with Doppler
  • Connected Doppler with the External Secrets Operator
  • Used a Doppler project secret within a Kubernetes Deployment

What's Next?

There's even more you can do with the External Secrets Operator and Doppler, so to learn more, dive into the documentation to learn how to use features like JSON processing, filtering, and more.

If you're actively using Kubernetes secrets stored in etcd today, you should also configure encryption at rest so that your secrets are secure no matter where they're stored. Whether you load them with kubectl or the External Secrets Operator, configuring encryption at rest is still important to address at every step of your secret management lifecycle.

Manage Kubernetes secrets and more, effortlessly, with Doppler >

Stay up to date with new platform releases and get to know the team of experts behind them.

Related Content

Explore More