Kubernetes

How to Manage Secrets in Kubernetes – A Complete Guide

Kubernetes secrets

Almost every software application is guaranteed to have some secret data. This secret data can range from database credentials to TLS certificates or access tokens to establish secure connections.

The platform you build your application on should provide a secure means for managing this secret data. This is why Kubernetes provides an object called Secret to store sensitive data you might otherwise put in a Pod specification or your application container image. 

In this article, you will learn what a Kubernetes Secret is, its built-in types, ways to create, view, decode, and edit them, and how to use them in Pods. Also, in the end, you will learn the best practices to follow when using Secrets. 

What are Kubernetes Secrets?

In Kubernetes, a Secret is an object that contains a small amount of sensitive data such as login usernames and passwords, tokens, keys, etc. The primary purpose of Secrets is to reduce the risk of exposing sensitive data while deploying applications on Kubernetes.

To adequately explain what Kubernetes Secrets are, the following are key points about Kubernetes secrets:

  • You create Secrets outside of Pods — you create a Secret before any Pod can use it.
  • When you create a Secret, it is stored inside the Kubernetes data store (i.e., an etcd database) on the Kubernetes Control Plane. 
  • When creating a Secret, you specify the data and/or stringData fields. The values for all the data field keys must be base64-encoded strings. Suppose you don’t want to convert to base64. In that case, you can choose to specify the stringData field instead, which accepts arbitrary strings as values.
  • When creating Secrets, you are limited to a size of 1MB per Secret. This is to discourage the creation of very large secrets that could exhaust the kube-apiserver and kubelet memory. 
  • Also, when creating Secrets, you can mark them as immutable with immutable: true. Preventing changes to the Secret data after creation. Marking a Secret as immutable protects from accidental or unwanted updates that could cause application outages.
  • After creating a Secret, you inject it into a Pod either by mounting it as data volumes, exposing it as environment variables, or as imagePullSecrets. You will learn more about this later in this article.

Types of Kubernetes Secrets

The following are several types of Kubernetes Secrets:

  • Opaque Secrets: Opaque Secrets are used to store arbitrary user-defined data. Opaque is the default Secret type, meaning that when creating a Secret and you don’t specify any type, the secret will be considered Opaque.
  • Service account token Secrets: You use a Service account token Secret to store a token credential that identifies a service account. It is important to note that when using this Secret type, you must ensure that the kubernetes.io/service-account.name annotation is set to an existing service account name.
  • Docker config Secrets: You use a Docker config secret to store the credentials for accessing a container image registry. You use Docker config secret with one of the following type values:
    • kubernetes.io/dockercfg
    • kubernetes.io/dockerconfigjson
  • Basic authentication Secret: You use this Secret type to store credentials needed for basic authentication.  When using a basic authentication Secret, the data field must contain at least one of the following keys:
    • username: the user name for authentication
    • password: the password or token for authentication
  • SSH authentication secrets: You use this Secret type to store data used in SSH authentication. When using an SSH authentication, you must specify a ssh-privatekey key-value pair in the data (or stringData) field as the SSH credential to use.
  • TLS secrets: You use this Secret type to store a certificate and its associated key typically used for TLS. When using a TLS secret, you must provide the tls.key and the tls.crt key in the configuration’s data (or stringData) field. 
  • Bootstrap token Secrets: You use this Secret type to store bootstrap token data during the node bootstrap process. You typically create a bootstrap token Secret in the kube-system namespace and named it in the form bootstrap-token-<token-id>.

To learn more about these Secret types, check out their documentation.

Ways to Create Kubernetes Secrets

To create Kubernetes Secrets, you can use one of the following methods:

Prerequisites

Before you learn how to use each of the above methods, ensure you have the following prerequisites:

For demo purposes, the Secrets you will create below will store hypothetical credentials (username — admin and password — password) required by Pods to access a database. 

Also, create a namespace to store the demo resources for easy cleanup:

$ kubectl create namespace secrets-demo

Creating Kubernetes Secrets Using kubectl

There are two ways of providing the Secret data to kubectl when creating Secrets using Kubectl, and there are:

  • Providing the secret data through a file using the --from-file=<filename> tag or
  • Providing the literal secret data using the --from-literal=<key>=<value> tag

This article will use the file method.

It is important to note that when providing the secret data --from-literal=<key>=<value> tag, special characters such as $, \, *, =, and ! require escaping. However, you can easily escape in most shells with single quotes (‘).

To start creating a Secret with kubectl providing the Secret data from a file in any directory of your choice. Create files to store the hypothetical user credentials with the following command:

$ echo -n 'admin' > username.txt
$ echo -n 'password' > password.txt

The -n flag in the above command ensures that no newline character is added at the end of the text. This is crucial since kubectl will encode the extra newline character if present when it reads the file and turns the content into a base64 string.

After running the above commands, you can verify that the password and username were written to the file with the cat command, as in the image below.

kubernetes secrets verify password

Now, create the Kubernetes Secret with the files using the kubectl command below:

$ kubectl create secret generic database-credentials \  
    --from-file=username.txt \ 
    --from-file=password.txt \
    --namespace=secrets-demo

The generic subcommand tells kubectl to create the Secret with Opaque type. The above command will output the following:

kubernetes secrets Opaque type

Note: When using the above command, the key of your secret data will be the filename (username.txt and password.txt) by default.  To provide keys for the Secret data, use the following syntax --from-file=[key=]source, for example:

kubectl create secret generic database-credentials \
--from-file=username=username.txt \
--from-file=password=password.txt \
--namespace=secrets-demo

To verify the Secret creation, run the following command:

$ kubectl -n secrets-demo get secrets

The above command will show an output similar to the image below.

Creating Secrets Using a Manifest File

Before you create a Secret using a manifest file, you must first decide how you want to add the Secret data using the data field and/or the stringData field.

Using the data field, you must encode the secret data using base64.  To convert the username and password to base64, run the following command:

echo -n 'admin' | base64
echo -n 'password' | base64

After running the above command, you will get an output similar to the image below. Copy the base64 values and store them as that is what you will put in your manifest file.

kubernetes secrets base64

Now create a demo-secret.yaml manifest file using any means you prefer (text editor, vim or nano, etc.) and add the following configuration.

apiVersion: v1
kind: Secret
metadata:
  name: demo-secret
type: Opaque
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQ=

In the above manifest file, the username and password values in the data field are the base64 encoded values of the original credentials. 

When using the stringData field, the manifest file will be:

apiVersion: v1
kind: Secret
metadata:
  name: demo-secret
type: Opaque
stringData:
  username: admin
  password: password

To create the Secret, run the following command:

$ kubectl -n secrets-demo apply -f demo-secret.yaml

After running the above command, you should get an output similar to the image below.

Creating Secrets Using a Generator Like Kustomize

Using a resource Generator like Kustomize can help you create Kubernetes Secrets quickly.

To create a Secret using Kustomize, first create a kustomization.yaml file. In that file, define a secretGenerator to reference one of the following:

  • Files that store the secret data,
  • The unencrypted literal version of the secret data values,
  • Environment variable (.env) files.

You don’t need to base64 encode the values with all these methods.

When referencing Secret data files, you define the secretGenerator like:

secretGenerator:
- name: database-credentials
  files:
  - username.txt
  - password.txt

When using the literal version of the data values, you define the secretGenerator like:

secretGenerator:
- name: database-credentials
  literals:
  - username=admin
  - password=password

When using .env files, you define the secretGenerator like:

secretGenerator:
- name: database-credentials
  envs:
  - .env.secret

Create the kustomization.yaml file, and paste either of the first two options.

Then in the same directory as the file, generate the Secret with the following kubectl command:

$ kubectl -n secrets-demo apply -k .

After running the above command, you should see an output similar to the image below.

create kubernetes secrets with kustomize

So far, you’ve learned what Kubernetes secrets are, its built-in types, and the methods you can use to create them. Next, you will learn how to describe a Secret, decode a Secret, edit Secret values, and finally, how to use a Secret in Pods. 

Describing a Kubernetes Secret Using kubectl describe

Using the kubectl describe, you can view some basic information about Kubernetes objects. To use it to view the description of one of the Secrets you’ve created in the article, run:

$ kubectl -n secrets-demo describe secrets/database-credentials

After running the above command, you will get an output similar to the image below.

kubernetes secret kubectl describe

If you notice, the above output doesn’t show the contents of the Secret. This is to protect the Secret from being exposed or logged in the terminal. 

To view the Secret data, you will need to decode the secret. 

Decoding a Kubernetes Secret

To view the data of the Secret you created, run the following command:

$ kubectl -n secrets-demo get secret database-credentials -o jsonpath='{.data}'

After running the above commands, it will output the encoded key-value pairs of the secret data as in the image below. 

kubernetes secrets encoded key-value pairs

To decode the encoded strings, you can use the following command:

$ echo 'YWRtaW4=' | base64 --decode
$ echo 'cGFzc3dvcmQ=' | base64 --decode

After running the above commands you should see an output similar to the image below.

kubernetes secrets decode strings

Note: If you do the above, you could store the Secret data in your shell history. To avoid that, combine the previous two steps into one command like the one below.

$ kubectl get secret database-credentials -o jsonpath='{.data.password}' | base64 --decode

Editing a Kubernetes Secret

To edit the content of the Secret you created, run the following kubectl command:

$ kubectl -n secrets-demo edit secrets database-credentials

The above command will open your terminal’s default editor to allow you to update the base64 encoded Secret data in the data field as in the image below.

editing kubernetes secrets

Note: It is important to note that when you set a Secret as immutable upon creation, you can’t edit it. Nonetheless, you can edit any existing mutable Secret to make it immutable by adding immutable: true in the manifest file like the following:

apiVersion: v1
kind: Secret
metadata:
  ...
data:
  ...
immutable: true

How to Use Kubernetes Secrets

The following are the three main ways a Pod can use a Secret:

  • As container environment variables 
  • As files in a volume mounted on one or more of its containers.
  • By the kubelet when pulling images for the Pod — imagePullSecrets.

Using Secret data as container environment variables

For demo purposes, below is a Pod manifest with the Kubernetes Secret data you created exposed as environment variables. Create a secret-test-env-pod.yaml and paste the configuration in it.

apiVersion: v1
kind: Pod
metadata:
  name: env-pod
spec:
  containers:
    - name: secret-test
      image: nginx
      command: ['sh', '-c', 'echo "Username: $USER" "Password: $PASSWORD"']
      env:
        - name: USER
          valueFrom:
            secretKeyRef:
              name: database-credentials
              key: username.txt
        - name: PASSWORD
          valueFrom:
            secretKeyRef:
              name: database-credentials
              key: password.txt

Create the Pod using the following kubectl command:

$ kubectl -n secrets-demo apply -f secret-test-evn-pod.yaml

To verify that Kubernetes mounted the Secret on the Pod, describe the Pod with the following command:

# "env-pod" is the Pod name as in the above manifest file
$ kubectl -n secrets-demo describe pod env-pod

After running the above command, you should see an output similar to the image below.

kubernetes secrets env-pod

Also, seeing the echo command in the Pod manifest file, you can verify by checking the logs of the Pod with:

$ kubectl -n secrets-demo logs env-pod

The above command will output the Secret data as in the image below.

Using Secret data as files in a volume mounted on a Pod’s container(s)

For demo purposes, below is a Pod manifest with the Kubernetes Secret data you created as files in a volume mounted on the Pod’s containers. 

Create a secret-test-volume-pod.yaml and paste the configuration in it.

apiVersion: v1
kind: Pod
metadata: 
  name: volume-test-pod
spec:
  containers:
  - name: secret-test
    image: nginx
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/config/secret
  volumes:
  - name: secret-volume
    secret:
      secretName: database-credentials

Create the Pod using the following kubectl command:

$ kubectl -n secrets-demo apply -f secret-test-volume-pod.yaml

To verify that the Pod can access the Secret data, connect to the container and run the following commands in the volume directory:

$ kubectl -n secrets-demo exec volume-test-pod -- cat /etc/config/secret/username.txt

$ kubectl -n secrets-demo exec volume-test-pod -- cat /etc/config/secret/password.txt

$ kubectl -n secrets-demo exec volume-test-pod -- ls /etc/config/secret

The above commands will have an output similar to the image below.

volume mounted on a Pod’s container

kubelet using Secret data when pulling images for a Pod

You use this way when you want to fetch container images from a private repository. These secrets are configured at the Pod level.

To use this method, you configure the Secret data in the imagePullSecrets field of your Pod manifest file. The kubelet on each node will authenticate to that repository. 

apiVersion: v1
kind: Pod
metadata:
  ...
spec:
  containers:
    ...
  imagePullSecrets:
    - name: <your-registry-key>

To learn more about using Secret data when pulling images for a Pod and how to specify imagePullSecrets, check out this documentation.

Cleaning up

Clean up the entire setup by deleting the namespace which deletes all the secrets and Pods you created with the following command:

$ kubectl delete ns secrets-demo

Best Practices to Follow When Using Kubernetes Secrets

By default, Kubernetes stores Secrets unencrypted in the etcd data store. To safely use Kubernetes Secrets, take at least the following steps:

  1. Enable Encryption at Rest for Secrets.
  2. Set least-privilege access to Secrets as the default setting with RBAC rules.
  3. Only allow certain containers to have access to a certain Secret.
  4. Use third-party Secret store providers, if possible.

For more guidelines on how to safely use Secrets, read the following documentation:

You can also take a look at how Spacelift helps you manage the complexities and compliance challenges of using Kubernetes. Anything that can be run via kubectl can be run within a Spacelift stack. Find out more about how Spacelift works with Kubernetes, and get started on your journey by creating a free trial account.

Key Points

In this article, you learned what a Kubernetes Secret is, its built-in types, ways to create, view, decode, and edit them, and how to use them in Pods. Also, you learned the best practices to follow when using Secrets.

To dive deeper into Kubernetes Secrets, check out its API reference

The most Flexible CI/CD Automation Tool

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities s for infrastructure management.

Start free trial