Kubectl is the default tool for directly accessing your Kubernetes cluster from the command line. It allows you to create, update, and delete Kubernetes resources.
Commands such as ‘kubectl get’ give you a brief summary of your workloads, but they don’t really dive deep into what’s happening in the backend and give you the full breakdown. However, ‘kubectl describe’ can show you detailed information about your cluster’s workloads.
In this article, we will examine the kubectl describe command and demonstrate how to use it for Kubernetes pods, deployments, services, nodes, and more. We will also look at some common use cases and best practices for running this command.
What we will cover:
The kubectl describe
command retrieves and displays detailed data from a single resource or multiple resources in your cluster. This command can be helpful when you are debugging specific issues in your Kubernetes workloads and need to see more than just a brief summary of your resources.
Here’s the basic syntax:
kubectl describe <resource> <resource-name> [flags]
The output you receive from kubectl describe
shows the resource’s current state, namespace, start times, labels, events, replicas, resource configurations, environment variables, and other details specified in the YAML manifest.
kubectl describe vs. kubectl logs
You might wonder why you can’t just use kubectl logs
to troubleshoot an issue with a pod.
kubectl logs
command provides in-depth logging information that can take time to read and understand what is really wrong with your resource.
kubectl describe
, on the other hand, gives you a set of events that you can quickly read through and get to the root cause right away. This makes ‘kubectl describe’ a better first step to debug and troubleshoot your Kubernetes resources.
kubectl describe
can be used with a wide range of Kubernetes resource types such as pods, nodes, Deployments, Services, ReplicaSets, DaemonSets, and more. The describe
output varies for each resource type. Some kubectl describe
outputs are not that useful and running the describe
command against another resource type retrieves more information.
For example, kubectl describe deployment deployment-name
might not have all the information you are seeking. In that case, use the describe command against the individual pod in that deployment by running kubectl describe pod pod-name
. This enables you to retrieve concise information directly from the affected specific resource instead of getting minimal information from the deployment as a whole.
Let’s explore some examples of using kubectl describe
against different resource types and see the outputs it generates.
kubectl describe pod
The kubectl describe pod
command is provides detailed information about a specific pod running in your Kubernetes cluster. This information could include the pod’s metadata, status, events, and resource usage.
kubectl describe pod <pod-name> -n <namespace>
- <pod-name>: The name of the pod you want to inspect.
- -n <namespace>: The namespace where the pod is running. If you don’t specify the namespace, it will default to the current namespace or the default namespace.
It provides information on:
- Name and Namespace: The name and namespace of the pod
- Priority: The priority of the pod
- Start Time: The timestamp indicating when the pod was created
- Labels and Annotations: Any key-value pairs for identification (labels) and additional metadata (annotations) related to the pod
- Node: The node where the pod is currently running
- Status: The current state of the pod (e.g., Running, Pending, Failed)
- IP Addresses: The pod’s internal and external IP addresses
- Containers: Each container’s image, state, readiness probes, resource limits, and more — plus, events like crashes or restarts, including the last state of the container
- Volumes: Information about any volumes attached to the pod
- Events: Recent events that have affected the pod, such as scheduling issues, restarts, or failures
Example 1
In the following example, we are using kubectl describe pod nginx-pod
to retrieve more in-depth information about the pod. Your Nginx pod is running fine, and there are no issues to report.
As you can see, the State
is set to Running
, all Conditions
are set to True
, and nothing has been picked up on the Events
section.
~ ➜ kubectl describe pod nginx-pod
Name: nginx-pod
Namespace: default
Priority: 0
Node: node01/192.1.169.10
Start Time: Tue, 03 Sep 2024 03:03:17 +0000
Labels: run=nginx-pod
Annotations: <none>
Status: Running
IP: 10.244.1.2
IPs:
IP: 10.244.1.2
Containers:
nginx-pod:
Container ID: containerd://973aa7b5391c3042f83d88ead6efe820614c3a8e852bffab95ca7e3530f19c65
Image: nginx
Image ID: docker.io/library/nginx@sha256:447a8665cc1dab95b1ca778e162215839ccbb9189104c79d7ec3a81e14577add
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 03 Sep 2024 03:03:22 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7p6gq (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-7p6gq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30m default-scheduler Successfully assigned default/nginx-pod to node01
Normal Pulling 30m kubelet Pulling image "nginx"
Normal Pulled 30m kubelet Successfully pulled image "nginx" in 4.277827536s
Normal Created 30m kubelet Created container nginx-pod
Normal Started 30m kubelet Started container nginx-pod
Example 2
Let’s now review another kubectl describe pod
example with errors, so we can compare and see how utilizing a kubectl describe with pods can be helpful.
Here, you can see the ‘Status’ is set to Pending
, which points out that there is something with the pod that is preventing it from going into a running state.
The State
section shows ErrImagePull
, another clear identifier that the issue is related to this pod’s image.
The Conditions
section also gives us some details on how the status of the container and pod is set to ‘False’, therefore unable to run.
We can also review the Events
section, which will provide the most details regarding the errors generated from the pod and why it failed to run. The error message says failed to pull and unpack image
because we entered an invalid image name, which Kubernetes was unable to find (image name = ‘nginxxxx’).
~ ➜ kubectl describe pod nginx-pod-error
Name: nginx-pod-error
Namespace: default
Priority: 0
Node: node01/192.1.169.10
Start Time: Tue, 03 Sep 2024 03:41:32 +0000
Labels: run=nginx-pod-error
Annotations: <none>
Status: Pending
IP: 10.244.1.6
IPs:
IP: 10.244.1.6
Containers:
nginx-pod-error:
Container ID:
Image: nginxxxx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7qfz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-s7qfz:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16s default-scheduler Successfully assigned default/nginx-pod-error to node01
Normal Pulling 16s kubelet Pulling image "nginxxxx"
Warning Failed 15s kubelet Failed to pull image "nginxxxx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginxxxx:latest": failed to resolve reference "docker.io/library/nginxxxx:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning Failed 15s kubelet Error: ErrImagePull
Normal BackOff 15s kubelet Back-off pulling image "nginxxxx"
Warning Failed 15s kubelet Error: ImagePullBackOff
kubectl describe deployment
Now, we will review the details you receive from running kubectl describe
against deployments. The output is a bit more concise than the output you receive from describing pods because the majority of the information is on the pod level.
kubectl describe deployment
provides a detailed overview of a Kubernetes deployment’s configuration and status. It offers insights into key deployment attributes, making it useful for understanding the overall state and behavior of a deployment.
kubectl describe deployment <deployment-name> -n <namespace>
This command provides information on:
- Creation Timestamp: When the deployment was first created
- Labels: Key-value pairs associated with the deployment
- Selector: Indicates the strategy being used for updates providing insight into how new versions of applications are deployed
- Replica status: The current status of replicas, including how many are desired, available, and up-to-date
- Rolling Update Strategy: Details on how updates to pods are managed incrementally to minimize downtime
- Pod Template: Details about the pod template used by the deployment, such as labels, annotations, container specifications (image, ports, environment variables), and resource requests/limits
- Image: The container image version being used in the deployment
- Volumes: Any volumes attached to the deployment
- Conditions: The current state of the deployment, including whether it is progressing as expected or encountering any issues
- Events: Recent events related to the deployment, such as scaling actions or failures
For troubleshooting more granular issues at the container or pod level (like crashes or resource limits), it is recommended to use kubectl describe pod
command, as it provides more detailed information specific to individual pods.
Let’s look at an example:
~ ➜ kubectl describe deployment nginx-deploy
Name: nginx-deploy
Namespace: default
CreationTimestamp: Tue, 03 Sep 2024 22:06:59 +0000
Labels: app=nginx-deploy
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=nginx-deploy
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx-deploy
Containers:
nginx:
Image: nginx
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deploy-99976564d (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 21s deployment-controller Scaled up replica set nginx-deploy-99976564d to 3
You can also try using kubectl describe replicaset
, but it will provide more or less be the same information as you received from kubectl describe deployment.
kubectl describe node
The kubectl describe node
command offers an in-depth view of the current state, health, and resource allocation of a specific node in the Kubernetes cluster. The output includes detailed node conditions, like whether the node is ready, disk pressure, memory pressure, and network issues.
kubectl describe node <node-name>
The output includes:
- Node Name and Labels: The unique name of the node, along with any labels associated with it
- Annotations: Any annotations that provide additional metadata about the node
- Addresses: Various addresses associated with the node, including internal IP and hostname
- Conditions: The current conditions of the node, such as:
- Ready: Whether the node is ready to accept pods
- Memory Pressure: If the node is experiencing memory pressure
- Disk Pressure: If there are issues related to disk space
- Capacity and Allocatable Resources: Details the total resources available on the node (CPU, memory) and what is allocatable for pods
- System Info: Information about the operating system, kernel version, and container runtime being used on the node
- Pods: Pods currently running on this node, including their statuses and resource usage
- Events: Recent events related to the node, such as scheduling failures or resource allocation issues
Let’s see an example output:
~ ➜ kubectl describe node controlplane
Name: controlplane
Roles: control-plane
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=controlplane
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
node.kubernetes.io/exclude-from-external-load-balancers=
Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"96:7f:cc:c3:40:d2"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.41.220.3
kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 05 Sep 2024 20:45:43 +0000
Taints: node-role.kubernetes.io/control-plane:NoSchedule
node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: controlplane
AcquireTime: <unset>
RenewTime: Thu, 05 Sep 2024 22:29:08 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 05 Sep 2024 20:46:06 +0000 Thu, 05 Sep 2024 20:46:06 +0000 FlannelIsUp Flannel is running on this node
MemoryPressure False Thu, 05 Sep 2024 22:28:14 +0000 Thu, 05 Sep 2024 20:45:37 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 05 Sep 2024 22:28:14 +0000 Thu, 05 Sep 2024 20:45:37 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 05 Sep 2024 22:28:14 +0000 Thu, 05 Sep 2024 20:45:37 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 05 Sep 2024 22:28:14 +0000 Thu, 05 Sep 2024 20:46:09 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.41.220.3
Hostname: controlplane
Capacity:
cpu: 18
ephemeral-storage: 1016057248Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 98888308Ki
pods: 110
Allocatable:
cpu: 18
ephemeral-storage: 936398358207
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 98785908Ki
pods: 110
System Info:
Machine ID: 15c49d4f47e04095aa13764b7e5c23f7
System UUID: bbdbe79a-1085-7763-21af-4585c812332a
Boot ID: ce8ebee8-715f-43ec-b865-96399fa78301
Kernel Version: 5.4.0-1106-gcp
OS Image: Ubuntu 18.04.6 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.6
Kubelet Version: v1.24.0
Kube-Proxy Version: v1.24.0
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (8 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-flannel kube-flannel-ds-h7xfr 100m (0%) 100m (0%) 50Mi (0%) 300Mi (0%) 103m
kube-system coredns-6d4b75cb6d-5vfrg 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 103m
kube-system coredns-6d4b75cb6d-7zssh 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 103m
kube-system etcd-controlplane 100m (0%) 0 (0%) 100Mi (0%) 0 (0%) 103m
kube-system kube-apiserver-controlplane 250m (1%) 0 (0%) 0 (0%) 0 (0%) 103m
kube-system kube-controller-manager-controlplane 200m (1%) 0 (0%) 0 (0%) 0 (0%) 103m
kube-system kube-proxy-p8n7w 0 (0%) 0 (0%) 0 (0%) 0 (0%) 103m
kube-system kube-scheduler-controlplane 100m (0%) 0 (0%) 0 (0%) 0 (0%) 103m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 950m (5%) 100m (0%)
memory 290Mi (0%) 640Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
kubectl describe service
In Kubernetes, a service is used to assign a name and unique IP address to a group of pods, allowing you to route traffic properly to your pods.
The kubectl describe service
command provides detailed information about a Kubernetes Service object, including metadata, specifications, and the current status of the service. It is useful for debugging and checking the configuration and status of the service.
kubectl describe service <service-name> -n <namespace>
The key sections of the output are:
- Name and Namespace: The name of the service and the namespace the service is running in
- Labels and Annotations: Any labels and annotations applied to the service
- Selector: The label selector that matches the Pods this service is directing traffic to
- Type: The type of the service (ClusterIP, NodePort, LoadBalancer, etc.)
- IP: The internal ClusterIP assigned to the service
- Ports: The ports that the service is exposing, including:
- Protocol (TCP/UDP)
- TargetPort (the port on the Pods)
- NodePort (if the service is of type NodePort).
- Endpoints: Lists the Pod IPs and ports the service is routing traffic to
- Session Affinity: Whether the service is sticky (affinitized) to a particular client
In the example below, Endpoints
are pointing to the internal IP address of the pod:
~ ➜ kubectl describe service redis
Name: redis
Namespace: default
Labels: app=redis
Annotations: <none>
Selector: app=redis
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.103.92.91
IPs: 10.103.92.91
Port: <unset> 6379/TCP
TargetPort: 6379/TCP
Endpoints: 10.244.1.2:6379
Session Affinity: None
Events: <none>
kubectl describe statefulset
Kubernetes StatefulSets are useful when you have specific workloads you need more control over in terms of consistency and data. StatefulSets allows you to have static workloads with a persistent storage and networking configuration and also set up ordered scaling and rolling updates, which prevents random issues from arising.
The kubectl describe statefulset
command provides a detailed description of a StatefulSet resource in your cluster. It generates similar results to what you would get from running kubectl describe against deployments but for persistent pods deployed through StatefulSets. The output includes the current state, configuration details, and events related to the StatefulSet.
kubectl describe statefulset <statefulset_name> -n <namespace>
Key sections of the output include:
- Name and Namespace: The name of the StatefulSet and the namespace where the StatefulSet resides
- Labels and Annotations: Labels and any additional metadata stored as annotations attached to the StatefulSet
- Selector: The label selector used to match pods with this StatefulSet
- Replicas: The number of desired and current replicas
- Pod Management Policy: The ordering and parallelism in pod creation
- Update Strategy: The strategy used for updating StatefulSet pods (e.g., RollingUpdate or OnDelete)
- Volume Claims: Information about persistent volume claims (PVCs) associated with each pod
- Pod Template: Details about how the pods are created, including container specifications, environment variables, and volumes
- Conditions: Any conditions related to the StatefulSet’s status (e.g., Available or Progressing)
- Events: Recent events related to the StatefulSet, such as scaling or failures
Let’s examine the example below:
~ ➜ k describe statefulset web
Name: web
Namespace: default
CreationTimestamp: Fri, 06 Sep 2024 22:56:42 +0000
Selector: app=nginx
Labels: <none>
Annotations: <none>
Replicas: 3 desired | 1 total
Update Strategy: RollingUpdate
Partition: 0
Pods Status: 0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: registry.k8s.io/nginx-slim:0.24
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/usr/share/nginx/html from www (rw)
Volumes: <none>
Volume Claims:
Name: www
StorageClass: my-storage-class
Labels: <none>
Annotations: <none>
Capacity: 1Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 13m statefulset-controller create Claim www-web-0 Pod web-0 in StatefulSet web success
Normal SuccessfulCreate 13m statefulset-controller create Pod web-0 in StatefulSet web successful
kubectl describe daemonset
Daemonset is a Kubernetes resource that allows you to run pods as a ‘daemon’ on your Kubernetes nodes. This gives you more control over which nodes your pods are distributed across. Daemonsets are mostly used to run pods that are always needed to run on each node, such as cluster storage, log collection agents, and monitoring systems.
When you run kubectl describe daemonset
it will give you a descriptive output for the specified DaemonSet, includes details like the pods it manages, its status, events, and more.
kubectl describe daemonset <daemonset-name> -n <namespace>
The key sections of the output:
- Name and Namespace: The name of the DaemonSet and the namespace where the DaemonSet is running
- Selector: The label selector used to determine which pods are managed by this DaemonSet
- Labels and Annotations: Any labels and annotations assigned to the DaemonSet
- Desired Number: The total number of pods the DaemonSet should be running
- Current Number: The current number of pods being managed by the DaemonSet
- Ready Number: The number of pods that are ready
- Updated Number: The number of pods that are updated to the desired specification
- Pod Template: The pod template used by the DaemonSet to create new pods; this section will show details of the pod spec, such as the containers, environment variables, volume mounts, etc.
- Events: Any recent events related to the DaemonSet, such as pod scheduling, updates, or errors; this helps in debugging if there are any issues.
Let’s see an example:
~ ➜ kubectl describe daemonset -n kube-system
Name: fluentd-elasticsearch
Selector: name=fluentd-elasticsearch
Node-Selector: <none>
Labels: k8s-app=fluentd-logging
Annotations: deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 2
Current Number of Nodes Scheduled: 2
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 2
Number of Nodes Misscheduled: 0
Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: name=fluentd-elasticsearch
Containers:
fluentd-elasticsearch:
Image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
Port: <none>
Host Port: <none>
Limits:
memory: 200Mi
Requests:
cpu: 100m
memory: 200Mi
Environment: <none>
Mounts:
/var/log from varlog (rw)
Volumes:
varlog:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 94s daemonset-controller Created pod: fluentd-elasticsearch-s9nmv
Normal SuccessfulCreate 94s daemonset-controller Created pod: fluentd-elasticsearch-g85cp
These are just some of the main kubernetes workloads you can run kubectl describe against. However, a wide selection of kubernetes resources will give you useful information when running kubectl describe, such as Jobs, CronJobs, PVs, PVCs, ConfigMaps, Ingress, Network Policies, and so on.
We have already reviewed a few outputs, but now we will explore all the specific details they provide. In this example, we will describe the output of a kubectl describe pod
from top to bottom.
Let’s start off with the first section:
Name: nginx-pod
Namespace: default
Priority: 0
Node: node01/192.1.169.10
Start Time: Tue, 03 Sep 2024 03:03:17 +0000
Labels: run=nginx-pod
Annotations: <none>
Status: Running
IP: 10.244.1.2
IPs:
IP: 10.244.1.2
- Name: The name of the pod
- Namespace: The namespace in which the pod is running (default in this case)
- Priority: The priority is set to determine the pod’s importance compared to other pods being deployed. The default priority is
0
. The higher the priority, the first the pod is deployed, and the lower priority pods are removed to make room for the higher priority pods. - Node: This shows the IP address of the node the pod is deployed to.
- Start Time: Indicates the time/date the pod was deployed
- Labels: Key-value pairs of the label assigned to the pod; the
run=nginx-pod
is a default label set when it was deployed. Labels are useful if you want to organize and group your resources together. - Annotations: These are non-identifying metadata that are attached to the pod. They are typically used to attach additional information to resources, such as version information, configuration details, etc.
- Status: The current state of the Pod:
- ‘Running’ indicates the pod and all of its containers are running as expected.
- ‘Failed’ would state that the Pod is terminated due to an error.
- Pending’ is when a pod is waiting to be scheduled.
- ‘Unknown’ when the status cannot be determined, and lastly
- ‘Succeeded’ shows that the task the Pod is performing has been completed successfully.
- IP: The internal IP address assigned to the pod; this is the IP address used to communicate with other pods in the cluster and is not accessible externally.
- IPs: This field provides you with a list of IP addresses associated with the pod. It is used more for complex networking configurations that require your pod to have more flexibility with multi-network.
Containers section
Containers:
nginx-pod:
Container ID: containerd://973aa7b5391c3042f83d88ead6efe820614c3a8e852bffab95ca7e3530f19c65
Image: nginx
Image ID: docker.io/library/nginx@sha256:447a8665cc1dab95b1ca778e162215839ccbb9189104c79d7ec3a81e14577add
Port: <none>
Host Port: <none>
State: Running
Started: Tue, 03 Sep 2024 03:03:22 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7p6gq (ro)
The Containers section includes all the following fields related to the specific container nginx-pod
; if there was a second container in the pod, it would have the same fields underneath.
- Nginx-pod: This is the name of the Container in the Pod.
- Container ID: This is the unique identifier for the container. This is provided to you by the container runtime your pod is using. In this situation, the container runtime is
containerd
. - Image: This is the Container image being used, pulled from the container registry. In this case, the Nginx image is being pulled from the default Docker Hub repository for nginx.
- Image ID: When you pull the image, a specific digest of the image must be used across your environment, and this ID uniquely identifies the image version you are using.
- Port: This will show all the ports that the container is exposed to. It shows none because no ports are being exposed for external access for this specific container in the pod.
- Host Port: This will show the corresponding ports on the host. If any ports were being exposed, you would see a corresponding port listed here.
- State: This is an indicator that shows if the specific container is running in the pod.
You can also receive other statuses such as Waiting
or Terminated
. The state of this container is running
, so you receive a field underneath that shows Started
with a timestamp. This would not populate if the state of the container was not started. If it was stuck in Waiting
or Terminated
, you would receive a Reason
field under it with a description or error message.
- Ready: This indicates the container has passed all readiness checks and is prepared to serve and receive traffic. If this was set to
False
, the container might be starting up or facing some health checks. - Restart Count: The number of times the container was restarted to reach its current state can indicate issues with the container, such as crashes, failures, or specific problems with the application’s configuration.
- Environment: This would show you all the environment variables that have been set for this container.
- Mounts: This lists all the volumes mounted to the container and the paths they are mounted to. If this populates, that means you have another section below that would specify more details about the volume that is being mounted to the container.
The volume name in this container is ”kube-api-access-7p7gq” and is mounted to the location path “/var/run/secrets/kubernetes.io/serviceaccount” with ready-only (ro) rights. This is a default mount in kubernetes, as you need access to the service account credentials stored in this path to be able to interact with the Kubernetes API securely.
Conditions section
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
The Conditions section provides statuses of different condition types regarding the overall health and status of the pod. These conditions must be met in order for the pod to be fully operational.
Here is a breakdown of all the Conditional Types that must be met for this pod:
- Initialized: This indicates if all the init containers have initialized successfully. The init containers are special containers that run before the main application container. These are often used for specific tasks such as initial setup tasks, creating directories, pulling specific data, etc. Once this is set to true, the main application container is ready to kick off and attempt to run.
- Ready: This shows if the pod is ready to serve requests and traffic. It reflects the readiness of the individual containers within the pod and whether they have passed their readiness checks such as HTTP/TCP checks — a requirement for the container to receive traffic.
- ContainersReady: This indicates whether all containers in the pod are fully ready. It ensures all the main application containers are running and have passed all the readiness checks.
- PodScheduled: Every pod needs to be assigned to a node in a Kubernetes cluster. This indicates if the pod has been successfully assigned to a node by the scheduler component. It also takes into account any resource requirements you might have, taints, tolerations, and affinity/anti-affinity rules. If this is set to False, it could be due to a lack of resources available in the cluster, node affinity rules not being met, or other constraints.
Volumes section
Volumes:
kube-api-access-7p6gq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
The Volumes section provides information about the volumes mounted within the pod. These volumes are basically storage units accessed by the containers running within the pod, used to store data, configuration files, etc.
- Volume Name:
kube-api-access-7p6gq
is the name of the volume, which is automatically created by Kubernetes when creating a pod. - Type: This is the type of volume being mounted. In this pod, the type is set to ‘Projected’, which is a special volume type in Kubernetes that allows you to combine multiple volume sources such as service accounts, ConfigMaps, secrets, tokens, and downward API resources. This is a default volume used for service account purposes to access the Kubernetes API, so this storage is mostly to manage credentials and configuration data.
- TokenExpirationSeconds: This is time in seconds until the service account token expires. Kubernetes regularly rotates these tokens for security purposes.
- ConfigMapName: This is the name of the configmap being used for this volume. ConfigMaps stores key-value configuration data. In this case, the configuration data is the Kubernetes Cluster Root CA certificate that allows the pod to communicate securely with the Kubernetes API.
- ConfigMapOptional: This just indicates if the configmap is optional. The value <nil> means that true or false was not specified for this pod. If the value was set to ‘true’, configmap would not be required in order for the volume to be mounted. However, because it is not set (<nil>), the configmap is expected and should be present.
- DownwardAPI: This points out if a Downward API is enabled for this volume. The Downward API allows containers to have information about the containers or the pods, such as annotations, resource limits, labels, etc. It projects the information to a file within the container which can be useful for pods when they need to be aware of the environment and configuration settings.
QoS Class, Node-Selectors, and Tolerations
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
The QoS Class stands for Quality of Service classification for a pod. This mainly focuses on the priority of the pod in terms of resource allocation. This aligns directly with the CPU/Memory resource requests and limits you set up for the pod/container. You can get one of the following classes depending on the configuration of your Pod/Container:
- BestEffort: This will be set if you do not have any resource requests/limits set up for the containers in the pod. This means this pod will be the last to receive resources from the cluster and the first to get evicted if there is any high resource utilization in the cluster. This is the default class that is set.
- Guaranteed: This will be set if your memory/CPU requests and limits are equal. This means this specific pod will be ‘guaranteed’ the resources they requested and will be the last to get evicted when there is heavy resource utilization in the cluster.
- Burstable: This will be set if your CPU and memory usage limit is higher than your request. Therefore the pods can use more resources than requested when there are resources available in the cluster.
Node-Selectors and Tolerations are specific towards scheduling what node the pod can be scheduled and under what conditions the pod can tolerate specific states of the node.
Node-Selectors – This is a key-value pair used to restrict the pod to only run on specific nodes that match the labels listed here.
For this specific pod, there are none, so nothing is listed here, and Kubernetes can schedule the pod in any node. However, if you have a node with the labels env=prod
, you can set a node selector in the pod as env: prod
, and that will schedule the pod to only run under that node.
Tolerations – This is used with Taints/Tolerations. If a specific node has a taint to allow (tolerate) this pod to be scheduled, you would see it listed here. This is to ensure specific pods get assigned to specific nodes.
In our example, we have the following tolerations listed:
- node.kubernetes.io/not-ready:NoExecute op=Exists for 300s: This toleration will allow the pod to stay on a node marked as
not-ready
for up to 300 seconds after the time is up. If the node is still innot-ready
, the pod will be evicted. - node.kubernetes.io/unreachable:NoExecute op=Exists for 300s: This toleration will allow the pod to stay on the node marked as
unreachable
for up to 300 seconds. And after that time is up and if the node is stillunreachable
, the pod will be evicted.
Events section
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30m default-scheduler Successfully assigned default/nginx-pod to node01
Normal Pulling 30m kubelet Pulling image "nginx"
Normal Pulled 30m kubelet Successfully pulled image "nginx" in 4.277827536s
Normal Created 30m kubelet Created container nginx-pod
Normal Started 30m kubelet Started container nginx-pod
The Events section gives a chronological list of key events that took place over the lifecycle of the pod. This includes events the Kubernetes cluster took to schedule, start, manage, and debug any rising issues related to the pod. Each event includes:
- Type: This indicates the severity of the event. You will either get ‘Normal’ or ‘Warning’.
- Reason: This provides you with a short reason for what the event is related to.
- Age: The time since the event started
- From: The key component that generated the event
- Message: Description of what the event is
Our example shows fairly common events that take place when you deploy a simple Nginx pod.
Labels are key-value pairs attached to kubernetes resources that help you organize and categorize resources throughout your cluster. Selectors are expressions/queries that can also filter and select Kubernetes resources based on their labels.
kubectl describe
has the option to filter by labels and selectors, which is useful when you have a large number of workloads running on your cluster and need to easily retrieve details from specific labeled resources.
You can use two flags with kubectl describe
to retrieve extra information: the -l
flag or --selector
followed by key=value
; you can also use the flags with equality-based selector (=
, or !=
) or set-based selectors (in
, notin
, and exists
)
Here are some examples of using labels with different Kubernetes resources:
Using kubectl describe labels with Pods
The following is a simple example of using kubectl describe with a single label:
kubectl describe pod -l key=value
Using labels with Deployments
The following example utilizes multiple labels to retrieve specific resources:
kubectl describe deployment -l env=prod, zone=eastus2
Using labels with Services
The following example uses some extra logic to filter out your kubectl describe
results:
kubectl describe services —-selector 'env in (prod, dev), app notin (nginx)'
kubectl describe
is a very powerful command; however, the output it generates has some limitations. It is very static and cannot aggregate, filter, format, or customize the output.
Here is a list of some of the limitations you may face while using ‘kubectl describe’:
- Filtering limitations: You can utilize the
-l
flag or--selector
to filter the KLubernetes resources you want to runkubectl describe
against, but this is more for applying the describe command against a filtered set of resources instead of modifying and filtering out the output you receive.
It can become challenging if you try to filter out the resources beyond the labels and selectors or try to use logical operators to combine different conditionals and queries.
- Lack of aggregation:
kubectl describe
does not have the full ability to aggregate and summarize data such as the total resource usage across multiple resources such as nodes, pods, deployments, etc. - JSON/YAML support:
kubectl describe
also does not support structured outputs such as JSON/YAML, so you cannot pipe the output to other tools for analysis or processing. - Lack of output customization: You cannot customize the output you receive from
kubectl describe
. You will get a static detailed output with all the information you are requesting, so it might become complex for you to parse through it for automation purposes.
kubectl describe alternatives
You can use other tools to help you get better outputs you can utilize for automation or for more clarity. Here is a list of some alternatives:
kubectl get pods
command: Using this command with the-o
flag allows you to get structured outputs that can be sent to either JSON or YAML format, which you can easily parse through and use within your automation scripts or tools.
kubectl get pods -o json
kubectl get pods -o yaml
kubectl logs
: This command is useful ifkubectl describe
command is not generating all the details you need, and you need more real-time logging information.
kubectl logs pod-name
kubectl events
command: This is similar tokubectl logs
and can also allow you to filter out certain events and retrieve the correct output.
kubectl get events —-sort-by='{.lastTimestamp}'
- K9s or Kubernetes Dashboard: This will give you an actual GUI that you can use to get more visuals and mainly use an interactive approach to managing your Kubernetes resources. In the GUI, you have more access to filter, describe, and analyze resources.
- Prometheus and Grafana: Utilize monitoring and debugging tools such as Prometheus to retrieve metrics from your cluster and Grafana to visualize the data. This allows for powerful filtering capabilities, aggregation, and visualization options.
- CRD (Custom Resource Definition): You always have the option to create and customize your own resource when
kubectl describe
,kubectl get
,kubectl logs
orkubectl get events
have failed. You can use this option to create a CRD that includes more specialized and relevant information. This approach would be more tailored to your specific requirements and provide more relevant information based on your specific needs.
Now we will review a few use cases where we can maximize our use of the ‘kubectl describe’ command and also follow best practices:
1. Debugging a pod
In this scenario, we have a Pod that is stuck on Pending
, CrashLoopBackOff
, or Error
state. We can start by running the kubectl describe
command against the pod that is problematic:
kubectl describe pod pod-name
What should we inspect?
- Status: The Pod status is the first place to check.
- Containers Section: You should then review the container and validate the state of the container.
- Conditions Section: Third, you should check the Conditions section to ensure there are no ‘False’ statuses for each of the requirements listed.
- Events: The events can be the first thing you look at to get straight to the point or the last thing if you want to drill down slowly into the issue. You should check for any events that have error messages or warnings. If the pod is not running, there should be an error message here that would identify the issue.
Best practices
By this point, troubleshooting the events and other values given to you from kubectl describe
is not helpful.
You can start running other kubernetes tools to dig deeper, such as kubectl get events
for an in-depth series of events that took place on the pod or kubectl logs
for extra details.
kubectl get events --sort-by='{.lastTimestamp}'
kubectl logs
2. Networking issues with Services
In this scenario, the service is not routing traffic correctly to the endpoints. Right away, you can quickly run the kubectl describe
command against the service in question and check what endpoints it is pointing towards. This can easily give you a starting point to troubleshoot this issue.
kubectl describe service service-name
What should we inspect?
- Endpoints: In the described output, you can scroll down to the ‘Endpoints’ section and check if the correct pod’s internal IP address is listed as an endpoint.
- Types and Ports: Ensure the service type is correct and the ports match your configuration
- Selector: You probably used some sort of labels to deploy your resources, ensuring the services selector matches the labels on the target pods.
Best practices
- Instead of validating endpoints through
kubectl describe service
, you can also use the kubernetes tool,kubectl get endpoints
. This will directly verify that the endpoints are corresponding to healthy pods. Kubectl get service -o wide
can also provide you with a summary view
3. Node issues in the cluster
In this scenario, the node in your cluster is stuck in the NotReady
state, and you might be wondering where you should start to troubleshoot. The kubectl describe
is a great starting point to get visuals on the error messages and status of the Node.
kubectl describe node node-name
What should we inspect?
- Status: Verify the status is showing
NotReady
. - Conditions: Here, you can get a vague idea of the issue you might be dealing with, such as
MemoryPressure
,DiskPressure
, orPIDPressure
. - Capacity: Verify the CPU, Memory and other resource capacity here.
- Allocatable: Check if CPU, Memory, and other resources are not maxed out.
- Events: Check through the event logs for error messages related to disk, network, or other failures.
Best practices
- Node health is very important for your cluster to run properly with all of its workloads; therefore, it is important to be proactive and detect issues.
- Utilizing a monitoring tool such as Prometheus or Grafana to retrieve and visualize metrics can help detect Node failures early on in the process.
4. Debugging pods and nodes with taints/tolerations
In some cases, pods can be assigned to the wrong nodes based on the taints on your nodes and the tolerations/selectors on your pods. In these situations, it is best to evaluate the affected Pods and Nodes individually to get to the root cause.
You can start off by inspecting the affected Pod:
kubectl describe pod pod-name
What should we inspect?
- Tolerations: In the Tolerations sections, you can check what taints the pod can tolerate and for how long
- Node Affinity: Review each node affinity rule and ensure which nodes the pod is allowed to run on based on the labels.
- Node Selectors: Check the Node-Selectors section to see if the pod is limited to run on nodes with specific labels.
- Events: Check for any potential events that could have caused any of these mismatches with the Nodes. Look for ‘Warning’ events that have ‘Scheduling’ messages or something along the lines of nodes not matching the pod affinity rules.
Now let’s inspect the affected Node:
kubectl describe node node-name
What should we inspect?
- Taints: Go through the taints and make sure they match the tolerations specified on the pod.
- Events: Check if there are any node mismatch messages in event logs.
Best practices
- Ensure all Taints and Tolerations match across your nodes and pods. Taints with ‘NoSchedule” or ‘NoExecute’ effects require the correct tolerations.
- Review Node-Affinity and Anti-Affinity rules and make sure they don’t conflict.
- Utilize
kubectl logs
on the kube-scheduler to get more insight into the mismatch issue. - Use
kubectl get events —-sort-by=’.lastTimestamp’
to check for all events in the cluster.
If you need assistance managing your Kubernetes projects, look at Spacelift. It brings with it a GitOps flow, so your Kubernetes Deployments are synced with your Kubernetes Stacks, and pull requests show you a preview of what they’re planning to change.
To take this one step further, you could add custom policies to reinforce the security and reliability of your configurations and deployments. Spacelift provides different types of policies and workflows that are easily customizable to fit every use case. For instance, you could add plan policies to restrict or warn about security or compliance violations or approval policies to add an approval step during deployments.
You can try Spacelift for free by creating a trial account or booking a demo with one of our engineers.
The kubectl describe
command is useful for managing and troubleshooting Kubernetes resources like Pods, Nodes, and Deployments. It provides detailed information such as status, conditions, and events. Although it offers limited customization, it can be supplemented with tools like kubectl logs
, kubectl get
, and monitoring solutions like Prometheus and Grafana.
Manage Kubernetes Easier and Faster
Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.