Going to AWS re:Invent 2024?

➡️ Book a meeting with Spacelift

Kubernetes

Kubectl Describe Command – Pods, Deployments & More

kubectl describe pod

Kubectl is the default tool for directly accessing your Kubernetes cluster from the command line. It allows you to create, update, and delete Kubernetes resources. 

Commands such as ‘kubectl get’ give you a brief summary of your workloads, but they don’t really dive deep into what’s happening in the backend and give you the full breakdown. However, ‘kubectl describe’ can show you detailed information about your cluster’s workloads.

In this article, we will examine the kubectl describe command and demonstrate how to use it for Kubernetes pods, deployments, services, nodes, and more. We will also look at some common use cases and best practices for running this command.

What we will cover:

  1. What is the kubectl describe command?
  2. How to use kubectl describe with different resources
  3. Kubectl describe output
  4. How to use kubectl describe with labels and selectors
  5. Kubectl describe limitations and other options 
  6. Practical use cases & best practices

What is the kubectl describe command?

The kubectl describe command retrieves and displays detailed data from a single resource or multiple resources in your cluster. This command can be helpful when you are debugging specific issues in your Kubernetes workloads and need to see more than just a brief summary of your resources. 

Here’s the basic syntax:

kubectl describe <resource> <resource-name> [flags]

The output you receive from kubectl describe shows the resource’s current state, namespace, start times, labels, events, replicas, resource configurations, environment variables, and other details specified in the YAML manifest.

kubectl describe vs. kubectl logs

You might wonder why you can’t just use kubectl logs to troubleshoot an issue with a pod. 

 

kubectl logs command provides in-depth logging information that can take time to read and understand what is really wrong with your resource.

 

kubectl describe, on the other hand, gives you a set of events that you can quickly read through and get to the root cause right away. This makes ‘kubectl describe’ a better first step to debug and troubleshoot your Kubernetes resources. 

How to use kubectl describe with different resources

kubectl describe can be used with a wide range of Kubernetes resource types such as pods, nodes, Deployments, Services, ReplicaSets, DaemonSets, and more. The describe output varies for each resource type. Some kubectl describe outputs are not that useful and running the describe command against another resource type retrieves more information. 

For example, kubectl describe deployment deployment-name might not have all the information you are seeking. In that case, use the describe command against the individual pod in that deployment by running kubectl describe pod pod-name. This enables you to retrieve concise information directly from the affected specific resource instead of getting minimal information from the deployment as a whole. 

Let’s explore some examples of using kubectl describe against different resource types and see the outputs it generates. 

kubectl describe pod

The kubectl describe pod command is provides detailed information about a specific pod running in your Kubernetes cluster. This information could include the pod’s metadata, status, events, and resource usage.

kubectl describe pod <pod-name> -n <namespace>
  • <pod-name>: The name of the pod you want to inspect.
  • -n <namespace>: The namespace where the pod is running. If you don’t specify the namespace, it will default to the current namespace or the default namespace.

It provides information on:

  • Name and Namespace: The name and namespace of the pod
  • Priority: The priority of the pod
  • Start Time: The timestamp indicating when the pod was created
  • Labels and Annotations: Any key-value pairs for identification (labels) and additional metadata (annotations) related to the pod
  • Node: The node where the pod is currently running
  • Status: The current state of the pod (e.g., Running, Pending, Failed)
  • IP Addresses: The pod’s internal and external IP addresses
  • Containers: Each container’s image, state, readiness probes, resource limits, and more — plus, events like crashes or restarts, including the last state of the container
  • Volumes: Information about any volumes attached to the pod
  • Events: Recent events that have affected the pod, such as scheduling issues, restarts, or failures

Example 1

In the following example, we are using kubectl describe pod nginx-pod to retrieve more in-depth information about the pod. Your Nginx pod is running fine, and there are no issues to report. 

As you can see, the State is set to Running, all Conditions are set to True, and nothing has been picked up on the Events section. 

~ ➜  kubectl describe pod nginx-pod 

Name:         nginx-pod
Namespace:    default
Priority:     0
Node:         node01/192.1.169.10
Start Time:   Tue, 03 Sep 2024 03:03:17 +0000
Labels:       run=nginx-pod
Annotations:  <none>
Status:       Running
IP:           10.244.1.2
IPs:
  IP:  10.244.1.2
Containers:
  nginx-pod:
    Container ID:   containerd://973aa7b5391c3042f83d88ead6efe820614c3a8e852bffab95ca7e3530f19c65
    Image:          nginx
    Image ID:       docker.io/library/nginx@sha256:447a8665cc1dab95b1ca778e162215839ccbb9189104c79d7ec3a81e14577add
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 03 Sep 2024 03:03:22 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7p6gq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-7p6gq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  30m   default-scheduler  Successfully assigned default/nginx-pod to node01
  Normal  Pulling    30m   kubelet            Pulling image "nginx"
  Normal  Pulled     30m   kubelet            Successfully pulled image "nginx" in 4.277827536s
  Normal  Created    30m   kubelet            Created container nginx-pod
  Normal  Started    30m   kubelet            Started container nginx-pod

Example 2

Let’s now review another kubectl describe pod example with errors, so we can compare and see how utilizing a kubectl describe with pods can be helpful. 

Here, you can see the ‘Status’ is set to Pending, which points out that there is something with the pod that is preventing it from going into a running state. 

The State section shows ErrImagePull, another clear identifier that the issue is related to this pod’s image. 

The Conditions section also gives us some details on how the status of the container and pod is set to ‘False’, therefore unable to run. 

We can also review the Events section, which will provide the most details regarding the errors generated from the pod and why it failed to run. The error message says failed to pull and unpack image because we entered an invalid image name, which Kubernetes was unable to find (image name = ‘nginxxxx’).

~ ➜  kubectl describe pod nginx-pod-error 
Name:         nginx-pod-error
Namespace:    default
Priority:     0
Node:         node01/192.1.169.10
Start Time:   Tue, 03 Sep 2024 03:41:32 +0000
Labels:       run=nginx-pod-error
Annotations:  <none>
Status:       Pending
IP:           10.244.1.6
IPs:
  IP:  10.244.1.6
Containers:
  nginx-pod-error:
    Container ID:   
    Image:          nginxxxx
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s7qfz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-s7qfz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  16s   default-scheduler  Successfully assigned default/nginx-pod-error to node01
  Normal   Pulling    16s   kubelet            Pulling image "nginxxxx"
  Warning  Failed     15s   kubelet            Failed to pull image "nginxxxx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginxxxx:latest": failed to resolve reference "docker.io/library/nginxxxx:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     15s   kubelet            Error: ErrImagePull
  Normal   BackOff    15s   kubelet            Back-off pulling image "nginxxxx"
  Warning  Failed     15s   kubelet            Error: ImagePullBackOff

kubectl describe deployment

Now, we will review the details you receive from running kubectl describe against deployments. The output is a bit more concise than the output you receive from describing pods because the majority of the information is on the pod level. 

kubectl describe deployment provides a detailed overview of a Kubernetes deployment’s configuration and status. It offers insights into key deployment attributes, making it useful for understanding the overall state and behavior of a deployment.

kubectl describe deployment <deployment-name> -n <namespace>

This command provides information on:

  • Creation Timestamp: When the deployment was first created
  • Labels: Key-value pairs associated with the deployment
  • Selector: Indicates the strategy being used for updates providing insight into how new versions of applications are deployed
  • Replica status: The current status of replicas, including how many are desired, available, and up-to-date
  • Rolling Update Strategy: Details on how updates to pods are managed incrementally to minimize downtime
  • Pod Template: Details about the pod template used by the deployment, such as labels, annotations, container specifications (image, ports, environment variables), and resource requests/limits
  • Image: The container image version being used in the deployment
  • Volumes: Any volumes attached to the deployment
  • Conditions: The current state of the deployment, including whether it is progressing as expected or encountering any issues
  • Events: Recent events related to the deployment, such as scaling actions or failures

For troubleshooting more granular issues at the container or pod level (like crashes or resource limits), it is recommended to use kubectl describe pod command, as it provides more detailed information specific to individual pods.

Let’s look at an example:

~ ➜  kubectl describe deployment nginx-deploy 
Name:                   nginx-deploy
Namespace:              default
CreationTimestamp:      Tue, 03 Sep 2024 22:06:59 +0000
Labels:                 app=nginx-deploy
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx-deploy
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx-deploy
  Containers:
   nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-deploy-99976564d (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  21s   deployment-controller  Scaled up replica set nginx-deploy-99976564d to 3

You can also try using kubectl describe replicaset, but it will provide more or less be the same information as you received from kubectl describe deployment. 

kubectl describe node

The kubectl describe node command offers an in-depth view of the current state, health, and resource allocation of a specific node in the Kubernetes cluster. The output includes detailed node conditions, like whether the node is ready, disk pressure, memory pressure, and network issues. 

kubectl describe node <node-name>

The output includes:

  • Node Name and Labels: The unique name of the node, along with any labels associated with it
  • Annotations: Any annotations that provide additional metadata about the node
  • Addresses: Various addresses associated with the node, including internal IP and hostname
  • Conditions: The current conditions of the node, such as:
    • Ready: Whether the node is ready to accept pods
    • Memory Pressure: If the node is experiencing memory pressure
    • Disk Pressure: If there are issues related to disk space
  • Capacity and Allocatable Resources: Details the total resources available on the node (CPU, memory) and what is allocatable for pods
  • System Info: Information about the operating system, kernel version, and container runtime being used on the node
  • Pods: Pods currently running on this node, including their statuses and resource usage
  • Events: Recent events related to the node, such as scheduling failures or resource allocation issues

Let’s see an example output:

 ~ ➜  kubectl describe node controlplane 
Name:               controlplane
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=controlplane
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"96:7f:cc:c3:40:d2"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.41.220.3
                    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 05 Sep 2024 20:45:43 +0000
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
                    node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  controlplane
  AcquireTime:     <unset>
  RenewTime:       Thu, 05 Sep 2024 22:29:08 +0000
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 05 Sep 2024 20:46:06 +0000   Thu, 05 Sep 2024 20:46:06 +0000   FlannelIsUp                  Flannel is running on this node
  MemoryPressure       False   Thu, 05 Sep 2024 22:28:14 +0000   Thu, 05 Sep 2024 20:45:37 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 05 Sep 2024 22:28:14 +0000   Thu, 05 Sep 2024 20:45:37 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 05 Sep 2024 22:28:14 +0000   Thu, 05 Sep 2024 20:45:37 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 05 Sep 2024 22:28:14 +0000   Thu, 05 Sep 2024 20:46:09 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.41.220.3
  Hostname:    controlplane
Capacity:
  cpu:                18
  ephemeral-storage:  1016057248Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             98888308Ki
  pods:               110
Allocatable:
  cpu:                18
  ephemeral-storage:  936398358207
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             98785908Ki
  pods:               110
System Info:
  Machine ID:                 15c49d4f47e04095aa13764b7e5c23f7
  System UUID:                bbdbe79a-1085-7763-21af-4585c812332a
  Boot ID:                    ce8ebee8-715f-43ec-b865-96399fa78301
  Kernel Version:             5.4.0-1106-gcp
  OS Image:                   Ubuntu 18.04.6 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.6
  Kubelet Version:            v1.24.0
  Kube-Proxy Version:         v1.24.0
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
Non-terminated Pods:          (8 in total)
  Namespace                   Name                                    CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                    ------------  ----------  ---------------  -------------  ---
  kube-flannel                kube-flannel-ds-h7xfr                   100m (0%)     100m (0%)   50Mi (0%)        300Mi (0%)     103m
  kube-system                 coredns-6d4b75cb6d-5vfrg                100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     103m
  kube-system                 coredns-6d4b75cb6d-7zssh                100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     103m
  kube-system                 etcd-controlplane                       100m (0%)     0 (0%)      100Mi (0%)       0 (0%)         103m
  kube-system                 kube-apiserver-controlplane             250m (1%)     0 (0%)      0 (0%)           0 (0%)         103m
  kube-system                 kube-controller-manager-controlplane    200m (1%)     0 (0%)      0 (0%)           0 (0%)         103m
  kube-system                 kube-proxy-p8n7w                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         103m
  kube-system                 kube-scheduler-controlplane             100m (0%)     0 (0%)      0 (0%)           0 (0%)         103m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                950m (5%)   100m (0%)
  memory             290Mi (0%)  640Mi (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>

kubectl describe service

In Kubernetes, a service is used to assign a name and unique IP address to a group of pods, allowing you to route traffic properly to your pods. 

The kubectl describe service command provides detailed information about a Kubernetes Service object, including metadata, specifications, and the current status of the service. It is useful for debugging and checking the configuration and status of the service.

kubectl describe service <service-name> -n <namespace>

The key sections of the output are:

  • Name and Namespace: The name of the service and the namespace the service is running in
  • Labels and Annotations: Any labels and annotations applied to the service
  • Selector: The label selector that matches the Pods this service is directing traffic to
  • Type: The type of the service (ClusterIP, NodePort, LoadBalancer, etc.)
  • IP: The internal ClusterIP assigned to the service
  • Ports: The ports that the service is exposing, including:
    • Protocol (TCP/UDP)
    • TargetPort (the port on the Pods)
    • NodePort (if the service is of type NodePort).
  • Endpoints: Lists the Pod IPs and ports the service is routing traffic to
  • Session Affinity: Whether the service is sticky (affinitized) to a particular client

In the example below, Endpoints are pointing to the internal IP address of the pod:

~ ➜  kubectl describe service redis
Name:              redis
Namespace:         default
Labels:            app=redis
Annotations:       <none>
Selector:          app=redis
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.103.92.91
IPs:               10.103.92.91
Port:              <unset>  6379/TCP
TargetPort:        6379/TCP
Endpoints:         10.244.1.2:6379
Session Affinity:  None
Events:            <none>

kubectl describe statefulset

Kubernetes StatefulSets are useful when you have specific workloads you need more control over in terms of consistency and data. StatefulSets allows you to have static workloads with a persistent storage and networking configuration and also set up ordered scaling and rolling updates, which prevents random issues from arising. 

The kubectl describe statefulset command provides a detailed description of a StatefulSet resource in your cluster. It generates similar results to what you would get from running kubectl describe against deployments but for persistent pods deployed through StatefulSets.  The output includes the current state, configuration details, and events related to the StatefulSet.

kubectl describe statefulset <statefulset_name> -n <namespace>

Key sections of the output include:

  • Name and Namespace: The name of the StatefulSet and the namespace where the StatefulSet resides
  • Labels and Annotations: Labels and any additional metadata stored as annotations attached to the StatefulSet
  • Selector: The label selector used to match pods with this StatefulSet
  • Replicas: The number of desired and current replicas
  • Pod Management Policy: The ordering and parallelism in pod creation
  • Update Strategy: The strategy used for updating StatefulSet pods (e.g., RollingUpdate or OnDelete)
  • Volume Claims: Information about persistent volume claims (PVCs) associated with each pod
  • Pod Template: Details about how the pods are created, including container specifications, environment variables, and volumes
  • Conditions: Any conditions related to the StatefulSet’s status (e.g., Available or Progressing)
  • Events: Recent events related to the StatefulSet, such as scaling or failures

Let’s examine the example below:

~ ➜  k describe statefulset web
Name:               web
Namespace:          default
CreationTimestamp:  Fri, 06 Sep 2024 22:56:42 +0000
Selector:           app=nginx
Labels:             <none>
Annotations:        <none>
Replicas:           3 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        registry.k8s.io/nginx-slim:0.24
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:
      /usr/share/nginx/html from www (rw)
  Volumes:  <none>
Volume Claims:
  Name:          www
  StorageClass:  my-storage-class
  Labels:        <none>
  Annotations:   <none>
  Capacity:      1Gi
  Access Modes:  [ReadWriteOnce]
Events:
  Type    Reason            Age   From                    Message
  ----    ------            ----  ----                    -------
  Normal  SuccessfulCreate  13m   statefulset-controller  create Claim www-web-0 Pod web-0 in StatefulSet web success
  Normal  SuccessfulCreate  13m   statefulset-controller  create Pod web-0 in StatefulSet web successful

kubectl describe daemonset

Daemonset is a Kubernetes resource that allows you to run pods as a ‘daemon’ on your Kubernetes nodes. This gives you more control over which nodes your pods are distributed across. Daemonsets are mostly used to run pods that are always needed to run on each node, such as cluster storage, log collection agents, and monitoring systems. 

When you run kubectl describe daemonset it will give you a descriptive output for the specified DaemonSet, includes details like the pods it manages, its status, events, and more.

kubectl describe daemonset <daemonset-name> -n <namespace>

The key sections of the output:

  • Name and Namespace: The name of the DaemonSet and the namespace where the DaemonSet is running
  • Selector: The label selector used to determine which pods are managed by this DaemonSet
  • Labels and Annotations: Any labels and annotations assigned to the DaemonSet
  • Desired Number: The total number of pods the DaemonSet should be running
  • Current Number: The current number of pods being managed by the DaemonSet
  • Ready Number: The number of pods that are ready
  • Updated Number: The number of pods that are updated to the desired specification
  • Pod Template: The pod template used by the DaemonSet to create new pods; this section will show details of the pod spec, such as the containers, environment variables, volume mounts, etc.
  • Events: Any recent events related to the DaemonSet, such as pod scheduling, updates, or errors; this helps in debugging if there are any issues.

Let’s see an example:

~ ➜ kubectl describe daemonset -n kube-system
Name:           fluentd-elasticsearch
Selector:       name=fluentd-elasticsearch
Node-Selector:  <none>
Labels:         k8s-app=fluentd-logging
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 2
Current Number of Nodes Scheduled: 2
Number of Nodes Scheduled with Up-to-date Pods: 2
Number of Nodes Scheduled with Available Pods: 2
Number of Nodes Misscheduled: 0
Pods Status:  2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  name=fluentd-elasticsearch
  Containers:
   fluentd-elasticsearch:
    Image:      quay.io/fluentd_elasticsearch/fluentd:v2.5.2
    Port:       <none>
    Host Port:  <none>
    Limits:
      memory:  200Mi
    Requests:
      cpu:        100m
      memory:     200Mi
    Environment:  <none>
    Mounts:
      /var/log from varlog (rw)
  Volumes:
   varlog:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:  
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  94s   daemonset-controller  Created pod: fluentd-elasticsearch-s9nmv
  Normal  SuccessfulCreate  94s   daemonset-controller  Created pod: fluentd-elasticsearch-g85cp

These are just some of the main kubernetes workloads you can run kubectl describe against. However, a wide selection of kubernetes resources will give you useful information when running kubectl describe, such as Jobs, CronJobs, PVs, PVCs, ConfigMaps, Ingress, Network Policies, and so on. 

Understanding the kubectl describe output

We have already reviewed a few outputs, but now we will explore all the specific details they provide. In this example, we will describe the output of a kubectl describe pod from top to bottom. 

Let’s start off with the first section:

Name:         nginx-pod
Namespace:    default
Priority:     0
Node:         node01/192.1.169.10
Start Time:   Tue, 03 Sep 2024 03:03:17 +0000
Labels:       run=nginx-pod
Annotations:  <none>
Status:       Running
IP:           10.244.1.2
IPs:
  IP:  10.244.1.2
  • Name: The name of the pod
  • Namespace: The namespace in which the pod is running (default in this case)
  • Priority: The priority is set to determine the pod’s importance compared to other pods being deployed. The default priority is 0. The higher the priority, the first the pod is deployed, and the lower priority pods are removed to make room for the higher priority pods. 
  • Node: This shows the IP address of the node the pod is deployed to. 
  • Start Time: Indicates the time/date the pod was deployed 
  • Labels: Key-value pairs of the label assigned to the pod; the run=nginx-pod is a default label set when it was deployed. Labels are useful if you want to organize and group your resources together. 
  • Annotations: These are non-identifying metadata that are attached to the pod. They are typically used to attach additional information to resources, such as version information, configuration details, etc. 
  • Status: The current state of the Pod:
    • ‘Running’ indicates the pod and all of its containers are running as expected. 
    • ‘Failed’ would state that the Pod is terminated due to an error. 
    • Pending’ is when a pod is waiting to be scheduled. 
    • ‘Unknown’ when the status cannot be determined, and lastly 
    • ‘Succeeded’ shows that the task the Pod is performing has been completed successfully. 
  • IP: The internal IP address assigned to the pod; this is the IP address used to communicate with other pods in the cluster and is not accessible externally. 
  • IPs: This field provides you with a list of IP addresses associated with the pod. It is used more for complex networking configurations that require your pod to have more flexibility with multi-network. 

Containers section

 

Containers:
  nginx-pod:
    Container ID:   containerd://973aa7b5391c3042f83d88ead6efe820614c3a8e852bffab95ca7e3530f19c65
    Image:          nginx
    Image ID:       docker.io/library/nginx@sha256:447a8665cc1dab95b1ca778e162215839ccbb9189104c79d7ec3a81e14577add
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 03 Sep 2024 03:03:22 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7p6gq (ro)

The Containers section includes all the following fields related to the specific container nginx-pod; if there was a second container in the pod, it would have the same fields underneath.

  • Nginx-pod: This is the name of the Container in the Pod. 
  • Container ID: This is the unique identifier for the container. This is provided to you by the container runtime your pod is using. In this situation, the container runtime is containerd
  • Image: This is the Container image being used, pulled from the container registry. In this case, the Nginx image is being pulled from the default Docker Hub repository for nginx. 
  • Image ID: When you pull the image, a specific digest of the image must be used across your environment, and this ID uniquely identifies the image version you are using.
  • Port: This will show all the ports that the container is exposed to. It shows none because no ports are being exposed for external access for this specific container in the pod. 
  • Host Port: This will show the corresponding ports on the host. If any ports were being exposed, you would see a corresponding port listed here. 
  • State: This is an indicator that shows if the specific container is running in the pod.

You can also receive other statuses such as Waiting or Terminated. The state of this container is running, so you receive a field underneath that shows Started with a timestamp. This would not populate if the state of the container was not started. If it was stuck in Waiting or Terminated, you would receive a Reason field under it with a description or error message.

  • Ready: This indicates the container has passed all readiness checks and is prepared to serve and receive traffic. If this was set to False, the container might be starting up or facing some health checks. 
  • Restart Count: The number of times the container was restarted to reach its current state can indicate issues with the container, such as crashes, failures, or specific problems with the application’s configuration. 
  • Environment: This would show you all the environment variables that have been set for this container. 
  • Mounts: This lists all the volumes mounted to the container and the paths they are mounted to. If this populates, that means you have another section below that would specify more details about the volume that is being mounted to the container. 

The volume name in this container is ”kube-api-access-7p7gq” and is mounted to the location path “/var/run/secrets/kubernetes.io/serviceaccount” with ready-only (ro) rights. This is a default mount in kubernetes, as you need access to the service account credentials stored in this path to be able to interact with the Kubernetes API securely. 

Conditions section

 

Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 

The Conditions section provides statuses of different condition types regarding the overall health and status of the pod. These conditions must be met in order for the pod to be fully operational. 

Here is a breakdown of all the Conditional Types that must be met for this pod:  

  • Initialized: This indicates if all the init containers have initialized successfully. The init containers are special containers that run before the main application container. These are often used for specific tasks such as initial setup tasks, creating directories, pulling specific data, etc. Once this is set to true, the main application container is ready to kick off and attempt to run. 
  • Ready: This shows if the pod is ready to serve requests and traffic. It reflects the readiness of the individual containers within the pod and whether they have passed their readiness checks such as HTTP/TCP checks  — a requirement for the container to receive traffic. 
  • ContainersReady: This indicates whether all containers in the pod are fully ready. It ensures all the main application containers are running and have passed all the readiness checks. 
  • PodScheduled: Every pod needs to be assigned to a node in a Kubernetes cluster. This indicates if the pod has been successfully assigned to a node by the scheduler component. It also takes into account any resource requirements you might have, taints, tolerations, and affinity/anti-affinity rules. If this is set to False, it could be due to a lack of resources available in the cluster, node affinity rules not being met, or other constraints. 

Volumes section

 

Volumes:
  kube-api-access-7p6gq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true

The Volumes section provides information about the volumes mounted within the pod. These volumes are basically storage units accessed by the containers running within the pod, used to store data, configuration files, etc. 

  • Volume Name: kube-api-access-7p6gq is the name of the volume, which is automatically created by Kubernetes when creating a pod. 
  • Type: This is the type of volume being mounted. In this pod, the type is set to ‘Projected’, which is a special volume type in Kubernetes that allows you to combine multiple volume sources such as service accounts, ConfigMaps, secrets, tokens, and downward API resources. This is a default volume used for service account purposes to access the Kubernetes API, so this storage is mostly to manage credentials and configuration data. 
  • TokenExpirationSeconds: This is time in seconds until the service account token expires. Kubernetes regularly rotates these tokens for security purposes. 
  • ConfigMapName: This is the name of the configmap being used for this volume. ConfigMaps stores key-value configuration data. In this case, the configuration data is the Kubernetes Cluster Root CA certificate that allows the pod to communicate securely with the Kubernetes API. 
  • ConfigMapOptional: This just indicates if the configmap is optional. The value <nil> means that true or false was not specified for this pod. If the value was set to ‘true’, configmap would not be required in order for the volume to be mounted. However, because it is not set (<nil>), the configmap is expected and should be present.
  • DownwardAPI: This points out if a Downward API is enabled for this volume. The Downward API allows containers to have information about the containers or the pods, such as annotations, resource limits, labels, etc. It projects the information to a file within the container which can be useful for pods when they need to be aware of the environment and configuration settings. 

QoS Class, Node-Selectors, and Tolerations

 

QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

The QoS Class stands for Quality of Service classification for a pod. This mainly focuses on the priority of the pod in terms of resource allocation. This aligns directly with the CPU/Memory resource requests and limits you set up for the pod/container. You can get one of the following classes depending on the configuration of your Pod/Container:

  • BestEffort: This will be set if you do not have any resource requests/limits set up for the containers in the pod. This means this pod will be the last to receive resources from the cluster and the first to get evicted if there is any high resource utilization in the cluster. This is the default class that is set. 
  • Guaranteed: This will be set if your memory/CPU requests and limits are equal. This means this specific pod will be ‘guaranteed’ the resources they requested and will be the last to get evicted when there is heavy resource utilization in the cluster. 
  • Burstable: This will be set if your CPU and memory usage limit is higher than your request. Therefore the pods can use more resources than requested when there are resources available in the cluster. 

Node-Selectors and Tolerations are specific towards scheduling what node the pod can be scheduled and under what conditions the pod can tolerate specific states of the node. 

Node-Selectors This is a key-value pair used to restrict the pod to only run on specific nodes that match the labels listed here. 

For this specific pod, there are none, so nothing is listed here, and Kubernetes can schedule the pod in any node. However, if you have a node with the labels env=prod, you can set a node selector in the pod as env: prod, and that will schedule the pod to only run under that node. 

Tolerations – This is used with Taints/Tolerations. If a specific node has a taint to allow (tolerate) this pod to be scheduled, you would see it listed here. This is to ensure specific pods get assigned to specific nodes.

In our example, we have the following tolerations listed: 

  • node.kubernetes.io/not-ready:NoExecute op=Exists for 300s: This toleration will allow the pod to stay on a node marked as not-ready for up to 300 seconds after the time is up. If the node is still in not-ready, the pod will be evicted. 
  • node.kubernetes.io/unreachable:NoExecute op=Exists for 300s: This toleration will allow the pod to stay on the node marked as unreachable for up to 300 seconds. And after that time is up and if the node is still unreachable, the pod will be evicted. 

Events section

 

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  30m   default-scheduler  Successfully assigned default/nginx-pod to node01
  Normal  Pulling    30m   kubelet            Pulling image "nginx"
  Normal  Pulled     30m   kubelet            Successfully pulled image "nginx" in 4.277827536s
  Normal  Created    30m   kubelet            Created container nginx-pod
  Normal  Started    30m   kubelet            Started container nginx-pod

The Events section gives a chronological list of key events that took place over the lifecycle of the pod. This includes events the Kubernetes cluster took to schedule, start, manage, and debug any rising issues related to the pod. Each event includes:

  • Type: This indicates the severity of the event. You will either get ‘Normal’ or ‘Warning’. 
  • Reason: This provides you with a short reason for what the event is related to.
  • Age: The time since the event started
  • From: The key component that generated the event
  • Message: Description of what the event is 

Our example shows fairly common events that take place when you deploy a simple Nginx pod. 

Using kubectl describe with Labels and Selectors

Labels are key-value pairs attached to kubernetes resources that help you organize and categorize resources throughout your cluster. Selectors are expressions/queries that can also filter and select Kubernetes resources based on their labels. 

kubectl describe has the option to filter by labels and selectors, which is useful when you have a large number of workloads running on your cluster and need to easily retrieve details from specific labeled resources. 

You can use two flags with kubectl describe to retrieve extra information: the -lflag or  --selector followed by key=value; you can also use the flags with equality-based selector (=, or !=) or set-based selectors (in, notin, and exists)

Here are some examples of using labels with different Kubernetes resources:

Using kubectl describe labels with Pods

The following is a simple example of using kubectl describe with a single label:

kubectl describe pod -l key=value

Using labels with Deployments

The following example utilizes multiple labels to retrieve specific resources:

kubectl describe deployment -l env=prod, zone=eastus2

Using labels with Services

The following example uses some extra logic to filter out your kubectl describe results:

kubectl describe services —-selector 'env in (prod, dev), app notin (nginx)'

Kubectl describe limitations and other options

kubectl describe is a very powerful command; however, the output it generates has some limitations. It is very static and cannot aggregate, filter, format, or customize the output. 

Here is a list of some of the limitations you may face while using ‘kubectl describe’:

  • Filtering limitations: You can utilize the -l flag or --selector to filter the KLubernetes resources you want to run kubectl describe against, but this is more for applying the describe command against a filtered set of resources instead of modifying and filtering out the output you receive.

It can become challenging if you try to filter out the resources beyond the labels and selectors or try to use logical operators to combine different conditionals and queries. 

  • Lack of aggregation: kubectl describe does not have the full ability to aggregate and summarize data such as the total resource usage across multiple resources such as nodes, pods, deployments, etc. 
  • JSON/YAML support: kubectl describe also does not support structured outputs such as JSON/YAML, so you cannot pipe the output to other tools for analysis or processing. 
  • Lack of output customization: You cannot customize the output you receive from kubectl describe. You will get a static detailed output with all the information you are requesting, so it might become complex for you to parse through it for automation purposes. 

kubectl describe alternatives

You can use other tools to help you get better outputs you can utilize for automation or for more clarity. Here is a list of some alternatives:

  • kubectl get pods command: Using this command with the -o flag allows you to get structured outputs that can be sent to either JSON or YAML format, which you can easily parse through and use within your automation scripts or tools. 
kubectl get pods -o json 
kubectl get pods -o yaml 
  • kubectl logs: This command is useful if kubectl describe command is not generating all the details you need, and you need more real-time logging information. 
kubectl logs pod-name
  • kubectl events command: This is similar to kubectl logs and can also allow you to filter out certain events and retrieve the correct output. 
kubectl get events —-sort-by='{.lastTimestamp}'
  • K9s or Kubernetes Dashboard: This will give you an actual GUI that you can use to get more visuals and mainly use an interactive approach to managing your Kubernetes resources. In the GUI, you have more access to filter, describe, and analyze resources. 
  • Prometheus and Grafana: Utilize monitoring and debugging tools such as Prometheus to retrieve metrics from your cluster and Grafana to visualize the data. This allows for powerful filtering capabilities, aggregation, and visualization options. 
  • CRD (Custom Resource Definition): You always have the option to create and customize your own resource when kubectl describe, kubectl get, kubectl logs or kubectl get events have failed. You can use this option to create a CRD that includes more specialized and relevant information. This approach would be more tailored to your specific requirements and provide more relevant information based on your specific needs.

Practical use cases & best practices for the kubectl describe command

Now we will review a few use cases where we can maximize our use of the ‘kubectl describe’ command and also follow best practices:

1. Debugging a pod

In this scenario, we have a Pod that is stuck on Pending, CrashLoopBackOff, or Error state. We can start by running the kubectl describe command against the pod that is problematic:

kubectl describe pod pod-name

What should we inspect?

  • Status: The Pod status is the first place to check.
  • Containers Section: You should then review the container and validate the state of the container. 
  • Conditions Section: Third, you should check the Conditions section to ensure there are no ‘False’ statuses for each of the requirements listed. 
  • Events: The events can be the first thing you look at to get straight to the point or the last thing if you want to drill down slowly into the issue. You should check for any events that have error messages or warnings. If the pod is not running, there should be an error message here that would identify the issue.

Best practices

By this point, troubleshooting the events and other values given to you from kubectl describe is not helpful. 

You can start running other kubernetes tools to dig deeper, such as kubectl get events for an in-depth series of events that took place on the pod or kubectl logs for extra details. 

kubectl get events --sort-by='{.lastTimestamp}'
kubectl logs

2. Networking issues with Services

In this scenario, the service is not routing traffic correctly to the endpoints. Right away, you can quickly run the kubectl describe command against the service in question and check what endpoints it is pointing towards. This can easily give you a starting point to troubleshoot this issue. 

kubectl describe service service-name

What should we inspect?

  • Endpoints: In the described output, you can scroll down to the ‘Endpoints’ section and check if the correct pod’s internal IP address is listed as an endpoint. 
  • Types and Ports: Ensure the service type is correct and the ports match your configuration
  • Selector: You probably used some sort of labels to deploy your resources, ensuring the services selector matches the labels on the target pods. 

Best practices

  • Instead of validating endpoints through kubectl describe service, you can also use the kubernetes tool, kubectl get endpoints. This will directly verify that the endpoints are corresponding to healthy pods. 
  • Kubectl get service -o wide can also provide you with a summary view

3. Node issues in the cluster

In this scenario, the node in your cluster is stuck in the NotReady state, and you might be wondering where you should start to troubleshoot. The kubectl describe is a great starting point to get visuals on the error messages and status of the Node. 

kubectl describe node node-name 

What should we inspect?

  • Status: Verify the status is showing NotReady.
  • Conditions: Here, you can get a vague idea of the issue you might be dealing with, such as MemoryPressure, DiskPressure, or PIDPressure
  • Capacity: Verify the CPU, Memory and other resource capacity here.
  • Allocatable: Check if CPU, Memory, and other resources are not maxed out. 
  • Events: Check through the event logs for error messages related to disk, network, or other failures. 

Best practices

  • Node health is very important for your cluster to run properly with all of its workloads; therefore, it is important to be proactive and detect issues. 
  • Utilizing a monitoring tool such as Prometheus or Grafana to retrieve and visualize metrics can help detect Node failures early on in the process.

4. Debugging pods and nodes with taints/tolerations

In some cases, pods can be assigned to the wrong nodes based on the taints on your nodes and the tolerations/selectors on your pods. In these situations, it is best to evaluate the affected Pods and Nodes individually to get to the root cause. 

You can start off by inspecting the affected Pod:

kubectl describe pod pod-name 

What should we inspect?

  • Tolerations: In the Tolerations sections, you can check what taints the pod can tolerate and for how long
  • Node Affinity: Review each node affinity rule and ensure which nodes the pod is allowed to run on based on the labels. 
  • Node Selectors: Check the Node-Selectors section to see if the pod is limited to run on nodes with specific labels.
  • Events: Check for any potential events that could have caused any of these mismatches with the Nodes. Look for ‘Warning’ events that have ‘Scheduling’ messages or something along the lines of nodes not matching the pod affinity rules. 

Now let’s inspect the affected Node:

kubectl describe node node-name

What should we inspect?

  • Taints: Go through the taints and make sure they match the tolerations specified on the pod. 
  • Events: Check if there are any node mismatch messages in event logs. 

Best practices

  • Ensure all Taints and Tolerations match across your nodes and pods. Taints with ‘NoSchedule” or ‘NoExecute’ effects require the correct tolerations. 
  • Review Node-Affinity and Anti-Affinity rules and make sure they don’t conflict. 
  • Utilize kubectl logs on the kube-scheduler to get more insight into the mismatch issue. 
  • Use kubectl get events —-sort-by=’.lastTimestamp’ to check for all events in the cluster. 

Managing Kubernetes with Spacelift

If you need assistance managing your Kubernetes projects, look at Spacelift. It brings with it a GitOps flow, so your Kubernetes Deployments are synced with your Kubernetes Stacks, and pull requests show you a preview of what they’re planning to change. 

To take this one step further, you could add custom policies to reinforce the security and reliability of your configurations and deployments. Spacelift provides different types of policies and workflows that are easily customizable to fit every use case. For instance, you could add plan policies to restrict or warn about security or compliance violations or approval policies to add an approval step during deployments. 

You can try Spacelift for free by creating a trial account or booking a demo with one of our engineers.

Key points

The kubectl describe command is useful for managing and troubleshooting Kubernetes resources like Pods, Nodes, and Deployments. It provides detailed information such as status, conditions, and events. Although it offers limited customization, it can be supplemented with tools like kubectl logs, kubectl get, and monitoring solutions like Prometheus and Grafana. 

Manage Kubernetes Easier and Faster

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.

Learn more

The Practitioner’s Guide to Scaling Infrastructure as Code

Transform your IaC management to scale

securely, efficiently, and productively

into the future.

ebook global banner
Share your data and download the guide