Solving the DevOps Infrastructure Dilemma: Enabling developer velocity with control 💡

Register for the webinar here →

Kubernetes

How to Troubleshoot Kubernetes ImagePullBackOff

How to Troubleshoot Kubernetes ImagePullBackOff

This article will guide you through understanding what ImagePullBackOff means, why it happens, and how to fix it. You’ll be able to solve this error without waiting for another developer or administrator. Let’s get started.

We will cover:

  1. What does ImagePullBackOff mean?
  2. What causes ImagePullBackOff errors?
  3. What happens during an ImagePullBackOff?
  4. How does ImagePullBackOff relate to ErrImagePull?
  5. How to troubleshoot the ImagePullBackOff error?

What does ImagePullBackOff mean?

ImagePullBackOff is a common Kubernetes error which occurs when your cluster is unable to pull a container image required by one of your Pods. This error message can be confusing, but you can debug it by following a methodical troubleshooting process.

ImagePullBackOff means there was a problem pulling a container image required by your Pod. Kubernetes “backs off” before retrying, adding a delay that’s intended to provide time for any temporary issues to resolve themselves.

The error is generated by the Kubelet process running on the Kubernetes Node that hosts your Pod. Nodes are responsible for pulling and storing the container images their Pods require. When a Pod is scheduled to a Node, the Node first has to pull the image for each of the Pod’s containers, as defined by the spec.containers[].image fields in your Pod’s manifest. If an image can’t be pulled, then an ImagePullBackOff error will occur.

ImagePullBackOff can appear even when the Node has already stored an image matching the requested tag. Whether this will happen depends on the imagePullPolicy assigned to the Pod. When the Always policy is used, then Kubelet will try to pull the image regardless of an existing version being available. This ensures the latest image version is used, but will trigger an ImagePullBackOff error if the image can’t be pulled.

What causes ImagePullBackOff errors?

ImagePullBackOff occurs whenever Kubernetes is unable to fetch an image it needs. Unfortunately, Kubernetes can’t always tell you the exact cause. There are several possible reasons, ranging from user error to network conditions:

1. Invalid image name

Accidentally referencing the wrong image, or inserting a typo, will cause your Pod to try to pull an image that doesn’t exist. This is an easy mistake to make, and it could occur in any part of the image name—either the repository (example.com), image name (my-app), or tag (latest).

2. Image removed from the registry 

Sometimes, you might believe you’re using the correct image name, but the image has, in fact, been removed from the registry. Kubernetes can’t pull something that doesn’t exist.

3. Missing credentials for the image registry

You need to supply access credentials when you’re using images from a private registry. Missing credentials will lead to access denied errors that prevent the image being pulled.

4. Hitting registry rate limits 

Image pulls can be blocked when the registry implements rate limiting and you’ve used up your request quota.

Notably, Docker Hub significantly reduced its rate limit quotas in 2020; active Kubernetes clusters that pull many Docker Hub images without authenticating may hit the cap of 100 requests per six hours.

5. Registry inaccessible

ImagePullBackOff can point to the registry being inaccessible, such as when it’s offline for maintenance or your cluster’s network connectivity is degraded.

You’ll notice that none of these situations are particularly complicated. ImagePullBackOff is one of the simplest Kubernetes problems to resolve, once you’ve worked out which root cause you’re facing.

What happens during an ImagePullBackOff?

When a Pod enters the ImagePullBackOff status, Kubernetes will periodically retry the image pull after an exponentially increasing delay. First, it will wait five seconds, then 10 seconds, then 20 seconds, and so on, up to a maximum of five minutes between attempts.

The increasing delay is designed to give transient issues such as flaky network connectivity a chance to resolve themselves. If the error is being caused by one of these temporary problems, then the image will eventually be pulled successfully and the Pod startup procedure will continue. But if the error is actually due to a permanent issue—such as a typo in the image’s name–then the Pod will be stuck displaying ImagePullBackOff forever, until you take action to resolve the error.

How does ImagePullBackOff relate to ErrImagePull?

ImagePullBackOff is closely related to the ErrImagePull error. You may observe Pods first displaying ErrImagePull, then switching to ImagePullBackOff.

Both errors have the same possible root causes. ErrImagePull occurs the first time an image pull fails. It’s then replaced by ImagePullBackOff once a retry has also failed, causing Kubernetes to begin the back-off cycle.

Consequently, you can treat ErrImagePull and ImagePullBackOff as equivalent to each other when debugging. You’re more likely to see ImagePullBackOff because ErrImagePull will be quickly replaced after Kubernetes performs the first retry attempt.

How to troubleshoot ImagePullBackOff error?

Let’s see how to troubleshoot an ImagePullBackOff error.

1. Create an example Pod manifest

First, copy the following Pod manifest and save it to pod.yaml in your working directory:

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
spec:
  containers:
    - name: nginx
      image: nginx:lates

The spec.containers.image field intentionally contains a typo. Kubernetes will try to pull a non-existent image, which will lead to an ImagePullBackOff error.

2. Apply the manifest to your Kubernetes cluster

Use Kubectl to apply the manifest to your Kubernetes cluster:

$ kubectl apply -f pod.yaml
pod/demo-pod created

3. Run get pods command

Next, run Kubectl’s get pods command to check whether your Pod is running:

$ kubectl get pods
NAME       READY   STATUS         RESTARTS   AGE
demo-pod   0/1     ErrImagePull   0          14s

The Pod shows the ErrImagePull status.

Wait a minute, then repeat the get pods command:

$ kubectl get pods
NAME       READY   STATUS             RESTARTS   AGE
demo-pod   0/1     ImagePullBackOff   0          71s

Now the Pod has the ImagePullBackOff status.

4. Inspect you Pod

To inspect why this has happened, you can use the describe pod command to view the Pod’s event history. The events are displayed in a table at the bottom of the command’s output:

$ kubectl describe pod demo-pod
...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  67s                default-scheduler  Successfully assigned default/demo-pod to heronweb-basic-4vcpu-8gb-y6m2k
  Normal   Pulling    26s (x3 over 67s)  kubelet            Pulling image "nginx:lates"
  Warning  Failed     25s (x3 over 66s)  kubelet            Failed to pull image "nginx:lates": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:lates": failed to resolve reference "docker.io/library/nginx:lates": docker.io/library/nginx:lates: not found
  Warning  Failed     25s (x3 over 66s)  kubelet            Error: ErrImagePull
  Normal   BackOff    14s (x3 over 65s)  kubelet            Back-off pulling image "nginx:lates"
  Warning  Failed     14s (x3 over 65s)  kubelet            Error: ImagePullBackOff

The event history explains what happened to the Pod. You can see that the image pull fails with a NotFound message, indicating the image name is invalid. The ErrImagePull status is assigned, then a retry occurs. This also fails, and Kubernetes moves the Pod to the ImagePullBackOff status.

You should view your Pod’s event history as the first troubleshooting step whenever you’re diagnosing an ImagePullBackOff error. It will usually tell you what’s happened or point to the underlying cause. For example, if the event message reports being unable to reach the registry server, your Node’s outbound network connectivity could have failed, or the registry might be offline.

Fixing ImagePullBackOff Error

Here’s a cheatsheet for how to resolve specific ImagePullBackOff causes.

1. Invalid image name, or image not in the registry

Issues such as the one shown above, where the image can’t be pulled because it doesn’t exist, give you two ways to fix the problem:

  1. Adjust your Pod manifest to remove the typo or reference a valid image.
  2. Push the expected image into the registry so that Kubernetes can pull it.

If the problem was a typo and you need to make manifest changes, you’ll need to reapply the fixed manifest to your cluster.

Try editing the manifest shown above so that the image field is amended to nginx:latest:

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
spec:
  containers:
    - name: nginx
      image: nginx:latest

Then use Kubectl to apply the updated manifest:

$ kubectl apply -f pod.yaml
pod/demo-pod configured

When you run the get pods command, you should see the Pod is now Running as Kubernetes has been able to pull the corrected image tag.

$ kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
demo-pod   1/1     Running   0          37m

Alternatively, if you resolve the issue by pushing the missing image into the registry, you should be able to wait and watch as Kubernetes automatically starts the Pod. Once the image is in the registry, Kubernetes will successfully pull it on the next retry attempt once the back-off delay has expired.

2. Missing registry credentials

Image pulls that fail because you need to authenticate to a private registry will generate a Pod event that looks similar to the following:

Failed to pull image "docker.io/user/image": rpc error: code = Unknown desc = Error response from daemon: pull access denied for user/image, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

If you see a requested access to the resource is denied message like this, then you need to supply registry credentials so Kubernetes can pull the image.

To do this, you must create a Kubernetes secret that contains your registry user credentials:

$ kubectl create secret docker-registry docker-registry-credentials \
  --docker-server=https://index.docker.io/v1/
  --docker-username=user
  --docker-password=password
  --docker-email=example@example.com

The command shown above will create a secret called docker-registry-credentials. Next, you must adjust your Pod’s manifest so it references the secret within the spec.imagePullSecrets field:

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
spec:
  containers:
    - name: nginx
      image: nginx:latest
  imagePullSecrets:
    - name: docker-registry-credentials

Kubernetes will now use the credentials in your secret when it pulls the Pod’s images. Your Node will correctly authenticate to the image registry, allowing the image to be pulled, and fixing the ImagePullBackOff error.

3. Network connectivity issues

Your remediation options can be limited when an ImagePullBackOff error appears to be caused by network problems.

In this instance, you can try to pull the image from a different machine on a separate physical network. If the pull also fails there, then you can assume the remote image registry is down. You could try to restart the registry server if you’re in control of it, but otherwise, you’ll have to wait for the provider to recover. Once the registry is back up, Kubernetes should successfully pull the image and start your Pod.

When the image pulls successfully from another machine, the cause could be due to impaired network connectivity within your cluster or on the Node that’s running the Pod. To dive deeper into this problem, you’ll need to retrieve the Node’s logs to inspect what’s going on.

Key points

Kubernetes Pods fail with an ImagePullBackOff error when repeated attempts to pull a container image have been unsuccessful. Kubernetes steadily increases the delay between retry attempts, “backing off” each time a failure occurs.

To resolve an ImagePullBackOff error, use the steps shown above to identify why the image is inaccessible. It’s often something simple such as a typo in the image’s tag, or missing credentials for its registry. Alternatively, ImagePullBackOff can occur because the image has been removed from the registry or there’s a network connectivity problem.

ImagePullBackOff is one of several Pod errors you might encounter when using Kubernetes. Check out our other Kubernetes articles to help you troubleshoot more cryptic messages, such as CrashLoopBackOff and OOMKilled.

If you need any assistance with managing your Kubernetes projects, take a look at Spacelift. It brings with it a GitOps flow, so your Kubernetes Deployments are synced with your Kubernetes Stacks, and pull requests show you a preview of what they’re planning to change. It also has an extensive selection of policies, which lets you automate compliance checks and build complex multi-stack workflows. You can check it for free by creating a trial account.

Manage Kubernetes Easier and Faster

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.

Start free trial

The Practitioner’s Guide to Scaling Infrastructure as Code

Transform your IaC management to scale

securely, efficiently, and productively

into the future.

ebook global banner
Share your data and download the guide