This article will guide you through understanding what ImagePullBackOff means, why it happens, and how to fix it. You’ll be able to solve this error without waiting for another developer or administrator. Let’s get started.
We will cover:
ImagePullBackOff is a common Kubernetes error which occurs when your cluster is unable to pull a container image required by one of your Pods. This error message can be confusing, but you can debug it by following a methodical troubleshooting process.
ImagePullBackOff means there was a problem pulling a container image required by your Pod. Kubernetes “backs off” before retrying, adding a delay that’s intended to provide time for any temporary issues to resolve themselves.
The error is generated by the Kubelet process running on the Kubernetes Node that hosts your Pod. Nodes are responsible for pulling and storing the container images their Pods require. When a Pod is scheduled to a Node, the Node first has to pull the image for each of the Pod’s containers, as defined by the spec.containers[].image fields in your Pod’s manifest. If an image can’t be pulled, then an ImagePullBackOff error will occur.
ImagePullBackOff can appear even when the Node has already stored an image matching the requested tag. Whether this will happen depends on the imagePullPolicy assigned to the Pod. When the Always policy is used, then Kubelet will try to pull the image regardless of an existing version being available. This ensures the latest image version is used, but will trigger an ImagePullBackOff error if the image can’t be pulled.
ImagePullBackOff occurs whenever Kubernetes is unable to fetch an image it needs. Unfortunately, Kubernetes can’t always tell you the exact cause. There are several possible reasons, ranging from user error to network conditions:
1. Invalid image name
Accidentally referencing the wrong image, or inserting a typo, will cause your Pod to try to pull an image that doesn’t exist. This is an easy mistake to make, and it could occur in any part of the image name—either the repository (example.com), image name (my-app), or tag (latest).
2. Image removed from the registry
Sometimes, you might believe you’re using the correct image name, but the image has, in fact, been removed from the registry. Kubernetes can’t pull something that doesn’t exist.
3. Missing credentials for the image registry
You need to supply access credentials when you’re using images from a private registry. Missing credentials will lead to access denied errors that prevent the image being pulled.
4. Hitting registry rate limits
Image pulls can be blocked when the registry implements rate limiting and you’ve used up your request quota.
Notably, Docker Hub significantly reduced its rate limit quotas in 2020; active Kubernetes clusters that pull many Docker Hub images without authenticating may hit the cap of 100 requests per six hours.
5. Registry inaccessible
ImagePullBackOff can point to the registry being inaccessible, such as when it’s offline for maintenance or your cluster’s network connectivity is degraded.
You’ll notice that none of these situations are particularly complicated. ImagePullBackOff is one of the simplest Kubernetes problems to resolve, once you’ve worked out which root cause you’re facing.
When a Pod enters the ImagePullBackOff status, Kubernetes will periodically retry the image pull after an exponentially increasing delay. First, it will wait five seconds, then 10 seconds, then 20 seconds, and so on, up to a maximum of five minutes between attempts.
The increasing delay is designed to give transient issues such as flaky network connectivity a chance to resolve themselves. If the error is being caused by one of these temporary problems, then the image will eventually be pulled successfully and the Pod startup procedure will continue. But if the error is actually due to a permanent issue—such as a typo in the image’s name–then the Pod will be stuck displaying ImagePullBackOff forever, until you take action to resolve the error.
ImagePullBackOff is closely related to the ErrImagePull error. You may observe Pods first displaying ErrImagePull, then switching to ImagePullBackOff.
Both errors have the same possible root causes. ErrImagePull occurs the first time an image pull fails. It’s then replaced by ImagePullBackOff once a retry has also failed, causing Kubernetes to begin the back-off cycle.
Consequently, you can treat ErrImagePull and ImagePullBackOff as equivalent to each other when debugging. You’re more likely to see ImagePullBackOff because ErrImagePull will be quickly replaced after Kubernetes performs the first retry attempt.
Let’s see how to troubleshoot an ImagePullBackOff error.
1. Create an example Pod manifest
First, copy the following Pod manifest and save it to pod.yaml in your working directory:
apiVersion: v1
kind: Pod
metadata:
name: demo-pod
spec:
containers:
- name: nginx
image: nginx:lates
The spec.containers.image field intentionally contains a typo. Kubernetes will try to pull a non-existent image, which will lead to an ImagePullBackOff error.
2. Apply the manifest to your Kubernetes cluster
Use Kubectl to apply the manifest to your Kubernetes cluster:
$ kubectl apply -f pod.yaml
pod/demo-pod created
3. Run get pods command
Next, run Kubectl’s get pods command to check whether your Pod is running:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
demo-pod 0/1 ErrImagePull 0 14sThe Pod shows the ErrImagePull status.
Wait a minute, then repeat the get pods command:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
demo-pod 0/1 ImagePullBackOff 0 71sNow the Pod has the ImagePullBackOff status.
4. Inspect you Pod
To inspect why this has happened, you can use the describe pod command to view the Pod’s event history. The events are displayed in a table at the bottom of the command’s output:
$ kubectl describe pod demo-pod
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 67s default-scheduler Successfully assigned default/demo-pod to heronweb-basic-4vcpu-8gb-y6m2k
Normal Pulling 26s (x3 over 67s) kubelet Pulling image "nginx:lates"
Warning Failed 25s (x3 over 66s) kubelet Failed to pull image "nginx:lates": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:lates": failed to resolve reference "docker.io/library/nginx:lates": docker.io/library/nginx:lates: not found
Warning Failed 25s (x3 over 66s) kubelet Error: ErrImagePull
Normal BackOff 14s (x3 over 65s) kubelet Back-off pulling image "nginx:lates"
Warning Failed 14s (x3 over 65s) kubelet Error: ImagePullBackOffThe event history explains what happened to the Pod. You can see that the image pull fails with a NotFound message, indicating the image name is invalid. The ErrImagePull status is assigned, then a retry occurs. This also fails, and Kubernetes moves the Pod to the ImagePullBackOff status.
You should view your Pod’s event history as the first troubleshooting step whenever you’re diagnosing an ImagePullBackOff error. It will usually tell you what’s happened or point to the underlying cause. For example, if the event message reports being unable to reach the registry server, your Node’s outbound network connectivity could have failed, or the registry might be offline.
Here’s a cheatsheet for how to resolve specific ImagePullBackOff causes.
1. Invalid image name, or image not in the registry
Issues such as the one shown above, where the image can’t be pulled because it doesn’t exist, give you two ways to fix the problem:
- Adjust your Pod manifest to remove the typo or reference a valid image.
- Push the expected image into the registry so that Kubernetes can pull it.
If the problem was a typo and you need to make manifest changes, you’ll need to reapply the fixed manifest to your cluster.
Try editing the manifest shown above so that the image field is amended to nginx:latest:
apiVersion: v1
kind: Pod
metadata:
name: demo-pod
spec:
containers:
- name: nginx
image: nginx:latestThen use Kubectl to apply the updated manifest:
$ kubectl apply -f pod.yaml
pod/demo-pod configured
When you run the get pods command, you should see the Pod is now Running as Kubernetes has been able to pull the corrected image tag.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
demo-pod 1/1 Running 0 37mAlternatively, if you resolve the issue by pushing the missing image into the registry, you should be able to wait and watch as Kubernetes automatically starts the Pod. Once the image is in the registry, Kubernetes will successfully pull it on the next retry attempt once the back-off delay has expired.
2. Missing registry credentials
Image pulls that fail because you need to authenticate to a private registry will generate a Pod event that looks similar to the following:
Failed to pull image "docker.io/user/image": rpc error: code = Unknown desc = Error response from daemon: pull access denied for user/image, repository does not exist or may require 'docker login': denied: requested access to the resource is deniedIf you see a requested access to the resource is denied message like this, then you need to supply registry credentials so Kubernetes can pull the image.
To do this, you must create a Kubernetes secret that contains your registry user credentials:
$ kubectl create secret docker-registry docker-registry-credentials \
--docker-server=https://index.docker.io/v1/
--docker-username=user
--docker-password=password
--docker-email=example@example.comThe command shown above will create a secret called docker-registry-credentials. Next, you must adjust your Pod’s manifest so it references the secret within the spec.imagePullSecrets field:
apiVersion: v1
kind: Pod
metadata:
name: demo-pod
spec:
containers:
- name: nginx
image: nginx:latest
imagePullSecrets:
- name: docker-registry-credentialsKubernetes will now use the credentials in your secret when it pulls the Pod’s images. Your Node will correctly authenticate to the image registry, allowing the image to be pulled, and fixing the ImagePullBackOff error.
3. Network connectivity issues
Your remediation options can be limited when an ImagePullBackOff error appears to be caused by network problems.
In this instance, you can try to pull the image from a different machine on a separate physical network. If the pull also fails there, then you can assume the remote image registry is down. You could try to restart the registry server if you’re in control of it, but otherwise, you’ll have to wait for the provider to recover. Once the registry is back up, Kubernetes should successfully pull the image and start your Pod.
When the image pulls successfully from another machine, the cause could be due to impaired network connectivity within your cluster or on the Node that’s running the Pod. To dive deeper into this problem, you’ll need to retrieve the Node’s logs to inspect what’s going on.
Kubernetes Pods fail with an ImagePullBackOff error when repeated attempts to pull a container image have been unsuccessful. Kubernetes steadily increases the delay between retry attempts, “backing off” each time a failure occurs.
To resolve an ImagePullBackOff error, use the steps shown above to identify why the image is inaccessible. It’s often something simple such as a typo in the image’s tag, or missing credentials for its registry. Alternatively, ImagePullBackOff can occur because the image has been removed from the registry or there’s a network connectivity problem.
ImagePullBackOff is one of several Pod errors you might encounter when using Kubernetes. Check out our other Kubernetes articles to help you troubleshoot more cryptic messages, such as CrashLoopBackOff and OOMKilled.
If you need any assistance with managing your Kubernetes projects, take a look at Spacelift. It brings with it a GitOps flow, so your Kubernetes Deployments are synced with your Kubernetes Stacks, and pull requests show you a preview of what they’re planning to change. It also has an extensive selection of policies, which lets you automate compliance checks and build complex multi-stack workflows. You can check it for free by creating a trial account.
Manage Kubernetes Easier and Faster
Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.
