What is ImagePullBackOff? | DevOps Dictionary

ImagePullBackOff — Why Your Pod Can't Start

What Is ImagePullBackOff in Simple Terms?

Before a pod can run, Kubernetes must pull the container image onto the node. ImagePullBackOff means that pull is failing — and Kubernetes is retrying with increasing wait times (10s, 20s, 40s... up to 5 minutes) rather than hammering the registry endlessly.

◈ DIAGRAM

+------------------------------------------+
| Pod scheduled on mumbai-worker-1         | <- Step 1: Scheduler picks a node
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Kubelet tries to pull the image          | <- Step 2: Contacts the registry
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Pull fails                               | <- Step 3: Registry rejects or unreachable
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Status: ErrImagePull                     | <- Step 4: First failure label
+------------------------------------------+
                    |
                    v
+------------------------------------------+
| Kubelet retries with backoff timer       | <- Step 5: Retry at 10s, 20s, 40s...
| Status: ImagePullBackOff                 |
+------------------------------------------+

REMEMBER THIS
**Remember:** `ErrImagePull` and `ImagePullBackOff` are the same root problem. `ErrImagePull` is the first attempt, `ImagePullBackOff` is what you see once Kubernetes starts spacing out retries. Always fix the underlying cause — retries will not resolve it on their own.

The Four Root Causes — In Order of Frequency

◈ DIAGRAM

+------------------------------------------+
| Cause 1: Wrong image name or tag         | <- Most common. Typo in deployment YAML
+------------------------------------------+
| Cause 2: Image does not exist in registry| <- Tag was deleted or never pushed
+------------------------------------------+
| Cause 3: Missing registry credentials   | <- Private registry, no imagePullSecret
+------------------------------------------+
| Cause 4: Registry unreachable from node | <- Network issue, VPC firewall, DNS
+------------------------------------------+

How to Diagnose — Step by Step

Bash

# Step 1 — Read the exact error message from pod events
kubectl describe pod <pod-name> -n production
 
# Look for this section at the bottom of the output:
# Events:
#   Type     Reason     Message
#   ----     ------     -------
#   Warning  Failed     Failed to pull image "registry.razorpay.in/api:v2.5":
#                       rpc error: code = Unknown desc = pulling image:
#                       unauthorized: authentication required
 
# Step 2 — Confirm what image name is in the spec
kubectl get pod <pod-name> -n production \
  -o jsonpath='{.spec.containers[*].image}'
# Output: registry.razorpay.in/api:v2.5

Fix 1 — Wrong Image Name or Tag

The most common cause. A tag that does not exist on the registry returns a manifest unknown error.

YAML

# deployment.yaml — verify these three parts exactly
spec:
  containers:
    - name: api
      image: registry.razorpay.in/api:v2.5.1   # registry / name / tag
      #       ^^^^^^^^^^^^^^^^^  ^^^  ^^^^^
      #       registry host      repo  tag — all three must be exact

Bash

# Verify the tag exists in the registry before deploying
# For Docker Hub:
docker pull nginx:1.25-alpine
 
# For a private ECR registry:
aws ecr describe-images \
  --repository-name api \
  --region ap-south-1 \
  --query 'imageDetails[*].imageTags'

Fix 2 — Private Registry Needs Authentication

When pulling from a private registry (ECR, GCR, self-hosted Harbor), the node needs credentials. These are stored as a Kubernetes Secret of type kubernetes.io/dockerconfigjson and referenced in the pod spec as imagePullSecrets.

Bash

# Create the registry credential Secret
kubectl create secret docker-registry ecr-credentials \
  --docker-server=905418385260.dkr.ecr.ap-south-1.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password --region ap-south-1) \
  -n production

YAML

# deployment.yaml — reference the Secret in the pod spec
spec:
  imagePullSecrets:
    - name: ecr-credentials      # Must exist in the same namespace as the pod
  containers:
    - name: api
      image: 905418385260.dkr.ecr.ap-south-1.amazonaws.com/api:v2.5.1

COMMON MISTAKE / WARNING
**Security:** ECR tokens expire every 12 hours. For production clusters at Razorpay or Zerodha, use the `amazon-ecr-credential-helper` or an IRSA-based solution instead of a static Secret — static tokens cause ImagePullBackOff every 12 hours when the token silently expires.

Fix 3 — Registry Unreachable from Node

If the image name and credentials are correct but the pull still fails, the node cannot reach the registry. This is common when private registries are inside a VPC and worker nodes are in a different subnet.

Bash

# SSH into the affected node and test registry connectivity directly
ssh rahul@10.0.1.50
 
# Test TCP connectivity to the registry
nc -zv registry.razorpay.in 443
# Expected: Connection to registry.razorpay.in 443 port [tcp/https] succeeded!
 
# Test DNS resolution of the registry hostname
nslookup registry.razorpay.in
# If this fails, check your VPC DNS settings and /etc/resolv.conf on the node
 
# Attempt a manual pull from the node
crictl pull registry.razorpay.in/api:v2.5.1
# The error output here is far more detailed than what kubectl shows

Quick Troubleshooting Reference

Error Message	Root Cause	Fix
`manifest unknown`	Tag does not exist in the registry	Push the correct tag or fix the tag name in the deployment
`unauthorized: authentication required`	Missing or expired credentials	Create or refresh `imagePullSecrets`
`pull access denied`	Image is private, wrong credentials	Verify username and password in the docker-registry Secret
`no such host`	Registry DNS not resolving on the node	Check node `/etc/resolv.conf` and VPC DNS configuration
`connection refused`	Registry unreachable on port 443	Check firewall rules and security group for the node
`context deadline exceeded`	Network timeout to registry	Check VPC route tables, NAT gateway, or proxy config

Bash

# Full diagnostic sequence — run these in order
# 1. Read the exact error
kubectl describe pod <pod-name> -n production | grep -A 10 Events
 
# 2. Confirm the image string in the spec
kubectl get pod <pod-name> -n production -o jsonpath='{.spec.containers[*].image}'
 
# 3. Check if imagePullSecret exists in the right namespace
kubectl get secret ecr-credentials -n production
 
# 4. Test the secret content is valid
kubectl get secret ecr-credentials -n production \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq
 
# 5. Force a fresh pull attempt by deleting and recreating the pod
kubectl delete pod <pod-name> -n production
# The deployment controller will recreate it immediately

COMMON MISTAKE / WARNING
**Common Mistake:** Creating the `imagePullSecrets` Secret in the `default` namespace but deploying the pod to `production`. Secrets are namespace-scoped — the Secret must exist in the same namespace as the pod that references it. A Secret in `default` is completely invisible to pods in `production`.

PLACEMENT PRO TIP
**Tip:** On EKS clusters, attach the `AmazonEC2ContainerRegistryReadOnly` IAM policy to your node group's IAM role. Worker nodes will authenticate to ECR automatically without any `imagePullSecrets` — ECR credentials are handled transparently via instance metadata. This is the cleanest solution for Hotstar or Swiggy scale where dozens of services pull from the same ECR registry. ===