WARNINGKubernetesContainer Orchestration

ImagePullBackOff when rolling out a new container image

kubernetesregistryimagepullbackoffsecrets
Symptoms
  • Pods stay in ImagePullBackOff or ErrImagePull
  • kubectl describe shows 'unauthorized: authentication required' or 'manifest unknown'
  • New deployments never become Ready
Root Cause
  • Registry credentials missing or expired in the namespace
  • Image tag does not exist or was overwritten
  • Private registry not whitelisted on the nodes
Diagnosis
  • kubectl describe pod <pod> and read the Events section
  • Verify the image exists: `docker pull <image>` from a worker node
  • Inspect the imagePullSecret: kubectl get secret regcred -o yaml
Fix
  • Recreate the pull secret:
  • kubectl create secret docker-registry regcred \
      --docker-server=registry.example.com \
      --docker-username=$USER \
      --docker-password=$TOKEN \
      --namespace=prod
    
  • Reference it in the service account or pod spec:
  • imagePullSecrets:
      - name: regcred
    
    Prevention
    • Pin images by digest (sha256) instead of mutable tags
    • Rotate registry tokens via a secret manager with expiry alerts
    • Mirror critical images to an internal registry