CRITICALNginxNetworking

Nginx returns 502 Bad Gateway under load

nginxproxy502networkingperformance
Symptoms
  • Nginx error log shows 'upstream prematurely closed connection'
  • 502 rate spikes in sync with request volume
  • Backend service logs look healthy
Root Cause
  • keepalive_requests on upstream exhausted and Nginx re-opens connections aggressively
  • Backend timeout is shorter than Nginx proxy_read_timeout
  • Backpressure from a slow dependency
Diagnosis
  • tail -f /var/log/nginx/error.log | grep upstream
  • ss -s on the Nginx host - look at TIME_WAIT count
  • Compare upstream_response_time vs request_time
Fix
  • Tune upstream keepalive:
  • upstream app {
      server app:8080;
      keepalive 64;
      keepalive_requests 1000;
      keepalive_timeout 60s;
    }
    server {
      location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_read_timeout 30s;
      }
    }
    
  • Ensure backend keepalive timeout > Nginx keepalive_timeout
  • Prevention
    • Load test before launch and alert on 5xx > 1%
    • Emit upstream_response_time to Prometheus
    • Use circuit breakers in the application tier