Nginx

Configures and debugs nginx: reverse proxy, load balancing, SSL/TLS termination, caching, redirects, and static file serving. Use when writing or reviewing nginx.conf, server blocks, locations, upstreams, or proxy_pass, when a site behind nginx throws 502, 504, 413, 403, or a redirect loop, when WebSockets, SSE, or gRPC break through the proxy, when a certificate works in curl but warns in browsers, when requests hit the wrong location or the backend sees the wrong path, when tuning workers, buffers, gzip, or proxy cache, when rate limiting or blocking abuse, when proxying raw TCP/UDP, or when nginx runs in Docker or Kubernetes. Not for certificate issuance or renewal (ACME, Let's Encrypt) — that is the ssl skill.

User preferences and memory live in ~/Clawic/data/nginx/ (see setup.md on first use, memory-template.md for the file format). If you have data at an old location (~/nginx/ or ~/clawic/nginx/), move it to ~/Clawic/data/nginx/.

Configuration

User-dependent variables. Defaults apply until the user states a preference; store them in ~/Clawic/data/nginx/config.yaml.

Variable	Type	Default	Effect
os_family	debian | rhel | alpine	debian	Selects package/config layout (sites-enabled vs conf.d), makes SELinux the first 502/403 suspect on rhel (debug.md), and sets module install paths
deployment	systemd | docker | kubernetes	systemd	Selects reload commands and resolver address; docker/kubernetes routes advice through containers.md
edge_position	standalone | behind-cdn-lb	standalone	behind-cdn-lb turns on realip guidance, switches redirect logic from $scheme to X-Forwarded-Proto, and flags rate-limit keying on the LB address

Preference areas to record as the user reveals them:

• tooling — OSS vs Plus, mainline vs stable channel, dynamic modules in use (brotli, headers-more, njs); governs which directives are assumed available
• conventions — config layout (conf.d vs sites-enabled vs single file), snippet/include organization, upstream and zone naming; governs where examples place directives
• safety posture — 302-before-301 rollout ramp, HSTS ramp speed, confirm-before-reload on production hosts; governs pacing in redirects.md and ssl.md
• platform — IPv6 listeners, HTTP/2/3 adoption, which CDN/LB sits in front; governs listen directives and header-trust guidance

When To Use

• Writing or reviewing nginx config: server blocks, locations, proxy_pass, upstreams
• Debugging 502/504/413/403/redirect loops behind an nginx reverse proxy
• SSL/TLS termination, HTTP/2, WebSocket, gRPC, or SSE through nginx
• Tuning: workers, buffers, gzip, proxy cache, rate limiting
• Proxying raw TCP/UDP (databases, TLS passthrough, syslog) and operating nginx (upgrades, monitoring, log rotation)
• Not for certificate issuance/renewal itself (ACME, Let's Encrypt) — nginx consumes certs; issuance is the ssl skill

Quick Reference

Situation	Play
Random 502 after a container/backend redeploy	DNS cached at startup — use variable in proxy_pass + resolver (proxy.md)
502 immediately vs 504 after ~60s	502 = refused/reset/bad response; 504 = timeout. Different fixes (proxy.md)
413 on uploads	client_max_body_size — default is 1m; raise in http or the exact server/location
WebSocket connects then dies, or never upgrades	Upgrade trio + timeout (proxy.md)
SSE/streaming arrives all at once	proxy_buffering off for that location (proxy.md)
gRPC calls fail through nginx, direct works	grpc_pass, not proxy_pass — trailers and HTTP/2 (proxy.md)
Wrong file served / 403 on aliased path	root vs alias semantics and the alias-traversal slash bug (semantics.md)
Request hits wrong location block	Re-derive with the matching algorithm below; nginx -T to see effective config
Backend receives wrong path (/api/api/... or missing prefix)	proxy_pass trailing-slash rules (below)
Browser cert warning, curl works	Missing intermediates — serve fullchain (ssl.md)
80→443 redirect loop behind a CDN/LB	Trust X-Forwarded-Proto, don't redirect on $scheme alone (ssl.md)
Redirect fixed in config but browser still loops	Cached 301 — test in curl or a private window (redirects.md)
PHP blank page, "File not found", or browser downloads .php source	SCRIPT_FILENAME and location order (fastcgi.md)
nginx container exits instantly, or template renders empty values	Foreground mode and the envsubst collision (containers.md)
Proxy a database / route TLS by SNI without terminating / forward syslog	stream {} block, ssl_preread (stream.md)
Upgrade the nginx binary or a module without dropping connections	USR2/WINCH signal sequence (operations.md)
"Is nginx overloaded?" / capacity monitoring	stub_status and what its numbers mean (operations.md)
Slow under load, high CPU or connection errors	performance.md
Security headers vanished on some routes	add_header inheritance trap (semantics.md)
Config behaves unlike it reads (if, variables, includes, root/alias)	semantics.md
Anything else	Debugging Order below, then the closest file above

Depth on demand: debug.md startup failures, status-code decoder, tracing · proxy.md 502/504, DNS trap, WebSocket, gRPC, buffering, retries · semantics.md root/alias, inheritance, if, variables, includes, server selection · redirects.md return/rewrite, status codes, canonical host · ssl.md chain, baseline, HSTS, OCSP, mTLS, HTTP/3 · performance.md workers, buffers, gzip, proxy cache · security.md rate/conn limits, auth, hardening · fastcgi.md PHP-FPM · containers.md Docker/K8s · stream.md TCP/UDP, TLS passthrough · operations.md signals, upgrades, monitoring, log rotation.

Core Rules

1. nginx -t && nginx -s reload — never restart to apply config; reload is graceful (old workers finish in-flight requests). Test first: a bad config on restart takes the site down; on reload it's rejected.
2. Read the effective config with nginx -T, not the files — includes, inheritance, and distro defaults (/etc/nginx/conf.d/*) mean the file you're editing may not be what runs.
3. One canonical Host line: proxy_set_header Host $host;. Without it the backend sees the upstream name from proxy_pass — breaks virtual hosts, redirects, and anything that reads Host.
4. proxy_set_header in a location wipes ALL inherited proxy headers from server/http level, same for add_header. Inheritance is all-or-nothing per level: if you set one header in a location, re-declare the full set there.
5. Sizing: max concurrent proxied clients ≈ worker_processes × worker_connections / 2 (each proxied request holds a client fd and an upstream fd). 4 workers × 1024 connections → ~2048 clients. Set worker_rlimit_nofile ≥ 2× worker_connections.
6. Never put logic in if beyond return/rewrite — if in location context creates a pseudo-location where other directives misbehave. Use map for conditionals.
7. Diagnose from the log split: log $request_time and $upstream_response_time together. High request_time + low upstream_time = slow client or buffering problem; both high = slow backend. Without both numbers you're guessing which side is slow.

Location Matching (the real algorithm)

Common misreading is "regex beats prefix". Actual order:

1. Exact = /path — match ends immediately.
2. Find the LONGEST matching prefix (order in file irrelevant for prefixes).
3. If that prefix is marked ^~ — use it, skip regex entirely.
4. Otherwise try regex locations ~ / ~* in FILE ORDER — first regex match wins.
5. No regex matched — fall back to the longest prefix from step 2.

Consequences:

• location /api also matches /api-v2, /apiary — prefix is string prefix, not path segment. Use location /api/ plus location = /api if you need the segment.
• A short regex declared early beats your long careful prefix — ^~ on static asset prefixes is the standard defense.
• location /api/ does not match /api (no trailing slash) — pair with exact match or accept the 404.

proxy_pass Path Rules

• proxy_pass http://backend; (no URI part) → request path passed unchanged: /api/users → /api/users.
• proxy_pass http://backend/; (any URI part, even just /) → matched location prefix is REPLACED by that URI: location /api/ + .../ → /api/users becomes /users.
• URI part inside a regex location or inside if = config error at startup — rewrite instead, or drop the URI part.
• With a variable in proxy_pass (set $up http://backend; proxy_pass $up;) path handling changes again: nginx passes the URI as given in the directive; combine with $request_uri explicitly if needed.
• Verify with the backend's access log or curl -v against the backend directly — not by reasoning about the config.

Proxy Headers & Real IP

Canonical block (re-declare wholesale wherever any proxy_set_header appears — rule 4):

nginx

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

• $host = lowercase, no port; $http_host = raw header with port. APIs that generate absolute URLs usually need $http_host.
• X-Forwarded-For is client-spoofable. If nginx is behind a trusted LB/CDN, use the realip module: set_real_ip_from <LB subnet>; real_ip_header X-Forwarded-For; real_ip_recursive on; — otherwise rate limits and logs key on the LB's IP.
• Headers with underscores are silently dropped by default (underscores_in_headers on to keep) — a classic "auth works with curl -H, fails through nginx" cause.

Upstream & Keepalive

nginx

upstream backend {
    server 10.0.0.2:3000 max_fails=3 fail_timeout=10s;
    server 10.0.0.3:3000 max_fails=3 fail_timeout=10s;
    keepalive 32;
}

• keepalive 32 does NOTHING alone. The trio: keepalive N in upstream + proxy_http_version 1.1; + proxy_set_header Connection ""; in the location. Missing either latter directive = a new TCP (and TLS) handshake per request, silently.
• keepalive N = idle connections kept per worker, not a connection limit.
• Defaults: max_fails=1 fail_timeout=10s. fail_timeout is dual-purpose: the window for counting failures AND the ban duration. max_fails=0 disables marking down entirely.
• These are passive checks (real requests fail first). Active health checks are nginx Plus only — in OSS, put a real health endpoint behind your monitoring instead.
• Server without a port = port 80 — a common surprise when the app listens on 3000.
• Retries: on error/timeout nginx tries the next upstream server. Non-idempotent methods (POST, PATCH, LOCK) are not retried since nginx >=1.9.13 unless you set proxy_next_upstream non_idempotent — do not set it for endpoints with side effects.

try_files & Static

• SPA: try_files $uri $uri/ /index.html; — file, then directory (needs index), then internal fallback. Last arg is a redirect/code, not a checked file: =404 to error instead.
• try_files + proxy_pass in one location: try_files controls; route to the proxy via a named location — try_files $uri @app; + location @app { proxy_pass ...; }. This is the canonical "static if present, else app" pattern.
• Static asset locations: ^~ prefix, access_log off;, expires 30d; + add_header Cache-Control "public, immutable"; for hashed filenames (expires alone without Cache-Control gets ignored by some clients).
• sendfile on; tcp_nopush on; together — sendfile without tcp_nopush leaves the kernel optimization on the table.

SSL/TLS Essentials

• ssl_certificate takes the FULLCHAIN (leaf + intermediates, leaf first, no root). Leaf-only "works" in browsers with cached intermediates and fails on fresh clients — the classic "works for me, warning for users".
• Baseline: ssl_protocols TLSv1.2 TLSv1.3; and start from the Mozilla SSL config generator (intermediate profile) rather than hand-picking ciphers.
• ssl_ciphers only governs ≤TLS1.2; TLS1.3 suites need ssl_conf_command Ciphersuites (OpenSSL) — a source of "my cipher config does nothing".
• ssl_prefer_server_ciphers off is the modern recommendation (client-hardware-aware selection); on was TLS1.2-era advice.
• ssl_session_cache shared:SSL:10m; — ~4000 sessions per MB, shared across workers. Skipping it costs a full handshake per returning client.
• Redirect loops, HSTS rollout, OCSP, client certs → ssl.md.

Rate Limiting

nginx

limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
location /api/ { limit_req zone=api burst=20 nodelay; limit_req_status 429; }

• rate=10r/s is enforced per-millisecond: 1 request per 100ms. Two instant requests without burst = second one rejected. Always pair rate with a burst sized for legitimate client behavior (a browser page load fires 10-30 parallel requests).
• nodelay = serve the burst immediately, refill over time; without it, burst requests queue and add latency.
• Default rejection status is 503 — set 429 so clients and monitoring can tell throttling from outage.
• Behind a CDN/LB without realip configured, $binary_remote_addr is the LB's address — you rate-limit everyone as one client.
• Zone sizing, tiered keys, connection/bandwidth limits → security.md.

Debugging Order

1. nginx -t — config valid? nginx -T — what's actually loaded?
2. tail -f the error log; raise level only on the suspect vhost. Grep-to-cause:

Error log message	Meaning
connect() failed (111: Connection refused)	Backend down or wrong port → 502
no live upstreams	All upstream servers marked failed (max_fails tripped) → 502
upstream prematurely closed connection	Backend crashed mid-response or app timeout shorter than nginx's → 502
upstream timed out (110)	nginx waited proxy_read_timeout (default 60s) → 504
client intended to send too large body	client_max_body_size (default 1m) → 413
worker_connections are not enough	Connection budget exhausted → see Core Rule 5
Anything else	Reproduce with curl -v against nginx AND directly against the backend; the diff localizes the fault

3. A timeout in nginx does not cancel the backend request — the backend keeps burning CPU on a request nobody will read. Fix the slow endpoint, don't just raise the timeout.

Output Gates

Before emitting an nginx config or config advice, verify:

• Apply instructions end with nginx -t then reload — never a restart?
• Every location that sets any proxy_set_header or add_header re-declares the full inherited set (rule 4)?
• proxy_pass URI part checked against the trailing-slash rules — the backend receives the path you intend?
• Upload-handling routes have client_max_body_size above the real payload size?
• New permanent redirects shipped as 302 first, 301 only after verification (redirects.md ramp)?
• If edge_position is behind-cdn-lb: realip configured before anything keys on client IP, and redirects read X-Forwarded-Proto?

Traps

Trap	Why it fails	Do instead
worker_processes auto in containers	Reads host CPU count, not cgroup quota — 64 workers on a 2-CPU limit	Set explicitly to the container's CPU limit
Editing config, reloading, no change	Distro includes another file that wins	nginx -T | less, find who owns the directive
Hostname in proxy_pass to dynamic infra	Resolved once at startup, cached forever	Variable + resolver (proxy.md)
gzip on for images/zip/woff2	Recompressing compressed data: CPU spent, bytes gained	gzip_types with text formats only
if for routing logic	Pseudo-location; directives inside behave unpredictably	map + variable, or separate locations
Log rotation without signal	nginx keeps writing to the deleted inode; disk fills with no visible file	nginx -s reopen (USR1) in the rotate script (operations.md)
Reload "not taking" with WebSockets	Old workers stay alive until long-lived connections close	worker_shutdown_timeout 30s; to bound the drain
No default_server defined	First server block silently catches all unmatched Hosts	Explicit listen 80 default_server; return 444; catch-all (semantics.md)

Where Experts Disagree

• Mainline vs stable channel. The nginx team recommends mainline for most users; "stable" means fewer feature changes, not more reliability. Default: mainline from nginx.org repos when you control the host; pinned distro/stable inside images where reproducibility wins (operations.md).
• conf.d vs sites-enabled. Debian's symlink pattern adds an explicit enable/disable step; flat conf.d is simpler. Either works — the failure mode is mixing both and losing track of what's live; nginx -T is the referee (rule 2).
• Terminate TLS at the CDN/LB or at nginx. Edge termination centralizes cert management; nginx termination keeps encryption to the box. The boundary is compliance scope and who owns header trust — whoever terminates must set X-Forwarded-Proto and realip correctly (ssl.md).

Related Skills

More Clawic skills, get them at https://clawic.com/skills/nginx (install if the user confirms):

• ssl — certificate issuance, renewal, and TLS debugging beyond nginx directives
• docker — nginx in containers: images, networks, and the resolver at 127.0.0.11
• caddy — when automatic HTTPS and a simpler config beat nginx's control
• vps — server provisioning and hardening around the nginx install
• dns — records and propagation issues upstream of the proxy

Feedback

• If useful, star it: https://clawic.com/skills/nginx
• Latest version: https://clawic.com/skills/nginx

Part of Clawic, the verified skill library. Get this skill: https://clawic.com/skills/nginx.