Reverse Proxy and Load Balancing

A reverse proxy receives requests on behalf of a service and then initiates exchanges with the backend; load balancing decides which available target receives a connection or request. The two are often implemented in a single component, but they are not the same concept: you may need a reverse proxy even with only one backend, and a Layer 4 load balancer may not parse HTTP at all.

Taking https://api.example.com/orders as an example, a user request might first pass through an edge entry point, then to a regional proxy, and finally to an order instance. Each hop has its own connection, timeout, and identity boundaries; a 502 seen by the client might correspond to a backend request that was already submitted but failed during the response transmission.

Two Connections, Three Easily Confused Identities

Layer 7 proxies typically terminate the downstream TLS and establish an independent upstream connection. The client verifies the public service certificate provided by the proxy; if the proxy accesses the backend via HTTPS, it must also verify the backend's identity. The TLS client seen by the backend might be the proxy, not the end user.

You must also record separately: the authority of the client's original request, the TLS ServerName when the proxy connects to the upstream, and the external service name used by the application to generate absolute URLs. They may be the same or different. Changing the Host to 127.0.0.1:3000 everywhere will cause some applications to generate incorrect redirects; keeping the external Host might not meet the virtual host requirements of the hosted upstream.

A forward proxy represents the client choosing the target, while a reverse proxy represents the service providing the entry point; "hiding addresses" is a possible outcome, not the definition. A proxy may also actively pass the original address, so you cannot simply define them by who is anonymized.

L4 / L7: The Unit of Selection Matters More Than Speed Labels

An HTTP/2 connection can carry many requests. Assigning this connection to one backend via L4 does not mean each request within it is load-balanced separately; L7 can select the upstream per request, but is still constrained by connection pools, stream concurrency, and affinity policies.

Least connections is not exclusive to L7; for example, NGINX stream upstream supports least_conn. Conversely, few connections do not necessarily mean low business load: one connection may carry many HTTP/2 streams or one expensive long-running task. NGINX stream upstream

L4 not parsing HTTP usually reduces some processing overhead, but you cannot declare that all L4 deployments are faster than L7 without considering TLS, copying, cache hits, and hardware paths. WebSockets do not require L4 only; HTTP proxies that support handshake and tunnel forwarding can also carry them.

Balancing Algorithms: Ask First What Is Being Balanced—Count, Connections, or Cost

For example, if A is assigned 100 requests of 2ms each, and B is assigned 100 requests of 200ms each, the scheduling count is completely uniform, but the serial work volume differs by 100 times. A weight of 3:1 only constrains the selection share under its specific selection model; it does not guarantee three times the bytes or three times the CPU usage.

If healthy instances have a combined average processing capacity of 1000 requests/second, but the entry point continuously receives 1200, changing the balancing algorithm cannot eliminate the continuously growing queue. You need queue limits, backpressure, rate limiting, or scaling, and must leave budget for retries. NGINX HTTP upstream

Client Address and Forwarding Headers: Trusted Source Before Field Values

Forwarded, X-Forwarded-For, X-Forwarded-Proto, etc., can express information observed by the previous hop, but external clients can also construct fields with the same names. The backend cannot unconditionally take the leftmost address for auditing, rate limiting, or access control.

You should define which directly connected sources are trusted proxies, as well as the rules for cleaning, appending, and overriding at each layer. A multi-layer chain can parse forward from the known trusted end, stopping at untrusted boundaries; private network segments do not naturally equate to controlled proxies. Forwarded itself does not provide cryptographic signatures. RFC 7239

The PROXY protocol is another mechanism for passing metadata at the beginning of a connection; it is not an HTTP header and does not automatically authenticate the sender; it must be enabled by both parties on a limited entry point. Sending it to a listener expecting standard TLS or HTTP will cause parsing failures.

Minimal Caddy Example: Preserving Default Forwarding Rules

The following configuration is provided for offline understanding: route to two local instances at the local port 8080 entry point by path, preserving the /api/ prefix. There is no public certificate application, nor does it rely on a real domain name.

http://127.0.0.1:8080 {
    handle /api/* {
        reverse_proxy 127.0.0.1:9001 127.0.0.1:9002 {
            lb_policy round_robin
            health_uri /readyz
            health_interval 10s
            health_timeout 2s
            health_fails 2
            health_passes 2
            transport http {
                dial_timeout 2s
                response_header_timeout 5s
            }
        }
    }
    handle {
        respond "not found" 404
    }
}

handle does not strip the matching prefix; if changed to handle_path, you must confirm that the URI expected by the backend changes accordingly. /api/* does not automatically include the exact path /api; the entry contract should be explicit.

Caddy handles X-Forwarded-For/Proto/Host by default and ignores corresponding values passed from untrusted entries; what needs additional configuration is the trusted proxy boundary, not unconditionally overwriting all forwarding fields. This uses the HTTP upstream; the Host behavior for HTTPS upstreams varies by version. Starting from Caddy 2.11.0, it can automatically match the upstream Host; older versions should explicitly configure the actual name. Caddy reverse_proxy

This example has been adapted and checked offline with Caddy 2.10.2; passing the configuration syntax does not mean the backend /readyz implementation is correct, nor does it mean network traffic has been verified. Automatic HTTPS, TLS trust, and production exposure addresses should be configured separately based on deployment.

Timeouts and Retries: Constrain All Attempts with the Same Business Deadline

Connection timeout limits connection establishment, response header timeout limits waiting for headers, read idle timeout limits the interval between two progress points, and end-to-end deadline limits the total business time. They cannot replace each other. For example, NGINX proxy_read_timeout constrains the wait between consecutive reads, not the maximum duration of the entire response. NGINX proxy timeout

After the first attempt has already taken 220ms, the second attempt cannot reset the full 850ms budget again. You must also consider proxy queuing, response return, and the client's remaining wait time. Whether the backend task is cancelled after a timeout depends on whether the cancellation signal continues to propagate to the application and downstream.

Retriable conditions must be met simultaneously: the request body must be replayable, the failure phase must allow it, business execution uncertainty must be acceptable, and the remaining budget must be sufficient. After partially sending a response to the client, it is usually not possible to transparently switch to another backend to replay the entire response. POST may rely on explicit idempotency keys; "receiving a 502" does not automatically prove safe repetition.

budget, elapsed, first_attempt, reserve = 850, 140, 220, 100
remaining = max(0, budget - elapsed - first_attempt - reserve)
assert remaining == 390
# Teaching model: independent failure probability p per attempt, max 3 attempts.
p = 0.2
expected_attempts = 1 + p + p**2
assert abs(expected_attempts - 1.24) < 1e-12
assert abs(1000 * expected_attempts - 1240) < 1e-9
# Three layers, max 3 attempts each, worst case amplifies to 27 downstream attempts.
assert 3**3 == 27
print('Remaining ms / Expected attempts per sec / Nested worst attempts:', remaining, 1000 * expected_attempts, 3**3)

The independent failure above is just a teaching assumption; real failures are often correlated, and retries may further collapse dependencies. Limiting the number of attempts, total duration, and additional retry traffic, and using appropriate backoff and jitter, is necessary to avoid amplification from layered retries.

How Much Longer Does It Take to Reset the Full Timeout for Each Retry?

The time bars share the same start point and deadline; the first failure does not shift them. The "available budget" here only answers whether there is still time, not whether it is safe to repeat the business operation; retries should only occur when both judgments are passed.

Buffering, Streaming, and Backpressure

Response buffering allows the proxy to read the upstream faster, cache the content, and slowly send it to the client, reducing long-term occupation of the upstream by slow readers; the cost is memory/disk budget and potential first-byte or chunking latency. Request buffering affects how slow uploads occupy the backend and whether the request body can be replayed after failure.

SSE, model streaming output, and large downloads need flushing configured according to delivery requirements. Relying solely on the application calling flush does not guarantee that every proxy and compressor outputs immediately; turning off all buffering is not a free optimization, as slow clients may propagate backpressure upstream and occupy resources for a long time. NGINX proxy buffering

Connection pool reuse reduces handshakes, but you must coordinate the lifespan of idle upstream connections. Pool connections that the backend has already closed but the proxy has not yet detected will cause failures upon reuse; whether to retry still depends on request semantics and the sending phase, and infinite retries cannot mask pool configuration issues.

Health Checks and Publish Drain

Active checks periodically send probes, while passive checks observe real request failures; the two answer different questions. /readyz returning 200 can indicate that this instance is ready to receive new work, but should not unconditionally diffuse transient jitter of all shared dependencies into the simultaneous removal of the entire instance pool.

Detection period, failure threshold, and probe timeout jointly determine discovery latency. Probing every 10 seconds and removing only after two consecutive failures does not mean the instance is cut off exactly 20 seconds after a failure occurs; it is also affected by occurrence phase, probe duration, and scheduling. Upon recovery, traffic can be gradually increased, and caches or connections can be warmed up.

Decommissioning first stops accepting new work, then waits for already accepted requests to finish, and finally closes residual connections at the deadline. HTTP/2 and HTTP/3 can use the respective GOAWAY, while long connections like WebSockets/SSE need explicit maximum drain time and client reconnection behavior. Simply removing the instance from the discovery list will not make all existing connections disappear immediately.

Blue-green or weight-based canary deployments also need to consider session affinity, request cost, and sample distribution. 1% of requests do not necessarily cover 1% of users, and may miss large tenants or specific paths; metrics should be observed by version, route, and error type simultaneously.

Observability: Mapping a Failure to a Specific Attempt

Access logs should at least be able to associate request identifiers, routes, selected upstream, attempt count, status, and key latencies. Breaking down the client's total latency into queuing, connection setup, waiting for headers, and response transmission can distinguish between "upstream didn't respond," "client read slowly," and "proxy pool waiting."

Caching, rate limiting, and authorization should also indicate which layer executed them; if a cache hit bypasses the backend, the backend log having no request is normal. Do not mix user tokens, original certificate private keys, or sensitive content into debug logs; diagnostics should rely on structured results and necessary identifiers.

Continue Reading

• mTLS: Identity boundaries after proxy termination.
• HTTP/2, HTTP/3: Multiplexing, exit, and flow control.
• REST and API Design: Retries and business idempotency.
• WebSocket and SSE: Long connections and streaming delivery.