Rate Limiting Is the Wrong Control for HTTP/2 Memory

Every edge tier I have shipped carries a requests-per-second dial, and most teams treat it as the throttle of last resort. It measures the wrong unit: a server does not run out of requests, it runs out of resident memory.

The graph starts flat and then bends. Resident memory on one edge worker climbs through an afternoon with no matching rise in traffic, until the kernel picks the process and kills it. Nothing in the access log accounts for the climb. The connections holding that memory were open the whole time — moving bytes, returning no errors, healthy by every signal you instrument. Requests per second never left its normal band, so the dial you would have reached for was never going to move.

A limit is a number and a unit, and the unit is what decides whether the limit defends anything. Requests per second counts arrivals. What a machine runs out of is bytes it is still holding, and HTTP/2 separated those two the day it shipped: one connection carries many concurrent streams, each with its own state, and RFC 9113 sets no ceiling on how many — it only recommends staying at or above 100, so that parallelism is not capped needlessly. nginx documents 128 as its default. The resident cost of a connection is therefore concurrent streams times per-stream allocation, and the count of open connections multiplies all of it again. Arrivals per second appear nowhere in that product.

Per-stream allocation is the term that spoils the product. HPACK's dynamic table belongs to the connection rather than to a request, and RFC 7541 prices an entry as its name, plus its value, plus 32 octets of bookkeeping. A client seeds one entry and then emits indexed references to it: on the wire a reference is a single byte, inside the server it is a full header field with that bookkeeping attached. A cap on decoded header size does not close the gap, because it bounds what one header set expands to, while what you pay for here is the separate allocation standing behind each of several thousand references.

Calif's 2 June disclosure measured the ratio on shipped builds: roughly 70 bytes of allocation per wire byte on nginx 1.29.7, roughly 4,000 on Apache httpd 2.4.67, and roughly 5,700 on Envoy 1.37.2 — the disclosure puts Envoy at about 3,800:1 across streams, and up to 5,700:1 on a single stream once allocator overhead piles on top. Eight kilobytes of headers is an unremarkable request, and it is about eight thousand references — half a megabyte pinned by one stream at nginx's figure, about 47 MB at Envoy's. Take the low figure and nginx's default of 128 streams: one connection holds about 70 MB, and a 32 GB worker is exhausted somewhere under five hundred connections. On Calif's bench, Envoy 1.37.2 reached 32 GB in about ten seconds. Five hundred sockets sit well inside what a single laptop opens, and well below anything a rate limiter is configured to notice.

Allocation alone would drain. The stall is what makes it stick. The client advertises a zero-byte receive window, so the response cannot be written out and its buffers stay resident; then it drips a WINDOW_UPDATE now and again, and every idle timer you own starts its count over. That is why the connection read as healthy: bytes were moving, the far side answered, and the request counter sat near zero because one request was still, technically, in progress. Both halves are ordinary use of the protocol — a slow reader and a client that compresses headers — so the defaults have no grounds on which to refuse them.

Set the budget explicitly at every hop that speaks the protocol: concurrent streams, header table size, maximum connection lifetime, and a timeout that measures progress rather than packets. The load balancer, the reverse proxy, the sidecar and the application server each terminate HTTP/2 on their own, each keeps its own decoder state, and each ships with defaults tuned for parallelism. "We are behind a CDN" does not remove a hop: the CDN defends its own edge, then opens fresh HTTP/2 connections to your origin from ranges you allowlisted and your rate limiter will never throttle. The hop that usually has no dial at all is the innermost one, where the state is largest and the traffic looks most trusted.

None of this is particular to HTTP/2; it follows from multiplexing. Once many requests share one connection, the connection is what holds the state, and any limit still written per request is counting in a unit the resource does not use. HTTP/3 carries the same shape into QUIC: windows the far side can hold at zero, and a QPACK dynamic table with the same reference economics. It ships with defaults chosen, reasonably enough, for parallelism. A limit written in requests is as blind against QUIC as it is against HTTP/2, and it is blind at every hop that terminates the connection, not only at the one with a dashboard in front of it. The unit is fixed by what the resource is made of, and the memory is held per connection whether or not anything counts it that way.