Bounded admission: why an inference server buckles under load

An inference server accepts every request by default. Under moderate load that works well: throughput and latency stay stable, and the GPU operates in its efficient range. But once the number of concurrent requests exceeds what the GPU can handle, the behaviour changes abruptly — not linearly, but with a knee.

The reason is simple queueing logic: as long as requests arrive faster than they are processed, the queue grows. And the longer the queue, the longer every single request waits. Just before the capacity limit, latencies do not rise gently but steeply — the server is not yet “full” by the numbers, yet to the user it already feels frozen.

With LLM inference, several effects combine to sharpen the knee:

• Shared memory: every active request occupies GPU memory for its context (KV cache). Too many concurrent requests cause memory pressure — requests get evicted or aborted.
• Batching has limits: modern engines bundle requests for high throughput. Above a certain number, though, the response time per user drops noticeably.
• Timeouts create retries: when a request times out, the client often resends it — extra load at exactly the moment the system is already overloaded.

The result is a feedback loop: overload creates timeouts, timeouts create retries, retries create more overload.

An unprotected server has no defence against this feedback loop. It keeps accepting every request until memory is exhausted or the queue grows so long that practically all requests time out. For the user, the difference between “slow” and “down” then disappears — both feel the same. In distributed systems, an overloaded node can also drag others down when requests are routed to it.

The gateway sits as a gatekeeper in front of the model and limits how many requests enter inference at once. Instead of flooding the server beyond its efficient range, it keeps it right inside it. Concretely:

• Admission control: only as many requests enter inference at once as the hardware can serve efficiently.
• Bounded queue: excess requests wait briefly and predictably — not indefinitely.
• Controlled shedding: if the queue is full too, requests are rejected cleanly and immediately instead of hanging for minutes.
• Backpressure: the “full right now” signal reaches the caller quickly, so it can react sensibly.

Errors are absorbed before the user sees them — and the server stays in the range where it works fastest and most reliably.

Under extreme load, the gateway achieves an 18× higher success rate than the bare server. “Success rate” here means the share of requests that receive a valid response instead of running into a timeout or error. Importantly: this is not because the gateway computes faster — the actual inference is identical. It is because it meters the load so that the requests that do get through are actually answered.

Equally important is what does not change: the per-user token rate stays the same. The operating layer therefore costs no generation performance — it only orders who goes when.

This robustness costs practically no performance. The platform layer adds around 56 milliseconds of additional first-token latency — the pure network path through the gateway. That is a one-off, barely perceptible surcharge on the first token, not ongoing compute. You pay it once per request and get, in return, a system that does not buckle under extreme load.

Both limit requests — but in fundamentally different ways. Rate limiting hands out fixed quotas per client (say “100 requests per minute”), regardless of how busy the system currently is. Admission control follows actual capacity dynamically: when there is headroom it lets requests through, under overload it meters. A pure rate limit protects against individual heavy talkers, but not against collapse when many legitimate users arrive at once. Bounded admission guards exactly against that.

Capacity spikes are the normal case, not the exception: the Monday morning, the quarterly report, the use case that suddenly becomes popular internally. A system that stays reliable exactly then is the difference between “AI in a pilot” and “AI the company relies on”. And because behaviour under load is a property of the operating layer, not of the model, you get it regardless of which model you deploy tomorrow.