LLM Latency SLAs: Architecting for Guaranteed Response Times

As soon as an AI feature becomes customer-facing, "fast enough" stops being a feeling and starts being a number you must hit. An LLM latency SLA is that number written down: a time-to-first-token budget, a tokens-per-second floor, and a target for full completion. The hard part is that LLM providers are shared infrastructure — latency drifts with their load, not yours.

You cannot control the provider's queue, but you can architect your own layer so that variance does not reach your users. This guide walks through the metrics to contract on, the latency budget, and the patterns — streaming, caching, autoscaling, and fallback tiers — that turn an LLM latency SLA from a hope into a guarantee.

The three metrics that actually matter

1. Time to first token (TTFT): the wait between request and the first visible character. This is the metric users feel — sub-second feels instant, two seconds feels broken.
2. Tokens per second (TPS): how fast the response streams after the first token. Slow TPS makes long answers feel like a typewriter with a stutter.
3. Time to last token (TTLT): full completion time. Matters when downstream work — parsing JSON, triggering actions — only starts after the stream ends.

Build a latency budget, not a latency hope

Add up every hop between the user and the model: client network, your API edge, auth, gateway routing, prompt assembly, model queue, generation, streaming path, and UI rendering. Most teams discover their "3 second" response is really 300ms of their own code plus 2.7s of model time — and that model time is the only part a gateway can influence directly.

Streaming is non-negotiable

A non-streaming endpoint hides TTFT until the whole answer is ready, so a slow model becomes a blank spinner. With streaming, the first token arrives in a fraction of the total time and users start reading immediately. Your gateway must pass server-sent events through without buffering the response — the most common failure is a proxy that collects the entire stream before forwarding it, which silently destroys the entire benefit.

Caching layers for repeated work

• Prompt caching: providers reuse stable prefixes at lower cost and faster processing, so cached requests skip the longest part of prompt prefill.
• Semantic response cache: store the response for queries whose embedding is within a similarity threshold of a previous one — excellent for FAQ-style and support traffic.
• Template cache: pre-render and cache system prompts, tool schemas, and RAG snippets so assembly adds no serialization overhead.

Autoscaling: traffic, not tokens

LLM APIs throttle by requests-per-minute and tokens-per-minute, so a sudden spike in interactive traffic can turn into a wall of 429s — which then becomes latency you can see from space. Autoscaling your gateway fleet on request volume is only half the story; the other half is pre-warming token quotas and, where possible, reserving concurrency for interactive traffic while pushing batch work to lower-priority pools.

Fallback tiers for the SLA floor

1. Tier 1 (primary): the frontier model you benchmarked, for the best quality.
2. Tier 2 (fallback): a comparable model on a second provider that absorbs provider outages and rate spikes.
3. Tier 3 (degraded): a small, fast model or a cached answer that keeps the feature responsive even when everything else fails.

Measure the percentile, not the average

Average latency hides the tail that destroys trust. Track p50, p95, and p99 for TTFT and TPS separately per model and per feature, and alert when p95 drifts past your SLA. Publish internal latency SLAs per feature — "p95 TTFT < 900ms, streaming" — and treat violations as incidents, not as data points to sigh at.

FAQ