The LLM Routing Formula: Cost × Latency × Quality, Explained

The LLM routing formula is how you decide which model handles a request: cost, latency, and quality all matter, and the right answer changes per request. This guide breaks the formula into a practical scoring system you can apply today.

The three factors

• Cost: price per million input/output tokens for the model you call.
• Latency: how fast the model responds — matters for chat, agents, and API callers.
• Quality: correctness on your task — judge with evals, not intuition.

Every routing decision is a trade between these three. A flash model is cheap and fast but may fail complex tasks; a frontier model nails hard tasks but costs 10-30x more and responds slower.

The formula

Step 1: Classify your requests

1. Critical: coding reasoning, legal/medical content, production logic — quality first.
2. Standard: summaries, drafts, extraction — balance quality and cost.
3. Cheap: classifications, formatting, simple Q&A — cost and latency first.

Step 2: Map models to tiers

Typical tiers: cheap/fast (flash-class models like GPT-4o-mini or Gemini Flash), mid (balanced), and frontier (GPT-5, Claude Opus-class, Gemini Pro). Route critical requests to frontier, cheap requests to flash, and benchmark mid-tier for everything else.

Step 3: Add fallbacks

Routing is not a single pick — it's a ladder. Try cheap first; escalate to a bigger model when confidence is low or the task needs it. Some gateways do this automatically with threshold rules.

What this saves in practice

Teams that route aggressively cut LLM spend by 50-80% because most requests are simple. The key is measuring quality so you don't route away correctness. Run the same prompt set through each tier once a month and compare.

Tools for routing

LayerFlow gives you side-by-side comparison to build the evidence base for routing, plus budgets so the experiment stays cheap. For production, use its OpenAI-compatible gateway with provider keys you bring yourself.

FAQ