Choosing an LLM: A Decision Framework for Task, Latency, Cost, Quality

Picking an LLM feels like comparing spec sheets, but the right model depends on your task, not the leaderboard. A frontier model wastes money on simple classification, while a small model falls over on complex reasoning. This framework turns model choice into a checklist: task type, quality bar, latency, cost, and compliance — evaluated in an order that prevents expensive mistakes.

Work through these dimensions in order and you will land on a defensible choice — and know exactly when to revisit it.

Step 1: What kind of task is this?

Classify the workload before comparing models. Structured tasks — classification, extraction, summarization of short text — need less reasoning power and often fit a small model. Open-ended generation, multi-step reasoning, and agentic loops demand a frontier model. A task that is 80 percent extraction and 20 percent reasoning can be split: a cheap model handles the extraction, and the frontier model only appears when the hard case shows up.

Step 2: Set the quality bar with your own data

Vendor benchmarks rarely match your domain. Build a small eval set of 50 to 200 real inputs with known-correct outputs and score candidate models on it. Include the edge cases that hurt you: messy input, ambiguous phrasing, adversarial formatting. A model that wins on general benchmarks can still lose on your specific format, vocabulary, or user base.

• Assemble 50 to 200 real inputs with known-good outputs.
• Score candidates on accuracy, not on feel.
• Include the messy inputs that break your current solution.

Step 3: Set the latency budget

Time-to-first-token and throughput matter differently per use case. A chatbot needs fast first tokens; a batch document job needs throughput. Small models and specialized fast variants win on latency; reasoning models are the slowest. Decide your deadline in seconds — interactive under one, near-interactive under five, background anything goes — and filter the field before comparing quality.

Step 4: Compare cost per successful result

Compare cost per completed task, not per token, because a frontier model may finish in one call where a cheap model needs three retries. Account for context size: a model that is cheap per token but needs a huge prompt is not cheap per task. And model prices change monthly — a pricing comparison from January can be financially wrong by August.

• Compare cost per completed task, including retries.
• Factor in prompt size and cache hit rates.
• Recheck pricing quarterly; it changes.

Step 5: Compliance and data handling

If your data is regulated — healthcare, finance, EU users — compliance outranks quality and cost. Confirm where the provider processes data, whether training on your inputs can be opted out, and what contractual protections exist. On-prem or self-hosted open-source models become attractive when data cannot leave your boundary, even at a real quality cost. Get this wrong and no latency or price matters.

Putting it together: the decision matrix

Score each candidate against your thresholds per dimension, then apply this rule: eliminate on compliance first, then on latency, compare cost per task among the survivors, and let quality break ties. Write the choice down with the eval scores and the date, because the answer expires. Revisit whenever a dimension changes — a model release, a pricing update, or a shift in your workload.

FAQ