LLM Quantization Explained: INT8, FP8, INT4 and Quality Trade-offs

Quantization shrinks a model by storing weights in fewer bits per parameter. A 70B model in FP16 needs about 140 GB of memory; the same model in 4-bit fits in roughly 35 GB. That is the difference between renting a cluster and running on a single GPU — but the savings come with measurable quality loss.

In 2026, INT8 and FP8 quantization are close to lossless in practice, while INT4 powers most on-device and consumer deployments. This guide explains the precision ladder, what the benchmarks actually show, and how to decide where to quantize.

What quantization does to a model

A weight stored as FP16 is a 16-bit floating-point number. Quantization maps ranges of weight values onto a smaller set of discrete levels — 8-bit integers, 4-bit integers, or 8-bit floats. The model runs faster and uses less memory because smaller values transfer and multiply faster, but each rounding introduces error that compounds across layers.

The precision ladder: FP16 to INT4

• FP16/BF16: the original weights; no compression, highest memory cost.
• INT8: 4x smaller weights; quality loss often under 1-2% on standard evals.
• FP8: similar footprint to INT8 but smoother rounding for transformer layers.
• INT4: 4x smaller than INT8; quality loss grows, but the memory savings are enormous.
• INT2 and ternary: mostly research; visible quality collapse outside narrow tasks.

What quality trade-offs look like in practice

Benchmark deltas are the average story; your workload is the real story. INT4 models often keep most of their reasoning ability but show degraded code generation, more factual slips in long generations, and weaker low-resource-language performance. Math and multi-step tasks degrade first. A quantized model that passes an evaluation suite can still fail specific production prompts.

When quantization is worth it

1. Running open-weights models on your own GPUs or edge devices.
2. Latency-sensitive serving where smaller weights cut memory bandwidth stalls.
3. Batch inference at scale where per-request cost matters.
4. On-device and mobile use where RAM is fixed and small.
5. Prototyping many models on one machine before committing to a provider.

When to skip quantization

• You use hosted APIs — the provider already handles precision internally.
• Quality is non-negotiable and your evals show real INT4 regressions.
• Your workload is mathematically exact (calculations, structured parsing).
• You rarely serve the model, so the memory savings never pay back.

The cost calculus

Quantization does not change token prices at most providers, but it changes your infrastructure bill: a 4-bit model lets you serve more concurrent requests per GPU and cuts the number of machines you rent. The real comparison is quality per dollar per request, not bits per weight.

FAQ