Prompt Evaluation Metrics: Measuring What Matters

You cannot improve a prompt you cannot measure. Prompt evaluation is the discipline of scoring your prompt on real inputs and using the numbers to decide what to change. The metrics that matter fall into five buckets: accuracy, faithfulness, format compliance, and the two everyone forgets — cost and latency.

Accuracy: is the answer right

Accuracy is how often the model returns the correct answer. It needs ground truth: a set of test inputs with known-correct outputs. Measure exact match for structured answers, and semantic similarity or an LLM judge for open-ended ones. Track accuracy against your test set after every prompt change — it is the guardrail that stops sounds better from becoming works worse.

Faithfulness: is the answer grounded

Faithfulness is whether the answer is supported by the provided context — critical for RAG and document work where the model should answer from sources, not from memory. The classic check is an NLI-style evaluation: does the answer contradict the source or lack support in it? Hallucination is not just an accuracy problem; a confident wrong answer can look accurate and still be unfaithful.

Format compliance: does it parse

• Valid JSON or output matching the declared schema.
• All required fields present with the right types.
• No extra prose wrapped around a structured output.
• Enum values drawn from the allowed set.
• Items correctly ordered when order matters.

Cost and latency: the forgotten metrics

A prompt that scores two points better but costs three times more and adds a second of latency is a bad trade at volume. Measure tokens per request — input, output, and total — and time-to-first-token on your real infrastructure. Normalize quality per dollar and per millisecond, and let that ratio, not raw accuracy, pick the winner between two similar prompts.

Building a lightweight eval harness

1. Collect 30-50 real requests with ground truth where it exists.
2. Run them against the current prompt and the candidate prompt.
3. Score accuracy, faithfulness, and format automatically.
4. Log tokens and latency for every run.
5. Compare on a scorecard and pick the prompt that wins the ratio, not just accuracy.
6. Re-run the suite on every prompt edit and every model upgrade.

FAQ