RAG Evaluation Basics: The Metrics That Matter

Most RAG systems get measured by a single number, and that number hides the two failures that actually matter. A pipeline can retrieve the wrong documents and still sound confident, or retrieve the right ones and still answer wrong. If your dashboard shows one score, you can't tell those apart, and you can't fix what you can't separate.

Two questions, not one

Every RAG answer is really two steps: find the evidence, then use it. Grade them apart, because a retrieval bug and a generation bug need different fixes.

• Did retrieval surface the chunks that contain the answer?
• Did the answer follow from those chunks, and only those chunks?

Collapse the two into one score and a retriever that missed the evidence looks identical to a model that ignored it. Keep them separate and the failure points to its own fix.

Retrieval quality

Before the model writes a word, the retriever has already decided whether a correct answer is even possible. Three measurements tell you how well it did:

Metric	What it catches
Recall@k	Did the right chunk make the shortlist at all?
Precision@k	How much noise came along with it?
First-hit rank	How far down the list the useful chunk sits.

Answer quality

Once the evidence is on the table, grade what the model did with it: faithfulness (is every claim grounded in the retrieved context), answer relevance (does it address the question that was asked), and completeness (did it use the evidence that was there). These are about the generation step, and they only make sense once retrieval is measured separately.

You can't fix what you can't separate. Grade retrieval and generation apart, or you are guessing at which half broke.

The gap between them

Here is the number most teams never track: an answer can be perfectly faithful to its context and still wrong, because retrieval never surfaced the evidence. Measure the gap between what the retriever made possible and what the model actually got right. When the gap is wide, fix retrieval. When it is narrow but accuracy is low, fix generation. The gap tells you where to spend the week.

What to measure, and when

In development, run recall@k and faithfulness against a labeled set so a change shows up as a number, not a vibe. In production, sample real queries into an environment with an answer key and re-run the same set on every deploy, so a regression surfaces before a customer finds it.