RAG evaluation: how to know your AI answers are actually correct

Your retrieval-augmented answer system demos beautifully. It pulls from your documents, it writes fluent answers, everyone in the room nods. Then it ships, and a customer asks a question it answers confidently and wrongly — citing a policy that was retired last year, or inventing a detail that was in none of the retrieved documents. The fluency was never evidence of correctness. Knowing whether your answers are actually right is a separate discipline, and skipping it is how good demos become support tickets.

This is about that discipline: how to measure whether a RAG system is telling the truth, and how to keep measuring it as the system changes.

Why RAG is confidently wrong

A language model produces text that is plausible. Plausible and correct usually coincide, which is exactly what makes the gap dangerous — the failures look identical to the successes. Retrieval is supposed to anchor the model to your real documents, but it introduces two independent places to fail.

Retrieval can miss. If the right passage never makes it into the context, the model answers from its training or its imagination. The most common cause of a wrong RAG answer is not a bad model; it is that the relevant chunk was not retrieved in the first place.

Generation can drift. Even when the right passage is retrieved, the model can ignore it, blend it with prior knowledge, or overstate what it says. The answer cites a real document but claims something the document does not.

Because these two failures are separate, you have to measure them separately. “The answer was wrong” is not actionable. “The answer was wrong because retrieval returned nothing relevant” and “the answer was wrong because the model contradicted a correctly retrieved passage” point at completely different fixes.

Build an evaluation set first

You cannot improve what you cannot measure, and you cannot measure without a fixed set of questions with known-good answers. This eval set is the single most valuable artifact in a serious RAG project, and it is worth building by hand.

• Draw questions from reality. Use the questions your users actually ask, or would ask. If the system is live, mine the logs. If it is not, sit with the people who field these questions today and write down the real ones.
• Cover the awkward cases on purpose. Include questions the documents answer clearly, questions they answer only partially, and questions the documents do not answer at all. That last category matters most, because the correct behaviour there is to decline — to say the information is not available rather than invent it. A system that never says “I do not know” is a system that will confidently lie.
• Record the ground truth. For each question, note the correct answer and, critically, which document passages it should come from. Knowing the expected source is what lets you grade retrieval and generation independently.
• Keep it a realistic size. A few dozen well-chosen questions beats hundreds of careless ones. Fifty to a couple hundred, spanning the categories above, is plenty to start and small enough to maintain.

This set does not need to be large. It needs to be honest, representative, and stable, so that a score today means the same thing as a score next month.

The metrics that matter

Grade retrieval and generation as two separate stages.

Retrieval quality. Given a question, did the system fetch the passages that actually contain the answer? Because you recorded the expected source passages in your eval set, you can check whether they appear in what was retrieved. Two practical measures: whether the correct passage was retrieved at all within the top results, and how highly it ranked. A correct passage sitting at position twenty when you only feed the model the top five is a retrieval failure, even though the passage technically “matched.” Fix these upstream — chunking, embeddings, how many results you retrieve — before touching prompts.

Faithfulness, or groundedness. Given the retrieved passages, does the answer stay within what they actually support? A faithful answer makes no claim that is not backed by the provided context. This is the metric that catches confident invention. You check it by asking, for each claim in the answer, whether the retrieved passages support it. An answer can be faithful and still unhelpful, but an unfaithful answer is a liability regardless of how good it sounds.

Answer correctness. Does the final answer match the ground truth you recorded? This is the end-to-end measure users feel. It can fail because retrieval missed, because generation drifted, or because the question had no good answer and the system pretended otherwise — which is why you keep the first two metrics alongside it to tell you why.

A subtlety worth naming: an answer can be perfectly faithful to the retrieved context and still wrong, because the retrieved context was the wrong context. Faithfulness measures the generator; retrieval quality measures the retriever. You need both, or you will keep fixing the wrong stage.

Using a model as a judge, carefully

Grading faithfulness and correctness by hand across a whole eval set every time you make a change does not scale, so teams increasingly use a language model to grade the outputs — a “judge” model scoring whether an answer is grounded in its context. This works, with caveats. The judge has its own biases, it can be fooled by a confident tone, and it should be checked against human judgement rather than trusted blindly. Use it to grade at volume, and use humans to audit the judge. It is a force multiplier, not a replacement for looking at the output yourself.

Human spot-checks stay in the loop

No metric replaces a knowledgeable person reading a sample of real answers. Automated scores tell you the shape of the problem; a human reading twenty answers a week catches the specific, embarrassing failure that no metric was watching for. Put someone who knows the subject matter in front of a rotating sample of production answers and give them an easy way to flag bad ones. Those flags become new entries in your eval set, which is how the set stays honest as reality shifts.

Regression testing, so fixes stay fixed

Every change to a RAG system — a new model version, a different chunking strategy, a tweaked prompt, an updated document set — can improve some answers and quietly break others. Without a regression test, you will not know until a user tells you.

Run your eval set on every meaningful change and compare the scores to the last run. When a change raises overall correctness but drops faithfulness, you want to see that trade before you ship, not after. This is ordinary software engineering discipline applied to a probabilistic system: a fixed test suite, run on every change, gates what goes to production. It is exactly the kind of observability we build into the RAG systems we develop, and it is the natural companion to the debuggable, framework-free pipeline we described in building production RAG on Cloudflare without LangChain.

One more honest point. Sometimes the right result of an evaluation is deciding the task should not use a language model at all — when the accuracy you can reach is not good enough for what the answer is used for. Knowing that boundary is its own skill, and we wrote about it separately in when not to use an LLM.

Send us two paragraphs about what your AI system answers and how wrong answers would hurt, and we will reply in writing within one business day.