Most retrieval-augmented generation projects we are asked to rescue have the same symptom: it demoed beautifully and it is wrong often enough in production that nobody trusts it. In almost every case the generation step is fine. The retrieval step is returning plausible-looking chunks that do not contain the answer, and the model is doing what models do - writing a fluent paragraph anyway.
You cannot fix what you do not measure, and vibes-based evaluation stops scaling at about twenty test questions. Here is the harness we build on week one of any AI & cloud solutions engagement.
Separate retrieval metrics from generation metrics
Grading the final answer tells you that something is broken. It does not tell you what. Score the two stages independently and the debugging loop collapses from days to minutes.
Retrieval: recall@k is the number that matters
Build a golden set where each question is annotated with the document chunks that genuinely contain the answer. Then measure what fraction of questions have at least one correct chunk in the top k results. If recall@10 is below about ninety percent, no amount of prompt engineering will save the system, because the answer is simply not in the context window.
recall@k - did the right chunk make it into the context at all?
Mean reciprocal rank - how far down the list was it?
Context precision - what share of retrieved tokens were actually relevant?
Chunk coverage - for multi-hop questions, did you retrieve every required chunk?
Generation: faithfulness before fluency
Once retrieval is healthy, grade the answer against the retrieved context rather than against the ground truth. The question is not "is this correct" but "is every claim in this answer supported by something the model was given". Unsupported claims are hallucinations even when they happen to be true, because they are not reproducible.
faithfulness = supported_claims / total_claims # per answer, LLM-judged, human-spot-checkedBuild the golden set from real questions
Synthetic questions generated from your own documents produce flattering scores, because they are phrased the way the documents are phrased. Real users ask about things the documentation never quite says. Pull the first two hundred questions from support tickets, sales calls and search logs, and annotate them by hand. It is two days of unglamorous work and it is the highest-leverage two days in the project.
A golden set that your domain experts disagree about is more valuable than one they all accept. The disagreements are exactly where your product will fail.
Run evaluation in CI, not in a notebook
Chunking strategy, embedding model, reranker, prompt and model version all interact. Change one and the others shift. We wire the harness into continuous integration so every pull request that touches the pipeline reports recall@k and faithfulness deltas against the golden set, and we fail the build on regressions beyond a threshold.
Cost and latency are evaluation metrics too
A reranker that adds four points of recall and eight hundred milliseconds of latency is a product decision, not an engineering one. Report quality, p95 latency and cost per query on the same dashboard so the trade-off is visible when it is made rather than discovered after launch.
What good looks like
For an enterprise knowledge assistant we target recall@10 above ninety-two percent, faithfulness above ninety-five percent on the golden set, p95 latency under three seconds, and an explicit abstention path so the system says it does not know instead of guessing. The abstention rate is itself a metric - if it is near zero, the model is not abstaining when it should.
