Most RAG teams test before they ship. They run queries, read outputs, confirm the results look reasonable, maybe compute a faithfulness score. What most teams don’t do is systematically evaluate retrieval configuration — testing one variable at a time against a ground-truth dataset to understand which design decisions actually drive quality, cost, and latency. That gap is where the largest optimization opportunities hide.
This paper is written for product managers who own or influence RAG-powered features and want a practical framework for evaluating retrieval quality. The framework is general-purpose; the examples are drawn from a D&D content generation system where this approach improved retrieval precision by 85% and reduced token cost by 65%.
When to invest in RAG evaluation
Not every RAG project needs a formal evaluation framework. For a quick prototype where quality is “good enough” and cost is negligible, a vibe check genuinely suffices. But there are clear signals that you’ve crossed the threshold where systematic evaluation pays for itself: you’re making configuration decisions with cost or latency implications, you’re going to production (you need a regression baseline), you’re A/B testing models or chunk strategies, or your corpus is too large or heterogeneous for spot-checking to cover it. If any of those apply, the setup investment is typically 15–30 hours, and the return compounds across every tuning decision afterward.
The framework in four steps
1. Build the ground-truth test set
Aim for 80–100 questions tied to specific documents in your corpus, distributed across six question types that probe different failure modes: direct lookup, attribute filter, semantic similarity, cross-category, boundary/negative, and generation quality. The mix matters. If you only test direct lookups, you’ll optimize for exact-match retrieval and miss semantic or cross-category failures. If you only test generation quality, you won’t know whether problems originate in retrieval or in the LLM.
Each entry records the question, expected answer, expected source documents, category, question type, and difficulty — stored as structured JSON. This structure pays off immediately: your evaluation harness can score retrieval by comparing retrieved document IDs against expected source documents, no manual review required. This step is the most labor-intensive and also the most valuable. Without it, every metric you compute is measuring against nothing.
2. Choose your metrics — three categories, all three matter
RAG metrics fall into three categories: retrieval metrics (did you find the right context?), generation metrics (did the LLM use it well?), and operational metrics (cost and speed). Retrieval metrics — especially Precision@k — are the most discriminative for tuning decisions. LLM-judge scores like faithfulness and answer relevancy are better suited as regression detectors; they tell you when something is badly broken, but they don’t help distinguish between a good config and a slightly better one. Operational metrics (tokens/query, end-to-end latency) keep you honest about what you’re spending.
A common mistake is to rely solely on LLM-judge scores. Use retrieval metrics for tuning; use judge scores for guardrails.
3. Structure a sequential configuration sweep
The most important principle: change one variable at a time. When you change two variables simultaneously, you cannot attribute the result to either one. If quality goes up, you don’t know which change helped. If it goes down, you don’t know which hurt. You’ve spent the compute without learning anything.
A sequential sweep carries the winner from each phase into the next. The recommended order starts with the cheapest, highest-leverage variables and works toward diminishing returns: metadata filtering → chunk size → embedding model → search method → top-k → reranking. Metadata filtering and chunk size are typically highest-leverage and cheapest to test; embedding model and search method are moderate; top-k and reranking are refinements. Testing in this order means you spend most of your compute on the decisions that matter most.
4. Interpret the results
Completed sweeps often contain surprises. A few patterns from the D&D project that generalize:
- Precision and cost move together. Switching from 1024-token to 256-token chunks improved Precision@k by 78% and cut token usage by 61%. Smaller chunks are more targeted, so the context window contains less noise. Counterintuitive — PMs often assume more context is better — but in structured-data domains, less context is often both cheaper and higher-quality.
- Pre-trained components carry domain assumptions. A cross-encoder reranker trained on web search added 79% latency to the pipeline with zero quality gain. Before adopting any pre-trained component, ask: was this trained on data that looks like mine?
- Metadata filtering has outsized impact on generation quality. Filtering retrieval by content category improved faithfulness and answer relevancy more than any embedding or search-method change. Scoping retrieval eliminates cross-category noise from the context window.
- “Hybrid search didn’t help” is a narrower finding than it sounds. BM25 didn’t help in a corpus where queries are conceptual and vocabulary is homogeneous — but strongly-typed attribute pre-filters would likely have. Evaluation tells you exactly what you tested, not the more general thing it sounds like.
- LLM-judge scores plateau quickly. Faithfulness and answer relevancy were broadly stable across configurations. Valuable as guardrails, not sensitive enough to guide optimization.
Headline result
Applied to the D&D content generation system: retrieval precision rose 85% over baseline, per-query token cost fell 65%, and faithfulness held steady at production-grade levels. The compounding gain came from the discipline, not from any single sophisticated technique.
Who this is for
Product managers who own or influence a RAG-powered feature; tech leads standing up an evaluation practice for the first time; anyone making “should we try X?” decisions (different embedding model, reranker, hybrid search) and wanting a measurement framework stronger than vibes. Prior IR or ML background isn’t required — the paper grounds each metric in a plain-English definition before reaching for formulas.
The minimal checklist
If you want the bare-minimum version: write 80–100 ground-truth questions tied to specific sources, instrument your pipeline to report Precision@k, Recall@k, tokens per query, and end-to-end latency, run a baseline against your current production config, sweep one variable at a time starting with metadata filtering and chunk size, and document findings as a delta table from baseline. The full paper walks through each step in detail with worked examples.
Read the full paper
The PDF covers each step in depth: ground-truth test-set design tips, plain-English definitions of every metric (Precision@k, Recall@k, MRR, NDCG@k, faithfulness, answer relevancy) with formulas, the full phase-by-phase sweep plan with the holding-constant rationale for each variable, and the caveat about dependency chains that bit me midway through the project.