RAG with retrieval evaluation

Result: the pipeline is built in stages, and retrieval evaluation is the gate before generation. The production score is add nDCG@10 from the MIRACL run.

Problem

A RAG demo can look good because the answer is fluent, even when the retriever returned the wrong chunks. I wanted the opposite order: prove retrieval first, then let the generator answer with citations.

Data

The benchmark is MIRACL. The corpus size is add indexed document and chunk counts. The evaluation query set and relevance labels are add source, size, and freeze date. The eval set is versioned separately from the index so a reindex cannot silently change the score.

Approach and why

The build has four stages:

A lexical baseline. BM25 gives a number to beat and makes the first run cheap.
A dense retriever over chunked documents. Chunk size, overlap, embedding model, and top k are logged per run.
A reranker on the top candidates. The eval reports retriever only and retriever plus reranker separately so the reranker cannot hide a weak first stage.
A cited answer. The generator receives only the selected chunks and must return chunk identifiers with the answer.

I chose this order because retrieval is measurable without trusting generated prose. The generator is the last variable, not the first.

Results

Stage	nDCG@10	Recall@10	Notes
BM25 baseline	add score	add score	Lexical baseline
Dense retriever	add score	add score	add model and top k
Dense plus reranker	add score	add score	add reranker

Evaluation

Leakage: the index and the eval queries are separate. A check asserts that no eval query text appears in an indexed chunk.
Baseline: BM25 and a no retrieval condition.
Validation: a frozen query set with relevance labels. The split and label source are documented.
Metric: nDCG@10 for ranking quality, recall@10 for coverage. Both are reported.
Reproducibility: pinned dependencies, logged chunk settings, index version, and a fixed eval set.

A reviewer should challenge any score without the baseline, the corpus size, and the chunk settings. Those are tracked in MISSING_FACTS.md.

Limitations

The benchmark does not prove performance on a private corpus. Chunking is sensitive to document structure. The reranker adds latency and cost that are not yet measured. The current stage is add current stage and next gate.

Next steps

Add the eval numbers, run a reranker ablation, measure retrieval latency and cost, and publish the failure cases where the right chunk was ranked below the cutoff.