RAG lessons: evaluate retrieval first

The trap

A RAG demo can answer one question well and fail on the next twenty. The generator makes the failure hard to see because the prose still sounds confident.

The order I use now

1. Freeze a query set with relevance labels.
2. Score BM25.
3. Score the dense retriever.
4. Add the reranker and score it again.
5. Only then connect the generator.

The metric

nDCG@10 is the main ranking metric. Recall@10 is the coverage check. If recall@10 is poor, the generator never had a chance.

def recall_at_k(ranked_ids, relevant_ids, k=10):
    top = set(ranked_ids[:k])
    return len(top & set(relevant_ids)) / max(len(relevant_ids), 1)

The failure I look for

The eval query is also in the indexed corpus. The score goes up, the product does not. Split the index and the eval set, and add a check that fails the run when they overlap.

Lesson

Evaluate the retriever before you tune the prompt. A prompt cannot fix a missing chunk.