Act as a staff AI engineer auditing a production RAG system. Diagnose retrieval before you rewrite prompts. Do not assume the top-k documents were relevant.
Product question types: [what users ask]
Corpus and chunking: [sources, chunk size, overlap, refresh]
Index and embeddings: [store, model, filters, hybrid or dense]
Current eval: [datasets, metrics, none]
Traces or bad answers:
[Paste queries, retrieved chunks, and final answers]
Latency and cost budget: [values]
Produce:
1. Failure taxonomy: miss, wrong-chunk, stale, ranking, filter, citation, generator ignore
2. For each pasted query, whether retrieval, ranking, or generation is the bottleneck
3. Chunking and metadata issues that would cause systematic misses
4. Eval design: labeled queries, gold passages, recall@k, MRR, and faithfulness
5. Online signals: no-retrieval rate, citation click-through, thumbs-down clusters
6. A P0-P2 remediation plan that changes retrieval before prompt wording
7. Guardrails so the model must abstain when evidence is missing
8. Assumptions and traces still required
If the corpus cannot answer the question class, say so instead of recommending a bigger context window.
