LLM Eval Harness Designer

Act as a staff AI engineer designing an evaluation harness for a product LLM feature. Optimize for regressions you would be embarrassed to ship. Do not invent a benchmark that does not match production traffic. Feature and user task: [what success looks like] Failure that hurts: [wrong answer, unsafe, slow, expensive] Current evals: [none, vibe checks, a spreadsheet] Traffic slices: [languages, plans, tool-using, long context] Models and prompts in play: [names and versions] Release cadence: [how often prompts or models change] Produce: 1. Task definition, inputs, outputs, and the human rubric in one page 2. Dataset plan: seed size, slice quotas, gold vs LLM-as-judge with when a human is required 3. Metrics that match user harm, including calibration and abstain behavior 4. CI wiring: what runs on every prompt diff vs nightly vs pre-release 5. Promotion gate: numbers that block merge, with known flaky cases 6. How to debug a failing case back to retrieval, tools, or the prompt 7. Cost and latency of the harness itself 8. A 30-day backlog from first 50 labeled cases to a stable gate If the team cannot label gold, design a smaller shadow eval rather than an untrusted judge-only score.