LLM Eval Dataset Builder

Customizable · 6 variables

Build a labeled eval dataset from production traces and slice quotas so prompt changes are judged on real failures, not toy questions.

Prompt

Act as a staff AI engineer building a first production eval set. Prefer ugly real traffic over tidy synthetic questions. Do not leak PII into the dataset. Feature: [task] Production traces available: [yes/no, volume] Failure reports: [Paste clusters, tickets, or thumbs-down notes] Slices that must not be skipped: [language, plan, length, tools] PII rules: [what must be redacted] Target size: [for example 100 cases] Produce: 1. Schema: input, context, gold, slice tags, severity 2. Sampling plan from traces plus synthetic fillers only where traces are missing 3. Labeling guide a contractor could follow, including abstain 4. Duplicate, leakage, and train/eval contamination checks 5. Slice quotas and the minimum per high-risk slice 6. Redaction procedure 7. A starter list of 15 case outlines drawn from the pasted failures 8. How the set will be versioned when the product changes If traces are unavailable, say the eval will be weak and list the logging to add before claiming coverage.

LLM

Customize this prompt

6 variables

Progress: 0 of 6 variables filled0%

1Fill Variables

task*

yes/no, volume*

Paste clusters, tickets, or thumbs-down notes*

This field is required

language, plan, length, tools*

what must be redacted*

for example 100 cases*

2Generated Prompt

Act as a staff AI engineer building a first production eval set. Prefer ugly real traffic over tidy synthetic questions. Do not leak PII into the dataset. Feature: [task] Production traces available: [yes/no, volume] Failure reports: [Paste clusters, tickets, or thumbs-down notes] Slices that must not be skipped: [language, plan, length, tools] PII rules: [what must be redacted] Target size: [for example 100 cases] Produce: 1. Schema: input, context, gold, slice tags, severity 2. Sampling plan from traces plus synthetic fillers only where traces are missing 3. Labeling guide a contractor could follow, including abstain 4. Duplicate, leakage, and train/eval contamination checks 5. Slice quotas and the minimum per high-risk slice 6. Redaction procedure 7. A starter list of 15 case outlines drawn from the pasted failures 8. How the set will be versioned when the product changes If traces are unavailable, say the eval will be weak and list the logging to add before claiming coverage.

LLM Eval Dataset Builder

Customize this prompt

1Fill Variables

2Generated Prompt

Comments

Details