Evidence for LLM evaluation sets
Your first question: Does a generated evaluation set cover the cases I care about, without near-duplicates, and are its labels right?
Short answer, from one 200-row run per request: planning a taxonomy of the domain, with a few scenario variants per leaf, reached more leaves of an independent taxonomy on 3 of 4 requests and spread texts more evenly across its leaves on 4 of 4, with near-duplicates at or near zero. A difficulty dial made “hard” rows look harder but broke their labels, so it isn't offered. Taxonomy coverage is now coverage mode: opt-in from the request panel, and you confirm the taxonomy before it's applied.
Coverage
Leaves of a held-out taxonomy reached by the written text
A blind judge assigns each text to a leaf of a taxonomy that was never used to generate anything. Share of leaves with at least one text. Higher is better.
| Request | Coverage mode off | Coverage mode on |
|---|---|---|
| LLM eval set (bank support) | 84.4% | 90.6% |
| Support tickets | 76.9% | 92.3% |
| E-commerce reviews | 61.1% | 94.4% |
| Patient visit notes | 86.2% | 86.2% |
| Request | Leaves, today | Leaves, prototype | ≥3 texts, today → prototype | Effective leaves, today → prototype |
|---|---|---|---|---|
| LLM eval set (bank support) | 27 of 32 | 29 of 32 | 22 → 25 | 20.4 → 23.5 |
| Support tickets | 20 of 26 | 24 of 26 | 15 → 22 | 12.2 → 19.8 |
| E-commerce reviews | 11 of 18 | 17 of 18 | 8 → 14 | 4.8 → 10.6 |
| Patient visit notes | 25 of 29 | 25 of 29 | 16 → 19 | 15.2 → 19.8 |
Coverage costs realism: the leaf floor moves the top-level mix (e-commerce “delivered” fell from 147 to 118 of 200 rows). That is right for an evaluation set that asked for coverage and wrong for data meant to look like production. Coverage is likely but not guaranteed: leaves are weights, not quotas.
Label agreement
| Request | In the assigned leaf |
|---|---|
| LLM eval set (bank support) | 96% |
| Support tickets | 97% |
| E-commerce reviews | 85% |
| Patient visit notes | 95% |
Near-duplicates
| Request, arm | Near-duplicates |
|---|---|
| LLM eval set, baseline | 1% |
| LLM eval set, taxonomy, no variants | 4.1% |
| LLM eval set, taxonomy + variants | 0% |
| Support tickets, baseline | 2.5% |
| Support tickets, taxonomy + variants | 0% |
| E-commerce reviews, baseline | 0% |
| E-commerce reviews, taxonomy + variants | 2% |
| Patient visit notes, baseline | 0% |
| Patient visit notes, taxonomy + variants | 0% |
A taxonomy without variants made texts within a leaf more alike (near-duplicates on the eval set rose to 4.1%); variants brought them back to zero. Variants ship with the taxonomy or not at all.
- How it was measured
- Four templates at 200 rows per arm; text written through the production path (Claude Opus with Ground's quality checks and personas). Judges are blind to the arm and the assigned labels: Haiku assigns leaves, Sonnet built the held-out taxonomy and checks labels.
- Uncertainty
- One run, 200 rows per arm, no confidence intervals: treat differences of one or two leaves as noise.
- Scored by
- Ground's assessment harness with LLM judges (self-scored). The held-out taxonomy comes from the same prompt on a different model, not from a different method.
- Raw output
- coverage-results.json
- Source
docs/assessments/taxonomy-and-difficulty.md
Measured no-go: a difficulty rubric
Labels still correct, by difficulty level (LLM eval set)
Share of rows whose assigned intent a Sonnet judge accepted as a correct reading of the text. Higher is better.
| Level | naive (level named, one-line hint) | difficulty rubric |
|---|---|---|
| Easy | 98.5% | 98.5% |
| Medium | 96.9% | 98.5% |
| Hard | 100% | 76.5% |
| Arm, level | Judged difficulty | Judge category accuracy | Answer score |
|---|---|---|---|
| naive (level named, one-line hint), easy | 1.93 | 98.5% | 3.43 |
| naive (level named, one-line hint), medium | 2.20 | 93.8% | 3.38 |
| naive (level named, one-line hint), hard | 3.15 | 94.1% | 3.21 |
| difficulty rubric, easy | 1.93 | 97% | 3.46 |
| difficulty rubric, medium | 2.20 | 92.3% | 3.66 |
| difficulty rubric, hard | 3.38 | 50% | 3.72 |
Before it is reconsidered it needs a verify-and-repair step per row, difficulty measured on the target model itself, and length held constant so “hard” doesn't just mean “longer”.
Leakage guard
AI-written text must not give the label away. Prompts forbid stating it, a per-column list of label words rewrites any value that uses them, and each export's text-quality summary compares a text-only classifier for the label with one trained on the row's other columns. Published leakage measurement on a public evaluation set (text-only AUC): pending first independent run.
Next: methods and reproducibility, or defects and edge cases for test data.