Evidence for ML engineers

Short answer, from one benchmark run: on 4 public classification tables a model trained only on Ground's rows scored within 0.007 AUC of the same model trained on the real rows, tested on real rows neither had seen. On 2 regression tables it lost 0.058–0.060 R², a larger gap you should check on your own target. With the rare-cases option on, a rare class shows up in every preview and exact counts per combination hold to the row; adding rows where your model fails did not help.

Train on synthetic, test on real

How it was measured

Each public table is shuffled with seed 7 and split 80/20. Every synthesizer trains on the same 80% (at most 30,000 rows); a random forest is trained on its synthetic rows and tested on the real 20%, next to one trained on the real 80%. Details on the benchmark overview.

Uncertainty

One run with fixed seeds, no confidence intervals. Differences of a point or two in Overall, or a few thousandths of AUC, are within noise.

Scored by

Ground's own report (self-scored). Independent scoring is pending first independent run.

Limits

SDV CTGAN ran 50 epochs (default 300) and MOSTLY AI had a 5-minute training cap, so their numbers understate what tuned runs could do.

Raw output

benchmark-results.json · evidence.json

Source

apps/web/lib/benchmark-results.json; docs/benchmarks.md

Rare classes you can see and count

A 25-row preview of card transactions (is_fraud (2.9%)) has no fraud row in 47% of seeds today. The rare-cases option swaps rows so the rare class is always in the preview, and the exported file keeps the real rate once it passes about 1,000 rows. It is behind options that are off by default, with no UI yet.

Searching for rows that meet a combination keeps what the other columns do inside it, where forcing the values breaks it (table above). What this doesn't show: whether a model trained on rebalanced rare-class rows does better on real rare cases. That measurement is pending first independent run.

How it was measured

200 preview seeds and 20 count seeds per dataset, on the card-transactions and support-tickets templates and the loan sample fitted as an upload. TVD against two unconstrained runs with different seeds is the noise floor.

Uncertainty

Shares over 200 seeds; the baseline matches the closed form (1 − 0.971²⁵ = 0.52). Count errors are the largest over 20 seeds.

Scored by

Ground's assessment harness (self-scored).

Raw output

rare-cases.json

Source

docs/assessments/rare-cases.md

Measured no-go: generating rows where your model fails

How it was measured

Same six tables and split as the benchmark. Gradient-boosted trees (or a regression forest) trained on 1,000 or 5,000 real rows; five-fold out-of-fold errors find the worst slice (at least 50 rows); four arms add the same number of rows: none, plain synthetic, failure-sliced synthetic, or repeated real slice rows.

Uncertainty

sd of paired differences over 5 seeds. The test set is the same in every seed, so small test slices (insurance smokers: 56 rows) are less certain than the sd suggests.

Scored by

Ground's assessment harness (self-scored).

Raw output

failures-results.json · failures-results-boost3.json (tripled dose)

Source

docs/assessments/failure-driven-generation.md

Check it on your own task: the notebook

Benchmarks on public tables are not your data. Most exports of an uploaded dataset (single-table files, not zip or related-table exports) come with a ready-to-run notebook: it trains a model on the synthetic rows and tests it on the real rows Ground held back when it learned your data, next to a model trained on the other real rows and one trained on both. It runs in Jupyter, VS Code or Google Colab with pandas and scikit-learn. Its result is the number that matters for your model; Ground doesn't publish it.

Next: what the same synthetic rows reveal about real people, or read synthetic data for machine learning.