Build with evidence.

Prompt testing by hand does not survive production.

Teams ship model and prompt changes weekly. Spot-checking a handful of outputs in a playground tells you almost nothing about what changed, for whom, or whether it is safe to release.

Oloproof replaces that with a measured record: the same cases, the same judges, the same seeds, run on every change.

Versioned datasets, pinned seeds, and declarative judges. Every run reproduces exactly, months later.

Deltas with confidence intervals and thresholds, so a two-point move is never mistaken for a win.

Clustered failures by segment, intent, and tool path, each linked to the exact traces behind it.

A release recommendation with the evidence attached — exportable for review, audit, and customers.

Every interval crosses the dashed zero line. Read off the point estimates alone, the reranker would have shipped.

Define datasets, judges, and thresholds in version control. Review evaluation changes the way you review code.

Programmatic checks, LLM judges, trained classifiers that score text on your own machine, and human labels in one pipeline. Each judge's agreement with people is measured with an interval, and one that has not cleared its bar cannot decide a release.

Fail a pull request when a metric crosses its threshold. The gate reports which cases moved and by how much.

Every score links to inputs, outputs, tool calls, retrieved context, and judge rationale. Nothing is a black box.

Quality is never the only axis. Track spend and tail latency alongside correctness, under the same thresholds.

Share the run, diagnostics, and recommendation without changing the measured decision it contains.

Three steps, then it runs itself.

$ oloproof run
Run run_01M3B0CCPA0JFQQ7HWRKYNWFJC [DECIDED/COMPLETE]
Gate: ALLOW (exit 0)

  Rule         Metric       State  Reasons
  label-floor  exact_label  PASS   lower_bound_meets_minimum

  Metric       Estimate  Interval          N
  exact_label  100.0%    [88.4%, 100.0%]   30 / 30 observed · 0 missing

Cache: execution 0 hit/30 miss; judgment 0 hit/30 miss

$ oloproof run
Run run_01M3B0CXTTVPTBXAFK3WNGFX42 [DECIDED/COMPLETE]
Gate: ALLOW (exit 0)
Cache: execution 30 hit/0 miss; judgment 0 hit/30 miss

Every number traces back to a case.

Runs are immutable and dated. Datasets, prompts, judges, and configuration are versioned together, so any result can be reproduced or challenged.

Read off the point estimates, the reranker delivers +12.7 points of hit@1. Read honestly, this panel cannot tell whether it helps or hurts.