Errors

The distinction this page exists for: a run that failed and a run that errored are different things, and presenting one as the other tells a developer their change is bad when the truth is that the harness fell over.

A failure is a result. An error is not.

FAIL means a gate was evaluated and the evidence was below the threshold. That is a quality result about the change.

RUN_ERROR means the run did not complete. No gate was evaluated and nothing about the change has been measured. It is not a decision state, and neither are PARTIAL or CANCELLED — they describe what happened to a run, not what was decided about a system.

What an incomplete run reports

A gate over an incomplete run exits 5, not 1. The evidence is missing rather than bad, and the two need different responses: a failing gate asks you to look at your change, an incomplete one asks you to look at your infrastructure.

Cases that could not be measured

A case that errored is bounded, not dropped. Every metric records four counts — total, eligible, observed and missing — and the interval accounts for the cases it could not see.

This matters more than it sounds. A metric that counts an ungraded case as a zero charges a system's own outages to the model as quality failures, and a denominator that quietly shrinks when a system starts failing is how a failing system reports a rising score. Both were observed on a live assistant, where 13.8% of calls returned HTTP 500.

Transient failures

A system can declare that a failure was transient, and a transient failure is retried rather than recorded as a quality result. A provider timeout is not evidence about a prompt.

The declaration is explicit: a callable system raises TransientError with retryable=True. retryable defaults to False, and a TransientError without it is recorded on the case like any other exception, so the case is missing rather than retried. An HTTP system and a judge provider need no declaration: a timeout, a dropped connection, a 429 or a 5xx is retried for them. Either way a call is tried at most four times, with backoff, and a case that still fails is missing. Progress and concurrency shows both.

Configuration errors

Exit code 2 means the invocation or the configuration was wrong and nothing ran. A quota refusal is one of these: it is a configuration error with a named reason, refused before the first case executes, and it is never a decision state. A commercial fact must not arrive wearing a quality verdict.

Diagnosing what did fail

oloproof diagnose RUN_ID --intervention gold-context --criterion answer_correct

The intervention is what it re-runs the failed cases under — gold-context, top-k or reranker — and the criterion says which evaluator's failures to take. A diagnosis reports failure factors with counts, the scenario ids behind them and the assumptions it made. It names a factor, not a cause: the engine does not assert that one thing produced another, and neither does any screen that renders it.

Where to go next

• Core concepts covers decision states and denominators.
• Gating CI lists every exit code.
• RAG evaluation walks through a diagnosis end to end.