Skip to main content
The judge is in beta. The concept below is stable; the interface for reporting a failure yourself is still being finalised, so talk to us before building against it.
A task’s average score tells you how it is doing. It does not tell you what to fix. For that you need the calls that failed, grouped by how they failed. Protégé tracks failures per task. That is the point of the label: a failure is only meaningful relative to a definition of correct, and the definition lives on the task.

​
Where a failure comes from

The eval contract

A scored example the route got wrong. This is the strongest signal, because the correct answer is known.

The judge

A model scoring production outputs against the task’s definition of correct, where no reference answer exists. In beta.

Your application

A call your own code or your users rejected: a retry, an edit, an escalation to a human, a thumbs-down.

Hard failures

Malformed output, a schema that did not parse, a tool call that could not be executed. No judgement needed, these are simply wrong.

​
The judge

Most production calls have no reference answer. Nobody wrote down what the correct reply to a specific customer message was, so a score cannot be computed by comparison. The judge fills that gap: a separate model that reads the task’s definition of correct, the input and the output, and decides whether the output satisfies it. It runs over traces out of band, so it does not add latency to your calls, and it never gates a production response.
A judge is a model, so it is wrong sometimes. Treat judge scores as a way to find candidates worth looking at, not as ground truth. Anything that drives a route change still goes through the eval contract, which is human-authored.
This is why it is in beta. We publish what a judge agrees with humans on before we ask anyone to trust it, in the same way the benchmarks publish the runs that lost.

​
Why failures are the training signal

A task that routes down to a cheaper model and stays correct is finished. A task that cannot route down is the interesting one, and the question is always the same: what specifically does the cheap model get wrong that the expensive one gets right? Failures answer it. Clustered failures are a repair map:
  • Clustered by input shape. One document layout, one language, one edge case. Often fixable with output constraints or a prompt change, no training run.
  • Clustered by output mode. Valid content in an invalid shape. Usually the cheapest class of failure to fix.
  • Scattered with no pattern. The model does not have the capability. This is the case where a specialist model trained on the task’s own traces is the answer.
That last case is what a training run is for, and the failure set is what it trains against. A model trained on a task’s successes learns to repeat what already worked. Trained against its failures, it learns the part it was getting wrong.

​
What this looks like in practice

Failures accumulate per task alongside the trace corpus. When a task has enough of them to be worth acting on, they come back as a grouped set with the pattern named, rather than as a list of individual bad calls. Nothing here changes your integration. Keep sending labelled completions.

How routing uses all this

Why a route is never promoted on price alone.