Skip to main content
An eval contract is a fixed definition of correct for one task, with scored examples behind it. It is the gate every route change passes through. Time: a few hours, mostly spent arguing about edge cases. No GPU cost.
1

Pick the task, not the workflow

One task from your cost split, usually the most expensive one. If you cannot write the rule below in one sentence, the task is still too broad. Go back to Split a workflow.
2

Write the rule

One sentence, in the language of the domain, that a non-engineer who does this job would agree with.The strong versions are checkable by someone who is not on the ML team. That is the bar.
3

Collect examples, including the hard ones

Pull real inputs from the task’s traces. Aim for coverage of the shapes you actually see, not a round number.
A set of easy examples produces a contract everything passes, which gates nothing. Deliberately include the cases that caused a retry, an escalation or a complaint. Those are the ones a cheaper model will fail on.
4

Score the incumbent

Run your current model against the set and record the score with its spread and its cost. This is the number every candidate has to beat, and without it “cheaper” means nothing.

​
What the contract buys you

Once it exists, routing can move the workload, because there is something to gate the move on. Until it exists, it stays on the model you name. It is also yours. The contract and its examples leave with you if you leave, in the same way the weights do.

Next: route a task down

Use the contract to move the task to a cheaper model.