The evaluation harness is the deliverable nobody asks for

No statement of work mentions an evaluation harness. It appears in no screenshot, it demos badly, and it cannot be priced as a line item. It is also the only artifact that lets a customer and an FDE disagree about quality without the conversation turning into opinion — which is what happens in every project where acceptance is a demo and a nod.

Build the ruler before the pitch

The cases come from questions the system will actually be asked, taken from the customer's own history rather than from the team's imagination. Sort them by requirement point, and for each point write three kinds: the normal case, the reverse case, and the boundary case. Then add a deliberately hard group — questions with the model name missing, two requests in one message, a question the material does not answer.

Each row carries five fields: the input, the expected outcome or the expected handling, the evidence that makes it correct, whether it is hard or easy, and who owns it. Without the fifth field, cases rot silently; nobody notices a case that stopped being relevant in month three.

The split that matters is between cases with a determinate answer and cases without. One small-team deliverer describes designing about 85% of knowledge-base questions to have a determinate answer, holding the rest as recorded judgement. That ratio is a design target, not a law; what matters is that the ratio is chosen on purpose.

Determinate covers refusals too

A determinate case does not have to have an answer in the document. "Ask a follow-up" and "transfer to a human" are determinate outcomes, and in most deployments they are the outcomes that decide whether the thing is safe.

This is where the eval set and the metric definition table meet. A metric without an exclusion rule is a number someone will argue about; a case without an expected handling is a case that will be marked pass by whoever runs it last. The stop condition you wrote into the requirements is, in the harness, an ordinary row.

The remainder is a log, not noise

When two reasonable reviewers disagree about a case, the honest record is the disagreement itself: who read it, on what date, against which version of the rubric, and what was decided. What must not happen is averaging it into a percentage. A permission leak does not disappear behind forty good answers, and a wrong price does not behave like a rounding error.

Two rules keep this from becoming bureaucracy. A disagreement that appears twice becomes a rubric line or a defect — never a third unresolved argument. And a case whose failure would cause harm, in access, money or compliance, is never averaged with anything; it is pass or fail on its own.

What a customer can extend

The harness is a deliverable, so it has to be usable by the customer after the engagement ends. That means one file with five parts, none of which requires reading the model's code.

Cases in a plain table the customer can add a row to. A runner that says pass, fail or needs-judgement per case, and never just an accuracy figure. Field checks written as code, for amounts, statuses and identifiers — those are not judgement and should not be scored by one. A diff against the previous run that names the cases that changed, so a regression has a name. And a hold-out flag on part of the set.

A customer who can add a case and get a verdict without the vendor in the room has an acceptance mechanism. A customer who receives a spreadsheet and a promise has a souvenir.

Wire the score, don't display it

Scores on a dashboard are read for a week. The same score wired into the alerting path changes behaviour, because a regression on a held-out case is then an event with an owner, an escalation route and a timestamp — the same machinery as an incident, described in escalation and incident response.

The threshold should be blunt: a held-out case moving from pass to fail blocks the change. Not "reviewed", blocked. Anything softer gets negotiated at the exact moment when nobody wants to slow down.

The failure you will see, and the criterion

The usual failure is a test set built by the vendor from questions the system already answers. It passes before go-live, the percentage is quoted in the readout, and in week two the first real question fails with no baseline to compare against. The customer remembers the demo passing; nobody can say which cases it covered. A second version is subtler and more damaging: a single accuracy percentage presented to the business, which looks rigorous and cannot be decomposed, so the one category that is actually broken stays invisible.

The criterion is short. The harness is finished when a business reviewer who has never seen the code can add a case, run the set, and get a verdict without you in the room — and when every miss from that run has been written into the method library with its trigger, failure mode, signal, avoiding action and boundary condition, so the next delivery starts from a slightly better set than this one did. That last step is what the feedback loop is for; without it the harness is a receipt, not an asset.