Evaluation harness design for customer-specific models
Building the test set, the scoring rules and the disagreement process that lets a customer decide whether a model is good enough — before anyone tunes a prompt.
Needed in the first two weeks of any deployment where accuracy is the argument being made. Provable by: a harness the customer's own analysts can extend without help, a scored baseline on real data, and a written record of where the labels disagreed. Why it is scarce: it looks like data work, so it gets assigned to whoever is free; it is judgment work, and the deployments that slip usually slip here.