The measurement has to exist before the model does
A team that cannot tell improvement from variance will ship the version that happened to demo well.
What was observed
Ask a team midway through an AI project how they know the current version is better than last month’s, and the answer is usually a demonstration. Someone opens a terminal, types an input they have typed before, and the output is good. Everyone agrees it has improved.
That is not a measurement. It is a recollection compared against a demonstration, and both halves are selected by the person making the argument. The inputs that were typed are the inputs that worked. The ones that failed last week are not in the comparison, because nobody wrote them down.
The consequence follows directly, and it is rarely recognised as a consequence. With no instrument that could settle the question, every opinion about whether the system is working becomes equally valid. So the argument gets settled by seniority instead, or by whoever won the knowledge contest most recently.
What it means
An evaluation harness is not quality assurance arriving late. It is the thing that makes the rest of the work decidable. Without it a team cannot answer the only question that matters between two versions: is this better, or is this different.
The comparison with instrumented systems elsewhere holds exactly. In a production trading system nobody argues about whether a change improved execution, because the system records what it did and the record is not selected by the person proposing the change. That discipline is neither unusual nor expensive. It is simply assumed, because the cost of being wrong is immediate and arrives in a statement.
AI work has the same need and rarely inherits the same habit, because the outputs are language and language is easy to read charitably. A number that is wrong looks wrong. A paragraph that is wrong looks like a paragraph.
What to do next
Build the harness before the second version of the model, not after the fifth. It needs three things, and none of them is sophisticated: a fixed set of inputs that includes the ones that have failed, an expected answer or written criteria that somebody other than the author can apply, and a record that persists between runs.
Then adopt one rule. A change ships when the harness says it is better, not when a demonstration says it is. The first time that rule blocks a change everyone liked is the moment the harness starts earning its cost.