The ERP as the answer key
Nobody had time to label data. The filed paperwork already paired every purchase order with the order the ERP booked. Whole-order accuracy went from 18% to 64% in nine days.
The platform reads dealers' purchase orders with two readers: a rules engine that parses the PDF's text, and a language model. When they disagree on a quantity, arithmetic decides. Quantity times unit price must equal the line total, and the side that satisfies it wins.
In early October I wanted to know how often the result was completely right. Not one field: the whole order.
The answer key was already filed
Nobody had time to label training data. But after every invoice the team files one merged PDF: the dealer's purchase order together with the ERP's own printouts of the order. That is a labelled pair, made by people doing their normal jobs.
A nightly scorer pairs each purchase order with the order the ERP actually booked and scores eight fields. A replay tool re-scores every historical order in about a minute. A held-out set is kept that nobody opens and nobody tunes against.
The rules of a round
- Every fix lands first as a failing test.
- No field that was right on any order may turn wrong, unless a named decision says it should.
- A person's later correction never becomes a rule by itself. The engine is code. It does not learn on its own, and that is on purpose.
Five rounds in nine days
| Training set | Held-out set | |
|---|---|---|
| Start | 18% | 17% |
| After five rounds, 7–15 Oct | 64.4% | 61.5% |
The held-out set moved with the training set, which is the sign that the fixes were general and not fitted to the examples. In one round, the questions waiting for a person fell from 230 to 70.
What did not work
- A later round aimed for five more points and found 0.28, with no field worse. A new model prompt tried in the same round did about 15 points worse on addresses and item matches, so the engine was left as it was. A non-result is a result.
- For a week the language-model reader was not reading anything, while its status line still said it was working. "No error" in a status file is a claim. The status now records the last call and its result, and a card says in words when a file could not be read.
A default is a bet
Some dealers almost always pick up their orders, so the engine defaulted their orders to pickup. When an order was really a delivery and nothing flagged it, the scorer now counts that as the reader's mistake. Every default has a cost when it is wrong, and that cost has to be counted.
What it says about AI and rules
The language model reads what the rules engine cannot, like scanned pages. Deciding stays with code that can be tested. A model's reading is a proposal: it never names the customer account on its own, and the account is confirmed by a person. That split is why the accuracy number means something.