The long version.

01 · The problem, in the operators' words

Large retailers publish a rulebook for how a trading partner's documents must be formatted. The rulebook runs 20 to 200 pages[C](Client-stated). An analyst opens a document, opens the rulebook, and checks one against the other line by line. It takes 30 to 50 minutes, and up to about three hours for someone new[C](Client-stated). Around seventy analysts do this five or six times a day[C](Client-stated).

The cost of getting it wrong is not abstract. A broken document means a delayed shipment, a chargeback, and a strained relationship with a customer the company cannot afford to annoy.

Three things a pilot analyst said, in interview. These are the only real quotes in this project, and there are no testimonials:

02 · The pipeline, and where the model is not

Eight stages. Six make no model call at all, one is a human gate, one calls the model.

There is one route per stage and deliberately no run-all endpoint. The analyst advances the work one step at a time and can stop between any two. That costs convenience and buys visibility, which for a first deployment on protected data is the right trade.

The boundaries, and whether each is real

I separate constraints that are structural, meaning the thing cannot happen, from ones that are advisory, meaning the model was asked nicely. Publishing the difference matters more than the list.

Structural. The model never receives an original sensitive value, because both documents are tokenized before the prompt is assembled. Envelope segments, formatting-only differences and count mismatches are resolved by rule and never reach the model at all. Unmasking happens locally after the response, against a mapping the model never sees. Barcode validity is checksum arithmetic. Delimiter detection, parsing and difference classification involve no model anywhere. Output must satisfy a fixed response schema.

Human gate. Nothing is sent until a person approves the exact payload. The approval screen shows the masked text, the residual leak scan, the rows already settled by rule, the verbatim prompt, and its fingerprint. Nothing reaches the model that a person has not read first.

Advisory, and therefore the weakest link. In its default mode the model is instructed to ignore its own general knowledge of the format and treat the supplied rulebook as the sole source of truth. That is a prompt instruction. I cannot make it structural, and I am not going to present it as though I had. What I did instead was make the consequence structural: if the model answers from general knowledge, it cannot cite a passage, and an uncited pass gets downgraded automatically. The instruction can fail. The check behind it does not.

03 · Redaction design

Mask by structural position, not by pattern

A value is sensitive because of where it sits in the record, not because of what it looks like. The sensitivity map covers 8 segment types and 13 element positions across 9 categories: party names, party identifiers, addresses, city, state, postcode, item descriptions, contact names and phone numbers, plus general identifiers.

Two consequences that took a mistake each to learn.

Qualifiers stay readable. In a party segment, the code saying which party this is carries no identity and must survive, or the model loses the structure it needs to reason about the record. Element 0 stays, element 1 goes.

Registration is document-wide, not difference-only. A party name identical on both sides of the comparison still gets masked. Masking only what changed would leak everything that did not.

The two failures

Over-masking. My first pass masked anything that looked like a code. Short tokens broke it: "or" is Oregon and "in" is Indiana, so both were masked as state codes, including inside the rulebook's own prose. With them gone, the structure the model needed went too, and the verdicts degraded. Masking by string was the bug. Masking strictly by structural slot was the fix, plus a case-sensitive rule for short codes and a floor that refuses to mask anything under two characters. Over-masking is not caution. It is a quieter way to be wrong.

Under-masking. The opposite mistake was more dangerous because it was invisible. Product barcodes look like ordinary numbers, so my first pass left them in. They are not ordinary. One barcode can re-identify an entire anonymous file. I only caught it by attacking my own work: I handed a fully masked file to a stronger model and asked it to re-identify the data. It went straight for the barcode.

Barcodes are now detected by check-digit arithmetic rather than by appearance, and replaced with synthetic substitutes drawn from a restricted range that cannot collide with a real brand-owner prefix. The same product keeps the same token across packaging levels, so the model can still reason about it.

What I would do differently: type the barcodes first. That failure was findable on day one with the same adversarial pass I eventually ran. I ran it late because I was confident, which is the exact condition under which you should run it early.

One more masking bug worth naming

An early implementation replaced values token by token, which let a later replacement match text inside a token already inserted. A single longest-match-first pass fixed it. It is a small bug with a large lesson: the redaction layer needs the same adversarial attention as the model, and it is easier to reason about when it runs once rather than iteratively.

04 · Refusal, and why silence is the dangerous case

Five verdicts. The one that matters most is the one that declines to answer.

The failure mode I care about is not a wrong verdict. It is a confident pass on a rule nobody actually checked, because that is the one nobody catches. So the system flags by default: anything not positively sanctioned goes to a human. This sends more rows to people than a looser design would. That is not inefficiency. It is the price of never turning silence into false certainty.

The citation check. A pass verdict only stands if it quotes a real passage. After the model answers, code verifies that the cited text is non-empty and is not one of the sentinel values meaning "not in the rulebook." Anything failing that check is silently downgraded to human review. Missing and extra segments can never be auto-passed at all. The downgrade only ever moves toward caution: reject and clarification_required are never touched.

05 · Provenance instead of confidence

The first result screen showed a verdict and a confidence label beside it. Watching analysts use it, "medium" did nothing. Nobody could act differently on it. It performed nuance while sending every reviewer back into the rulebook to find the rule themselves, so the screen had added a step rather than removing one. It was also a second probabilistic signal sitting next to an already probabilistic verdict, which confuses rather than calibrates.

I removed it. The column is gone from the schema entirely.

Two more shipped controls went with it: a verdict-override dropdown that read as a duplicate of the reviewer disposition control beside it, and a per-row verdict badge, because "breaks the rules" is a top-level count and not a marker on every row.

What replaced confidence is provenance: the exact passage, the page it came from, the verbatim prompt, and a fingerprint of what was sent. Calibration here is provenance, not a score.

06 · Evaluation: what exists, and what does not

What exists. Eight test files, covering difference classification, barcode masking and check digits, model-response parsing including truncation salvage and invalid-verdict coercion, ambiguous short-code masking, party and address masking, parser terminator handling, rule-engine auto-resolution, and the full flag-by-default verdict matrix.

Note the shape of that list: every structural boundary in section 2 has a test pinning it open. The tests test the constraints, not the model. That is deliberate. The constraints are the part I can guarantee.

The adversarial re-identification pass in section 3 is the only true red-team evaluation, and it found a real leak.

A code audit run against the shipped product produced a trust-gap table. It caught, among other things, the two failures in section 8.

What does not exist, and should. No golden verdict dataset. No answer key. No precision, recall or inter-rater study. No regression detection beyond unit tests. No calibration study, and the only calibration argument in the project is the one against scores.

Five human overrides were logged across 7,724 analyzed differences[M](Measured). That is a usage signal. It is not an accuracy measurement, there is no denominator behind it that means "correct," and I will not present it as one.

The measurement I did not build. I can tell you the model stage runs in minutes. I cannot tell you from telemetry how long a reviewer now spends confirming a cited verdict against how long they used to spend hunting through the rulebook, and that is the number this whole design is actually about. It rests on observation rather than logging. That is a gap I built, and it is the first thing I would instrument next.

07 · The numbers, with their provenance

Pilot telemetry, one snapshot dated 26 June 2026[M](Measured)

The number missing from this table is the one I most want: of those 7,724 differences, how many were settled deterministically and never reached the model? That is the denominator behind "spend the model sparingly," it exists in the pilot database, and it is not yet written down.

Time

30 to 50 minutes by hand[C](Client-stated). 8 minutes with the tool. About 30 minutes saved per document, which is the conservative low end of that range and the figure the projections below are built on[C](Client-stated).

The two projections, side by side

Both use the same hourly rate[C](Client-stated) and the same 30 minutes saved per document[C](Client-stated). They differ on one variable only: assumed volume.

One inconsistency I inherited and did not hide: the conservative rows convert hours to people at a lower annual-hours figure than the saturation row does. That widens the gap slightly in the saturation model's favour. Reconciling both to one figure is worth doing before either number is quoted anywhere that matters.

The base case is the one the client already used in their own planning[C](Client-stated), which is why I anchor on it. The saturation model assumes total adoption from day one, which does not happen. Its number is 17 times larger than the base case and considerably weaker as an argument, and publishing only the larger one is how figures like these stop being believed.

These are planning models, not measured results. No measured dollar saving exists.

08 · Where I was wrong

Six items. The first two are the ones I would lead with in an interview.

• 01My README claimed a safety feature that did not exist. It stated that high-severity leak warnings block the send button. At audit time the backend computed those warnings and the interface never rendered or gated on them. The claim was false. It is now true, and it is enforced. This is the most damaging thing the audit found and the best evidence that running the audit was worth it.
• 02The audit trail's prompt fingerprint did not match what was sent. The hash was logged over the aggregate package while the model was called per batch, so the provenance mechanism, the thing whose entire job is proving what happened, was itself wrong. It now logs both the aggregate and every per-batch hash.
• 03Over-masking, then under-masking. Section 3. Both mine, both shipped, both found by me rather than by a user.
• 04A shipped feature that measured nothing. The confidence score. Section 5.
• 05Reviewer identity is not authenticated. The approving name was hardcoded at audit time; it now accepts a supplied value and falls back to a default. There is no user table and no real authentication, so whatever string the client sends is what gets logged. That is a partial fix, not a fix, and it is a real limitation of a single-user pilot.
• 06Dead types the code claims to support. Three difference types are defined in the type union and referenced nowhere in the engine. They document intent rather than behaviour. That is documentation debt, and it is the kind of thing that makes a reader trust the rest of the types less, which is fair.

I also removed an entire rule-extraction subsystem after concluding that sending the rulebook text directly to the model was better than pre-parsing it into structured rules. That was not a failure so much as a design I outgrew, but it cost real time.

09 · Published limits

• 01The residual leak scan is a heuristic, not a guarantee. It is tuned for one document type, and the badge in the product says so rather than leaving the user to assume.
• 02This is a single-user pilot, not a multi-tenant production system. Reviewer identity is a self-entered name.
• 03It compares two documents against a rulebook. A violation present identically in both is out of scope by design.
• 04The design covers one transaction type. Which others matter was raised early and never answered.
• 05There is no cost measurement. No token count, no bill, no cost per comparison. The current design re-sends the rulebook with every batch, which works and is wasteful for no gain in judgment. I know it, I can price the fix, and I have not shipped it.
• 06The only measured numbers here are the pilot counts, dated 26 June 2026.

10 · How it was built

One model call is not what happens. There is one model stage, one prompt contract, and rows go out in batches of twenty through a small concurrency pool, so a run makes as many calls as it has batches. Each batch is fingerprinted individually. An earlier draft of this case study said "exactly one call." That was wrong, and it is the kind of wrong that sounds better than the truth, which is why it is corrected here rather than quietly dropped.

Around that: one retry with a short backoff, then a fallback provider. Streaming progress to the interface. Successes and failures counted separately, so a partial run can never render as a clean success. A stop control that halts new work between batches. Two distinct failure messages, because "this batch failed, re-run it" and "the model returned no verdict, review this manually" are different problems for the person reading the screen. A salvage parser, so one malformed response never wipes a batch.

The delivery method. A planning model turns one decision at a time into a self-contained brief. I answer a short lettered gate, sometimes literally "Q1 A, Q2 B." A coding agent executes that brief against the live repository. Screenshots come back, the planner verifies against a written definition of done, and anything missed folds into the next brief. Twelve briefs, executed in strict order.

The product shipped across one working day of commits, and the last commit in the repository is the fix for my own over-masking bug.