Extraction works on the clean draft and fails on the signed executed copy

The pipeline scores 90-something on clean drafts and then loses clause boundaries entirely on the file that was actually signed. Clause 8.2 comes back containing half of 8.3, the liability cap is attributed to the wrong schedule, and a document any of the deal team can read in 20 seconds returns an extraction nobody can use.

The cause is that these are two different artefacts. A draft is born digital, with a numbering tree the file itself carries, one continuous body, and no marks on it. The executed copy is a photograph of a stack of paper that a person assembled, initialled, stamped and fed through a scanner. Every pathology below comes from that assembly step, and none of them can be reproduced by testing on drafts.

Diff the clause index against the version that went out for signature

1. Recover the signature-ready draft from the deal folder or the signing platform. It is the last version anyone had in a structured format.
2. Extract the clause index from both: every heading number, its text, and the page it begins on. Ignore content at this stage.
3. Align the two indexes in order and mark the first divergence. Everything after it is downstream noise, so only the first point matters.
4. Note where it sits. Diverging inside the body points at segmentation or a spliced page; diverging only at the first schedule points at assembly order.
5. Count pages against the index and the internal cross-references. A scan with fewer pages than the executed bundle is a document you cannot answer absence questions on at all.
6. Run a handwriting or non-print-stroke detector per page and list every page it flags. Those pages are quarantined until a person reconciles them.

This takes an hour on 5 documents and settles arguments that otherwise run for weeks, because it separates the model question from the document question. If the recovered index matches the draft and the extracted values are still wrong, you have a model problem. If the index itself has diverged, no amount of prompt work will help.

Six things that only happen to a document once it is signed

The margin edit that changes the deal and leaves no text behind

A late change at the signing table is often made on paper: a number struck through, a replacement written above it, both parties' initials in the margin. The parties ordinarily intend that change to have effect, and whether it does in a given instrument is a legal question. What is not in question is what extraction does with it — the printed characters underneath are still the clearest thing on the page, so the pipeline reads the superseded number with high confidence and reports it cleanly.

That is the worst failure mode available. A missing value produces an empty field somebody chases. A confidently extracted superseded value produces a repository that is wrong and looks fine, and everything built on top inherits it — including any review that judges new clauses against what the firm has previously signed, which is one of the costs weighed up in judging a clause against a playbook or against your own past deals.

Execution in counterparts, and the body those pages land in

Where an agreement is executed in counterparts, each party signs a separate copy and someone assembles the complete record afterwards. Assembled well, the signature pages sit together after the body. Assembled by whoever had the scanner, they can appear after each party's own copy of the execution block, between the body and the schedules, or — with a stapled bundle fed straight through — in the middle of a clause.

The signal is reliable: a clause numbering sequence that runs 12.1, 12.2, then a page with a signature block, then 12.3. Naive segmenters treat the interruption as the end of a section and start a new one, which is why the clause after a spliced page is so often truncated. The same rule that governs how a contract is cut into clauses applies here with one addition: a page whose layout is a signature block is not a boundary, it is an interruption, and the clause continues across it.

When the exhibit stack is not the order the index promised

The index says Schedule 1 is the statement of work, Schedule 2 the service levels, Schedule 3 the charges. In the bundle, the charges were printed last-minute and went on top. Nothing in the scan says so, and every schedule-scoped extraction now attributes values to the wrong document — service levels read as charges, a liability figure lifted from the wrong annex.

• Resolve schedules by their own title page, never by position. A schedule with no title page is a finding to report rather than an ordering assumption to make.
• Reconcile the count. 3 schedules in the index and 4 appended documents means 1 is an attachment nobody listed, usually a certificate or an approval.
• Treat a schedule that appears twice as unresolved. 2 versions of the charges annex is exactly where guessing is worse than escalating.
• Carry the resolution through to drafting. If the next amendment is going to be built from clause blocks read out of this file, a mis-resolved schedule propagates — a risk that belongs in the choice between assembling a first draft and generating one.

Three answers the system should decline to give on a scan

1. Any absence claim, where the page count does not reconcile against the index and internal cross-references. Confirming a clause is not present requires the whole document, and half a document supports no such statement.
2. Any value from a page flagged for non-print strokes in a body region, until a person has reconciled it. Return the page image and the question, not a figure.
3. Any redline. Producing tracked changes requires a document structure the scan does not have, so the honest artefact is a page-anchored deviation report — the output question resolved in a deviation report or a marked-up draft.

A missing value gets chased. A confidently extracted superseded value gets relied on, and that is the failure worth engineering against.

What a better scanner still leaves broken

Higher resolution and deskewing fix legibility, not assembly. A perfect 600 dpi capture of a bundle whose schedules are in the wrong order is a perfectly legible wrong answer, and a flawless scan of an initialled amendment still shows the superseded number in print. Rescanning is worth doing where character confidence has collapsed under a stamp; it is worth nothing anywhere else on this list.

Nor does any of this make the clause judgement right. Once the operative text is recovered, whether a cap or an indemnity is acceptable is a scoring question, handled by the rubric method in turning negotiation instinct into a scoring rubric and the rest of the contract review and redlining topic. One further constraint is worth naming early: executed bundles carry wet signatures, personal data and sometimes identity documents in the schedules, which pushes the capture stage towards the deployment patterns in private LLM deployment. Building that pipeline is scope we take under AI agents and automation for legal teams.

• contract review
• document extraction
• executed documents
• OCR