AI Document Processing: OCR, Extraction, Classification, and Cost

AI document processing turns unstructured files — PDFs, scans, emails, forms — into structured data your systems can act on. The modern pipeline is simpler than the OCR-heavy stacks of the past: a multimodal model reads the document, an extraction model pulls the fields you need, and a validation layer catches mistakes before they reach your database. This guide covers OCR, extraction, classification, pipeline design, and cost.

The payoff is real — invoices, contracts, and applications processed in seconds instead of hours — but the pipeline has sharp edges. Here is how to design one that works.

Input handling and OCR: getting readable text

Text-based PDFs are already readable — skip OCR entirely and pass the text straight to the model. Scans and photos need OCR, and multimodal LLMs can read them directly as images. The modern approach sends images to a vision model and lets it transcribe while it extracts, which handles tables and handwriting better than classic OCR libraries. Keep original page images alongside the text, because layout matters for fields like totals and signatures.

• Detect whether each file is text-based or needs OCR.
• Send images to a multimodal model instead of pre-OCR where accuracy matters.
• Preserve page images for layout-dependent fields.

Extraction: from documents to fields

Extraction is a structured output task. Define the fields you need — vendor, invoice number, line items, dates, totals — and ask the model for JSON against that schema. Field definitions beat open-ended prompts: say 'total amount in USD, numeric' rather than 'get the total.' Few-shot examples help on messy documents, and a validator that rejects missing or out-of-range fields turns silent errors into visible ones.

1. Define the target schema field by field, with types and formats.
2. Extract with a structured output request and validate on receipt.
3. Reject documents that fail validation instead of accepting best-effort guesses.

Classification: routing before extraction

Most pipelines see many document types — invoice, receipt, contract, purchase order — and the fields to extract differ per type. Classify first, then extract with a type-specific schema. Classification is cheap: short prompts with high accuracy, especially batched. Getting it wrong wastes the expensive extraction call on the wrong schema. A confidence threshold with a human-review queue catches the few percent of genuinely ambiguous documents.

Designing the pipeline

Route bulk processing through a queue with retries and a dead-letter bucket for failures. Batch the small stuff — classifying a hundred files in one call with a list input costs far less than one call per file — and parallelize the extraction calls. Link every raw input and extracted output by a document ID so you can replay a single failed document without rerunning the whole batch.

• Queue documents, batch classifications, parallelize extractions.
• Keep a dead-letter bucket for validation failures.
• Store raw text plus extracted JSON per document ID for audit and replay.

Cost and the trade-offs

Cost scales with pages and complexity. Short clean forms on text PDFs are cheap; long contracts with tables and scans are expensive because images and dense text burn tokens fast. Three strategies help: use a small model for classification, extract only the needed pages instead of whole documents, and cache results by document hash so identical files never reprocess. Batch APIs cut price by up to half when your workload can tolerate hours of latency instead of seconds.

FAQ