Intelligent Document Processing with VLMs: An On-Prem Approach

Key Takeaways

• Intelligent document processing built on a vision-language model handles the failure modes that break template-based OCR — shifted layouts, handwriting, multi-column reading order — because the model reasons over the whole page instead of matching fixed coordinates
• Qwen2.5-VL reaches 96.4% accuracy on DocVQA, close to the 98.1% human baseline, and exceeds Gemini 1.5-Pro by 9.6–20.6% on OCRBench v2's English and Chinese tracks (OCRBench v2)
• The production pattern isn't OCR-only or VLM-only — it's hybrid: PaddleOCR handles layout detection and raw text extraction cheaply, Qwen2.5-VL handles structured interpretation where semantic understanding actually matters
• Prodinit built a fully air-gapped document extraction pipeline for a document AI client: PaddleOCR for text extraction, Qwen2.5-VL served via Ollama on NVIDIA NIM for structured interpretation — zero network egress from the inference layer
• Ollama fits this workload specifically because document processing is throughput-per-document, not massive concurrency — the same reason it fits low-concurrency vision-language serving generally

Template-based OCR fails the moment a vendor changes their invoice layout. Fixed-coordinate extraction rules break silently — a field that used to be in the top-right corner now overlaps a logo, and nobody notices until the output is subtly wrong. Vision-language models don't have that failure mode, because they read the document the way a person does: the whole page, at once, with the field labels still attached to their values.

Intelligent document processing with vision-language models replaces rigid coordinate-based extraction with a model that jointly reasons over layout, text, and visual structure in a single pass — reading invoices, forms, and scanned contracts the way a human would, and producing structured output even when the document layout has never been seen before. Qwen2.5-VL is the model driving most production deployments of this pattern in 2026.

Why VLMs Changed Intelligent Document Processing

Traditional OCR pipelines recognize characters, then apply post-processing rules and fixed coordinates to reconstruct meaning — a design where every stage's errors compound into the next, and unusual layouts, handwriting, or noisy scans degrade accuracy fast. A vision-language model instead processes visual and textual information jointly in one forward pass, which is why it holds up exactly where template extraction breaks: shifted layouts, multi-column reading order, and handwritten annotations that a coordinate-based rule has no way to handle.

The accuracy gap is measurable, not just qualitative. Qwen2.5-VL scores 96.4% on DocVQA — near the 98.1% human baseline — and 88.8% on OCRBench, while exceeding Gemini 1.5-Pro by 9.6% and 20.6% on OCRBench v2's English and Chinese tracks respectively (OCRBench v2). The official Qwen2.5-VL technical report specifically highlights "robust structured data extraction from invoices, forms, and tables" as a core capability, not an incidental one (Qwen2.5-VL Technical Report).

This doesn't retire OCR. Traditional OCR still extracts raw text at a fraction of the compute cost and with far more transparent, debuggable output — which matters when a document pipeline is processing millions of pages a month and structure genuinely doesn't vary. The choice between OCR-only, VLM-only, and hybrid comes down to document consistency and what "correct" actually requires downstream.

The Hybrid Architecture: PaddleOCR + Qwen2.5-VL

The production pattern that holds up at scale is neither pure OCR nor a VLM working from a raw image alone — it's a two-stage pipeline where each model does the part it's actually good at. PaddleOCR's PP-StructureV3 pipeline handles layout detection and raw text extraction, converting a document image or PDF into a structured intermediate representation; Qwen2.5-VL then interprets that output against the original image to resolve what the extracted fields actually mean.

Stage 1 — PaddleOCR layout and text extraction. PP-StructureV3 runs layout detection, general OCR, and — depending on the document — table recognition and formula recognition as optional sub-pipelines, converting the page into structured JSON or Markdown with text blocks tied to their positions. PaddleOCR's own technical report shows this pipeline reaching state-of-the-art results on OmniDocBench among pipeline-based tools, competitive with dedicated VLMs on structural parsing while running at a fraction of the compute cost (PaddleOCR 3.0 Technical Report).

Stage 2 — Qwen2.5-VL structured interpretation. The VLM receives both PaddleOCR's structured extraction and the original document image, then resolves the semantic layer that raw text extraction can't: which extracted field is the invoice total versus the subtotal, whether a handwritten annotation overrides a printed value, and how a multi-column table's cells actually associate when the layout detector's boxes are ambiguous. This is the step that turns "text that was on the page" into "data with a schema."

Running OCR first cuts the volume of image tokens the VLM needs to reason over and gives it a text scaffold to check its interpretation against — cheaper and more accurate than asking the VLM to read raw pixels cold on every page. Running the VLM after OCR is what catches the errors a fixed pipeline can't: reading order across columns, field disambiguation, and interpretation the coordinate stage has no way to encode.

Running the Pipeline On-Prem and Air-Gapped

Document processing is frequently the workload that forces an air-gapped architecture, because the documents themselves — contracts, medical records, financial statements — are exactly the data regulated organizations can't send to a third-party API. Prodinit built a fully air-gapped document extraction pipeline for a document AI client: PaddleOCR for text extraction and Qwen2.5-VL served via Ollama on NVIDIA NIM inside a private on-premises environment, processing document images with PaddleOCR for text extraction and the VLM for structured data interpretation, with zero network egress from the inference layer.

Ollama is the right serving choice here specifically because of the workload shape: document processing throughput is measured per-document, not by simultaneous concurrent users, and Ollama's design priorities — quantized models, single-binary deployment, low operational overhead — fit that pattern well below the concurrency level where vLLM's continuous batching would earn its added complexity. Pairing it with NVIDIA NIM's optimized inference path recovers throughput headroom without moving to a heavier serving stack the workload doesn't need.

The air-gapped architecture itself follows the same pattern as any zero-egress deployment: model weights downloaded once and stored privately, inference on private compute with no internet path, and every supporting service — the document store, the extraction queue, the output database — reached only through private networking. For regulated document workloads specifically, this is often the deciding factor between a VLM-based pipeline being viable at all versus a hosted API being a non-starter regardless of accuracy.

Building the Pipeline: Step by Step

1. Ingest and normalize. Convert incoming documents (scanned images, PDFs, photographed forms) to a consistent image format and resolution before either model sees them — inconsistent DPI is a common source of accuracy loss at both stages.
2. Run PaddleOCR (PP-StructureV3). Layout detection and general OCR produce structured text blocks with position metadata; enable the table or formula sub-pipelines only for document types that need them, since each adds latency.
3. Pass both outputs to Qwen2.5-VL. Prompt the VLM with the original image plus PaddleOCR's structured extraction, and ask it to resolve field-level meaning against a defined schema — not to re-read the page from scratch.
4. Validate against the schema. Reject or flag extractions that don't match expected types, ranges, or required fields, the same discipline as tool schema design for any agent-facing output — an unvalidated VLM output is exactly as risky as an unvalidated tool call.
5. Route low-confidence extractions to human review. Documents where the model's confidence falls below a threshold, or where required fields are missing, go to a review queue rather than silently producing a wrong value downstream.
6. Log every extraction for audit. Especially in regulated environments, keep a record of which model version produced which output for which document — the same audit requirement that applies to any private LLM deployment in a regulated industry.

When Each Approach Fits