Making Documents, PDFs and Scans Usable for AI Agents

Most of the knowledge an AI agent needs in a typical company does not live in a database. It sits in files: quotes exported as PDF, scanned delivery notes, inspection reports, datasheets, spreadsheets and engineering drawings. Whether an agent gives useful answers is often decided at the moment those files are read in. This guide walks through how a robust document pipeline is built and which decisions shape it.

What does it mean to make documents usable for an AI agent?

It means turning files into structured text that a language model can process reliably. Headings, paragraphs, tables, reading order and page references survive; scans become readable. Only then can an agent extract fields, answer questions or trigger the next step in a workflow.

This is harder than it sounds, because a PDF is a print description rather than a structured document. The pypdf documentation puts it plainly: the format exists to produce the desired visual result for printing, and PDF files don't contain a semantic layer. Headers, footers, page numbers and paragraphs are not labelled; text is stored as positioned glyphs. The same page notes that tables are typically just absolutely positioned text, which makes rows and columns hard to recover.

Where this matters in practice:

• Quotation work: reading requirement specifications, request-for-quote PDFs and drawing title blocks to prepare an offer (see AI agents for quotation).
• Inbound paperwork: splitting purchase orders, order confirmations and delivery notes into fields and matching them against the ERP system.
• Knowledge management: making manuals, work instructions and inspection reports searchable (see AI agents for knowledge management).
• Field and technical service: using maintenance logs and service manuals to answer diagnostic questions (see AI agents for technical service).

How does a document pipeline for agents work?

A dependable pipeline has four stages: identify the document type, parse it or run text recognition, convert the result into an intermediate format such as Markdown or JSON, and only then hand it to the agent. Each stage can be checked on its own, which is what makes errors traceable.

1. Intake and classification. Files arrive from a mailbox, a network share, a document management system or an upload form. The pipeline works out what it has received: a PDF with a text layer, a scanner-generated PDF, an Office file, an image or an email with attachments. A simple check for PDFs: if text extraction returns almost nothing per page, you are probably looking at a scan.

2. Parsing and text recognition. Digital PDFs and Office files are parsed directly. Scans need text recognition, either a classic OCR engine or a vision-capable model that reads the page as an image. Good parsers also detect layout: headings, tables, page headers and footers, figures and checkboxes.

3. Intermediate format. Store the output as Markdown or JSON, tagged with page number and source file. Markdown is easy for models to read and preserves tables and headings. This is also your main checkpoint: whatever is wrong here, the agent cannot fix later.

4. Handover to the agent. Either a model fills a fixed schema from the text, or the text is split into chunks, indexed and retrieved on demand through retrieval-augmented generation (RAG). Both options are compared below.

Parsing tools at a glance (as of October 2026)

• Docling is an open-source toolkit under the MIT licence, started by the AI for Knowledge team at IBM Research Zurich and now hosted by the LF AI & Data Foundation. According to its repository it handles PDF, DOCX, PPTX, XLSX, HTML, email and images, among others; it detects layout, reading order, table structure and formulas, provides OCR for scans, and runs in air-gapped environments. Its format overview lists Markdown, HTML, lossless JSON and RAG-ready chunks among the outputs.
• Tesseract is an Apache-2.0 OCR engine. Version 4 introduced an LSTM-based neural network engine, version 5 is the current major release, and the official language data covers more than 100 languages. It returns text, not table structure, so treat it as a component rather than a full document parser.
• Azure AI Document Intelligence is Microsoft's cloud service. Its layout model (prebuilt-layout, API version 2024-11-30 GA) extracts text, tables, selection marks and roles such as title, section heading, page header and footer, and can return the result as Markdown.
• Models with native PDF input: according to Anthropic's documentation, the Claude API converts each PDF page into an image and passes the page's extracted text alongside it, so charts and figures are read as well.

A minimal example based on the Docling quickstart, converting one file to Markdown (latest version on PyPI in October 2026: 2.138.0, requires Python 3.10 or later):

from docling.document_converter import DocumentConverter

# Local path or URL
source = "inbox/delivery_note_4711.pdf"
converter = DocumentConverter()
result = converter.convert(source)
print(result.document.export_to_markdown())

Which document types need which approach?

Difficulty varies a lot by document. Digital PDFs from business systems parse almost cleanly, scans need text recognition, tables need structure detection, and engineering drawings remain the hardest case. Design the pipeline around the document type you see most often.

Digital PDFs come from ERP systems, word processors or PDF printers, so the characters are embedded. Problems show up in reading order (multi-column layouts, margin notes), spacing and tables. The pypdf documentation explains that correct whitespace is hard to guarantee because spacing is encoded as position adjustments rather than actual space characters. A layout-aware parser handles this far better than plain text extraction.

Scans and photos contain only pixels; without text recognition the agent sees nothing. Quality depends heavily on the original: Tesseract's own guidance says it works best on images of at least 300 dpi, and recommends straightening skewed pages and removing dark scan borders. Stamps, handwritten notes and carbon copies stay error-prone whatever tool you use.

Tables cause most real-world trouble: line items on purchase orders, measurement tables in inspection reports, bills of materials. Without structure detection, columns shift and the agent assigns a quantity to the wrong line. Docling and the Azure layout model rebuild rows and columns and emit them as Markdown tables. Still spot-check merged cells and tables that run across page breaks.

Office files (Word, Excel, PowerPoint) are structurally simpler because they already know about paragraphs and tables. With spreadsheets the real question is what the agent should see: formulas or computed values, every sheet or one. Microsoft notes that the Azure layout model does not support table analysis for XLSX input, so reading the workbook directly with a spreadsheet library is often the better route.

Engineering drawings are the hardest case. A vision-capable model can usually read the title block, drawing number and material. Dimensions, tolerances and datum references in the drawing area are far less reliable. Use drawings for metadata and pre-screening, not as the sole basis for costing or manufacturing, and have a qualified person confirm any value that feeds into a quote.

Structured extraction or RAG?

The second key decision is what the agent does with the text. Both approaches rely on the same parsing stage but differ in purpose, testability and design.

With structured extraction you define a schema, for example order number, delivery date and line items with quantity and part number. The model fills it in and your code validates it. Vendors offer dedicated features: the Claude API's structured outputs use constrained decoding to return JSON that matches your schema. The documentation also lists exceptions: refusals or truncated responses can break the schema, and schema keywords such as minimum and maximum for numbers are not supported. Plausibility checks, such as line totals or valid part numbers, remain your job.

With RAG, documents are split into chunks, embedded as vectors and the relevant chunks are pulled into context for each question. It scales to large collections but depends on sensible chunking: a chunk that cuts a table in half produces wrong answers. That is why a parser that keeps headings and tables intact matters as much for RAG as for extraction.

Local or cloud processing?

The third decision is where processing happens. Cloud services spare you the infrastructure and come with specialised, continuously maintained models. Local tools keep confidential documents inside your own network and make it easier to set up a pipeline that can be operated in line with the GDPR, without involving another data processor.

You can decide each stage separately. A common pattern is to parse and OCR locally with Docling or Tesseract and send only the cleaned text to a hosted model. If you want the model in-house as well, see sovereign AI. With hosted models that accept PDFs directly, mind the limits: per Anthropic's documentation, Claude accepts requests up to 32 MB and 100 or 600 pages depending on the context window, and encrypted or password-protected PDFs are not supported.

What do you need before you start?

Three things: clear access to the document sources, a precise idea of which fields or questions the agent should handle, and a sample of real documents with known correct results. Without that sample you cannot measure quality, let alone improve it.

Data. Collect 30 to 100 typical documents per type, including the awkward ones: poor scans, tables spanning several pages, mixed-language files. Record the correct values for the fields that matter. This becomes your test set.

Systems. Establish where documents come from (mailbox, file share, SharePoint, document management system) and where results go (ERP, CRM, ticketing). The pipeline needs read access to the source and a defined write path to the target, ideally an API rather than a shared folder.

Organisation. Decide who reviews uncertain results and how quickly. An agent that hands low-confidence cases to a person is worth more in production than one that always answers. Also settle retention and deletion for the intermediate files, since they hold the same content as the originals.

How do you roll it out step by step?

Start with one document type and one measurable goal, build the pipeline against your test set, and expand only once results are stable. Tackling every document type at once makes it impossible to tell where errors come from.

1. Pick a document type. Choose the one with the highest volume and a clear structure, such as supplier order confirmations.
2. Build the test set. Real documents plus expected results, as described above.
3. Compare parsers. Convert the test set to Markdown with two tools and inspect tables, reading order and special characters by hand. Most problems surface here.
4. Set up extraction or RAG. Define the schema, or configure chunking and the index. Every output carries its page reference and source file.
5. Add validation rules. Plausibility checks, matching against master data, a threshold that routes cases to human review.
6. Measure. Field accuracy or answer quality against the test set, plus runtime and cost per document. For hosted models, Anthropic states that PDFs typically use 1,500 to 3,000 text tokens per page, plus image tokens, because every page is also processed as an image.
7. Connect it to the process. Only then write into the target system, reviewing every result at first and moving to sampling once the numbers hold.

What are the limits and risks?

The biggest risk is silent failure: shifted table columns, misread digits in scans, and fields the model fills in plausibly but wrongly. None of these crash the pipeline; they surface later in the business process. The counter-measures are page references, plausibility checks and human review whenever confidence is low.

• Invented fields. If a value is missing from the document, a model may infer or invent it. Allow empty values explicitly in the schema and require a source location for every field.
• Context limits. Long or dense documents can exceed a model's context window. Anthropic notes that dense PDFs with small fonts, complex tables or heavy graphics can fill the context window before the page limit is reached. Split such documents into sections.
• Handwriting and stamps. Handwritten notes, initials and stamps remain unreliable. Where they carry legal weight, for example on acceptance certificates, keep a person in the loop.
• Security. Documents from outside sources can contain text phrased as instructions to the model. Always treat document content as data, never as commands, and give the agent only the permissions the task requires.
• Drift. When a supplier changes its document layout, extraction quality can degrade without any error. Re-run your test set on a regular schedule.

A document AI pipeline is the wrong tool when the data already arrives in structured form, for example as EDI messages or structured e-invoices. Reading that structure directly is more reliable than any text recognition.

Where should you start?

Take 50 real documents of one type, convert them to Markdown with a local parser such as Docling, and read the output. Within an afternoon you will know whether your bottleneck is reading the documents or what happens afterwards.

That tells you where to invest. If reading is the problem, look at better scans, a stronger layout model or a vision-capable language model. If the problem comes later, tighten the schema or improve chunking. In our own operations, keeping the Markdown intermediate files permanently has proven worthwhile: every later answer from the agent can be traced back to the exact page.

For the kinds of agents that build on a document pipeline like this, from quotation to service, see AI agents by business function. More integration guides are collected in the guides section.