The documents care actually runs on
A care organisation runs on paper long after it has gone digital. A nurse's handwritten event note, a tablet-signed consent form, a scanned referral, a photographed medication chart, a spreadsheet of supports. When that content lands in our platform, something has to turn the pixels into text before anyone, or any agent, can search it, cite it, or reason over it. This is called optical character recognition, or OCR.
In an earlier role at Deloitte, I ran millions of financial documents through traditional OCR engines which are built for clean, printed pages laid out the same way every time. On these types of documents OCR is basically a solved problem, able to accurately extract printed text into something an AI agent can work with.
Care documents are the opposite of that. They are handwritten, rotated, signed over, photographed at an angle, laid out as forms and tables, scribbled in a hurry. When the OCR cannot read them the content becomes invisible and the agent cannot answer from it or point back to it. While traditional OCR engines have gotten better at handwriting recognition over time, they can still struggle with recognising the structure of a document. Think about a chart full of medications and doses; these can't just be extracted as a slab of text, they need to be kept in their proper place so an agent is working with the truth.
What were the options and how did I test them?
I tested eight engines across four kinds: the OCR we run today, a couple of other managed OCR services, general vision-language models (a large frontier one and a smaller, cheaper one), and open models small enough to self-host.
Every page is turned into an image at the same resolution, and every engine gets the same prompt: return the page as clean markdown. The main measure is content recall: of the words on the page, how many did the engine actually get? A word it misses or mangles is a word the agent cannot find later. Alongside that I looked at whether tables kept their structure, whether the engine invented text that was not there, how often it failed outright, cost per thousand pages, and how fast it runs.
The documents are three public datasets: business documents with known transcriptions, deliberately hard and degraded PDFs, and photographed handwritten notes.
What I found
On clean printed text there is barely a gap between old and new, which was expected. On the hard, degraded PDFs the legacy engine gets about a third of what the modern models get, and it returns no table structure at all: the numbers come out as one flat run of text with the rows and columns gone. That is the medication-chart problem from earlier, the values are there but not which dose lines up with which time. Both vision-language models handle what the legacy engine cannot.
| Measure | Legacy OCR | Frontier VLM | Efficient VLM |
|---|---|---|---|
| Hard, degraded PDFs (pass rate) | 0.31 | 0.68 | 0.66 |
| Table structure (cell F1) | 0.00 | 0.79 | 0.69 |
| Handwriting (content recall) | 0.86 | 0.87 | 0.87 |
| Relative cost per 1,000 pages | 1x | ~16x | ~1.3x |
~350 pages, three open document sets, single shared prompt. Higher is better except cost.
The handwriting result surprised me
I expected handwriting to be the legacy engine's worst category, but it wasn't. On clean, photographed handwriting it held up close to the modern models. Where it failed is the degraded, rotated, signed-over, form-and-table pages, which is what a real care record actually looks like.
The bigger surprise was the smaller, cheaper model beating the frontier one on handwriting. When I read the pages I could see why: the frontier model sometimes decides a scrap of text at the edge of the page is not worth transcribing, and the reference counts it, so being more literal scored better here.
The best model is not automatically the right one
The most capable model in the table costs about twelve times as much per page as the cheaper one, and the lead it buys is mostly table structure and a couple of points of recall. Extraction runs on every document a customer uploads, and we meter that cost back to them, so a twelve-times multiple is the difference between switching it on for everyone and rationing it.
Throughput matters just as much, and it is easy to miss. If the extraction model shares capacity with the live product, a big backfill of old documents can slow down the agent someone is typing to right now. A model on its own separate capacity does not. So I was not looking for the highest score. I was looking for the cheapest model that clears the bar and runs on capacity that does not come out of the product's budget.
How I think about the choice
Extraction is not one model, it is a pipeline of tasks. Spreadsheets get read directly, because a grid of cells does not turn into a page image sensibly. Everything else, including digital PDFs and Office files, is turned into page images and read by one vision-language model in a single call that returns the clean text and the tags the record needs, and it runs in-region so the data never leaves the customer's jurisdiction. I picked the model that clears the bar on the hard documents and is cheap and fast enough to run on everything, not just the easy pages.
Because we keep the extracted text instead of throwing it away, it is also something a person can see and fix. When the agent cites a document, you can open the page it read, and if it got a word wrong you can correct it, with the original sitting alongside. For care records that is as important as the score: the people using the information can see where it came from.
Limitations
These are open datasets, not care documents, so they calibrate the method more than they settle it, and real care records are messier and more varied. The ground truth is imperfect too, especially on handwriting: I corrected fifty of those transcriptions by hand and they still carry errors, which drags every engine's score down and makes the best ones look worse than they are. The metric has an edge as well: content recall counts a near-miss as a full miss, which is the right call for search, since a word that is slightly wrong will not be found, but it is harsher than a character-level score. And this is a snapshot. The models move fast, so I treat it as a test I can rerun, not a final answer.
Method and figures from my own extraction evaluation, run over three openly licensed document sets with one shared prompt.