What OCR Actually Gets Wrong on Handwritten Recipes

The demo is always a clean, printed business card. Or a crisp, typed invoice. The text is black, the background is white, and the OCR engine nails it. 100% accuracy. Looks easy.

Then you open a real shoebox. Your grandmother's recipe cards. Faded blue ballpoint on yellowed index cards. Stains from forty years of use. Pencil annotations in the margins, written by three different people.

This is the reality I hit building MoveMyRecipes, my service for digitizing these exact shoeboxes. Optical Character Recognition on handwriting is not a solved problem. And recipes are the hardest possible version of that problem.

It’s not just about turning squiggles into letters. The errors that OCR makes on recipes are uniquely damaging because they are subtle. They produce a result that looks plausible but is catastrophically wrong.

Here’s what the demos don't show you.

The most dangerous error is with quantities. An engine sees "1/2 cup flour" and reads it as "12 cup flour". It sees a nicely written "1¼ tsp" and returns "114 tsp". The OCR engine isn't trained on the typography of fractions. It sees separate characters - a 1, a slash, a 2 - and dutifully reports them. Or it sees a small, floating character next to a bigger one and just guesses.

This is the worst kind of failure. If the OCR misreads "flour" as "flower", you'll probably catch it. But misreading "1/2" as "12" is an invisible error that quietly ruins the dish. The structure of the output is valid, but the data is poison. This isn't a theoretical problem; it happens constantly.

Next are the abbreviations. Recipes are a dense, domain-specific language. tsp, Tbsp, c., pkg, oz, doz. They’re littered with shorthand like 350° or pan sizes like 9x13. Most OCR models are trained on prose - books, articles, the general web. Their language models see tsp and think, "That's probably a typo for top." So they "helpfully" correct it.

Again, the output looks right at a glance. But it’s wrong in a way that requires a human with cooking knowledge to fix.

Then there's structure. An ingredient list is not a paragraph. It's often two columns of data: quantity and ingredient. You might have "1 cup flour" on the left, and "1 tsp salt" on the right, on the same horizontal line. The spatial relationship is the only thing connecting the quantity to the item.

OCR engines flatten this. They read left-to-right, top-to-bottom. So "1 cup flour" and "1 tsp salt" on the same line becomes a single, useless string: "1 cup flour 1 tsp salt". The context is gone. Reconstructing that requires another layer of software that is, frankly, just as brittle as the OCR itself.

Of course, sometimes you just get nothing. A recipe written in pencil 50 years ago has very little contrast with the now-brown paper it's on. You run it through the engine and get back an empty string. The signal is just too weak for the sensor to pick up.

And the most human parts of the card? The parts that make it a treasured family document? Forget it. The cross-outs, the arrows pointing from a margin note ("Dad likes less cinnamon") to an ingredient line, the corrections stacked on top of each other - this is chaos to a machine. It's data, but not in a format any current model can understand.

So what actually works? After processing thousands of these cards, I’ve learned that the most important work happens before the OCR engine ever sees the image. This is a hard-won lesson from my broader work on OCR pipelines.

First, photograph better. This is the single highest-impact thing you can do. Lay the card perfectly flat. Light it evenly from all sides. A cheap desk lamp and a piece of white poster board to bounce the light works wonders. Make sure your phone or camera is directly overhead to avoid skew. No hand shadows. A clean, sharp, evenly lit photo is more valuable than any software setting.

Second, preprocess the image. Once you have a good source photo, you can make it even better for the machine. Deskew the image so the text is perfectly horizontal. Dramatically increase the contrast. Convert it to grayscale. These steps make the characters pop, giving the engine a much clearer signal to analyze.

Third, pick the right engine. For handwriting, especially messy cursive, Google Cloud Vision AI is the most capable of the three. Tesseract is fantastic for clean printed text but fails badly on handwriting. AWS Textract has a handwriting mode, but in my tests, it struggles more with the low-contrast and varied styles found in recipe cards. I've documented a full tool comparison if you want the details.

But picking the engine is only the first step, not the solution. "Most capable" is not the same as "good enough." Even Google Vision, the best of the three at handwriting, still misreads fractions and quietly "corrects" shorthand on a faded card. A better engine buys you a better first draft. It does not buy you a finished recipe.

After all that, you get your text output. And it will still have errors.

This is the final, most important step: human review. OCR is a first draft. It's a tool for getting the bulk of the text down, but it is not a finisher. You must have a person read the output and compare it against the original card, paying special attention to the quantities and units. There is no way around this.

On a clean, recent, clearly-written card, a good OCR process will get you 80-90% of the way there. On a faded, annotated, 60-year-old card, that number can drop to 50% or less. But the work is in that last 10-20%. That’s where the fractions are. That’s where the weird abbreviations live. It's the part that determines if the recipe actually works.

Anyone selling a fully automated, hands-off recipe digitization service is skipping that last, critical step. The machine can do the typing. Only a human can do the reading.

If you're staring at a shoebox of your own family's recipes and want it done right, get in touch.

I build OCR and document-automation tools for small businesses. If you have a messy, manual process you want to automate, I can probably help.