We Taught Our Backend to Read Handwriting (And Then Not Trust It)

Somewhere in every retail supply chain, there is a person typing.

A vendor fills out a paper order form by hand. Vendor name, GSTIN, then a table of rows: article, design, color, sizes, rate, quantity. That paper gets photographed or scanned and sent over. And then somebody sits down and retypes all of it into a system so a Purchase Indent or order can be created.

It works. It is also slow and boring.

So we built a feature that takes one input, a URL to a photo of that form, and returns structured JSON with every field already matched against our master data. Ready to prefill the creation screen.

This is a walkthrough of how it is put together, and more importantly, why each piece exists. Almost every design decision here came from a problem the previous decision created.

Why not just use OCR?

The obvious first instinct is a template parser. Find the boxes, read the boxes, done.

That falls apart immediately, because we do not control the layout. Every vendor writes differently. Some use a printed pad, some use a ruled notebook. Columns drift. Rows get squeezed into margins. There is no fixed geometry to anchor a template to.

A vision-capable language model does not care about geometry. So that is what we use, driven by a single fixed prompt and a strict JSON schema, so the output arrives as shape-valid JSON instead of prose that would need its own parser.

That solved the reading problem and immediately created the next one.

One prompt, three providers

We did not hardcode the model vendor.

Different clients come with their own AI vendor relationships and their own API keys. Some are already paying for OpenAI. Some are on Google Cloud. Some prefer Anthropic. Telling a client “you must buy credits from our vendor” is a bad conversation to have during onboarding.

So provider selection happens at runtime, per organisation, from a plain lookup table:

{ openai, anthropic, gemini } → adapter

Each adapter implements exactly three functions:

• buildRequest(...) turns the shared prompt, schema, and image into that provider's request shape. OpenAI gets chat/completions with response_format.json_schema. Anthropic gets the messages endpoint with a schema-shaped tool and a forced tool call via tool_choice. Gemini gets generateContent with a responseSchema.
• parseResponse(...) pulls the structured JSON back out of that provider's response envelope.
• parseUsage(...) normalises token counts for logging.

All three of those are documented, first-class ways to get schema-conformant JSON out of each of those APIs, which is the whole reason this abstraction is thin enough to be worth having.

The payoff is where the knowledge lives. What a vendor order form looks like is domain logic, and it is written down once, in one prompt and one schema. How to talk to a specific vendor’s HTTP API is plumbing, and it is isolated in one small file per vendor. Adding a fourth provider means one new adapter file and one new entry in that table. The controller does not change.

The config itself (provider, model, encrypted key, base URL) lives in a database row keyed by organisation, not in a repo-wide environment variable. There is no shared fallback key, on purpose. A misconfigured org gets a clear error instead of quietly billing somebody else’s account.

What actually happens in a request

The endpoint is one POST. Underneath, it is a chain where each link exists to contain a specific failure.

1. Session check. Runs before anything else, because every step after this one assumes the caller is who the token says they are. For enterprise users, it also confirms the token belongs to the session that is currently active, so a stale token from an old device stops working.

2. Read the auth context. Organisation ID, user ID, and enterprise ID come off the decoded token. Missing organisation means a fast 401, because every database query below is scoped by it.

3. Validate the image URL. This one is load-bearing. We check that the URL is present, valid, uses HTTPS, and does not point to a private or internal host.

Why so paranoid? Because the server fetches that URL in the next step. Without this check, any authenticated caller could hand us a URL pointing at our own internal network and use our backend as a probe. That is server-side request forgery, and this is the only place in the feature where a caller-supplied URL gets fetched, so this is the only place that needed the guard.

4. Load the org’s LLM config and decrypt the API key in memory only.

5. Prepare the image. Downloaded, then re encoded with sharp down to a maximum dimension of 1536 pixels at quality 70. Phone photos are routinely tens of megabytes, and providers bill and rate limit by payload and tokens. This bounds both cost and latency, and normalises the format so every adapter can embed it.

6. Build and send the provider request with a 60-second timeout.

7. Log the call. Both the success and the failure path record usage before touching the result. More on why below.

8. Parse the response. A provider that returned an HTTP error and a provider that returned a 200 containing something that is not valid JSON are two different problems, so they get two different error codes. Future you, reading logs at midnight, will care about this distinction.

9. Resolve the tenant schema. We run one Postgres schema per enterprise, so every master data query from here has to be schema qualified. If the schema cannot be resolved, extraction stops. It does not fall back to a default, because silently querying the wrong client’s data is far worse than an error message.

10. Match everything against master data.

11. Return. Every line item carries both the raw extracted text and a match object per field.

The part that makes it trustworthy: verbatim first, matching second

Here is the decision I would defend hardest.

The prompt tells the model to copy what is on the form exactly. Do not normalise. Do not fix spelling. Never guess or invent a value.

That sounds like we are throwing away a free upgrade. Models are good at guessing that “Rs Brothrs” means “RS Brothers”. Why not let it?

Because a silent correction is unfalsifiable. Once the model quietly cleans a value, nobody downstream can tell the difference between what the vendor wrote and what the model decided the vendor meant. The one thing a data entry replacement absolutely must preserve is the ability to check its work against the original.

So correction is a separate, visible step. Verbatim text goes into a matching layer that reconciles it against real system codes using Postgres trigram similarity from the pg_trgm extension, which gives us a similarity() score plus a distance operator built for exactly this kind of near-miss text.

Two thresholds, not one:

AUTO_MATCH_THRESHOLD = 0.55   confident enough to fill in
SHORTLIST_THRESHOLD  = 0.3    not confident, but worth showing
SHORTLIST_SIZE       = 5      how many candidates the user sees
CANDIDATE_POOL_SIZE  = 50     rows pulled before scoring

A single cutoff forces a bad trade. Set it low, and you auto-accept wrong matches. Set it high, and you dump every slightly imperfect but correct match onto a human. Splitting it gives the interface three honest states instead of a binary hit or miss:

• matched → fill it in
• ambiguous → show a short picklist and let the user choose
• no_match → fall back to the raw text the vendor wrote

The raw text is never discarded, even on a confident match. What we return is a suggestion, not a verdict.

CANDIDATE_POOL_SIZE is pure query economics. Ordering by trigram distance with a limit of 50 lets the index cheaply narrow a large table first, so the more expensive scoring only runs over 50 rows instead of all of them. (Worth knowing if you copy this pattern: distance ordering of that kind is supported by GiST trigram indexes, so check which index type you actually created.)

Line items are matched in batches of ten, in parallel inside each batch. A form with forty rows and six matchable fields per row is a lot of queries to fire at once, and batching caps how many are in flight without making the whole thing serial.

The tenant-specific wrinkle

Vendors and articles were easy. They live in fixed tables with fixed column names.

Design, color, and size were not. Those values all sit in one shared description table, filtered by a category key, and that key is different for every client, because every client defines its own category taxonomy.

So before matching a design value, we first ask: what key means “design” for this organisation? That answer comes from a config row, and it gets cached in Redis, because it changes approximately never but would otherwise be looked up once per line item per field.

Redis here is opportunistic. If it is down or empty, the lookup falls through to the database. A cache outage should slow the feature down, not break it.

Every call gets a receipt

One row per model call: provider, model, token counts, HTTP status, latency, error message, outcome. One row per config change: who changed it, what the provider and model were before and after, and only the last four characters of the API key. Never the full key, not even in an audit table.

This exists because model calls are the one part of the feature that costs real money per invocation and can fail for reasons that have nothing to do with our code. Provider outage. Blown quota. A key that got rotated on their side and not ours. Without the log, “why did extraction fail for this client yesterday?” has no answer at all.

And the logging is deliberately not awaited in the response path. It fires, and a failure gets caught and printed. An audit trail that can take down the feature it audits is worse than no audit trail.

Where we deliberately stop

This is the boundary I like most about the design.

The feature does not create the Proforma Invoice. It does not generate a PDF. It returns structured, matched data and stops. Document creation and rendering already existed as their own pipeline, and it stayed that way.

It also does not correct anything, as covered above. And it does not assume a single AI vendor.

Three deliberate non-goals, and each one keeps the feature small enough to reason about.

A few things that transfer

If you are building something similar, these are the parts I would carry over:

• A model call is one step, not the product. The extraction is maybe a third of this feature. Validation, matching, tenant resolution, logging, and error taxonomy are the rest, and they are what make the output usable.
• Never let the model silently correct. Extract literally, correct visibly, and always keep the original.
• Two thresholds beat one. “I am not sure; here are five options” is a genuinely useful answer, and a single cutoff cannot express it.
• If your server fetches a caller-supplied URL, you have an SSRF problem whether or not you have thought about it yet. A hostname and IP block check is the floor, not the ceiling, since a public name can still resolve inward.
• Log the expensive, external, flaky thing. Then make sure the logging cannot break the request.

The unglamorous truth is that the model was the easy part. Deciding how much to trust it took considerably longer.