Case study · Machine learning in a regulated domain
That was patient intake at a national DME platform — millions of faxed referrals, typed into the system one field at a time. I led the product side of an Intelligent Document Automation platform that reads the fax, extracts the fields, and creates the patient and the order automatically — from problem definition through launch, in a HIPAA-regulated pipeline where a wrong billing field is a denied claim.
Every patient starts with a document — a faxed referral, a prescription, a chart note. On this platform, roughly 4.5 million of them a year arrived as faxes, and medical admins keyed each one in by hand to start the patient and bill the order. Manual intake at that scale isn't just slow — it leaks money and risk from four directions at once:
I saw a workflow that was expensive, error-prone, and perfectly shaped for machine learning: high-volume, structured-enough documents doing a repetitive extraction job. So I took it on — product ownership of an ML platform to automate it, from the problem statement to the launch.
A platform play only gets funded if the money is real. I made the case on four fronts, and the labor math alone carried it.
Onboarding one patient took ~30 min of admin time — roughly $20 a patient. For a supplier onboarding 10,000 patients, that's ~$200K in manual-labor cost, straight off the top.
Automation isn't only a cost cut — it's a product. We could monetize it as a usage-priced add-on: a new revenue line that grows with every document a customer runs through it.
Inaccurate, hand-keyed data drove ~10% of rejected claims. Cleaner extraction protects the supplier's revenue — and reduces the rework and resubmission that eats their staff time.
A competitor was bringing this capability to market. Shipping it protected supplier satisfaction and retention — table stakes we couldn't afford to cede.
Illustrative. The assumptions are 30 minutes of admin time per patient at a loaded cost of about $20. One supplier is a six-figure saving, and the platform multiplies it.
The pipeline takes an unstructured document and turns it into two structured business objects: a patient record and an order. The model does the reading; a confidence threshold decides what a human still touches.
The data scientists built the model. I owned everything that decided whether the model became a product — what it extracts, where a human steps in, what "good enough to launch" means, and the trade-off that governs all of it.
What gets extracted and where each field populates — prioritizing the document types that start a patient and bill the order, because that's where the money and the risk concentrate.
PHI protected, every access audited. I owned the requirement; engineering built the security controls, and we validated them with the compliance team — not an afterthought, a launch gate.
When and where a reviewer steps in, keyed to the confidence threshold — the rule that decides which documents a human still verifies and which flow straight through.
The accuracy bar required to launch, the automation rate, and time saved — defined up front and tracked every release, so "is it working?" had a number, not an opinion.
Bias toward accuracy over automation — the call that shaped the threshold, the review policy, and the launch bar all at once.
"I biased the system toward accuracy over automation. A missed automation just means a human still does that one — a cost. But a wrong billing field is a denied claim — it breaks the customer's revenue and our trust. So I set the confidence bar high and sent anything uncertain to a reviewer. I'd rather automate less and be right."
In a regulated domain the errors are not symmetric, so I spent automation to buy correctness.
Intake time per patient, before and after — an ~80% reduction, with a review step still in the loop.
Five numbers, three jobs: one for model quality, one for business value, and a guardrail that keeps the automation honest.
| Metric | What it tells you | How it's calculated |
|---|---|---|
| Extraction accuracy | Model quality — is the data it pulls correct? correct fields ÷ total fields | Field-level, not document-level: of every field it needed to pull, the share it got right. ~95%. |
| Automation rate | Business value — how much work actually left the humans? auto-processed ÷ total docs | The share of documents processed with no human touch — i.e. everything that cleared the confidence threshold. |
| Error rate (guardrail) | Safety — is the automation trustworthy? errors ÷ auto-processed | The error rate on auto-processed docs only. This is the number that justifies the threshold — push automation too high and this creeps up. |
| Time to patient + PO | The outcome customers feel avg minutes, intake → order | Average time to create a patient record and a purchase order. The 30→5 min headline. |
| Add-on volume | Monetization — is it earning? billable docs × price | Usage of the add-on, charged by volume — ties the platform directly to revenue. |
Automation rate and error rate pull against each other — that tension is the accuracy-over-automation trade-off, made measurable. Raise the threshold and error rate falls but automation rate drops; lower it and you automate more but risk denied claims.
Four things moved at once: labor cost, compliance risk, protected revenue and competitive position. The decision underneath all of them was one threshold: accuracy over automation, with a person on everything uncertain.