Rules, NLP, machine learning or LLMs: how to identify patients in EHR data

Most teams don't choose a phenotyping algorithm; they inherit one. The cohort somebody built from an ICD-10 pull goes up on the screen in a review meeting, a clinician starts scanning the patient list, and a third of the "diabetics" turn out to be rule-out visits and coding artifacts. Somewhere in the silence that follows, the analytics lead realizes nobody ever actually chose this method. Our primer on computational phenotyping (Part 1) covers why defining cohorts computationally matters in the first place. This post assumes you're past why and stuck on how: which of the four main EHR phenotyping methods fits your condition, your data, and your regulatory posture.

Our short answer, before the taxonomy: start rule-based unless you have a specific reason not to, add NLP when the signal lives in notes, reach for ML when rules plateau, and use LLMs to speed up the work rather than to replace the validation. The rest of this post is the reasoning behind that answer.

What are the four main EHR phenotyping methods?

Every phenotyping algorithm in production today is some combination of four approaches: rule-based logic over structured data, NLP over clinical notes, supervised or weakly-supervised machine learning, and LLM-assisted phenotyping. The research literature usually collapses these into two buckets, rule-based versus machine-learned; we split them four ways because each one carries a different price for the thing a deploying team cares about. Head-to-head studies show real accuracy gaps, especially in sensitivity, but the differences that usually decide the choice are what each costs you to build, explain, and defend.

Rule-based phenotyping is deterministic logic over structured fields: ICD/CPT codes, medications, lab values, encounter types. "Two outpatient ICD codes for type 2 diabetes at least 30 days apart, OR one code plus a glucose-lowering prescription, OR an HbA1c ≥ 6.5%, excluding gestational diabetes codes" is a classic pattern, though real validated definitions add guards this simplified example omits (metformin, for instance, gets special handling because it's also prescribed for prediabetes and PCOS). The academic research networks maintain public libraries of these validated definitions you can adapt rather than invent. Rules are transparent, auditable line by line, and cheap to run.

But a well-validated definition takes clinician-informaticist iteration and chart review to build, and a naive one (a single ICD code) often has a positive predictive value far below what analytics teams assume. Codes are billing artifacts first and clinical truth second.

NLP-based phenotyping goes where structured data can't. Smoking status, ejection fraction, disease severity, symptom onset, family history: much of this exists only in notes, and for some phenotypes the structured record is close to silent. Classical clinical NLP (concept extraction, negation detection, section-aware parsing) plus rules over the extracted concepts remains a workhorse.

The trade-offs: notes are messy, negation and hedging ("no evidence of," "rule out," "family history of") are where naive pipelines die, and every extraction step adds error you must measure. We've written about where NLP earns its keep in healthcare more broadly; in phenotyping, it's usually a feature-extraction layer feeding one of the other approaches rather than a standalone method. That added error can be contained, too. In our post-acute work we cross-check detections against clinical instruments already in the record: a depression detection is checked against PHQ-9 scores and contradictory evidence is filtered out, not ignored. BIMS scores get their own check when the assessment itself is validated.

Supervised ML phenotyping trains a classifier on labeled patients, and labels usually mean expensive clinician chart review. Done well, ML captures interactions no rule author would write down and typically lifts sensitivity meaningfully at comparable PPV, especially for fuzzy phenotypes (heart failure subtypes, frailty, uncontrolled disease).

The lesson for a product leader is that the labeled-data barrier is much lower than it was five years ago, provided you have the engineering to exploit it. This is the same discipline that underpins any predictive analytics program in healthcare: the model is the small part; the label strategy and validation are the product.

LLM-assisted phenotyping is the newest layer, and the evidence for it is worth taking seriously as long as you read the caveats too. LLMs can read a chart the way an abstractor does: classify whether a note supports a diagnosis, extract criteria, handle negation and temporality that break classical NLP, and adapt to a new phenotype with prompt changes instead of retraining.

How do the four approaches actually compare?

Accuracy alone won't settle the choice. The comparison that matters runs across five trade-offs at once.

The table understates two things. First, explainability isn't academic: when a payer, an IRB, or a regulator asks why patient X is in the cohort, "the model scored her 0.83" is a very different meeting than "she met criteria 2 and 4."

Second, every number in the table is a fact about the site that measured it; performance shifts with local coding practices and documentation habits. Cross-site generalization is its own discipline, and it's the subject of our post on phenotype portability and multi-site validation (Part 3); for now, whatever you build, budget for re-validation anywhere you deploy it.

Which phenotyping approach should you choose?

Choose by situation, not by sophistication. This is the decision framework we use with clients, first as a flowchart, then in detail:

The prose version, with the reasoning:

• If a validated definition already exists (start with public phenotype libraries such as OHDSI's) and your phenotype is visible in coded data, adapt it, chart-review a sample locally, and ship. Many published definitions date from the eMERGE era, so maintain the code sets. This covers more ground than most teams expect, and it's the cheapest credible option.
• If the defining signal lives in notes (severity, status, symptoms, anything narrative), you need an NLP extraction layer feeding rules or a classifier, or manual abstraction. A structured proxy such as a medication or procedure can stand in only if it has been validated on its own.
• When rules exist but plateau, with decent PPV, poor sensitivity, and endless exception lists, that plateau is a reason to consider ML, once you have ruled out missing data, mapping errors and note-locked signal. Label-efficient methods, whether weak supervision such as anchors or unsupervised embeddings such as Phe2vec, should be your default before you commission thousands of chart reviews.
• If you need many phenotypes fast, or your bottleneck is human chart abstraction, LLM assistance is worth piloting: drafting criteria, pre-screening charts, classifying notes for human adjudication. Validate it on local data and keep the human sign-off. The SHREC-style evidence supports assistance, not autonomy.
• If the cohort feeds a regulatory submission, a trial, or a payer contract, favor definitions that are prespecified, traceable, reproducible and locally validated, because you will be asked to defend every inclusion. Rule-based logic is usually the easiest to defend, but validated NLP or ML is appropriate when its inputs, performance, version and decision process are documented.

In practice, don't pick just one. Production phenotyping is usually a layered hybrid: a rule-based backbone for transparency, NLP features for note-locked criteria, label-efficient ML to lift sensitivity, and LLMs that speed up chart abstraction and iteration. That pattern isn't a thought experiment. In our skilled nursing work, about 75 condition scanners in three families run side by side over assessments, diagnoses, notes and vitals: rule-based scanners over diagnosis records and structured assessments for conditions like diabetes, calculation-based scanners for weight loss (a 5% threshold over 30 days, 10% over 180 days, mirroring how CMS defines it), and LLM scanners over progress notes for conditions like depression and falls, using note embeddings for retrieval with an LLM classifier and a confidence threshold behind them. Nobody chose a single algorithm there either, and this time that was the point.

The algorithm choice is also only half the decision. The other half is whether it can survive contact with production (versioned definitions, drift monitoring, a validation trail), which is the same pilot-to-production gap we've written about for healthcare AI generally. A phenotype in a notebook is a paper; a phenotype in production is a system.