Why patient-finding algorithms break at the next hospital, and where OMOP helps

You built a phenotyping algorithm that finds your target patients beautifully. Chart review at the pilot site confirmed it: PPV north of 90%, clinicians nodding along. Then you deployed it at the second site and the cohort made no sense. Half the patients you expected were missing, and a chunk of the ones it found didn't have the condition at all.

The algorithm didn't break. It's doing exactly what it did at site one, on data that no longer behaves the same way, and nobody budgeted for that. (Our primer covers how phenotypes turn EHR data into patient cohorts (Part 1); this post deals with the harder problem of making one survive contact with a second health system.)

Why does a phenotyping algorithm degrade at a new site?

Because a phenotype is never pure clinical logic. It's clinical logic entangled with one institution's coding habits, documentation culture, lab conventions, and patient population. Move the logic without the entanglements and performance drops. Four mechanisms do most of the damage:

Coding variability. Two hospitals seeing identical patients produce different code trails. One codes uncontrolled diabetes aggressively because its billing team optimized for it; another leans on unspecified codes. One maintains its problem list; another lets it go stale and buries truth in encounter diagnoses. A rule keyed to "two outpatient E11.x (type 2 diabetes) codes plus a medication order" inherits every one of those habits.

Different EHR vendors, different data shapes. Epic, Oracle Health, Meditech, and athenahealth store data differently, and they generate different data too. Note types, flowsheets, and where a smoking status or ejection fraction lives all vary. The eMERGE network's NLP portability work found that simply identifying which documents to process was a per-site project requiring manual mapping of local note-type taxonomies. One site discovered its relevant dermatology records existed only on paper.

Missing data patterns. Patients get care in more than one place, and each site sees only its slice. A Mayo Clinic-linked study of care fragmentation found phenotypes for rheumatoid arthritis and type 2 diabetes carried false-negative rates of 26.6% and 14%: patients whose diagnoses lived in records the algorithm couldn't see. Different markets have structurally different missingness, and your sensitivity moves with it.

Population differences. Prevalence, acuity, referral patterns, demographics. Fixed thresholds calibrated on an academic medical center's referral-skewed population behave differently at a community hospital even when every code maps perfectly. PPV is prevalence-sensitive, so the same specificity yields more false positives where the condition is rarer.

Some algorithm styles also travel better than others. Transparent rule-based definitions can be inspected and re-mapped at a new site; models trained on one site's raw feature distributions tend to memorize local quirks. We cover the trade-offs in our phenotyping algorithm approaches post (Part 2); portability should inform which approach you pick in the first place.

How do you validate a phenotyping algorithm at each new site?

You re-measure it, every site, against clinician chart review. The pilot site's performance numbers are facts about the pilot site, not about the algorithm. The practice that networks like eMERGE settled on is straightforward and worth copying:

1. Run the algorithm at the new site and pull the candidate cohort.
2. Sample for chart review. A random sample of algorithm-flagged patients for PPV (roughly 50–100 charts is a common starting point, sized up for the precision you need), plus a sample of algorithm-negatives to estimate sensitivity: either representative, or enriched for likely cases with the sampling scheme accounted for in the estimate.
3. Have clinicians adjudicate against a written gold-standard rubric: the same one at every site, agreed before anyone opens a chart, with inter-reviewer agreement checked on an overlap set.
4. Compute PPV and sensitivity per site against pre-declared acceptance thresholds tied to the use case. The bar depends on what a false positive or a false negative costs in that workflow, and it is set before anyone sees results.
5. Diagnose the misses. Every false positive and false negative is a message about local data. Most transport failures fall into actionable categories: a missing mapping, unavailable data, an excluded note type, local coding behavior, temporal logic, or definition ambiguity, and classifying each one keeps the same failure from recurring unnoticed. The eMERGE experience is blunt about where the time goes. Data preparation, not algorithm logic, was consistently the most labor-intensive step.

Even "easy" phenotypes wobble enough to justify this. A multisite validation of a deliberately simple obstructive sleep apnea definition hit ≥90% PPV and NPV everywhere except one site, where PPV came in at 87.5%. You only find that site by measuring at that site. Chart review is clinician time, per site, indefinitely, so treat multi-site validation as a standing line item rather than a launch task.

What do PheKB and eMERGE teach about portable phenotype definitions?

Portability is achievable, but only when it's designed in, and even then there's work at the receiving end. The eMERGE network spent over a decade moving phenotypes between institutions and captured the results in PheKB, a public knowledgebase of algorithms built explicitly for transportability. Published PheKB algorithms hold up well across sites: median case PPV about 96% at authoring institutions and slightly higher, 97.5%, at implementing sites. Read those numbers for what they are. They were measured after each implementing site did the local mapping and tuning this section describes, and algorithms that fail portability tend not to reach publication. Those figures describe the survivors, post-adaptation, not out-of-the-box transport.

The durable lessons:

• Ship definitions against standard vocabularies, using the terminology for each domain (SNOMED CT for clinical concepts, LOINC for observations, RxNorm for medications), never local codes or one vendor's identifiers.
• Publish the full logic, human-readable, including intent, so an implementing site can see why a criterion exists and adapt the how without breaking it.
• Expect a local mapping and tuning phase. Every eMERGE implementation involved site informaticians mapping local data to the definition's inputs; the BPH case study shows even executable standards still lose sites to terminology gaps and local quirks.
• NLP components port worst. Note formats, section headers, and document taxonomies are the least standardized layer of the EHR, and small things break them. One eMERGE module misread "atrial fibrillation/flutter is no longer present" as a positive case.

If eMERGE's well-funded academic sites needed dedicated informatics effort per implementation, a digital health company onboarding its fourth health system should assume the same and engineer for it.

Does data normalization with OMOP or FHIR solve phenotype portability?

It solves the biggest, most mechanical part of the problem and leaves a residue you still have to manage. Normalizing each site's data into a common data model like OMOP CDM, or consuming it through FHIR, means phenotype logic is written once against standard vocabularies and stable structures instead of rewritten per EHR. That's the engineering answer to the "different vendors, different shapes" problem, and it's why an OHDSI network study can send one cohort definition, unchanged, to dozens of participating databases drawn from a network of hundreds. It's the same architectural argument we make about interoperability in healthcare generally: standardize at the data layer so the application layer stops absorbing every site's idiosyncrasies.

What normalization doesn't fix: it standardizes representation, not behavior. If a hospital never coded something, OMOP faithfully represents the absence. Coding culture, care fragmentation, and population shift pass straight through a perfectly mapped CDM; the OHDSI cross-site study above degraded within a shared common data model.

The ETL also becomes a new thing to validate, because a lab mapped to the wrong LOINC concept fails silently, at scale. Treat per-site ETL with the same rigor as the phenotype: data-quality checks, mapping coverage reports, human review of the unmapped local codes that matter to the phenotype. (For provider leaders earlier in this process, the groundwork is getting the EHR foundation itself into defensible shape first.)

Practical guidance we give clients: OMOP when the workload is analytic and cohort-scale; FHIR when you need near-real-time, patient-level triggers inside clinical workflows. Mature multi-site systems frequently need both.

How do you engineer a phenotyping system for multi-site deployment from day one?

Assume site two exists before you finish site one, and let that assumption shape the architecture. Concretely:

Separate the definition from the plumbing. The clinical logic (concept sets, temporal criteria, thresholds) should live as versioned configuration against standard vocabularies, not as code interwoven with one site's schema. New site, new config; same engine.

Make site adaptation config-driven. Local code mappings, note-type mappings, threshold calibrations, and known data gaps belong in a per-site configuration layer that's reviewable and diffable, so "tuning for Site 4" is a pull request with an informaticist or clinician signing off, not a fork. Anything that changes the phenotype's clinical meaning is a new definition version, not local configuration. (What that review roster owes a compliance audit is post 5's territory (Part 5).) The OHDSI lesson applies here too: what you replicate at each site is the recipe.

Build validation in as a product feature. Ship the chart-review sampling, the adjudication workflow, and the per-site PPV/sensitivity dashboard as part of the system. If validating a new site takes a bespoke research project, it won't happen at site six.

Monitor for drift after go-live. Portability isn't a launch-day property. EHR upgrades, coding-policy changes, and a new documentation template can degrade a validated phenotype months later. Watch cohort size, input frequencies, missingness, and feature availability per site; alert on meaningful shifts; triage each one, since a cohort can grow because flu season arrived, not because the pipeline broke; and let unexplained drift trigger targeted re-validation. This is the same discipline that separates demos from deployed systems in healthcare AI generally, where most of the work starts after the thing first works.

Budget the human loop. Per-site clinician chart review, a named owner for mapping decisions, a feedback path from false positives back into the config. The teams that scale phenotyping across ten sites are the ones who industrialized the boring parts, not the ones with the cleverest model.

Put together, the multi-site lifecycle is a loop rather than a pipeline; drift sends a site back to re-validation, not to square one:

That, candidly, is the difference between a phenotype that produced one good pilot and a phenotyping system a company can sell to its next twelve customers. We build the second kind, and most of the engineering is exactly what's above: normalization pipelines, config-driven onboarding, embedded validation, pre-committed re-validation triggers. None of it is glamorous, and all of it is what gives site two a real chance of working.