Use Safe Harbor when you can remove all 18 identifier types and structured fields dominate. Use Expert Determination when you need free-text notes, dates or geography for the use case. For real-time LLM calls under a BAA, de-identification is not legally required, but tokenizing identifiers before the model is still the cheapest way to cut exposure.
Teams building with LLMs hit de-identification from two directions. Some want to send clinical text to a model or tool that has no BAA. Others want to build training, evaluation or analytics datasets from notes. Both run into the same problem: HIPAA's de-identification standard was written for tables, and clinical text is not a table. This guide explains the two legal methods, why free text breaks naive approaches, how a tokenization and re-identification pipeline works, and how the main tools compare. Checked against primary sources on 12 November 2026.
What is the difference between Safe Harbor and Expert Determination?
Safe Harbor is a checklist: remove 18 kinds of identifiers and have no actual knowledge that what remains could identify someone. Expert Determination is a risk analysis: a qualified expert documents that the risk of re-identification is very small, using statistical methods. Both are in 45 CFR 164.514(b).
| Safe Harbor | Expert Determination | |
|---|---|---|
| Test | Remove 18 identifier types; no actual knowledge of re-identification | Expert concludes risk is very small, alone or combined with other available data |
| Dates | All date elements except year removed; ages over 89 grouped as 90+ | Shifted or generalized dates can be allowed if risk stays very small |
| Geography | Nothing below state, except first 3 ZIP digits if that area has more than 20,000 people | Can keep finer geography with controls |
| Free text | Every identifier in every note must go, including "any other unique identifying characteristic" | Expert assesses residual risk of the text corpus and recipient environment |
| Documentation | Your process and QA | Expert's written methods and results |
| Cost and time | Lower; can be automated with QA | Expert fees and weeks of analysis; renew when data or use changes |
| Best for | Structured extracts, short texts | Large note corpora, timelines, model training data |
The 18 Safe Harbor identifiers are: names; geography below state; dates (except year) and ages over 89; phone and fax numbers; email; Social Security numbers; medical record numbers; health plan beneficiary numbers; account numbers; certificate and license numbers; vehicle identifiers; device identifiers; URLs; IP addresses; biometric identifiers; full-face photos; and any other unique identifying number, characteristic or code. They apply to the patient and to relatives, employers and household members.
Do you need to de-identify PHI before sending it to an LLM?
Not if the model provider is your business associate under a signed BAA and you use only covered services. You do need to de-identify, or not send the data at all, when the destination has no BAA: a model endpoint outside the covered scope, an evaluation platform, a labeling vendor, or a developer laptop.
Even under a BAA, the minimum necessary standard in 45 CFR 164.502(b) still applies. A model that summarizes a lab trend does not need the patient's name, address or MRN. Tokenizing those before the call and restoring them after costs a few milliseconds and removes most of the damage if prompts or logs leak. Note that tokenized text is not automatically de-identified: it only counts as de-identified if what remains meets Safe Harbor or an Expert Determination. Treat tokenization as a security control first.
Check the provider's own rules too. OpenAI's July 2026 HIPAA implementation guide, for example, says customers may not use its services to maintain PHI as part of a designated record set. That shapes where your source of truth lives. We compare providers in which LLM providers sign a BAA in 2026.
Why is free text so hard to de-identify?
Because identifiers in notes are unbounded and contextual. A note can name a relative, an employer, a small town, a rare job or a news event, and Safe Harbor's eighteenth category covers "any other unique identifying characteristic". Automated detectors miss some of these, and a few misses across millions of notes is a lot of exposure.
Quasi-identifiers combine
The classic result is Latanya Sweeney's finding that 87% of the US population was likely uniquely identifiable from 5-digit ZIP, gender and full date of birth. Notes are full of such combinations: "62-year-old retired firefighter from [town] with a lung transplant". No single token is a direct identifier, but together they may be.
Detectors are probabilistic
Every tool says so. AWS's Comprehend Medical DetectPHI documentation notes that its entities do not map 1:1 to the Safe Harbor list and recommends human review or other checks for compliance use cases. Presidio states plainly that there is no guarantee it will find all sensitive information. Plan for misses.
"Actual knowledge" bites in context
Safe Harbor fails if you actually know the remaining data could identify someone. A case that was in the news, a celebrity patient, or a one-of-a-kind procedure at a small hospital can fail the test even after the 18 categories are removed.
LLMs make re-identification easier
A capable model can link scattered hints across a long record faster than a person can. That is a reason to be more conservative with free text that will be shared, and a reason Expert Determination analyses increasingly consider what a recipient could do with modern models.
How does a tokenization and re-identification pipeline work for LLMs?
Detect identifiers, replace each with a consistent placeholder or realistic surrogate, keep the mapping in a separate secured vault, send only the transformed text to the model, then restore identifiers in the output for authorized users. The mapping key must never reach the model or logs.
- Detect. Run structured rules (MRN formats, phone regexes, known patient demographics from the EHR) plus an NER model. Using the known demographics of the patient in context catches far more than generic NER alone.
- Transform. Replace with typed placeholders ("[PATIENT_1]", "[DATE_3]") or surrogates. Surrogates read naturally and hide misses better; Microsoft's de-identification service documentation recommends surrogation for that reason. Keep replacements consistent within a document so the model can follow who is who.
- Vault. Store the mapping encrypted, per request or per dataset, with a short lifetime for inference. Under 164.514(c), a re-identification code must not be derived from the individual's information, and the mechanism must not be disclosed or used for other purposes.
- Infer. The model sees only transformed text. Instruct it to keep placeholders verbatim, and validate that the output contains no new identifiers.
- Restore. Re-insert real values only for the authorized user's view. Log the restore event.
Two traps. Date shifting keeps intervals useful but is not Safe Harbor, because dates other than year must be removed; it can work under Expert Determination. And the output can leak too: if a model paraphrases "[PATIENT_1]'s daughter, a nurse at the county hospital", your restore step must not be the only check. Scan outputs as well as inputs. We cover the surrounding controls in our HIPAA-compliant AI agent architecture.
Which de-identification tools should you compare?
Shortlist by where your data lives. Presidio if you want open source you can run anywhere and customize. AWS Comprehend Medical or Azure Health Data Services if your stack is on that cloud and you want a managed, HIPAA-eligible service. Test all of them on your own notes; published accuracy does not transfer between note styles.
| Tool | Type | Operations | Notes |
|---|---|---|---|
| Presidio | Open source; originally from Microsoft, now community-maintained under Data Privacy Stack | Detect (analyzer), anonymize, image redaction, structured data | Fully customizable recognizers; you host it; states no guarantee of finding everything |
| AWS Comprehend Medical (DetectPHI) | Managed API | Detects PHI entities with confidence scores | Detection only; you build the replacement step; entities not 1:1 with Safe Harbor list |
| Azure Health Data Services de-identification | Managed API | Tag, redact, surrogate; 27 entity types incl. the 18 HIPAA identifiers | Consistent surrogates within a batch job; 50 KB per request limit; billed per MB |
| General-purpose LLM as detector | Prompted model | Detect or rewrite | Good at context, unpredictable at recall; needs the same evaluation and a BAA if it sees PHI |
Details come from Microsoft's de-identification service overview, the AWS and Presidio pages linked above. In practice, the strongest pipelines combine a deterministic layer seeded with the patient's known demographics, a managed or open-source NER model, and a final scan of the output.
How do you measure whether de-identification is good enough?
Measure recall per identifier type on an annotated sample of your own notes, because a missed name matters more than an over-redacted drug. Report misses per 1,000 notes, review every miss by hand, and repeat whenever note templates, sources or tools change.
- Gold set. Two annotators mark every identifier in a few hundred notes drawn from each source system and note type.
- Recall first. Track recall separately for names, dates, locations, IDs and "other". Precision matters less for privacy but affects usefulness.
- Leak audit. Look at what was missed. A missed street name in a social history is different from a missed MRN.
- Downstream utility. Check that the de-identified text still works for the task, such as summarization quality or extraction accuracy.
These are the same evaluation habits we recommend for any model in a regulated setting; see LLM testing for regulated industries.
When should you pay for Expert Determination?
When you will share or retain large free-text datasets, need dates or fine geography, plan to train or fine-tune a model on clinical text, or will release data outside your organization. For transient inference under a BAA, tokenization plus minimum necessary is usually enough.
Engage the expert early. They will ask who receives the data, what controls the recipient has, what other data the recipient could link to, and how you will monitor. Their answers often change the pipeline design, for example keeping data in a controlled environment rather than shipping a file. Document the determination, its scope and its expiry, and treat a new data source or use case as a reason to revisit it.
How BeevR builds this
We build de-identification and tokenization layers as part of healthcare AI systems: a deterministic layer seeded from EHR demographics, a managed or open-source detector, consistent surrogates, an encrypted short-lived vault, output scanning, and a gold-set evaluation that runs in CI so a template change cannot quietly drop recall. For training or analytics datasets we work alongside your chosen de-identification expert rather than claiming a determination ourselves. The model can run inside your own AWS account under your AWS BAA, and BeevR signs a BAA as your business associate.
We are a senior, founder-led team in Hanoi, rated 5.0 on Clutch, and we built a multi-tenant EHR now live across four US clinics at about 300 patients a day. We work in fixed phases, from $10K per phase, and you own the code from day one. See our healthcare AI agent development and how we approach security.
If you are deciding how much PHI your LLM pipeline should see, book a 30-minute call. Bring a sample note format and your target use case, and we will sketch the pipeline and its weak points.