PHI in Prompts: The HealthTech AI Mistake That Ends Deals
Last updated October 2026
Sending PHI in LLM prompts is lawful only when a signed business associate agreement covers the exact model endpoint and feature you call, and only the minimum necessary data goes in. Most HealthTech exposure happens after the prompt: in logs, traces, caches, vector stores, eval sets, and agent tool calls that no BAA covers.
Key takeaways
- A BAA with a model vendor covers named products and endpoints, not your whole AI stack. Both OpenAI and Anthropic publish lists of features that are excluded.
- HHS treats a cloud service that stores or processes ePHI as a business associate even if it only holds encrypted data and has no key. That reasoning reaches your observability and eval vendors too.
- The prompt is one hop. A production LLM feature copies the same text into several other systems on your side of the line, and none of them are covered by the model vendor’s BAA.
- Redaction before the model is useful but leaky. A 2026 Baylor and Texas Children’s study found even the best models missed a meaningful share of institution-specific identifiers.
- Enterprise buyers ask for the data flow diagram, not the BAA. If you cannot draw where PHI goes, the security review stalls.
Is it legal to put PHI in LLM prompts?
Yes, under conditions. HIPAA does not ban language models. It requires that any vendor who creates, receives, maintains, or transmits PHI on your behalf is bound by a business associate agreement, and that you disclose only what the task needs.
The HHS Office for Civil Rights spelled out how far that reaches in its guidance on HIPAA and cloud computing. A cloud service provider that handles ePHI is a business associate, and HHS states this is true “even if the CSP processes or stores only encrypted ePHI and lacks an encryption key for the data.” Using such a provider without a BAA puts the covered entity or business associate “in violation of the HIPAA Rules.” The conduit exception, which engineers sometimes lean on, is “limited to transmission-only services.” An LLM API that stores prompts for abuse monitoring, or a tracing tool that keeps them for 30 days, is not a conduit.
So the legal question for PHI in LLM prompts is rarely “may we use a model.” It is “which of the systems this text touches has signed a BAA, and for which features.”
What does a model vendor’s BAA actually cover?
Less than most product teams assume. The two largest API vendors both scope their BAAs to specific products, endpoints, and features, and both say so in writing.
OpenAI’s help center page on HIPAA eligible products and functionality lists the API as eligible only “with Modified Retention,” and says the organization must be provisioned with Modified Retention in addition to the executed BAA. In ChatGPT, it names features where PHI “should not be entered,” including improved memory, event-triggered scheduled tasks, and Codex in the cloud. It adds that “a feature’s availability does not, by itself, determine whether a particular use or connected third-party service is covered.”
Anthropic’s API and data retention documentation publishes a feature eligibility table for HIPAA-enabled organizations. Batch processing, the Files API, code execution, and the MCP connector are listed as not eligible, and beta features are “generally not covered under the BAA unless explicitly listed.” One line is easy to miss: for structured outputs, “PHI must not be included in JSON schema definitions.” A schema with an enum of real patient names or an example value copied from production breaks that rule even though the call itself is eligible.
The engineering consequence: your BAA is a list of permitted code paths. A developer who switches a nightly summarization job from the synchronous endpoint to a batch endpoint to save money may have just moved PHI out of coverage, and nothing in your CI will notice.
Where does PHI in LLM prompts actually end up?
This is the part the ranking guides skip. They stop at the vendor boundary. In a production feature, the prompt text is copied into many systems on your side of the line, and each is either covered, scrubbed, or a finding.
| Where the prompt text lands | Why it is easy to miss | Control that holds up in review |
|---|---|---|
| 1. Application logs | Default debug logging prints the full request body, including the assembled prompt. | Log request IDs and token counts only. Fail the build on log statements that serialize prompt objects. |
| 2. LLM tracing and observability tools | Tracing SDKs capture full inputs and outputs by default so engineers can debug. | Either sign a BAA with the tracing vendor or self-host, and mask fields before export. |
| 3. Error trackers | An exception thrown mid-call attaches local variables, which include the prompt. | Scrub rules on the error tracker, tested with a synthetic PHI string. |
| 4. Response and semantic caches | A cache keyed on the prompt stores PHI in Redis or a vector index with no expiry. | Short TTLs, encryption at rest, and the cache store listed in your risk analysis. |
| 5. Vector stores for RAG | Chunks of clinical notes are embedded and stored next to the original text. | Treat the index as a PHI datastore: access control per tenant, deletion that propagates. |
| 6. Eval and regression datasets | Engineers copy “interesting” production conversations into a test set in a spreadsheet or repo. | Evals built from synthetic or de-identified records only, with the de-identification method recorded. |
| 7. Agent tool calls | An agent passes patient context to a search tool, a calendar API, or an MCP server. | Allowlist tools per data class. Tools without a BAA never receive PHI-bearing arguments. |
| 8. Analytics and product telemetry | Front-end events capture the text a user typed into a chat box. | Block free-text fields from analytics payloads at the SDK level. |
Rows 1, 2, and 6 are the easiest to create by accident, because they come from engineers doing reasonable debugging work, not from a design decision anyone signed off on. AI coding assistants make this worse in a specific way: asked to “add logging so I can debug this,” they will log the whole payload. We cover the assistant side of that problem in can you use AI coding assistants on HIPAA-regulated code.
Can you redact PHI before it reaches the model?
You can and often should, but treat redaction as a layer, not a guarantee. HIPAA’s Safe Harbor method requires removing 18 categories of identifiers, and the hard ones are not names and dates. They are free-text references that only identify someone in context.
A study posted to arXiv in August 2026 by researchers at Baylor College of Medicine and Texas Children’s Hospital, Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss, tested de-identification on 100 pediatric oncology notes. The purpose-built Stanford TiDE tool recalled only 15.7 percent of spans in the “other unique identifier” category. The strongest model tested reached an F1 of 0.918 against TiDE’s 0.779, yet in its baseline run 29 percent of its misses were institution-specific identifiers such as facility abbreviations and building names. Naming those categories in the prompt recovered 79 percent of them. The authors also found the human gold standard itself was incomplete. It is a single-institution study, but the lesson travels: generic redaction misses your organization’s own vocabulary.
Practical rules that follow:
- Minimize before you redact. The cheapest PHI to protect is the PHI you never put in the prompt. Pass a visit ID and fetch only the fields the task needs.
- Tune redaction to your institution. Feed your redactor the clinic names, unit codes, and internal ID formats your notes actually contain.
- Keep the covered path anyway. Redact, then still send to a BAA-covered endpoint. Redaction lowers the blast radius of a miss. It does not make an uncovered vendor acceptable.
- Re-identify server side. If the response needs names, map placeholders back inside your own boundary, after the model call.
Why does PHI in LLM prompts end enterprise deals?
Because hospital and payer security teams do not grade the BAA. They grade the data flow. A vendor security review for an AI feature typically asks where prompts are stored, for how long, who can read them, which subprocessors see them, and whether customer data is used for evaluation or training. An answer of “we have a BAA with our model provider” covers one box on that diagram.
When the honest answer to “where are prompts stored” includes a tracing SaaS on a standard plan, or an eval spreadsheet in a shared drive, the review does not fail loudly. It pauses for remediation, and a pause in a procurement cycle is often a lost quarter. The fix is usually a few weeks of engineering. The cost is the timing.
The deeper fundamentals, including risk analysis and audit logging, are in our pillar guide to HIPAA-compliant software development. For the AI path specifically, these are the five artifacts worth having ready before the questionnaire arrives:
- A data flow diagram for every LLM feature, showing each store and vendor from the table above.
- A list of model endpoints and features in use, mapped to the vendor’s published eligibility list.
- BAAs, or a documented reason none is needed, for every observability, eval, and vector database vendor.
- Retention settings for each store, with evidence that deletion works.
- A test that fails CI when a synthetic PHI string appears in logs or traces.
Frequently asked questions
Is it a HIPAA violation to put PHI in ChatGPT?
Entering PHI into a consumer AI tool with no BAA is a disclosure to a vendor that is not bound as a business associate. OpenAI offers BAAs only for specific products, such as the API with Modified Retention and ChatGPT for Healthcare, and lists features where PHI should not be entered.
Does a BAA with the model vendor cover my logs and traces?
No. A model vendor’s BAA covers data the vendor processes. Your logging, tracing, error tracking, caches, and eval tooling are separate systems, and HHS guidance treats any cloud service that stores ePHI as a business associate needing its own BAA.
Are embeddings of clinical notes PHI?
Treat them as PHI. Most retrieval setups store the source text beside the embedding, so the index contains the note itself. Apply the same access control, retention, and deletion rules as the source record.
How TopDevz handles PHI in AI features
When we build AI features for regulated clients in our $24,500 AI Integration Sprint, the data flow diagram and endpoint eligibility map are delivered before any prompt is written, and the final milestone is waived if delivery misses day 30. For teams whose AI feature is already live, the $4,950 Rescue Audit traces every place prompt text lands and is credited toward remediation. The engineers doing that work average more than 10 years of experience.
Book a Code Review
If a health system’s security questionnaire is sitting in your inbox and you are not sure where your prompts end up, a senior engineer can trace the path through your code and tell you what to fix first. Book a Code Review.