Do I need a BAA with my AI provider?

Yes, if it processes PHI on your behalf it is a Business Associate and needs a signed BAA. Confirm the provider offers one and that your exact tier and endpoint is covered.

Can I use consumer AI tools like ChatGPT with patient data?

Not the consumer versions. HIPAA-eligible use generally requires a business or cloud tier the provider will sign a BAA for, plus settings that stop retention and training. As an example, one major model provider signs a BAA only for its API and Enterprise plans, not its consumer or console tiers, and vendor terms change often, so verify the current terms for your endpoint.

Is "zero data retention" required for HIPAA?

No. HIPAA requires a BAA and the Security Rule safeguards, not zero retention. Some HIPAA-covered configurations actually mandate a short retention window, and some clouds retain a sample for abuse monitoring by default. Know the retention behavior of the exact service you use.

Are AI prompts and outputs considered PHI?

Yes. A prompt containing a patient’s data and the model’s response are PHI. Encrypt, access-control, retain, and audit them the same way you would the record.

Does HIPAA require de-identifying data before sending it to AI?

HIPAA requires the minimum necessary, not always full de-identification. When you do de-identify, use a real method (Safe Harbor or Expert Determination). Be aware that de-identified data can sometimes be re-identified, and AI increases that risk.

Does HIPAA cover whether the AI’s clinical output is safe or accurate?

No. HIPAA governs the privacy and security of patient data, not clinical safety. AI that influences diagnosis or treatment can fall under FDA Software-as-a-Medical-Device rules, which are separate.

What is changing with the 2025 HIPAA Security Rule update?

HHS OCR proposed a rule in January 2025 that would make encryption of ePHI mandatory, remove the addressable category so nearly all safeguards are required, define multi-factor authentication, and require Business Associates to verify their safeguards annually. It is proposed, not yet final, but designing to meet it is prudent.

Reference

The HIPAA compliance checklist for building AI into healthcare software

A practical checklist for teams adding AI features to software that handles patient data. Open, no signup.

Alex Pavlov

CEO, HighCraft.io · Updated 2026-07-04

Most HIPAA problems with AI features do not come from the model. They come from the plumbing around it: what data reaches the model, who signed what agreement, and what gets logged on the way. If your software handles Protected Health Information (PHI) and you are adding an AI feature such as lab-result analysis, a clinical assistant, a chatbot, or note generation, every item below should be true before it touches a real patient record. This is the HIPAA compliance checklist we run ourselves.

First, know your role. The provider who holds the patient relationship is the Covered Entity. The platform handling PHI on their behalf is a Business Associate, and any development shop or subprocessor under it is a subcontractor. Under HITECH, Business Associates and their subcontractors are directly liable to HHS OCR for the Security Rule, breach notification, minimum-necessary, and the BAA chain, whatever the contract says.

Before any PHI reaches a model

You have a signed BAA with the AI provider, and your exact tier and endpoint is covered. A BAA-covered cloud does not make every service under it HIPAA-eligible. Confirm the specific endpoint you call is on the provider’s eligible list and that your plan tier is covered (§164.502(e), §164.308(b)).
You have checked the default retention and abuse-monitoring behavior. Covered by a BAA is not the same as zero retention. Some HIPAA-covered configurations mandate a retention window, and some platforms retain a sample of prompts and completions for abuse monitoring by default unless you request the modified setting. Read the data terms for the exact endpoint you use.
The provider does not train on your data outside BAA terms. Using PHI to train or improve a model beyond what the BAA permits is an impermissible use with direct OCR liability (§164.502(a)(3)). Confirm the no-train commitment in writing and enable it where it is a setting.
You send the minimum necessary in every prompt. Pass only the PHI the task needs (§164.514(d)). A lab-analysis prompt rarely needs the patient name, address, or full chart.
You de-identify when the feature allows it, using a real method. HIPAA recognizes two: Safe Harbor (remove the 18 identifier categories and have no actual knowledge the remainder could re-identify) or Expert Determination (a qualified expert documents a very small re-identification risk), §164.514(b). Once validly de-identified, the data is no longer PHI. Two traps: a limited data set is not de-identified and needs a Data Use Agreement, and AI raises re-identification risk over time.

Treat prompts and outputs as PHI

Prompts and outputs are PHI. A prompt containing a patient’s data, and the model’s response about it, are PHI. Store, encrypt, access-control, retain, and audit them like the rest of the record.
Encrypt PHI in transit and at rest, everywhere it flows. Including any queue, cache, or vector store in between (§164.312(a)(2)(iv), (e)(2)(ii)). Encryption is addressable today, but the January 2025 Security Rule proposal would make it mandatory, so design for that now.
No PHI in URLs or query strings. URLs land in server logs, referrer headers, analytics, and browser history. Keep PHI in request bodies, never the path or query.
No PHI leaks into third-party telemetry. Error trackers, analytics, and logging SDKs quietly capture request bodies. Confirm none are recording prompts or outputs that contain PHI.
PHI redaction fails closed. If the scrubber that strips PHI from logs errors, the request should fail rather than log unredacted. A fail-open redactor is a silent leak.

Access, audit, and identity

The AI feature enforces the same access controls as the record. A user who cannot see a chart must not be able to ask the AI about it (§164.312(a)(1)).
Audit every read of PHI, not just writes. Chart reads, document views, and AI analyses of a patient are disclosures. Log who, which record, and when (§164.312(b) audit controls, §164.528 accounting of disclosures). Server-side logging is authoritative.
The audit trail is immutable, stored apart from PHI, and kept 6+ years. So that compromising the data store cannot erase the proof of the compromise. §164.316(b)(2) sets the six-year floor.
Emergency, break-glass access is controlled. Documented reason, time-limited, fully logged, alerted, and reviewed after use (§164.312(a)(2)(ii)).
Sessions can be revoked instantly, and MFA guards access. Deactivation, role change, or credential change should kill active sessions immediately. The January 2025 proposal would also define and require multi-factor authentication.

The vendor and subcontractor chain

Every PHI-handling vendor has a signed BAA before production, tracked. New vendors go through a change-management step: security review (SOC 2 or HITRUST evidence preferred), officer sign-off, and a row in a subprocessor register (§164.308(b)).
BAA obligations flow down to sub-processors. Your AI vendor’s own model host or sub-processor counts. A Business Associate is directly liable for failing to sign BAAs with subcontractors and for failing to police their breaches (§164.502(e)(1)(ii), §164.504(e)).
Non-PHI vendors are carved out and verified clean. A payment processor can rely on the §164.502(e) carve-out only if zero PHI reaches it. Confirm no PHI in metadata, descriptors, or logs.
You know your obligations if the vendor is breached. A breached Business Associate must notify you, and you then handle individual, HHS, and media notice under the Breach Notification Rule (§164.400-414).

Operational

Your breach-response plan covers the AI path. The incident plan names the AI feature and vendor as a place PHI can be exposed.
The people who build and operate the feature are trained. They should know what PHI can and cannot go where.
You review this on a schedule. Vendors change terms and model defaults, and your feature grows. The January 2025 proposal would require Business Associates to re-verify their safeguards every 12 months with a subject-matter-expert certification, so an annual review is the direction of travel.

What this checklist does not cover

This is HIPAA scope only. It does not cover FDA rules for AI that influences diagnosis or treatment (if your AI is a medical device, Software-as-a-Medical-Device guidance applies), state privacy laws that can be stricter than HIPAA and can restrict even de-identified data, or general application security beyond PHI. Those are separate, and some matter just as much.

Sources

Frequently asked questions

Do I need a BAA with my AI provider?: Yes, if it processes PHI on your behalf it is a Business Associate and needs a signed BAA. Confirm the provider offers one and that your exact tier and endpoint is covered.
Can I use consumer AI tools like ChatGPT with patient data?: Not the consumer versions. HIPAA-eligible use generally requires a business or cloud tier the provider will sign a BAA for, plus settings that stop retention and training. As an example, one major model provider signs a BAA only for its API and Enterprise plans, not its consumer or console tiers, and vendor terms change often, so verify the current terms for your endpoint.
Is "zero data retention" required for HIPAA?: No. HIPAA requires a BAA and the Security Rule safeguards, not zero retention. Some HIPAA-covered configurations actually mandate a short retention window, and some clouds retain a sample for abuse monitoring by default. Know the retention behavior of the exact service you use.
Are AI prompts and outputs considered PHI?: Yes. A prompt containing a patient’s data and the model’s response are PHI. Encrypt, access-control, retain, and audit them the same way you would the record.
Does HIPAA require de-identifying data before sending it to AI?: HIPAA requires the minimum necessary, not always full de-identification. When you do de-identify, use a real method (Safe Harbor or Expert Determination). Be aware that de-identified data can sometimes be re-identified, and AI increases that risk.
Does HIPAA cover whether the AI’s clinical output is safe or accurate?: No. HIPAA governs the privacy and security of patient data, not clinical safety. AI that influences diagnosis or treatment can fall under FDA Software-as-a-Medical-Device rules, which are separate.
What is changing with the 2025 HIPAA Security Rule update?: HHS OCR proposed a rule in January 2025 that would make encryption of ePHI mandatory, remove the addressable category so nearly all safeguards are required, define multi-factor authentication, and require Business Associates to verify their safeguards annually. It is proposed, not yet final, but designing to meet it is prudent.

HIPAA-compliant software development

How we build encryption, audit logs, access controls, and a signed BAA into a project from the first commit.

Generative AI development

AI features built for regulated data, not a wrapper around a public API.

Healthcare automation

Where AI and automation fit into clinical and back-office workflows.

Building this and want a second set of eyes?

We build HIPAA-compliant healthcare software, including production AI features. If you are wiring AI into a system that handles patient data, we can review the plan or build it with you.

Book a scoping call See our HIPAA work