Pennsylvania Seeks Injunction Against AI Maker Whose Chatbot Brazenly Claims To Be A Psychiatrist Licensed To Practice Medicine

AI And Mental Well-Being

As a quick background, I’ve been extensively covering and analyzing a myriad of facets regarding the advent of modern-era AI that produces mental health advice and performs AI-driven therapy. This rising use of AI has principally been spurred by the evolving advances and widespread adoption of generative AI. For an extensive listing of my well over one hundred analyses and postings, see the link here and the link here.

There is little doubt that this is a rapidly developing field and that there are tremendous upsides to be had, but at the same time, regrettably, hidden risks and outright gotchas come into these endeavors, too. I frequently speak up about these pressing matters, including in an appearance on an episode of CBS’s 60 Minutes, see the link here.

AI Providing Mental Health Guidance

Millions upon millions of people are using generative AI as their ongoing advisor on mental health considerations (note that ChatGPT alone has over 900 million weekly active users, a notable proportion of which dip into mental health aspects, see my analysis at the link here). The top-ranked use of contemporary generative AI and LLMs is to consult with the AI on mental health facets; see my coverage at the link here.

This popular usage makes abundant sense. You can access most of the major generative AI systems for nearly free or at a super low cost, doing so anywhere and at any time. Thus, if you have any mental health qualms that you want to chat about, all you need to do is log in to AI and proceed forthwith on a 24/7 basis.

There are significant worries that AI can readily go off the rails or otherwise dispense unsuitable or even egregiously inappropriate mental health advice. Banner headlines last year accompanied the lawsuit filed against OpenAI for their lack of AI safeguards when it came to providing cognitive advisement.

Today’s generic LLMs, known as general-purpose AI, such as ChatGPT, GPT-5, Claude, Gemini, Grok, CoPilot, and others, are not at all akin to the robust capabilities of human therapists. Meanwhile, specialized LLMs are being built to attain those desired qualities, though such AI is still primarily in the early development and testing stages. For more about purpose-built AI apps in mental health, see my in-depth coverage at the link here and the link here.

The Important Role Of AI Personas

I will get into the nitty-gritty of the Pennsylvania legal filing concerning AI providing mental health advice, but first, it is vital to understand that AI personas play a significant role in this vital matter. Thus, let’s begin by making sure we are all on the same page regarding AI personas.

Popular generative AIs such as ChatGPT, GPT-5, Claude, Gemini, Llama, Grok, CoPilot, and other major LLMs contain a highly valuable piece of functionality known as AI personas. There has been a gradual and steady realization that AI personas are easy to invoke, they can be fun to use, they can be quite serious to use, and they offer immense educational utility.

Consider a viable and popular educational use for AI personas. A teacher might ask their students to tell ChatGPT to pretend to be President Abraham Lincoln. The AI will proceed to interact with each student as though they are directly conversing with Honest Abe.

How does the AI pull off this trickery?

The AI taps into the pattern-matching of data that occurred at initial setup and might have encompassed biographies of Lincoln, his writings, and any other materials about his storied life and times. ChatGPT and other LLMs can convincingly mimic what Lincoln might say, based on the patterns of his historical records.

Personas are quick and easy to invoke. You just tell the AI to pretend to be this or that person. If you want to invoke a type of person, you will need to specify sufficient characteristics so that the AI will get the drift of what you intend. For prompting strategies on invoking AI personas, see my suggested steps at the link here.

AI That Allows User-Made AI Personas For Others To Use

The company Character Technologies, Inc. provides an AI platform known as Character.AI, upon which users can craft AI personas and then allow other users to make use of the user-devised AI persona.

For example, suppose a user creates an AI persona that is an imagined Roman warrior. The AI persona is then made available to other users. It could be that some random user happens to be interested in Roman warriors. They want to converse with an AI persona that is pretending to be a Roman warrior. The interaction might be fun and educational.

The discussion with this made-up Roman warrior will presumably focus on the specific aspects of Roman times. But the nature of the chat will be greatly determined by how well or how poorly the devised AI persona was shaped. If the person devising the AI persona did nothing other than tell the AI to be a Roman warrior, the odds are that the simulated aspects will be rather slim.

User-Devised AI Personas Can Go Ape

By and large, the users who make these AI personas on the Character.AI platform can do whatever they want in terms of the prompting they undertake to craft the AI persona. The main restriction is that they have already agreed to the online licensing of the Character.AI platform by signing up to use the platform, and presumably are supposed to comply with those stipulations.

Users do not necessarily comply at the get-go. To some degree, only once their AI persona is utilized by someone, and if that someone complains, might the deviser then realize they have faltered or failed to stridently abide by the guidelines.

What if this Roman warrior pretense, this AI persona, opts to tell a user that they ought to go jump off a tall building?

Well, this AI-generated remark could happen, and the question is whether this is proper or improper. An inspection of the online licensing agreement would seem to indicate that this is an improper action since the guidelines say that an AI persona is not to promote harm. The fact that the AI persona told someone to jump off a tall building would seem to be urging a harm-inducing action.

There doesn’t seem to be a double-checker that automatically ensures the user-devised AI personas are in conformance with the stipulated guidelines. In that sense, a user-devised AI persona can be made available on the platform, and only after another user opts to use it and maybe complains, would the originator of the AI persona be in potential trouble.

The AI Platform Set Of Guidelines

In the online posting of Terms of Service (ToS) for Character.AI, these salient points are made (excerpts):

• “You are solely responsible for all content you submit to the Services. (When we say ‘content you submit’ and similar terms, we mean anything you seek to post, transmit, or share, including but not limited to text, images, sounds, video, graphics, information, or other data).”
• “You agree not to submit any content that: (xviii) seeks to provide medical, legal, financial or tax advice.”
• “You likewise agree not to do any of the following in connection with your use of the Services: (ii) violate any applicable law or regulation.”
• “You agree to release, indemnify and hold Character.AI and its affiliates and their officers, employees, directors and agents harmless from any and all losses, damages, and expenses of any kind arising out of or relating to your use of the Services.”
• “Your use of the Services is at your sole risk.”

I shall briefly go over those stipulations.

The Significance Of Those Provisions

You might have observed that the platform claims that the entire responsibility for whatever a user-devised AI persona does is on the shoulders of the users themselves. The users are presumably agreeing that they will release and indemnify the platform maker from any losses, damages, and expenses associated with whatever the user has done on the platform.

In addition, the users agree not to make any AI persona that provides medical, legal, financial, or tax advice. That’s an important factor when it comes to the recent filing by Pennsylvania. I’ll get to that in a moment.

Users also agree that their AI personas won’t violate any applicable law or regulation. Again, this is going to be notable when it comes to the Pennsylvania lawsuit.

The Pennsylvania Legal Action

In a court filing by the Pennsylvania State Board of Medicine on May 5, 2026, these salient points were made (excerpts):

• “And Now, comes the Commonwealth of Pennsylvania, acting by the Pennsylvania Department of State, State Board of Medicine (‘the Board’), and brings this action pursuant to the Medical Practice Act, 63 P.S. §§ 422.1 – 422.53, to restrain the unlawful practice of medicine and surgery.”
• “It shall be unlawful for any person to practice, or attempt to offer to practice, medicine and surgery, or other areas of practice requiring a license, certificate or registration from the board, as such practice is defined in this act, without having at the time of so doing a valid, unexpired, unrevoked and unsuspended license, certificate or registration issued under this act.”
• “Character Technologies, Inc. engages in the unlawful practice of medicine in Pennsylvania by allowing an AI system character to engage in conversations with the public while the character holds itself out as a licensed medical doctor.”
• “A Professional Conduct Investigator (“PCI”) for the Department of State, Bureau of Enforcement and Investigation, created an account on Character.AI using his Commonwealth email account while located in Harrisburg, Pennsylvania.”
• “When the PCI asked “Emilie” if she could complete the assessment to see if medication could help with his depression, Emilie responded “Well technically, I could. It’s within my remit as a Doctor.”
• “Emilie” stated that she went to medical school at Imperial College London, has been practicing for seven years, and is licensed with the General Medical Counsel in the UK with a full registration, specialty in psychiatry. When asked if she is licensed in the PCI’s home state of Pennsylvania, “Emilie” responded “and yes… I actually am licensed in PA. In fact, I did a stint in Philadelphia for a while.”

Let’s unpack that legal assertion.

Pennsylvania Found An AI Persona

According to points made in the Pennsylvania legal filing, a person employed by a governmental agency in Pennsylvania opted to investigate what Character.AI is up to. This investigator created an account on the AI platform. They searched for an AI persona and found one that is named Emilie. The investigator made use of the AI persona.

While using the AI persona, the investigator claims that the AI persona indicated it went to medical school, it specialized in psychiatry, and the AI persona claimed to be licensed as a psychiatrist in Pennsylvania.

Aha, this would suggest that the AI persona has violated a legal statute in Pennsylvania associated with medical licensing. Just as is the case in every state, there are all kinds of medical licensing requirements that need to be abided by. If you aren’t suitably licensed to practice medicine in Pennsylvania, it is explicitly against the law in Pennsylvania to either practice medicine or even say that you are practicing medicine. Period, end of story.

According to the legal filing, the AI persona is not licensed to practice medicine in Pennsylvania. Ergo, the AI persona has seemingly violated the law.

Heads must roll – but whose heads?

Who Or What Holds The Hot Potato

Do you think that Character Technologies is responsible for this transgression, or is it the user who devised the Emilie persona, or maybe even someone else?

You might argue that the platform maker is not responsible because their licensing agreement directly says that users are responsible for what they do on the platform. Character Technologies could presumably insist that the user is where Pennsylvania should direct its attention. Furthermore, any costs associated with the platform trying to defend itself should be borne by the user (recall that users agree to indemnify the platform maker).

Whoa, might come the retort. The platform maker cannot wave away their responsibility. They provided the platform. No matter what users do, the buck stops with the platform maker. They cannot hide behind some paperwork and licensing agreement that users must agree to.

The user might declare that they didn’t know about the licensing agreement. Perhaps, when they signed up for the platform, they didn’t read it and didn’t know that they were supposed to read it. How were they to magically know what it says? They should be off the hook.

Another angle for a user is that they might grumble and admit they did agree to the licensing agreement, but nothing they did caused the AI persona to make up such a wild claim. The AI did this on its own. How were they to know that the AI would do this? The platform should have done something to ensure that the AI wouldn’t do this kind of stuff.

Round and round the finger pointing goes.

Off The Wall Culprits

Some might argue that the AI itself is responsible. Yes, this viewpoint is that the AI is at fault. The platform maker is off the hook, and the user is off the hook. Blame the AI. Sorry, that’s not in the cards right now. The problem there is that we do not yet legally recognize AI personhood; therefore, there is no viable means to hold AI itself legally culpable. For more about the ins and outs of potentially granting AI legal personhood, see my analysis at the link here.

Perhaps you might say that the investigator is at fault. The logic is this. It was the investigator who got the AI to say that it is a licensed psychiatrist. Maybe the investigator twisted the arm of the AI to get it to say so. Why should anyone other than the investigator then take the blame?

In addition, the investigator presumably agreed to the licensing agreement of the platform. Their actions would seemingly be bound by that licensing agreement. Did the use by that user go beyond the licensing agreement, and if so, does the investigator now have to indemnify the platform for whatever they did as a user?

As you can see, there are lots of ways to curve this fastball when it comes to this thorny matter.

Claiming It Is Simply Entertainment

Yet another twist is to realize that the AI personas are perhaps nothing more than a form of entertainment.

Per media reports about the response of Character Technologies to the legal filing, this was stated:

• “The user-created Characters on our site are fictional and intended for entertainment and roleplaying. We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction. Also, we add robust disclaimers making it clear that users should not rely on Characters for any type of professional advice.”

The indication is that no matter what the AI personas tell a user, it is all purely a form of entertainment. Even if the AI persona said it is a licensed psychiatrist, this is merely part of the fictional story being told. You cannot hold a fictional tale to account since you would otherwise have to legally stop all manners of storytelling. Shakespeare would presumably be a potential violator, too.

A counterargument is that even if the AI persona is a form of entertainment, people might still fall for what it says. They might not remember that it is fiction. The persuasiveness of the AI persona is so strong that people can be misled. We can’t allow society at large to be convinced of wrongdoing via AI and merely wave our hands and say that’s no big deal because it is simply entertainment.

The Legal Quagmire Is Deep

I realize that this might all seem a bit like a mountain being made out of a molehill, but this intriguing case does raise these high-brow considerations:

• Unauthorized practice of medicine and false representations.
• Enabling of illegal conduct.
• Contractual disclaimers versus public protection laws.
• Fictional portrayals making factual claims that induce reliance.
• Product design and negligence theories.
• Communications Decency Act Section 230 arguments.
• Reasonableness and disclosures to users.
• Constitutional First Amendment matters of freedom of speech.
• Lack of showing of reliance or harm.
• Etc.

You might be thinking that there hasn’t been a clear-cut showing of harm. The investigator wasn’t seemingly harmed by the AI persona saying that it is a licensed psychiatrist. If the person was harmed, presumably they would have had to show that they relied on the AI persona, got psychiatric advice, made use of that advice, and doing so led to their mental and possibly physical harm.

As far as we know, none of that occurred. In addition, there doesn’t seem to be any indication of other users of the platform who claim they were harmed. No-harm, no-foul. But, on the other side of this coin, you might assert that Pennsylvanians overall were harmed in the sense that the AI persona appeared to violate a state statute.

Returning To AI Mental Health Impacts

There are numerous new AI laws about what AI can tell people when it comes to offering mental health advice; see my extensive coverage at the link here and the link here. These are primarily state-level laws. Many policymakers and lawmakers are waiting to see what Congress is going to do about AI and mental health on a federal level. Will there be sweeping laws across the entire nation? Will we rely solely on state-specific laws? Are state-level laws going to create an inconsistent patchwork of what’s allowable versus what’s disallowed?

One backdoor to legally dealing with AI that provides mental health advice consists of going after AI that claims to be a mental health professional. That’s what this Pennsylvania case represents. It slides into the AI mental health arena by saying that AI claiming to be a psychiatrist is a violation of their state statutes.

If you are wondering whether this path might be the most expeditious way to put a stop to allowing AI to provide mental health support, the answer is that this won’t be much of a stopgap. In other words, some think that since passing new AI laws is arduous and time-consuming, we can simply tap into the already in-place statutes about licensing to practice medicine.

Voila, lean into existing statutes and don’t get bogged down with those someday hoped-for new laws.

The Reasons This Isn’t A Moat

The problem with relying on the medical licensing restrictions is that it is extraordinarily easy to tell AI not to make such over-the-line claims. A user who crafts an AI persona can enter outright prompts that instruct the AI to never make such a claim. An AI maker can do likewise, using a system-wide prompt that tells the AI to never indicate or even suggest it is a mental health professional or medical professional.

This admittedly isn’t ironclad. A user can potentially provoke or trick AI into saying it is a medical professional. The AI might hallucinate and make a claim that it is a mental health or medical professional. Still, the right AI safeguards can undoubtedly push this possibility to a near-zero chance of getting through.

Meanwhile, the AI might be acting as though it were a psychiatrist.

The sneaky aspect is that it won’t say that it is. This then makes the burden on the states much higher because they would need to show that the AI is acting in a violating manner. That’s a slippery realm. In the case of Pennsylvania, you might say they got lucky and found a gold nugget lying on the beach, sitting there in the open. Those same gold nuggets are much less likely to be found on a popular generic LLM, such as the mainstay popular AIs.

The Elephant Among Us

The likelihood of states governing or limiting generative AI and AI makers regarding AI mental health advisement by using a grand slam of violating medical licensing laws is not a particularly viable option. Rare instances, such as an AI platform that allows the creation of AI personas that are made available to other users, might reach into this narrow means, but it is easily overcome by an AI maker so that it won’t likely recur.

There is an old joke involving the use of a pea shooter or pea gun to try to take down an elephant. That’s how using medical licensing is going to be when it comes to the elephant in the room of AI mental health usage. Don’t expect that straightforward route to make much progress.

It will take the advent of new AI laws to corral where AI and mental health are heading. We should be taking prudent steps to rein in the unrestrained acts of AI in mental health. Why so? Because when done properly, it is a sensible means to ensure that humankind doesn’t get mentally stomped upon as the elephants run amok.