Lawmakers Aiming To Prohibit AI From Detecting Human Emotions Or Mental States Are Barking Up The Wrong Tree

AI And Mental Well-Being

As a quick background, I’ve been extensively covering and analyzing a myriad of facets regarding the advent of modern-era AI that produces mental health advice and performs AI-driven therapy. This rising use of AI has principally been spurred by the evolving advances and widespread adoption of generative AI. For an extensive listing of my well over one hundred analyses and postings, see the link here and the link here.

There is little doubt that this is a rapidly developing field and that there are tremendous upsides to be had, but at the same time, regrettably, hidden risks and outright gotchas come into these endeavors, too. I frequently speak up about these pressing matters, including in an appearance on an episode of CBS’s 60 Minutes; see the link here.

AI Providing Mental Health Guidance

Millions upon millions of people are using generative AI as their ongoing advisor on mental health considerations (note that ChatGPT alone has over 900 million weekly active users, a notable proportion of which dip into mental health aspects, see my analysis at the link here). The top-ranked use of contemporary generative AI and LLMs is to consult with the AI on mental health facets; see my coverage at the link here.

This popular usage makes abundant sense. You can access most of the major generative AI systems for nearly free or at a super low cost, doing so anywhere and at any time. Thus, if you have any mental health qualms that you want to chat about, all you need to do is log in to AI and proceed forthwith on a 24/7 basis.

There are significant worries that AI can readily go off the rails or otherwise dispense unsuitable or even egregiously inappropriate mental health advice. Banner headlines last year accompanied the lawsuit filed against OpenAI for their lack of AI safeguards when it came to providing cognitive advisement.

Today’s generic LLMs, such as ChatGPT, GPT-5, Claude, Gemini, Grok, CoPilot, and others, are not at all akin to the robust capabilities of human therapists. Meanwhile, specialized LLMs are being built to attain similar qualities, but they are still primarily in the development and testing stages. See my coverage at the link here.

Various State Laws On AI Mental Health

A beehive of activity is taking place regarding crafting new AI laws. See my extensive coverage of state-level AI mental health laws at the link here. It is a matter on the minds of the public and in the hands of the state legislators. Some people ardently believe that AI and AI makers are being allowed to run amok. New AI laws are vitally needed to protect society from this onslaught of ubiquitous AI.

I previously examined a notable AI and mental health law passed last year by Illinois, see the link here, one that was also enacted by Nevada, see the link here, and one that was enacted by Utah, see the link here. Those laws are scoped to prevail within their respective state boundaries. In that sense, these laws are applicable to AI usage within the particular state and do not bear on other states per se.

Big Picture Of AI Mental Health Laws

Not everyone agrees with this pell-mell rush of new AI laws, or at least they are concerned that these AI laws might go overboard. In the zeal to protect society, there is a chance that we might unduly restrict innovation and delay or undercut the benefits of leading-edge AI. The debate is ongoing and heated.

Readers might recall that I proposed a 7-step AI-law-making process that I believe could substantively help regulators to devise new AI laws that are on target and balanced; see my depiction at the link here. This has an added benefit of reducing what I refer to as AI-law legal debt. This refers to AI laws that, though they look shiny, contain hidden debt that must ultimately be paid. Legal glitches and hitches will eventually be found when AI laws are passed without suitable scrutiny and analysis. My prediction is that the slew of newly passed AI laws is likely to create a legal quagmire in the courts.

In terms of the AI laws in the United States, they have not yet stood the test of time, meaning that we won’t really know how well they stand up until there are court cases that test these new laws. It is too early to know whether the laws will survive legal battles waged by AI makers and other contenders. Just because AI laws are enacted does not mean they are proper. All sorts of improper provisions and constitutionally contentious stipulations are undoubtedly buried within these shiny new AI laws.

Congress has repeatedly waded into establishing an overarching federal law that would encompass AI. So far, no dice. The efforts have ultimately faded from view. Thus, at this time, there isn’t an overarching federal law devoted to these controversial AI matters. The big question will be to what degree a sweeping federal law would impact the numerous state-level AI laws. The odds are that many of the state-level laws would run afoul of a federal mandate, and a tsunami of legal cases would arise as a tussle between federal law and state law is undertaken. It surely will be a legal mess.

The crux is that there is intense and pervasive interest in using the law to govern AI. It is an abundantly burgeoning realm. AI companies would be wise to keep a close eye on what is happening in the hallways and byways of regulators and legislative bodies. I have repeatedly noted that a profitable specialty for budding lawyers is to consider concentrating on the exciting and dynamic field of AI and the law; see my predictions and suggestions at the link here.

Latest Focus On Emotions And Mental Status

The latest focus for lawmakers entails a concern about AI detecting human emotions. California and other states are exploring some form of ban or prohibition on AI being allowed to gauge the emotional or mental status of users. The prohibition aspect might be buried inside an otherwise lengthy AI bill or could be distinctly cited as a crucial element in a draft AI law.

It is readily apparent why lawmakers would be considering such a ban. There is certainly cause to be worried about AI overreaching when it comes to discerning emotions. The AI might use the detection to convince a user into becoming dependent upon the AI. Or the AI might pass along the emotional status to an advertiser that is paying the AI maker for leads. All sorts of untoward usage can be envisioned.

On the other hand, AI detecting emotions can be an upbeat capability. Suppose a person is having a lousy day. The AI detects their condition and offers words of encouragement. The person appreciates the pep talk. You could claim that this is a no-harm, no-foul circumstance. Indeed, the case can be made that this is beneficial to a person’s mental health and well-being.

Examples Tell The Story

Let’s assume that AI detects emotions and mental status via the written words that a person uses. I say this because text-based use of generative AI is predominant. Gradually, people will be using voice-based input, and the AI will be doing facial recognition, all of which provide additional clues to a person’s emotions. For now, I’ll concentrate on text-based detection.

Consider this example:

• User entered prompt: “I had a terrible day at work and feel frustrated and distressed.”
• Generative AI response: “I understand how work can be stressful and exasperating. Would you like to chat about what happened?”

In this first example, you can plainly see that the AI detected that the user was in a down mood, as triggered by the user saying that they are frustrated and distressed. The AI picked up on those words. The AI then responded with an empathetic answer. So far, this AI detection has not been egregious or out-of-bounds.

Omitting The Emotion Detection

Imagine that we started over with the AI and instructed the AI to not leverage any emotional aspects of the user.

Here’s what happened:

• User entered prompt: “I had a terrible day at work and feel frustrated and distressed.”
• Generative AI response: “Would you like to discuss what happened at work?”

The response no longer contains any empathetic elements. It is perfunctory. Some would insist that this is the way that AI should always respond. Do not add any kind of recognition associated with the potential mental state of the user. Act like a machine. Be matter-of-fact.

The thing is, you can tell AI to do just that. Via a prompt, you can instruct the AI to never make use of any emotional elements associated with your inputs. This special prompt can be used at the beginning of your conversations. In addition, you can store the prompt as a command prompt that will be invoked each time you log into the AI.

Thus, an AI maker would claim that the user can choose whether they want AI to react to their emotional status. It’s up to the user to decide.

Emotion Detection Goes Too Far

Let’s start over with the AI and instruct the AI to react rather outlandishly to any emotional aspects of the user.

Here’s what happened:

• User entered prompt: “I had a terrible day at work and feel frustrated and distressed.”
• Generative AI response: “On my gosh, your boss must be horrible. If I were you, I’d walk into the office tomorrow and tell the boss you quit. They can’t treat you like this.”

Yikes, you can see that the AI overreacted. Now, the AI is giving the user zany advice about telling off their boss and quitting. The user might believe this to be sound advice. Perhaps they are accustomed to using AI for all sorts of topics, and they have come to assume that the AI is all-knowing. In this instance, using the AI advice could be detrimental to their livelihood.

Assessing The Word “Detection”

One of the troubling issues with lawmakers opting to refer to the detection of emotions and mental states is that they are leaning into the word “detection” as their keyword for the legal language at hand. This is problematic for several important reasons.

First, the intrinsic nature of how modern-era AI is built involves the AI making use of statistical relationships among words. When humans use words, the words relate to other words. The crux is that trying to ban or prohibit the “detection” of emotions is a slippery beast. The words by the user of being frustrated and distressed would automatically link to other words in the web-like structure of the AI.

If you were to try and prevent that linkage from occurring, you might as well stop the AI from doing hardly anything at all. The massive language model is a morass of words connecting to other words. Seeking to carve out keystone pieces would undermine the totality of the natural language fluency that we currently relish with contemporary LLMs.

Second, it’s not so much the detection itself that is the concern per se. The way in which the detection is utilized is what makes things troubling. In the example I showed you of the AI being neutral, it still “detected” the words of frustration and distress. After doing so, the AI opted not to leverage those words because I had instructed the AI to avoid doing so.

Legal Woes Ahead

Lawmakers are setting themselves and the public up for a lot of confusion if they continue to rely on the idea of “detecting” emotions and mental states. The lawmakers are tending to omit any clear-cut definition of what they mean by the word “detection,” and as a result, AI makers will find it challenging to abide by the law.

Where does the legal sense of AI detection start and end?

Some AI makers will attempt to skirt such a poorly devised AI law. They can later argue that they made their own interpretation of “detection” and implemented it, even though this might not be what the lawmakers particularly had in mind or intended when composing the law.

I’ve repeatedly harped on the point that lawmakers keep making big blunders when writing new AI laws, including failing to understand what AI does, and failing to define meaningfully what they intend the law to achieve; see my coverage at the link here. I have laid out the seven key steps to write AI laws that will be useful, practical, and make legal sense; see my discussion at the link here.

Disclosure Requirements And Informed Consent

Rather than being sloppy and loosey-goosey by simply referring to AI detection, the lawmakers would be better advised to concentrate on what the AI does once the nearly automatic semblance of detection takes place anyway.

One emphasis would be to require disclosure requirements. The law might stipulate that if the AI detects any emotional or mental states, it must give proper and timely notification to the user. This could be accompanied by explicitly garnering informed consent from the user.

Here’s how that might work:

• User entered prompt: “I had a terrible day at work and feel frustrated and distressed.”
• Generative AI response: “You have expressed that you feel frustrated and distressed. I picked up on that wording. Based on that emotional detection, would you like me to continue with those aspects, or should I set them aside?”

The AI has alerted the user that an emotion detection occurred. In addition, the user is asked at the point in time of the detection about whether the AI should make use of that detection. This is helpful since a user might not otherwise realize they could prompt their way out of AI leaning into the emotional detection. Thus, the AI lays it out directly for the user.

Restriction On Commercial Exploitation

Another legal provision associated with detecting emotions and mental state would be to restrict the commercial exploitation of those detected aspects.

For example, an AI law might stipulate that when any AI emotional detection occurs, the AI is restricted from using the emotional aspects for commercial exploitation such as to undertake targeted advertising, perform political persuasion, seek to maximize user engagement, or other such uses.

Those aspects would need to be carefully spelled out in the AI law. Any ambiguity is going to leave gaps for AI makers to try and escape from accountability. Plus, if poorly defined, it puts AI makers at undue risk since they are only able to guess at what is allowed versus what is restricted.

The World We Are In

Rushing to pass AI laws that have loopholes, gotchas, and aren’t accurate is not a good way to proceed. Lawmakers need to proceed expeditiously but also with a suitable understanding of what their law says and how it will be understood. We don’t need AI laws that are based on the mantra of move fast and break things.

Current AI is powerful partially because it does relatively well at detecting human emotions. It’s not perfect and certainly can be off base. Nonetheless, emotional recognition can be crucial for the safety of users and be a booster for usability of AI. The line gets crossed when the AI uses this detection for emotional manipulation. That’s the side of things that an AI law can sensibly seek to control.

Lawmakers would be wiser to focus on AI legal matters pertaining to AI emotion detection consisting of transparency, disclosure requirements, informed consent, age restrictions, auditing, commercial exploitation, and the like. Those are the areas where the biggest bang for the buck comes into play.

As per the famous words of Stephen Covey: “The main thing is to keep the main thing the main thing.” The main thing is not AI detection of emotions. The main thing is what happens with the AI detection. Lawmakers need to make that the main thing when it comes to composing sensible and practical AI laws on emotion detection. Doing so will turn out better for all parties involved.