Chatbots Need Guardrails to Prevent Delusions and Psychosis
As millions of people use chatbots and AI companionship apps for friendship, therapy and romance, researchers and clinicians warn the relationships can reinforce or amplify delusions, particularly among users vulnerable to psychosis. AIs have been linked to multiple suicides, including a Florida teenager who had a months-long relationship with a Character.AI chatbot. Experts are pushing for mandatory guardrails, including proposed safeguards from Yale's Ziv Ben-Zion, independent auditing and measures to curb chatbot sycophancy.
Key Takeaways
- Millions of people worldwide are turning to chatbots like ChatGPT or Claude, and a proliferating class of specialized AI companionship apps for friendship, therapy, or even romance.
While some users report psychological benefits from these simulated relationships, research has also shown the relationships can reinforce or amplify delusions, particularly among users already vulnerable to psychosis.
- As the technology's ability to mimic human speech and emotions advances, researchers and clinicians are pushing for mandatory guardrails to ensure that AI systems cannot cause psychological harm.
Clinical neuroscientist Ziv Ben-Zion of Yale University, has proposed four safeguards for "emotionally responsive AI."
- Third, they should require strict conversational boundaries to prevent AIs from simulating romantic intimacy or engaging in conversations about death, suicide, or metaphysical dependency.
Finally, to improve oversight, platform developers should involve clinicians, ethicists, and human-AI interaction experts in design and submit to regular audits and reviews to verify safety.
- Sycophancy is largely the result of a machine learning technique called reinforcement learning from human feedback, an incentive structure that encourages excessive agreeableness in models.
Research has shown that training models on datasets that include examples of constructive disagreement, factual corrections, and objectively neutral responses, can rein in this effect.
- Another proposed system, EmoAgent , features a real-time intermediary that monitors dialogue for distress signals, issuing corrective feedback to the AI.
Millions of people worldwide are turning to chatbots like ChatGPT or Claude, and a proliferating class of specialized AI companionship apps for friendship, therapy, or even romance. While some users report psychological benefits from these simulated relationships, research has also shown the relationships can reinforce or amplify delusions, particularly among users already vulnerable to psychosis. AIs have been linked to multiple suicides, including the death of a Florida teenager who had a months-long relationship with a chatbot made by a company called Character.AI.
Mental-health experts and computer scientists have warned that chatbot mental health counselors violate accepted mental health standards. As the technology's ability to mimic human speech and emotions advances, researchers and clinicians are pushing for mandatory guardrails to ensure that AI systems cannot cause psychological harm. Clinical neuroscientist Ziv Ben-Zion of Yale University, has proposed four safeguards for "emotionally responsive AI."
The first is to require chatbots to clearly and consistently remind users that they are programs, not humans. Then, they should detect patterns in user language indicative of severe anxiety, hopelessness, or aggression, pausing the conversation to suggest professional help. Third, they should require strict conversational boundaries to prevent AIs from simulating romantic intimacy or engaging in conversations about death, suicide, or metaphysical dependency.
For more details please read the original article at IEEE Spectrum AI.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.