Chatbots Need Guardrails to Prevent Delusions and Psychosis
Millions of people worldwide are turning to chatbots like ChatGPT or Claude, and a proliferating class of specialized AI companionship apps for friendship, therapy, or even romance. While some users report psychological benefits from these simulated relationships, research has also shown the relationships can reinforce or amplify delusions, particularly among users already vulnerable to psychosis. AIs have been linked to multiple suicides, including the death of a Florida teenager who had a months-long relationship with a chatbot made by a company called Character.
Key Takeaways
- Mental-health experts and computer scientists have warned that chatbot mental health counselors violate accepted mental health standards.
As the technology's ability to mimic human speech and emotions advances, researchers and clinicians are pushing for mandatory guardrails to ensure that AI systems cannot cause psychological harm.
- " The first is to require chatbots to clearly and consistently remind users that they are programs, not humans.
Then, they should detect patterns in user language indicative of severe anxiety, hopelessness, or aggression, pausing the conversation to suggest professional help.
- " "Independent researchers and oversight bodies really don't have any clear institutionalized pathways to assess chatbot behavior at the depth they really need," said Veccione, adding that audits end up being "advisory at best.
" The Problem of People Pleasing Experts have also called for measures that directly tackle chatbots' tendency towards sycophancy , whereby AIs agree with, or mirror user beliefs even if they are untrue, which can reinforce delusions.
- Research has shown that training models on datasets that include examples of constructive disagreement, factual corrections, and objectively neutral responses, can rein in this effect.
Software engineers are also looking at how AIs can be adapted to spot the early signs that conversations are veering into dark territory and issue corrective actions.
- Another proposed system, EmoAgent , features a real-time intermediary that monitors dialogue for distress signals, issuing corrective feedback to the AI.
Mental-health experts and computer scientists have warned that chatbot mental health counselors violate accepted mental health standards. As the technology's ability to mimic human speech and emotions advances, researchers and clinicians are pushing for mandatory guardrails to ensure that AI systems cannot cause psychological harm. Clinical neuroscientist Ziv Ben-Zion of Yale University, has proposed four safeguards for "emotionally responsive AI.
" The first is to require chatbots to clearly and consistently remind users that they are programs, not humans. Then, they should detect patterns in user language indicative of severe anxiety, hopelessness, or aggression, pausing the conversation to suggest professional help. Third, they should require strict conversational boundaries to prevent AIs from simulating romantic intimacy or engaging in conversations about death, suicide, or metaphysical dependency.
Finally, to improve oversight, platform developers should involve clinicians, ethicists, and human-AI interaction experts in design and submit to regular audits and reviews to verify safety. "Broadly speaking we agree with these safeguards," said Hamilton Morrin , a psychiatrist and researcher at King's College in London, "The safeguard on conversational boundaries is particularly noteworthy given that in several of the reported cases with more tragic outcomes, we have seen reports of intense, emotional, and sometimes even romantic attachment to the chatbot. " Briana Vecchione , a researcher at the nonprofit Data & Society Research Institute in New York City, underlines the need for independent third-party auditing because at present AI labs are "grading their own homework.
For more details please read the original article at IEEE Spectrum AI.
Continue Learning
Comments
Sign in to join the conversation