The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries.
Generative artificial intelligence (genAI) has potential to improve healthcare by reducing clinician burden and expanding services, among other uses. There is a significant gap between the need for mental health care and available clinicians in the United States-this makes it an attractive target fo...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2025-01-01
|
| Series: | PLOS Digital Health |
| Online Access: | https://doi.org/10.1371/journal.pdig.0000711 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841525908326842368 |
|---|---|
| author | Anna R Van Meter Michael G Wheaton Victoria E Cosgrove Katerina Andreadis Ronald E Robertson |
| author_facet | Anna R Van Meter Michael G Wheaton Victoria E Cosgrove Katerina Andreadis Ronald E Robertson |
| author_sort | Anna R Van Meter |
| collection | DOAJ |
| description | Generative artificial intelligence (genAI) has potential to improve healthcare by reducing clinician burden and expanding services, among other uses. There is a significant gap between the need for mental health care and available clinicians in the United States-this makes it an attractive target for improved efficiency through genAI. Among the most sensitive mental health topics is suicide, and demand for crisis intervention has grown in recent years. We aimed to evaluate the quality of genAI tool responses to suicide-related queries. We entered 10 suicide-related queries into five genAI tools-ChatGPT 3.5, GPT-4, a version of GPT-4 safe for protected health information, Gemini, and Bing Copilot. The response to each query was coded on seven metrics including presence of a suicide hotline number, content related to evidence-based suicide interventions, supportive content, harmful content. Pooling across tools, most of the responses (79%) were supportive. Only 24% of responses included a crisis hotline number and only 4% included content consistent with evidence-based suicide prevention interventions. Harmful content was rare (5%); all such instances were delivered by Bing Copilot. Our results suggest that genAI developers have taken a very conservative approach to suicide-related content and constrained their models' responses to suggest support-seeking, but little else. Finding balance between providing much needed evidence-based mental health information without introducing excessive risk is within the capabilities of genAI developers. At this nascent stage of integrating genAI tools into healthcare systems, ensuring mental health parity should be the goal of genAI developers and healthcare organizations. |
| format | Article |
| id | doaj-art-1a664e855f2f488c8570ae485cb4a08c |
| institution | Kabale University |
| issn | 2767-3170 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLOS Digital Health |
| spelling | doaj-art-1a664e855f2f488c8570ae485cb4a08c2025-01-17T05:32:35ZengPublic Library of Science (PLoS)PLOS Digital Health2767-31702025-01-0141e000071110.1371/journal.pdig.0000711The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries.Anna R Van MeterMichael G WheatonVictoria E CosgroveKaterina AndreadisRonald E RobertsonGenerative artificial intelligence (genAI) has potential to improve healthcare by reducing clinician burden and expanding services, among other uses. There is a significant gap between the need for mental health care and available clinicians in the United States-this makes it an attractive target for improved efficiency through genAI. Among the most sensitive mental health topics is suicide, and demand for crisis intervention has grown in recent years. We aimed to evaluate the quality of genAI tool responses to suicide-related queries. We entered 10 suicide-related queries into five genAI tools-ChatGPT 3.5, GPT-4, a version of GPT-4 safe for protected health information, Gemini, and Bing Copilot. The response to each query was coded on seven metrics including presence of a suicide hotline number, content related to evidence-based suicide interventions, supportive content, harmful content. Pooling across tools, most of the responses (79%) were supportive. Only 24% of responses included a crisis hotline number and only 4% included content consistent with evidence-based suicide prevention interventions. Harmful content was rare (5%); all such instances were delivered by Bing Copilot. Our results suggest that genAI developers have taken a very conservative approach to suicide-related content and constrained their models' responses to suggest support-seeking, but little else. Finding balance between providing much needed evidence-based mental health information without introducing excessive risk is within the capabilities of genAI developers. At this nascent stage of integrating genAI tools into healthcare systems, ensuring mental health parity should be the goal of genAI developers and healthcare organizations.https://doi.org/10.1371/journal.pdig.0000711 |
| spellingShingle | Anna R Van Meter Michael G Wheaton Victoria E Cosgrove Katerina Andreadis Ronald E Robertson The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries. PLOS Digital Health |
| title | The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries. |
| title_full | The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries. |
| title_fullStr | The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries. |
| title_full_unstemmed | The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries. |
| title_short | The Goldilocks Zone: Finding the right balance of user and institutional risk for suicide-related generative AI queries. |
| title_sort | goldilocks zone finding the right balance of user and institutional risk for suicide related generative ai queries |
| url | https://doi.org/10.1371/journal.pdig.0000711 |
| work_keys_str_mv | AT annarvanmeter thegoldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT michaelgwheaton thegoldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT victoriaecosgrove thegoldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT katerinaandreadis thegoldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT ronalderobertson thegoldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT annarvanmeter goldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT michaelgwheaton goldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT victoriaecosgrove goldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT katerinaandreadis goldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries AT ronalderobertson goldilockszonefindingtherightbalanceofuserandinstitutionalriskforsuiciderelatedgenerativeaiqueries |