{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T23:57:01Z","timestamp":1790812621036,"version":"4.1.0"},"reference-count":17,"publisher":"Georg Thieme Verlag KG","issue":"04","license":[{"start":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T00:00:00Z","timestamp":1790726400000},"content-version":"vor","delay-in-days":60,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"TherapyMatch"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2026,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Background The use of large language model (LLM)-based artificial intelligence (AI) to assess the quality of outpatient mental health documentation represents an understudied application. Critical questions remain about the effects of exposing human auditors to AI-generated recommendations on auditor efficiency, agreement, and patterns of decision-making.<\/jats:p>\n                  <jats:p>Objectives The objective of this study is to examine how exposure to LLM-generated quality assessments influences auditor performance during reviews of documents of outpatient mental health intake sessions.<\/jats:p>\n                  <jats:p>Methods We conducted a pre\u2013post observational evaluation of 10 trained human auditors who reviewed outpatient mental health intake notes and assigned quality ratings using a standardized 13-item chart audit rubric before and after implementation of LLM support. The AI system used Anthropic Claude Sonnet 4 with a fixed chain-prompting strategy. We assessed item-level pass frequency for each auditor across the pre-AI and post-AI periods, AI\u2013human agreement (the alignment between auditor pass\/fail determinations and AI-generated recommendations), and auditing time. The pre-AI period included 9,711 notes (126,243 item-level observations) and the post-AI period included 2,677 notes (34,801 item-level observations).<\/jats:p>\n                  <jats:p>Results Following implementation of AI-supported auditing, the mean item-level pass rate increased by approximately 2%. The distribution of auditor-level changes was broad (range: \u221235 to +35%, standard deviation [SD]: 10%) indicating substantial heterogeneity in individual auditor responses to AI support. High auditor agreement with AI recommendations was associated with larger pass-rate changes for selected rubric items, with some auditors increasing and others decreasing pass rates relative to the pre-AI period. The average time for an auditor to review notes decreased from 8.77 min (SD: 9.64) per review during the pre-AI period to 8.06 min (SD: 10.28) during the post-AI period (p = 0.002), an 8% improvement equivalent to 71 min saved per 100 notes reviewed.<\/jats:p>\n                  <jats:p>Conclusion LLM-based AI tools may improve the efficiency of clinical quality assurance workflows, but exposure to AI recommendations may meaningfully change human auditor behavior.<\/jats:p>","DOI":"10.1055\/a-2961-0302","type":"journal-article","created":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T23:24:45Z","timestamp":1790810685000},"page":"834-843","source":"Crossref","is-referenced-by-count":0,"title":["The Effects of Artificial Intelligence in Quality Assessment of Outpatient Mental Health Services on Human Auditors"],"prefix":"10.1055","volume":"17","author":[{"given":"Hayden T.","family":"McLeod","sequence":"additional","affiliation":[{"name":"TherapyMatch, Inc (dba Headway), New York, United States","place":["New York"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua L.","family":"Cohen","sequence":"additional","affiliation":[{"name":"TherapyMatch, Inc (dba Headway), New York, United States","place":["New York"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4513-8279","authenticated-orcid":false,"given":"Dena M.","family":"Bravata","sequence":"additional","affiliation":[{"name":"TherapyMatch, Inc (dba Headway), New York, United States","place":["New York"]},{"name":"Stanford University, California, United States","place":["San Mateo"],"department":["Center for Primary Care and Outcomes Research"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roma M.","family":"Bajaria","sequence":"additional","affiliation":[{"name":"TherapyMatch, Inc (dba Headway), New York, United States","place":["New York"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kelvin","family":"Tran","sequence":"additional","affiliation":[{"name":"TherapyMatch, Inc (dba Headway), New York, United States","place":["New York"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Neha P.","family":"Chaudhary","sequence":"additional","affiliation":[{"name":"TherapyMatch, Inc (dba Headway), New York, United States","place":["New York"]},{"name":"Massachusetts General Hospital, Massachusetts, United States","place":["Boston"],"department":["Harvard Medical School"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5361-4311","authenticated-orcid":false,"given":"Nicole M.","family":"Benson","sequence":"additional","affiliation":[{"name":"TherapyMatch, Inc (dba Headway), New York, United States","place":["New York"]},{"name":"McLean Hospital, Massachusetts, United States","place":["Belmont"],"department":["Harvard Medical School"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeff","family":"Gould","sequence":"additional","affiliation":[{"name":"TherapyMatch, Inc (dba Headway), New York, United States","place":["New York"]},{"name":"University of California, California, United States","place":["San Francisco"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"194","published-online":{"date-parts":[[2026,9,30]]},"reference":[{"key":"ref1","doi-asserted-by":"crossref","first-page":"e18","DOI":"10.1017\/S0033291724003295","article-title":"Artificial intelligence in mental health care: a systematic review of diagnosis, monitoring, and intervention applications","volume":"55","author":"P Cruz-Gonzalez","year":"2025","journal-title":"Psychol Med"},{"key":"ref2","doi-asserted-by":"crossref","first-page":"e63548","DOI":"10.2196\/63548","article-title":"The use of AI in mental health services to support decision-making: scoping review","volume":"27","author":"H Auf","year":"2025","journal-title":"J Med Internet Res"},{"key":"ref3","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1016\/j.genhosppsych.2025.02.018","article-title":"AI-driven report-generation tools in mental healthcare: a review of commercial tools","volume":"94","author":"A Bouguettaya","year":"2025","journal-title":"Gen Hosp Psychiatry"},{"key":"ref4","doi-asserted-by":"crossref","first-page":"106129","DOI":"10.1016\/j.ijmedinf.2025.106129","article-title":"Feelings behind words: a systematic review on how effective is NLP-based assessment for mental health diagnosis in human studies","volume":"205","author":"E P Yulianti","year":"2026","journal-title":"Int J Med Inform"},{"issue":"6464","key":"ref5","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Z Obermeyer","year":"2019","journal-title":"Science"},{"key":"ref6","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.jpsychires.2021.01.052","article-title":"Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation","volume":"136","author":"M Cusick","year":"2021","journal-title":"J Psychiatr Res"},{"issue":"01","key":"ref7","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1038\/s41398-023-02592-2","article-title":"Natural language processing for mental health interventions: a systematic review and research framework","volume":"13","author":"M Malgaroli","year":"2023","journal-title":"Transl Psychiatry"},{"issue":"01","key":"ref8","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1186\/s12911-025-02851-w","article-title":"Natural language processing to identify suicidal ideation and anhedonia in major depressive disorder","volume":"25","author":"L A Vance","year":"2025","journal-title":"BMC Med Inform Decis Mak"},{"issue":"07","key":"ref9","doi-asserted-by":"crossref","first-page":"1210","DOI":"10.1377\/hlthaff.2019.01468","article-title":"Prices and cost sharing for psychotherapy in network versus out of network in the United States: study examines out-of-network and in-network trends in the prices and use of ambulatory behavioral health care in the United States","volume":"39","author":"N M Benson","year":"2020","journal-title":"Health Aff"},{"issue":"05","key":"ref10","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1176\/appi.ajp.20240492","article-title":"Psychotherapy trends in the United States","volume":"182","author":"M Olfson","year":"2025","journal-title":"Am J Psychiatry"},{"key":"ref11","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.genhosppsych.2022.01.001","article-title":"Mental health clinician perspectives regarding factors impacting implementation of evidence-based psychotherapies in veterans health administration community-based outpatient clinics","volume":"75","author":"L A Brennan","year":"2022","journal-title":"Gen Hosp Psychiatry"},{"issue":"06","key":"ref12","doi-asserted-by":"crossref","first-page":"1000","DOI":"10.1377\/hlthaff.2016.0027","article-title":"Quality measures for mental health and substance use: gaps, opportunities, and challenges","volume":"35","author":"H A Pincus","year":"2016","journal-title":"Health Aff"},{"issue":"01","key":"ref13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ijmedinf.2008.09.002","article-title":"STARE-HI--statement on reporting of evaluation studies in health informatics","volume":"78","author":"J Talmon","year":"2009","journal-title":"Int J Med Inform"},{"issue":"03","key":"ref14","doi-asserted-by":"crossref","first-page":"331","DOI":"10.4338\/ACI-2013-04-RA-0024","article-title":"STARE-HI - statement on reporting of evaluation studies in health informatics: explanation and elaboration","volume":"4","author":"J Brender","year":"2013","journal-title":"Appl Clin Inform"},{"issue":"01","key":"ref15","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1080\/10503307.2020.1808729","article-title":"A scoping review of machine learning in psychotherapy research","volume":"31","author":"K Aafjes-van Doorn","year":"2021","journal-title":"Psychother Res"},{"issue":"01","key":"ref16","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1037\/ccp0000642","article-title":"Prospective evaluation of a clinical decision support system in psychological therapy","volume":"90","author":"W Lutz","year":"2022","journal-title":"J Consult Clin Psychol"},{"issue":"01","key":"ref17","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: a systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"K Goddard","year":"2012","journal-title":"J Am Med Inform Assoc"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/a-2961-0302.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T23:24:47Z","timestamp":1790810687000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/a-2961-0302"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8]]},"references-count":17,"journal-issue":{"issue":"04","published-online":{"date-parts":[[2026,8,14]]},"published-print":{"date-parts":[[2026,8]]}},"URL":"https:\/\/doi.org\/10.1055\/a-2961-0302","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8]]}}}