{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T19:45:02Z","timestamp":1782762302628,"version":"3.54.5"},"publisher-location":"New York, NY, USA","reference-count":71,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T00:00:00Z","timestamp":1782345600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,6,25]]},"DOI":"10.1145\/3805689.3812332","type":"proceedings-article","created":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T17:52:08Z","timestamp":1782755528000},"page":"3894-3920","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testing"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2001-3488","authenticated-orcid":false,"given":"Kiana","family":"Jafari Meimandi","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-8375-5617","authenticated-orcid":false,"given":"Paul Ulrich Nikolaus","family":"Rust","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2832-9711","authenticated-orcid":false,"given":"Duncan","family":"Eddy","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2277-0785","authenticated-orcid":false,"given":"Robbie","family":"Fraser","sequence":"additional","affiliation":[{"name":"University College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8917-9320","authenticated-orcid":false,"given":"Nina","family":"Vasan","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7012-6755","authenticated-orcid":false,"given":"Darja","family":"Djordjevic","sequence":"additional","affiliation":[{"name":"Darja@post.harvard.edu, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3595-3173","authenticated-orcid":false,"given":"Akanksha","family":"Dadlani","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6405-513X","authenticated-orcid":false,"given":"Max","family":"Lamparth","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9619-0768","authenticated-orcid":false,"given":"Eugenia","family":"Kim","sequence":"additional","affiliation":[{"name":"Microsoft, Redmond, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7238-9663","authenticated-orcid":false,"given":"Mykel","family":"Kochenderfer","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,25]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"American Psychiatric Association. 2013. Clinician-Rated Severity of Nonsuicidal Self-Injury. https:\/\/www.psychiatry.org\/File%20Library\/Psychiatrists\/Practice\/DSM\/APA_DSM5_Clinician-Rated-Severity-of-Non-Suicidal-Self-Injury.pdf. DSM-5 Emerging Measure; accessed 2026-01-13."},{"key":"e_1_3_2_1_2_1","unstructured":"American Psychiatric Association. 2013. DSM-5 Clinician-Rated Dimensions of Psychosis Symptom Severity. https:\/\/www.psychiatry.org\/ File%20Library\/Psychiatrists\/Practice\/DSM\/APA_DSM5_Clinician-Rated-Dimensions-of-Psychosis-Symptom-Severity.pdf. Accessed: 2026-01-13. \u00a9 2013 American Psychiatric Association; reproduced with permission for clinical\/research use."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2321"},{"key":"e_1_3_2_1_4_1","volume-title":"ACM Web Science Conference.","author":"Aroyo Lora","year":"2013","unstructured":"Lora Aroyo and Christopher Welty. 2013. Crowd Truth: Harnessing Disagreement in Crowdsourcing a Relation Extraction Gold Standard. In ACM Web Science Conference."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1176\/appi.books.9780890425596"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","unstructured":"Yuntao Bai Saurav Kadavath Sandipan Kundu Amanda Askell Jackson Kernion Andy Jones Anna Chen Anna Goldie Azalia Mirhoseini Cameron McKinnon Carol Chen Catherine Olsson Christopher Olah Danny Hernandez Dawn Drain Deep Ganguli Dustin Li Eli Tran-Johnson Ethan Perez Jamie Kerr Jared Mueller Jeffrey Ladish Joshua Landau Kamal Ndousse Kamile Lukosuite Liane Lovitt Michael Sellitto Nelson Elhage Nicholas Schiefer Noemi Mercado Nova DasSarma Robert Lasenby Robin Larson Sam Ringer Scott Johnston Shauna Kravec Sheer El Showk Stanislav Fort Tamera Lanham Timothy Telleen-Lawton Tom Conerly Tom Henighan Tristan Hume Samuel R. Bowman Zac Hatfield-Dodds Ben Mann Dario Amodei Nicholas Joseph Sam McCandlish Tom Brown and Jared Kaplan. 2022. Constitutional AI: Harmlessness from AI Feedback. arXiv preprint (2022). arXiv:2212.08073 doi:10.48550\/arXiv.2212.08073","DOI":"10.48550\/arXiv.2212.08073"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/9781118511800"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0037562"},{"key":"e_1_3_2_1_10_1","volume-title":"Daniel P Dickstein, Wayne Drevets, Kerri L Kim, Matthew F Pescosolido, Scott Rausch, Karen E Seymour, Yvette Sheline, et al.","author":"Botteron Kelly","year":"2012","unstructured":"Kelly Botteron, Cameron Carter, Francisco Xavier Castellanos, Daniel P Dickstein, Wayne Drevets, Kerri L Kim, Matthew F Pescosolido, Scott Rausch, Karen E Seymour, Yvette Sheline, et al. 2012. Consensus report of the APA work group on neuroimaging markers of psychiatric disorders. Am Psychiatr Assoc (2012)."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2510.23845"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i6.25840"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835540"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.15217"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484161"},{"key":"e_1_3_2_1_17_1","volume-title":"Carl Yan Shan, and Kevin Wadman","author":"Chatterji Aaron","year":"2025","unstructured":"Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. 2025. How people use chatgpt. Technical Report. National Bureau of Economic Research."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v7i1.14432"},{"key":"e_1_3_2_1_19_1","volume-title":"Advances in Neural Information Processing Systems","volume":"30","author":"Christiano Paul F","year":"2017","unstructured":"Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, Vol. 30. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/hash\/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1037\/1040-3590.6.4.284"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-015-9275-x"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.14387"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1037\/t03974-000"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.2196\/mental.7785"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-emnlp.120"},{"key":"e_1_3_2_1_26_1","volume-title":"Impact of Preference Noise on the Alignment Performance of Generative Language Models. In Conference on Language Modeling. https:\/\/openreview.net\/forum?id=nMAaCsCTCI","author":"Gao Yang","year":"2024","unstructured":"Yang Gao, Dana Alon, and Donald Metzler. 2024. Impact of Preference Noise on the Alignment Performance of Generative Language Models. In Conference on Language Modeling. https:\/\/openreview.net\/forum?id=nMAaCsCTCI"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.hcinlp-1.8"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1001\/archpsyc.1989.01810110048007"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502004"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1101\/2024.04.07.24305462"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.16349"},{"key":"e_1_3_2_1_32_1","volume-title":"Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society","volume":"8","author":"Iftikhar Zainab","year":"2025","unstructured":"Zainab Iftikhar, Amy Xiao, Sean Ransom, Jeff Huang, and Harini Suresh. 2025. How LLM counselors violate ethical standards in mental health practice: A practitioner-informed framework. In Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society, Vol. 8. 1311\u20131323."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.06674"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.2196\/12106"},{"key":"e_1_3_2_1_35_1","first-page":"176","article-title":"The Role of Diagnosis in Psychiatry. Blackwell Scientific Publications, Oxford","author":"Kendell Robert E.","year":"1975","unstructured":"Robert E. Kendell. 1975. The Role of Diagnosis in Psychiatry. Blackwell Scientific Publications, Oxford, England. Pages: viii, 176.","journal-title":"England. Pages"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-2958.2004.tb00738.x"},{"key":"e_1_3_2_1_37_1","volume-title":"Manu Sharma, Aryan Shrivastava, Nina Vasan, and Colleen Waickman.","author":"Lamparth Max","year":"2026","unstructured":"Max Lamparth, Declan Grabb, Amy Franks, Scott Gershan, Kaitlyn N Kunstman, Aaron Lulla, Monika Drummond Roots, Manu Sharma, Aryan Shrivastava, Nina Vasan, and Colleen Waickman. 2026. Moving Beyond Medical Exams: A Clinician-Annotated Fairness Dataset of Real-World Tasks and Ambiguity in Mental Healthcare. In The Fourteenth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=tSy7OtONsg"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0156322"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.acl-long.1169"},{"key":"e_1_3_2_1_40_1","volume-title":"Hashimoto","author":"Li Xuechen","year":"2023","unstructured":"Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https:\/\/github.com\/tatsu-lab\/alpaca_eval."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","unstructured":"Yahan Li Jifan Yao John Bosco S. Bunyi Adam C. Frank Angel Hwang and Ruishan Liu. 2025. CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering. arXiv:2506.08584 doi:10.48550\/arXiv.2506.08584","DOI":"10.48550\/arXiv.2506.08584"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.2196\/52095"},{"key":"e_1_3_2_1_43_1","volume-title":"A diagnostic meta-analysis of the Patient Health Questionnaire-9 (PHQ-9) algorithm scoring method as a screen for depression. General hospital psychiatry 37, 1","author":"Manea Laura","year":"2015","unstructured":"Laura Manea, Simon Gilbody, and Dean McMillan. 2015. A diagnostic meta-analysis of the Patient Health Questionnaire-9 (PHQ-9) algorithm scoring method as a screen for depression. General hospital psychiatry 37, 1 (2015), 67\u201375."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.1.1.30"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3715275.3732039"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsat.2016.01.001"},{"key":"e_1_3_2_1_47_1","volume-title":"Department of Veterans Affairs","author":"National Center for PTSD, U.S.","year":"2015","unstructured":"National Center for PTSD, U.S. Department of Veterans Affairs. 2015. Clinician-Administered PTSD Scale for DSM-5 (CAPS-5): Past Week Version. https:\/\/www.ptsd.va.gov\/professional\/assessment\/documents\/CAPS_5_Past_Week.pdf. Assessment instrument; accessed 2026-01-13."},{"key":"e_1_3_2_1_48_1","volume-title":"NICHQ Vanderbilt Assessment Scales. https:\/\/nichq.org\/wp-content\/uploads\/2024\/09\/NICHQ-Vanderbilt-Assessment-Scales.pdf. Assessment instrument","author":"National Institute for Children's Health Quality (NICHQ). 2002.","year":"2026","unstructured":"National Institute for Children's Health Quality (NICHQ). 2002. NICHQ Vanderbilt Assessment Scales. https:\/\/nichq.org\/wp-content\/uploads\/2024\/09\/NICHQ-Vanderbilt-Assessment-Scales.pdf. Assessment instrument; accessed 2026-01-13."},{"key":"e_1_3_2_1_49_1","unstructured":"National Institute of Mental Health. 2024. Depression. https:\/\/www.nimh.nih.gov\/health\/publications\/depression. NIH Publication No. 24-MH-8079."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.glmedi.2024.100099"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-2011"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00293"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","unstructured":"Ethan Perez Saffron Huang Francis Song Trevor Cai Roman Ring John Aslanides Amelia Glaese Nat McAleese and Geoffrey Irving. 2022. Red Teaming Language Models with Language Models. arXiv:2202.03286 doi:10.48550\/arXiv.2202.03286","DOI":"10.48550\/arXiv.2202.03286"},{"key":"e_1_3_2_1_54_1","unstructured":"Kelly Posner David Brent Christopher Lucas Madelyn Gould Barbara Stanley Gregory Brown Prudence Fisher Jamie Zelazny Ainsley Burke Maria Oquendo and John Mann. 2010. Columbia-Suicide Severity Rating Scale (C-SSRS): Pediatric - Since Last Contact -Communities and Healthcare. https:\/\/cssrs.columbia.edu\/wp-content\/uploads\/C-SSRS_Pediatric-SLC_11.14.16.pdf. Version 6\/23\/10; accessed 2026-01-13."},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.2196\/24850"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","unstructured":"Rafael Rafailov Archit Sharma Eric Mitchell Stefano Ermon Christopher D. Manning and Chelsea Finn. 2024. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290 doi:10.48550\/arXiv.2305.18290","DOI":"10.48550\/arXiv.2305.18290"},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ajp.2012.12070999"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"crossref","unstructured":"Tony Rousmaniere Xu Li Yimeng Zhang and Siddharth Shah. 2025. Large language models as mental health resources: Patterns of use in the united states.","DOI":"10.31234\/osf.io\/q8m7g_v2"},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.2196\/69329"},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-022-00593-2"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.425"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1037\/\/0033-2909.86.2.420"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.4103\/indianjpsychiatry.indianjpsychiatry_926_23"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03423-7"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2009.01325"},{"key":"e_1_3_2_1_66_1","volume-title":"A Practical Guide to Fine-Tuning Language Models with Limited Data. arXiv preprint","author":"Szep Marton","year":"2024","unstructured":"Marton Szep, Daniel Rueckert, R\u00fcdiger von Eisenhart-Rothe, and Florian Hinterwimmer. 2024. A Practical Guide to Fine-Tuning Language Models with Limited Data. arXiv preprint (2024). arXiv:2411.09539"},{"key":"e_1_3_2_1_67_1","volume-title":"International Journal of Methods in Psychiatric Research 13 (01","author":"Ventura Joseph","year":"1993","unstructured":"Joseph Ventura, D. Lukoff, Keith Nuechterlein, R. P. Liberman, Megan Green, and Andrew Shaner. 1993. Brief Psychiatric Rating Scale Expanded version 4.0: Scales anchor points and administration manual. International Journal of Methods in Psychiatric Research 13 (01 1993), 221\u2013244."},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1001\/archpsyc.62.6.603"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1192\/bjp.133.5.429"},{"key":"e_1_3_2_1_70_1","doi-asserted-by":"publisher","unstructured":"Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu Yonghao Zhuang Zi Lin Zhuohan Li Dacheng Li Eric P. Xing Hao Zhang Joseph E. Gonzalez and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 doi:10.48550\/arXiv.2306.05685","DOI":"10.48550\/arXiv.2306.05685"},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2400"}],"event":{"name":"FAccT '26: The 2026 ACM Conference on Fairness, Accountability, and Transparency","location":"Montreal QC Canada","acronym":"FAccT '26","sponsor":["ACM\/SIG"]},"container-title":["Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3805689.3812332","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T19:00:21Z","timestamp":1782759621000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3805689.3812332"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,25]]},"references-count":71,"alternative-id":["10.1145\/3805689.3812332","10.1145\/3805689"],"URL":"https:\/\/doi.org\/10.1145\/3805689.3812332","relation":{},"subject":[],"published":{"date-parts":[[2026,6,25]]},"assertion":[{"value":"2026-06-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}