{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T17:17:59Z","timestamp":1790615879708,"version":"4.1.0"},"reference-count":18,"publisher":"BMJ","issue":"1","license":[{"start":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T00:00:00Z","timestamp":1790467200000},"content-version":"unspecified","delay-in-days":26,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000025","name":"National Institute of Mental Health","doi-asserted-by":"publisher","award":["R01 MH135034"],"award-info":[{"award-number":["R01 MH135034"]}],"id":[{"id":"10.13039\/100000025","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Ment Health"],"accepted":{"date-parts":[[2026,9,8]]},"published-print":{"date-parts":[[2026,9]]},"abstract":"<jats:p>Medical artificial intelligence (AI) benchmarks are increasingly used to assess the readiness of large language models for health-related tasks, but aggregate performance scores can obscure clinically meaningful variation across domains. HealthBench, an open benchmark of 5000 multi-turn health conversations, represents a salient example: the score is difficult to interpret for decisions about specific clinical use cases, including those with mental health needs. Conversations involving suicidality, psychosis, eating disorders, trauma or relational dependence require recognising ambiguous disclosures, responding safely under uncertainty, avoiding reinforcement of harmful beliefs or over-reliance and knowing when to escalate to human-based crisis support. We propose that HealthBench and similar benchmarks report domain-specific results. To demonstrate feasibility and utility, we adopted OpenAI\u2019s classified dataset to identify 332 mental health conversations, validated against human review (sensitivity 0.980, specificity 1.000, positive predictive value 1.000, negative predictive value 0.980, F1 0.990, inter-rater agreement 99.0%). We analysed mental health domain performance separately from overall HealthBench performance using a stratified random sample of 500 conversations. Across deployments, mental health point estimates differed from overall estimates by \u22120.034 to +0.047. Mental health domain composition was also uneven: for example, postpartum depression represented 26.8% of mental health conversations compared with 4.5% for suicidal crisis, 0.9% for psychosis and 1.5% for eating disorders. The findings suggest benchmark averages can mask domain-specific insights relevant to clinical safety. We argue that interpretable medical AI evaluation requires transparent domain composition; subgroup and failure mode reporting; calibration against clinically meaningful anchors, including human expert performance; and reproducible versioned analyses as models, grader systems and deployment contexts evolve.<\/jats:p>","DOI":"10.1136\/bmjment-2026-302921","type":"journal-article","created":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T16:30:37Z","timestamp":1790613037000},"page":"e302921","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":0,"title":["Making medical AI benchmarks clinically interpretable: the case of mental health"],"prefix":"10.1136","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0073-0348","authenticated-orcid":false,"given":"Ryan K","family":"McBain","sequence":"first","affiliation":[{"name":"Harvard Medical School, Boston, Massachusetts, USA"},{"name":"RAND Corporation, Arlington, Virginia, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ellice","family":"Huang","sequence":"additional","affiliation":[{"name":"RAND, Santa Monica, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caroline","family":"Figueroa","sequence":"additional","affiliation":[{"name":"Stanford University, Palo Alto, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Li Ang","family":"Zhang","sequence":"additional","affiliation":[{"name":"RAND, Santa Monica, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan","family":"Cantor","sequence":"additional","affiliation":[{"name":"RAND, Santa Monica, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2026,9,28]]},"reference":[{"key":"2026092809300572000_29.1.e302921.1","unstructured":"OpenAI . Introducing HealthBench. Available: https:\/\/openai.com\/index\/healthbench\/ [Accessed 06 May 2026]."},{"key":"2026092809300572000_29.1.e302921.2","unstructured":"Arora RK , Wei J , Hicks RS , et al . HealthBench: evaluating large language models towards improved human health. arXiv 2025. doi:10.48550\/arXiv.2505.08775"},{"key":"2026092809300572000_29.1.e302921.3","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-024-01074-z","article-title":"Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI","volume":"7","author":"Abbasian","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2026092809300572000_29.1.e302921.4","doi-asserted-by":"crossref","unstructured":"McCoy LG , Swamy R , Sagar N , et al . Assessment of Large Language Models in Clinical Reasoning: A Novel Benchmarking Study. NEJM AI 2025;2. doi:10.1056\/AIdbp2500120","DOI":"10.1056\/AIdbp2500120"},{"key":"2026092809300572000_29.1.e302921.5","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1126\/science.adz9193","article-title":"Transforming mental health research and care through artificial intelligence","volume":"391","author":"Opel","year":"2026","journal-title":"Science"},{"key":"2026092809300572000_29.1.e302921.6","doi-asserted-by":"crossref","first-page":"100311","DOI":"10.1016\/j.chbah.2026.100311","article-title":"Talking about mental health with AI-based digital personas: Understanding what users disclose","volume":"8","author":"Carre","year":"2026","journal-title":"Comput Hum Behav Artif Hum"},{"key":"2026092809300572000_29.1.e302921.7","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1001\/jama.2026.11448","article-title":"Specialized or General-Purpose-The Wrong Question for Mental Health AI Safety","volume":"336","author":"Nelson","year":"2026","journal-title":"JAMA"},{"key":"2026092809300572000_29.1.e302921.8","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-02943-6"},{"key":"2026092809300572000_29.1.e302921.9","doi-asserted-by":"crossref","DOI":"10.1038\/s41598-025-17242-4","article-title":"Performance of mental health chatbot agents in detecting and managing suicidal ideation","volume":"15","author":"Pichowicz","year":"2025","journal-title":"Sci Rep"},{"key":"2026092809300572000_29.1.e302921.10","doi-asserted-by":"crossref","first-page":"944","DOI":"10.1176\/appi.ps.20250086","article-title":"Evaluation of Alignment Between Large Language Models and Expert Clinicians in Suicide Risk Assessment","volume":"76","author":"McBain","year":"2025","journal-title":"Psychiatr Serv"},{"key":"2026092809300572000_29.1.e302921.11","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1097\/PRA.0000000000000892","article-title":"Contextualizing Clinical Benchmarks: A Tripartite Approach to Evaluating LLM-Based Tools in Mental Health Settings","volume":"31","author":"Flathers","year":"2025","journal-title":"J Psychiatr Pract"},{"key":"2026092809300572000_29.1.e302921.12","doi-asserted-by":"crossref","DOI":"10.2196\/70014","article-title":"Evaluating Generative AI in Mental Health: Systematic Review of Capabilities and Limitations","volume":"12","author":"Wang","year":"2025","journal-title":"JMIR Ment Health"},{"key":"2026092809300572000_29.1.e302921.13","unstructured":"Making ChatGPT better for clinicians. OpenAI; 2026. Available: https:\/\/openai.com\/index\/making-chatgpt-better-for-clinicians\/ [Accessed 28 Apr 2026]."},{"key":"2026092809300572000_29.1.e302921.14","unstructured":"Hicks RS , Trofimov M , Lim D , et al . HealthBench professional: evaluating large language models on real clinician tasks. 2026. Available: https:\/\/cdn.openai.com\/dd128428-0184-4e25-b155-3a7686c7d744\/HealthBench-Professional.pdf"},{"key":"2026092809300572000_29.1.e302921.15","unstructured":"Introducing trusted contact in ChatGPT. 2026. Available: https:\/\/openai.com\/index\/introducing-trusted-contact-in-chatgpt\/ [Accessed 08 May 2026]."},{"key":"2026092809300572000_29.1.e302921.16","unstructured":"OpenAI . GPT-5.5 system card. OpenAI deployment safety hub. Available: https:\/\/deploymentsafety.openai.com\/gpt-5-5\/health [Accessed 06 May 2026]."},{"key":"2026092809300572000_29.1.e302921.17","unstructured":"openai\/healthbench \u00b7 Datasets at hugging face. Hugging Face; 2025. Available: https:\/\/huggingface.co\/datasets\/openai\/healthbench [Accessed 28 Apr 2026]."},{"key":"2026092809300572000_29.1.e302921.18","doi-asserted-by":"crossref","DOI":"10.2196\/92817","article-title":"AI Chatbot Suicide Risk Detection and Response: Human Validation Study of the Open-Source VERA-MH Safety Evaluation","volume":"5","author":"Bentley","year":"2026","journal-title":"JMIR AI"}],"container-title":["BMJ Mental Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjment-2026-302921","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T16:30:42Z","timestamp":1790613042000},"score":1,"resource":{"primary":{"URL":"https:\/\/mentalhealth.bmj.com\/lookup\/doi\/10.1136\/bmjment-2026-302921"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9]]},"references-count":18,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,9,28]]},"published-print":{"date-parts":[[2026,9]]}},"alternative-id":["10.1136\/bmjment-2026-302921"],"URL":"https:\/\/doi.org\/10.1136\/bmjment-2026-302921","relation":{},"ISSN":["2755-9734"],"issn-type":[{"value":"2755-9734","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9]]}}}