{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,14]],"date-time":"2026-06-14T23:08:26Z","timestamp":1781478506027,"version":"3.54.1"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,3,7]],"date-time":"2025-03-07T00:00:00Z","timestamp":1741305600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,7]],"date-time":"2025-03-07T00:00:00Z","timestamp":1741305600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Red teaming, the practice of adversarially exposing unexpected or undesired model behaviors, is critical towards improving equity and accuracy of large language models, but non-model creator-affiliated red teaming is scant in healthcare. We convened teams of clinicians, medical and engineering students, and technical professionals (80 participants total) to stress-test models with real-world clinical cases and categorize inappropriate responses along axes of safety, privacy, hallucinations\/accuracy, and bias. Six medically-trained reviewers re-analyzed prompt-response pairs and added qualitative annotations. Of 376 unique prompts (1504 responses), 20.1% were inappropriate (GPT-3.5: 25.8%; GPT-4.0: 16%; GPT-4.0 with Internet: 17.8%). Subsequently, we show the utility of our benchmark by testing GPT-4o, a model released after our event (20.4% inappropriate). 21.5% of responses appropriate with GPT-3.5 were inappropriate in updated models. We share insights for constructing red teaming prompts, and present our benchmark for iterative model assessments.<\/jats:p>","DOI":"10.1038\/s41746-025-01542-0","type":"journal-article","created":{"date-parts":[[2025,3,7]],"date-time":"2025-03-07T12:29:51Z","timestamp":1741350591000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Red teaming ChatGPT in medicine to yield real-world insights on model behavior"],"prefix":"10.1038","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0647-7040","authenticated-orcid":false,"given":"Crystal T.","family":"Chang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hodan","family":"Farah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0564-940X","authenticated-orcid":false,"given":"Haiwen","family":"Gui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shawheen Justin","family":"Rezaei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5497-9037","authenticated-orcid":false,"given":"Charbel","family":"Bou-Khalil","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ye-Jean","family":"Park","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3426-9289","authenticated-orcid":false,"given":"Akshay","family":"Swaminathan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jesutofunmi A.","family":"Omiye","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akaash","family":"Kolluri","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3438-3313","authenticated-orcid":false,"given":"Akash","family":"Chaurasia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6952-289X","authenticated-orcid":false,"given":"Alejandro","family":"Lozano","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alice","family":"Heiman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Allison Sihan","family":"Jia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amit","family":"Kaushal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angela","family":"Jia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-1327-2184","authenticated-orcid":false,"given":"Angelica","family":"Iacovelli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Archer","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arghavan","family":"Salles","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arpita","family":"Singhal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Balasubramanian","family":"Narasimhan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Belai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5550-3559","authenticated-orcid":false,"given":"Benjamin H.","family":"Jacobson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Binglan","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Celeste H.","family":"Poe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chandan","family":"Sanghera","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenming","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Conor","family":"Messer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Damien Varid","family":"Kettud","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deven","family":"Pandya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6123-7499","authenticated-orcid":false,"given":"Dhamanpreet","family":"Kaur","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diana","family":"Hla","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diba","family":"Dindoust","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dominik","family":"Moehrle","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Duncan","family":"Ross","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ellaine","family":"Chou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fateme Nateghi","family":"Haredasht","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ge","family":"Cheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Irena","family":"Gao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacob","family":"Chang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jake","family":"Silberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9316-5768","authenticated-orcid":false,"given":"Jason A.","family":"Fries","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiapeng","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joe","family":"Jamison","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9868-2366","authenticated-orcid":false,"given":"John S.","family":"Tamaresis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4387-8740","authenticated-orcid":false,"given":"Jonathan H.","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua","family":"Lazaro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Juan M.","family":"Banda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julie J.","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karen Ebert","family":"Matthys","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9759-616X","authenticated-orcid":false,"given":"Kirsten R.","family":"Steffner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lu","family":"Tian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luca","family":"Pegolotti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Malathi","family":"Srinivasan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maniragav","family":"Manimaran","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Schwede","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Minghe","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7149-849X","authenticated-orcid":false,"given":"Minh","family":"Nguyen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7962-9148","authenticated-orcid":false,"given":"Mohsen","family":"Fathzadeh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qian","family":"Zhao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rika","family":"Bajra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rohit","family":"Khurana","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ruhana","family":"Azam","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rush","family":"Bartlett","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sang T.","family":"Truong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Scott L.","family":"Fleming","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shriti","family":"Raj","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Solveig","family":"Behr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sonia","family":"Onyeka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sri","family":"Muppidi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tarek","family":"Bandali","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7084-9646","authenticated-orcid":false,"given":"Tiffany Y.","family":"Eulalio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenyuan","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuanyu","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8967-2139","authenticated-orcid":false,"given":"Yanan","family":"Ding","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3697-5155","authenticated-orcid":false,"given":"Ying","family":"Cui","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7200-3222","authenticated-orcid":false,"given":"Yuqi","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6695-8991","authenticated-orcid":false,"given":"Yutong","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9385-7158","authenticated-orcid":false,"given":"Nigam","family":"Shah","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7988-9356","authenticated-orcid":false,"given":"Roxana","family":"Daneshjou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,3,7]]},"reference":[{"key":"1542_CR1","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","volume":"3","author":"J Clusmann","year":"2023","unstructured":"Clusmann, J. et al. The future landscape of large language models in medicine. Commun. Med. 3, 141 (2023).","journal-title":"Commun. Med."},{"key":"1542_CR2","doi-asserted-by":"publisher","first-page":"e379","DOI":"10.1016\/S2589-7500(24)00060-8","volume":"6","author":"S Chen","year":"2024","unstructured":"Chen, S. et al. The effect of using a large language model to respond to patient messages. Lancet Digit. Health 6, e379\u2013e381 (2024).","journal-title":"Lancet Digit. Health"},{"key":"1542_CR3","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1038\/s41746-023-00939-z","volume":"6","author":"JA Omiye","year":"2023","unstructured":"Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023).","journal-title":"NPJ Digit. Med."},{"key":"1542_CR4","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","volume":"29","author":"AJ Thirunavukarasu","year":"2023","unstructured":"Thirunavukarasu, A. J. et al. Large language models in medicine. Nat. Med. 29, 1930\u20131940 (2023).","journal-title":"Nat. Med."},{"key":"1542_CR5","doi-asserted-by":"publisher","first-page":"2298","DOI":"10.1016\/j.jid.2024.03.028","volume":"144","author":"H Gui","year":"2024","unstructured":"Gui, H. et al. Dermatologists\u2019 Perspectives and Usage of Large Language Models in Practice: An Exploratory Survey. J. Invest. Dermatol. 144, 2298\u20132301 (2024).","journal-title":"J. Invest. Dermatol."},{"key":"1542_CR6","unstructured":"Fox, A. How Epic is using AI to change the way EHRs work. Healthcare IT News https:\/\/www.healthcareitnews.com\/news\/how-epic-using-ai-change-way-ehrs-work (2023)."},{"key":"1542_CR7","unstructured":"Oracle brings generative AI capabilities to healthcare. https:\/\/www.oracle.com\/news\/announcement\/ohc-oracle-brings-generative-ai-capabilities-to-healthcare-2023-09-18\/."},{"key":"1542_CR8","unstructured":"Diaz, N. Which Big Tech companies health systems are choosing for AI partnerships. https:\/\/www.beckershospitalreview.com\/innovation\/which-big-tech-companies-health-systems-are-choosing-for-ai-partnerships.html."},{"key":"1542_CR9","doi-asserted-by":"publisher","first-page":"e243201","DOI":"10.1001\/jamanetworkopen.2024.3201","volume":"7","author":"P Garcia","year":"2024","unstructured":"Garcia, P. et al. Artificial intelligence-generated draft replies to patient inbox messages. JAMA Netw. Open 7, e243201 (2024).","journal-title":"JAMA Netw. Open"},{"key":"1542_CR10","doi-asserted-by":"publisher","first-page":"e240357","DOI":"10.1001\/jamanetworkopen.2024.0357","volume":"7","author":"J Zaretsky","year":"2024","unstructured":"Zaretsky, J. et al. Generative artificial intelligence to transform inpatient discharge summaries to patient-friendly language and format. JAMA Netw. Open 7, e240357 (2024).","journal-title":"JAMA Netw. Open"},{"key":"1542_CR11","unstructured":"Introducing ChatGPT. https:\/\/openai.com\/blog\/chatgpt."},{"key":"1542_CR12","doi-asserted-by":"crossref","unstructured":"Feffer, M., Sinha, A., Lipton, Z. C. & Heidari, H. Red-Teaming for Generative AI: Silver Bullet or Security Theater? arXiv [cs.CY] (2024).","DOI":"10.1609\/aies.v7i1.31647"},{"key":"1542_CR13","doi-asserted-by":"publisher","first-page":"e12","DOI":"10.1016\/S2589-7500(23)00225-X","volume":"6","author":"T Zack","year":"2024","unstructured":"Zack, T. et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit Health 6, e12\u2013e22 (2024).","journal-title":"Lancet Digit Health"},{"key":"1542_CR14","doi-asserted-by":"crossref","unstructured":"Yang, Y., Liu, X., Jin, Q., Huang, F. & Lu, Z. Unmasking and Quantifying Racial Bias of Large Language Models in Medical Report Generation. arXiv [cs.CL] (2024).","DOI":"10.1038\/s43856-024-00601-z"},{"key":"1542_CR15","doi-asserted-by":"crossref","unstructured":"Zhang, A., Yuksekgonul, M., Guild, J., Zou, J. & Wu, J. C. ChatGPT Exhibits Gender and Racial Biases in Acute Coronary Syndrome Management. arXiv [cs.CY] (2023).","DOI":"10.1101\/2023.11.14.23298525"},{"key":"1542_CR16","first-page":"12","volume":"8","author":"AY Tsou","year":"2017","unstructured":"Tsou, A. Y. et al. Safe practices for copy and paste in the EHR. Systematic review, recommendations, and novel model for health IT collaboration. Appl. Clin. Inform. 8, 12\u201334 (2017).","journal-title":"Appl. Clin. Inform."},{"key":"1542_CR17","unstructured":"Zhao, J., Fang, M., Pan, S., Yin, W. & Pechenizkiy, M. GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models. arXiv [cs.CL] (2023)."},{"key":"1542_CR18","unstructured":"Sun, H., Zhang, Z., Deng, J., Cheng, J. & Huang, M. Safety Assessment of Chinese Large Language Models. arXiv [cs.CL] (2023)."},{"key":"1542_CR19","doi-asserted-by":"crossref","unstructured":"Omar, R., Mangukiya, O., Kalnis, P. & Mansour, E. ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots. arXiv [cs.CL] (2023).","DOI":"10.1145\/3588911"},{"key":"1542_CR20","unstructured":"Chiu, Y. Y. et al. CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs\u2019 (Lack of) Multicultural Knowledge. arXiv [cs.CL] (2024)."},{"key":"1542_CR21","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41591-024-03328-5","volume":"31","author":"S Johri","year":"2025","unstructured":"Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat. Med. 31, 77\u201386 (2025).","journal-title":"Nat. Med."},{"key":"1542_CR22","doi-asserted-by":"crossref","unstructured":"Wornow, M. et al. Zero-shot clinical trial patient matching with LLMs. arXiv [cs.CL] (2024).","DOI":"10.1056\/AIcs2400360"},{"key":"1542_CR23","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-53081-z","volume":"15","author":"Q Jin","year":"2024","unstructured":"Jin, Q. et al. Matching patients to clinical trials with large language models. Nat. Commun. 15, 9074 (2024).","journal-title":"Nat. Commun."},{"key":"1542_CR24","doi-asserted-by":"crossref","unstructured":"Shuster, K., Poff, S., Chen, M., Kiela, D. & Weston, J. Retrieval Augmentation Reduces Hallucination in Conversation. arXiv [cs.CL] (2021).","DOI":"10.18653\/v1\/2021.findings-emnlp.320"},{"key":"1542_CR25","unstructured":"Tian, K., Mitchell, E., Yao, H., Manning, C. D. & Finn, C. Fine-tuning Language Models for Factuality. arXiv [cs.CL] (2023)."},{"key":"1542_CR26","doi-asserted-by":"crossref","unstructured":"Dhuliawala, S. et al. Chain-of-Verification Reduces Hallucination in Large Language Models. arXiv [cs.CL] (2023).","DOI":"10.18653\/v1\/2024.findings-acl.212"},{"key":"1542_CR27","doi-asserted-by":"crossref","unstructured":"Chen, L., Zaharia, M. & Zou, J. How is ChatGPT\u2019s behavior changing over time? arXiv [cs.CL] (2023).","DOI":"10.1162\/99608f92.5317da47"},{"key":"1542_CR28","doi-asserted-by":"publisher","first-page":"e100444","DOI":"10.1136\/bmjhci-2021-100444","volume":"28","author":"S Reddy","year":"2021","unstructured":"Reddy, S. et al. Evaluation framework to guide implementation of AI systems into healthcare settings. BMJ Health Care Inform 28, e100444 (2021).","journal-title":"BMJ Health Care Inform"},{"key":"1542_CR29","doi-asserted-by":"crossref","unstructured":"Callahan, A. et al. Standing on FURM ground -- A framework for evaluating Fair, Useful, and Reliable AI Models in healthcare systems. arXiv [cs.CY] (2024).","DOI":"10.1056\/CAT.24.0131"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01542-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01542-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01542-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,13]],"date-time":"2025-11-13T06:34:21Z","timestamp":1763015661000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-01542-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,7]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["1542"],"URL":"https:\/\/doi.org\/10.1038\/s41746-025-01542-0","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,7]]},"assertion":[{"value":"16 May 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 February 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 March 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"RD has served as an advisor to MDAlgorithms and Revea and received. consulting fees from Pfizer, L\u2019Oreal, Frazier Healthcare Partners, and DWA, and research funding from UCB and declares no non-financial competing interests. All other authors declare no financial or non-financial competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"149"}}