{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T15:23:29Z","timestamp":1786116209163,"version":"build-2736575974"},"reference-count":34,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T00:00:00Z","timestamp":1783728000000},"content-version":"vor","delay-in-days":4,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,7,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To develop a pipeline for evaluating large language models (LLMs) on the task of capturing symptoms from clinical encounters.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We created a gold standard dataset of symptom annotations from simulated doctor-patient encounter excerpts (264 encounters; 16 symptoms; double-coded and adjudicated). Nine different LLMs from 4 vendors (OpenAI, Meta, DeepSeek, Moonshot AI) were used as examples to test our evaluation pipeline; outputs were assessed for correct structure and symptom information.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Of 3085 excerpts, 2087 (68%) contained symptoms. Pain, cough, and shortness of breath were most common; LLMs achieved F1 scores ranging 0.66-0.88 for these symptoms with minimal prompt engineering. Of tested models, GPT-4.1 demonstrated the best overall performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Our evaluation pipeline and benchmarking dataset are publicly available and applicable to various LLMs, including open-source models.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>This work supports the development and optimization of models that seek to improve patient symptom understanding.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooag134","type":"journal-article","created":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T11:45:12Z","timestamp":1783424712000},"source":"Crossref","is-referenced-by-count":0,"title":["Development of a benchmarking dataset for symptom detection using large language models"],"prefix":"10.1093","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7324-6018","authenticated-orcid":false,"given":"Joshua","family":"Davis","sequence":"first","affiliation":[{"name":"Department of Supportive Oncology, Dana-Farber Cancer Institute , Boston, MA, 02215,","place":["United States"]},{"name":"Albany Medical College , Albany, NY, 12208,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6036-1420","authenticated-orcid":false,"given":"Brigitte N","family":"Durieux","sequence":"additional","affiliation":[{"name":"Department of Supportive Oncology, Dana-Farber Cancer Institute , Boston, MA, 02215,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chloe","family":"Van Dongen","sequence":"additional","affiliation":[{"name":"Department of Supportive Oncology, Dana-Farber Cancer Institute , Boston, MA, 02215,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kate","family":"Sciacca","sequence":"additional","affiliation":[{"name":"Department of Supportive Oncology, Dana-Farber Cancer Institute , Boston, MA, 02215,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Juan Manuel","family":"Gutierrez","sequence":"additional","affiliation":[{"name":"Department of Informatics & Analytics, Dana-Farber Cancer Institute , Boston, MA, 02215,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charlotta","family":"Lindvall","sequence":"additional","affiliation":[{"name":"Department of Supportive Oncology, Dana-Farber Cancer Institute , Boston, MA, 02215,","place":["United States"]},{"name":"Harvard Medical School, Harvard University , Boston, MA, 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,7,10]]},"reference":[{"key":"2026080604265441600_ooag134-B1","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1200\/JCO.2015.63.0830","article-title":"Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial","volume":"34","author":"Basch","year":"2016","journal-title":"J Clin Oncol"},{"key":"2026080604265441600_ooag134-B2","doi-asserted-by":"publisher","first-page":"100576","DOI":"10.1016\/j.adro.2020.09.016","article-title":"Evaluation of mobile health applications to track patient-reported outcomes for oncology patients: a systematic review","volume":"6","author":"Lu","year":"2021","journal-title":"Adv Radiat Oncol"},{"key":"2026080604265441600_ooag134-B3","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1097\/SLA.0000000000005029","article-title":"A review of PROM implementation in surgical practice","volume":"275","author":"Sokas","year":"2022","journal-title":"Ann Surg"},{"key":"2026080604265441600_ooag134-B4","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1038\/s42254-022-00441-7","article-title":"Scientific machine learning benchmarks","volume":"4","author":"Thiyagalingam","year":"2022","journal-title":"Nat Rev Phys"},{"key":"2026080604265441600_ooag134-B5","doi-asserted-by":"publisher","first-page":"916","DOI":"10.1016\/j.ijrobp.2023.11.012","article-title":"Toward clinical-grade evaluation of large language models","volume":"118","author":"Moreno","year":"2024","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"2026080604265441600_ooag134-B6","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1038\/s41597-022-01423-1","article-title":"A dataset of simulated patient-physician medical interviews with a focus on respiratory cases","volume":"9","author":"Fareez","year":"2022","journal-title":"Sci Data"},{"key":"2026080604265441600_ooag134-B7","doi-asserted-by":"publisher","first-page":"AIcs2300191","DOI":"10.1056\/AIcs2300191","article-title":"GPT-4 in a Cancer Center\u2013Institute-wide deployment challenges and lessons learned","volume":"1","author":"Umeton","year":"2024","journal-title":"NEJM AI"},{"key":"2026080604265441600_ooag134-B8","author":"Liu","year":"2023"},{"key":"2026080604265441600_ooag134-B9","author":"Chase","year":"2022"},{"key":"2026080604265441600_ooag134-B10","author":"U.S. Department of Health and Human Services, National Institutes of Health, National Cancer Institute","year":"2009"},{"key":"2026080604265441600_ooag134-B11","author":"Tkachenko","year":"2020"},{"key":"2026080604265441600_ooag134-B12","doi-asserted-by":"publisher","first-page":"3","DOI":"10.2466\/pr0.1966.19.1.3","article-title":"The intraclass correlation coefficient as a measure of reliability","volume":"19","author":"Bartko","year":"1966","journal-title":"Psychol Rep"},{"key":"2026080604265441600_ooag134-B13","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ Psychol Meas"},{"key":"2026080604265441600_ooag134-B14","author":"Rao","year":"2023"},{"key":"2026080604265441600_ooag134-B15","author":"OpenAI","year":"2024"},{"key":"2026080604265441600_ooag134-B16","author":"Pydantic","year":"2024"},{"key":"2026080604265441600_ooag134-B17","doi-asserted-by":"publisher","first-page":"e53559","DOI":"10.2196\/53559","article-title":"The temperature feature of ChatGPT: modifying creativity for clinical research","volume":"11","author":"Davis","year":"2024","journal-title":"JMIR Hum Factors."},{"key":"2026080604265441600_ooag134-B18","doi-asserted-by":"publisher","first-page":"256","DOI":"10.2307\/2332378","article-title":"The comparison of percentages in matched samples","volume":"37","author":"Cochran","year":"1950","journal-title":"Biometrika"},{"key":"2026080604265441600_ooag134-B19","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1007\/BF02295996","article-title":"Note on the sampling error of the difference between correlated proportions or percentages","volume":"12","author":"McNemar","year":"1947","journal-title":"Psychometrika"},{"key":"2026080604265441600_ooag134-B20","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2300110","article-title":"CORAL: expert-curated medical oncology reports to advance language model inference","volume":"1","author":"Sushil","year":"2024","journal-title":"NEJM AI"},{"key":"2026080604265441600_ooag134-B21","doi-asserted-by":"publisher","first-page":"101519","DOI":"10.1016\/j.imu.2024.101519","article-title":"Leveraging mobile NER for real-time capture of symptoms, diagnoses, and treatments from clinical dialogues","volume":"48","author":"Rhouma","year":"2024","journal-title":"Inform Med Unlocked"},{"key":"2026080604265441600_ooag134-B22","doi-asserted-by":"publisher","first-page":"e2100136","DOI":"10.1200\/CCI.21.00136","article-title":"Deep learning for cancer symptoms monitoring on the basis of electronic health record unstructured clinical notes","volume":"6","author":"Lindvall","year":"2022","journal-title":"JCO Clin Cancer Inform"},{"key":"2026080604265441600_ooag134-B23","doi-asserted-by":"publisher","first-page":"ooad009","DOI":"10.1093\/jamiaopen\/ooad009","article-title":"Development of a keyword library for capturing PRO-CTCAE-focused \u201csymptom talk\u201d in oncology conversations","volume":"6","author":"Durieux","year":"2023","journal-title":"JAMIA Open"},{"key":"2026080604265441600_ooag134-B24","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1016\/j.jpainsymman.2021.10.014","article-title":"Identification of uncontrolled symptoms in cancer patients using natural language processing","volume":"63","author":"DiMartino","year":"2022","journal-title":"J Pain Symptom Manage"},{"key":"2026080604265441600_ooag134-B25","doi-asserted-by":"publisher","first-page":"948","DOI":"10.1016\/j.jpainsymman.2020.06.010","article-title":"Deep natural language processing to identify symptom documentation in clinical notes for patients with heart failure undergoing cardiac resynchronization therapy","volume":"60","author":"Leiter","year":"2020","journal-title":"J Pain Symptom Manage"},{"key":"2026080604265441600_ooag134-B26","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.ijmedinf.2019.02.008","article-title":"A systematic review of natural language processing and text mining of symptoms from electronic patient-authored text data","volume":"125","author":"Dreisbach","year":"2019","journal-title":"Int J Med Inform"},{"key":"2026080604265441600_ooag134-B27","doi-asserted-by":"publisher","first-page":"e58977","DOI":"10.2196\/58977","article-title":"Automated system to capture patient symptoms from multitype Japanese clinical texts: retrospective study","volume":"12","author":"Nishiyama","year":"2024","journal-title":"JMIR Med Inform."},{"key":"2026080604265441600_ooag134-B28","doi-asserted-by":"publisher","first-page":"310","DOI":"10.1002\/path.6232","article-title":"Extracting structured information from unstructured histopathology reports using generative pre-trained transformer 4 (GPT-4)","volume":"262","author":"Truhn","year":"2024","journal-title":"J Pathol"},{"key":"2026080604265441600_ooag134-B29","doi-asserted-by":"publisher","first-page":"e230725","DOI":"10.1148\/radiol.230725","article-title":"Leveraging GPT-4 for post hoc transformation of free-text radiology reports into structured reporting: a multilingual feasibility study","volume":"307","author":"Adams","year":"2023","journal-title":"Radiology"},{"key":"2026080604265441600_ooag134-B30","doi-asserted-by":"publisher","first-page":"190","DOI":"10.1016\/j.jpainsymman.2024.05.015","article-title":"Symptom-BERT: enhancing cancer symptom detection in EHR clinical notes","volume":"68","author":"Zeinali","year":"2024","journal-title":"J Pain Symptom Manage"},{"key":"2026080604265441600_ooag134-B31","doi-asserted-by":"publisher","DOI":"10.1016\/j.gastha.2024.10.003","article-title":"Large language models outperform traditional natural language processing methods in extracting patient-reported outcomes in inflammatory bowel disease","volume-title":"Gastro Hep Adv","author":"Patel","year":"2024"},{"key":"2026080604265441600_ooag134-B32","doi-asserted-by":"publisher","first-page":"e12","DOI":"10.1016\/S2589-7500(23)00225-X","article-title":"Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study","volume":"6","author":"Zack","year":"2024","journal-title":"Lancet Digit Health"},{"key":"2026080604265441600_ooag134-B33","first-page":"113","author":"Cirillo","year":"2022"},{"key":"2026080604265441600_ooag134-B34","doi-asserted-by":"publisher","first-page":"1174","DOI":"10.1038\/s41591-024-02885-z","article-title":"Demographic bias in misdiagnosis by computational pathology models","volume":"30","author":"Vaidya","year":"2024","journal-title":"Nat Med"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/4\/ooag134\/68757699\/ooag134.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/4\/ooag134\/68757699\/ooag134.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T08:26:58Z","timestamp":1786004818000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooag134\/8732327"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,7]]},"references-count":34,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,7,7]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooag134","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,8]]},"published":{"date-parts":[[2026,7,7]]},"article-number":"ooag134"}}