{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,21]],"date-time":"2026-11-21T17:25:16Z","timestamp":1795281916424,"version":"4.10.3"},"reference-count":55,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2024,6,20]],"date-time":"2024-06-20T00:00:00Z","timestamp":1718841600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000057","name":"National Institute of General Medical Sciences","doi-asserted-by":"publisher","award":["R15GM139094"],"award-info":[{"award-number":["R15GM139094"]}],"id":[{"id":"10.13039\/100000057","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>The integration of these preventive guidelines with Electronic Health Records (EHRs) systems, coupled with the generation of personalized preventive care recommendations, holds significant potential for improving healthcare outcomes. Our study investigates the feasibility of using Large Language Models (LLMs) to automate the assessment criteria and risk factors from the guidelines for future analysis against medical records in EHR.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We annotated the criteria, risk factors, and preventive medical services described in the adult guidelines published by United States Preventive Services Taskforce and evaluated 3 state-of-the-art LLMs on extracting information in these categories from the guidelines automatically.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We included 24 guidelines in this study. The LLMs can automate the extraction of all criteria, risk factors, and medical services from 9 guidelines. All 3 LLMs perform well on extracting information regarding the demographic criteria or risk factors. Some LLMs perform better on extracting the social determinants of health, family history, and preventive counseling services than the others.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>While LLMs demonstrate the capability to handle lengthy preventive care guidelines, several challenges persist, including constraints related to the maximum length of input tokens and the tendency to generate content rather than adhering strictly to the original input. Moreover, the utilization of LLMs in real-world clinical settings necessitates careful ethical consideration. It is imperative that healthcare professionals meticulously validate the extracted information to mitigate biases, ensure completeness, and maintain accuracy.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>We developed a data structure to store the annotated preventive guidelines and make it publicly available. Employing state-of-the-art LLMs to extract preventive care criteria, risk factors, and preventive care services paves the way for the future integration of these guidelines into the EHR.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae145","type":"journal-article","created":{"date-parts":[[2024,6,20]],"date-time":"2024-06-20T15:18:43Z","timestamp":1718896723000},"page":"1743-1753","source":"Crossref","is-referenced-by-count":20,"title":["Zero-shot learning to extract assessment criteria and medical services from the preventive healthcare guidelines using large language models"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3649-9785","authenticated-orcid":false,"given":"Xiao","family":"Luo","sequence":"first","affiliation":[{"name":"Department of Management Science and Information Systems, Spears School of Business, Oklahoma State University , Stillwater, OK 74078, United States"},{"name":"Department of Biostatistics and Health Data Science, School of Medicine, Indiana University , Indianapolis, IN 46202, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fattah Muhammad","family":"Tahabi","sequence":"additional","affiliation":[{"name":"Department of Management Science and Information Systems, Spears School of Business, Oklahoma State University , Stillwater, OK 74078, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tressica","family":"Marc","sequence":"additional","affiliation":[{"name":"Department of Computer Information Technology, Purdue School of Engineering and Technology, Indiana University-Purdue University Indianapolis , Indianapolis, IN 46202, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura Ann","family":"Haunert","sequence":"additional","affiliation":[{"name":"School of Nursing, Indiana University , Indianapolis, IN 46202, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0701-7465","authenticated-orcid":false,"given":"Susan","family":"Storey","sequence":"additional","affiliation":[{"name":"School of Nursing, Indiana University , Indianapolis, IN 46202, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,6,20]]},"reference":[{"key":"2024071907523700800_ocae145-B1","author":"Health and Economic Costs of Chronic Diseases","year":"2023"},{"key":"2024071907523700800_ocae145-B2","year":"2023"},{"key":"2024071907523700800_ocae145-B3","author":"Healthcare Effectiveness Data and Information Set (HEDIS) Technical Resources","year":"2023"},{"issue":"5","key":"2024071907523700800_ocae145-B4","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1016\/j.acap.2013.03.004","article-title":"Improving immunization delivery using an electronic health record: the ImmProve project","volume":"13","author":"Bundy","year":"2013","journal-title":"Acad Pediatr"},{"issue":"2","key":"2024071907523700800_ocae145-B5","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1016\/j.amepre.2019.03.016","article-title":"Use of a preventive index to examine clinic-level factors associated with delivery of preventive care","volume":"57","author":"Hatch","year":"2019","journal-title":"Am J Prev Med"},{"issue":"10","key":"2024071907523700800_ocae145-B6","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1001\/jama.291.10.1238","article-title":"Actual causes of death in the United States, 2000","volume":"291","author":"Mokdad","year":"2004","journal-title":"JAMA"},{"key":"2024071907523700800_ocae145-B7","year":"2023"},{"issue":"2","key":"2024071907523700800_ocae145-B8","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/j.artmed.2006.07.012","article-title":"Extraction and use of linguistic patterns for modelling medical guidelines","volume":"39","author":"Serban","year":"2007","journal-title":"Artif Intell Med"},{"issue":"1","key":"2024071907523700800_ocae145-B9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.artmed.2009.05.004","article-title":"Design patterns for clinical guidelines","volume":"47","author":"Peleg","year":"2009","journal-title":"Artif Intell Med"},{"key":"2024071907523700800_ocae145-B10","first-page":"514","author":"Shah","year":"2017"},{"key":"2024071907523700800_ocae145-B11","author":"Hu"},{"key":"2024071907523700800_ocae145-B12","author":"Nori","year":"2023"},{"key":"2024071907523700800_ocae145-B13","first-page":"385","author":"Ramachandran"},{"issue":"1","key":"2024071907523700800_ocae145-B14","doi-asserted-by":"crossref","first-page":"4171","DOI":"10.1038\/s41598-023-31223-5","article-title":"Medical image captioning via generative pretrained transformers","volume":"13","author":"Selivanov","year":"2023","journal-title":"Sci Rep"},{"key":"2024071907523700800_ocae145-B15","author":"Achiam","year":"2023"},{"key":"2024071907523700800_ocae145-B16","author":"Anil","year":"2023"},{"key":"2024071907523700800_ocae145-B17","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1016\/j.ypmed.2015.07.012","article-title":"Has recommended preventive service use increased after elimination of cost-sharing as part of the affordable care act in the United States?","volume":"78","author":"Han","year":"2015","journal-title":"Prev Med"},{"issue":"1","key":"2024071907523700800_ocae145-B18","doi-asserted-by":"crossref","first-page":"e7","DOI":"10.2105\/AJPH.2014.302289","article-title":"Clinical preventive services coverage and the affordable care act","volume":"105","author":"Fox","year":"2015","journal-title":"Am J Public Health"},{"issue":"7","key":"2024071907523700800_ocae145-B19","doi-asserted-by":"crossref","first-page":"e21929","DOI":"10.2196\/21929","article-title":"The fast health interoperability resources (FHIR) standard: systematic literature review of implementations, applications, challenges and opportunities","volume":"9","author":"Ayaz","year":"2021","journal-title":"JMIR Med Inform"},{"key":"2024071907523700800_ocae145-B20","first-page":"327","author":"De Castilho"},{"key":"2024071907523700800_ocae145-B21","first-page":"1476","author":"Bhate","year":"2023"},{"issue":"4","key":"2024071907523700800_ocae145-B22","doi-asserted-by":"crossref","first-page":"1737","DOI":"10.1109\/JBHI.2021.3123192","article-title":"A deep language model for symptom extraction from clinical text and its application to extract COVID-19 symptoms from social media","volume":"26","author":"Luo","year":"2021","journal-title":"IEEE J Biomed Health Inform"},{"key":"2024071907523700800_ocae145-B23"},{"key":"2024071907523700800_ocae145-B24","first-page":"3982","author":"Reimers"},{"issue":"5","key":"2024071907523700800_ocae145-B25","doi-asserted-by":"crossref","first-page":"705","DOI":"10.15288\/jsa.1999.60.705","article-title":"Screening for alcohol problems in the US general population: a comparison of the CAGE and TWEAK by gender, ethnicity, and services utilization","volume":"60","author":"Cherpitel","year":"1999","journal-title":"J Stud Alcohol"},{"issue":"2","key":"2024071907523700800_ocae145-B26","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1001\/jama.280.2.166","article-title":"Alcohol screening questionnaires in women: a critical review","volume":"280","author":"Bradley","year":"1998","journal-title":"JAMA"},{"key":"2024071907523700800_ocae145-B27","author":"Wang","year":"2023"},{"key":"2024071907523700800_ocae145-B28","author":"Packer","year":"2023"},{"key":"2024071907523700800_ocae145-B29","author":"Wang","year":"2024"},{"key":"2024071907523700800_ocae145-B30"},{"key":"2024071907523700800_ocae145-B31","first-page":"343","author":"Laskar"},{"key":"2024071907523700800_ocae145-B32","author":"Dong","year":"2023"},{"key":"2024071907523700800_ocae145-B33","author":"Jin","year":"2024"},{"key":"2024071907523700800_ocae145-B34","author":"Ding","year":"2024"},{"issue":"5","key":"2024071907523700800_ocae145-B35","doi-asserted-by":"crossref","first-page":"1197","DOI":"10.1161\/HYPERTENSIONAHA.121.17981","article-title":"Hypertension in diabetes: an update of basic mechanisms and clinical disease","volume":"78","author":"Jia","year":"2021","journal-title":"Hypertension"},{"key":"2024071907523700800_ocae145-B36"},{"key":"2024071907523700800_ocae145-B37","first-page":"1754","author":"Mitchell"},{"key":"2024071907523700800_ocae145-B38","author":"Jain","year":"2023"},{"key":"2024071907523700800_ocae145-B39"},{"key":"2024071907523700800_ocae145-B40","first-page":"18990","author":"Song","year":"2024"},{"key":"2024071907523700800_ocae145-B41"},{"key":"2024071907523700800_ocae145-B42","author":"Si"},{"key":"2024071907523700800_ocae145-B43","first-page":"8931","author":"Yukun"},{"issue":"8","key":"2024071907523700800_ocae145-B44","first-page":"Vol:825","article-title":"AI and ethics: a systematic review of the ethical considerations of large language model use in surgery research","volume":"12","author":"Pressman","year":"2024","journal-title":"In: Healthcare"},{"issue":"1","key":"2024071907523700800_ocae145-B45","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1186\/s13000-024-01464-7","article-title":"Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic medicine with a focus on digital pathology\u2013a recent scoping review","volume":"19","author":"Ullah","year":"2024","journal-title":"Diagn Pathol"},{"key":"2024071907523700800_ocae145-B46","doi-asserted-by":"crossref","first-page":"e48009","DOI":"10.2196\/48009","article-title":"Ethical considerations of using ChatGPT in health care","volume":"25","author":"Wang","year":"2023","journal-title":"J Med Internet Res"},{"key":"2024071907523700800_ocae145-B47","author":"Haltaufderheide","year":"2024"},{"key":"2024071907523700800_ocae145-B48","doi-asserted-by":"crossref","DOI":"10.1101\/2023.08.28.23294730","article-title":"Assessing racial and ethnic bias in text generation for healthcare-related tasks by ChatGPT1","author":"Hanna","year":"2023"},{"key":"2024071907523700800_ocae145-B49"},{"issue":"8","key":"2024071907523700800_ocae145-B50","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"issue":"10","key":"2024071907523700800_ocae145-B51","first-page":"875","article-title":"Contribution and performance of ChatGPT and other large language models (LLM) for scientific and research advancements: a double-edged sword","volume":"5","author":"Rane","year":"2023","journal-title":"Int Res J Mod Eng Technol Sci"},{"issue":"12","key":"2024071907523700800_ocae145-B52","doi-asserted-by":"crossref","first-page":"2647","DOI":"10.1007\/s10439-023-03284-0","article-title":"ChatGPT, bard, and large language models for biomedical research: opportunities and pitfalls","volume":"51","author":"Thapa","year":"2023","journal-title":"Ann Biomed Eng"},{"key":"2024071907523700800_ocae145-B53","author":"Luo","year":"2023"},{"key":"2024071907523700800_ocae145-B54"},{"key":"2024071907523700800_ocae145-B55","author":"Xie","year":"2024"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/8\/1743\/58591179\/ocae145.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/8\/1743\/58591179\/ocae145.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T07:53:03Z","timestamp":1721375583000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/8\/1743\/7696537"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,20]]},"references-count":55,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,6,20]]},"published-print":{"date-parts":[[2024,8,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae145","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,8]]},"published":{"date-parts":[[2024,6,20]]}}}