{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T20:48:40Z","timestamp":1783457320727,"version":"3.55.0"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2023,11,11]],"date-time":"2023-11-11T00:00:00Z","timestamp":1699660800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,1,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>We aim to build a generalizable information extraction system leveraging large language models to extract granular eligibility criteria information for diverse diseases from free text clinical trial protocol documents. We investigate the model\u2019s capability to extract criteria entities along with contextual attributes including values, temporality, and modifiers and present the strengths and limitations of this system.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>The clinical trial data were acquired from https:\/\/ClinicalTrials.gov\/. We developed a system, AutoCriteria, which comprises the following modules: preprocessing, knowledge ingestion, prompt modeling based on GPT, postprocessing, and interim evaluation. The final system evaluation was performed, both quantitatively and qualitatively, on 180 manually annotated trials encompassing 9 diseases.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>AutoCriteria achieves an overall F1 score of 89.42 across all 9 diseases in extracting the criteria entities, with the highest being 95.44 for nonalcoholic steatohepatitis and the lowest of 84.10 for breast cancer. Its overall accuracy is 78.95% in identifying all contextual information across all diseases. Our thematic analysis indicated accurate logic interpretation of criteria as one of the strengths and overlooking\/neglecting the main criteria as one of the weaknesses of AutoCriteria.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>AutoCriteria demonstrates strong potential to extract granular eligibility criteria information from trial documents without requiring manual annotations. The prompts developed for AutoCriteria generalize well across different disease areas. Our evaluation suggests that the system handles complex scenarios including multiple arm conditions and logics.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>AutoCriteria currently encompasses a diverse range of diseases and holds potential to extend to more in the future. This signifies a generalizable and scalable solution, poised to address the complexities of clinical trial application in real-world settings.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad218","type":"journal-article","created":{"date-parts":[[2023,11,12]],"date-time":"2023-11-12T16:20:31Z","timestamp":1699806031000},"page":"375-385","source":"Crossref","is-referenced-by-count":46,"title":["AutoCriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5634-2005","authenticated-orcid":false,"given":"Surabhi","family":"Datta","sequence":"first","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyeryoung","family":"Lee","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hunki","family":"Paek","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frank J","family":"Manion","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nneka","family":"Ofoegbu","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0322-4566","authenticated-orcid":false,"given":"Jingcheng","family":"Du","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ying","family":"Li","sequence":"additional","affiliation":[{"name":"Regeneron Pharmaceuticals , Tarrytown, NY 10591, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liang-Chin","family":"Huang","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingqi","family":"Wang","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Lin","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hua","family":"Xu","sequence":"additional","affiliation":[{"name":"Yale School of Medicine , New Haven, CT 06511, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaoyan","family":"Wang","sequence":"additional","affiliation":[{"name":"Melax Technologies , Houston, TX 77030, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2023,11,11]]},"reference":[{"issue":"11","key":"2024011907293363300_ocad218-B1","doi-asserted-by":"crossref","first-page":"706","DOI":"10.1016\/j.tips.2015.08.007","article-title":"Optimizing clinical research participant selection with informatics","volume":"36","author":"Weng","year":"2015","journal-title":"Trends Pharmacol Sci"},{"issue":"3","key":"2024011907293363300_ocad218-B2","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.cct.2010.03.005","article-title":"Automated matching software for clinical trials eligibility: measuring efficiency and flexibility","volume":"31","author":"Penberthy","year":"2010","journal-title":"Contemp Clin Trials"},{"key":"2024011907293363300_ocad218-B3","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1186\/s12911-021-01492-z","article-title":"Automated classification of clinical trial eligibility criteria text based on ensemble learning and metric learning","volume":"21(Suppl 2)","author":"Zeng","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"issue":"6","key":"2024011907293363300_ocad218-B4","doi-asserted-by":"crossref","first-page":"1062","DOI":"10.1093\/jamia\/ocx019","article-title":"EliIE: an open-source information extraction system for clinical trial eligibility criteria","volume":"24","author":"Kang","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2024011907293363300_ocad218-B5","first-page":"25968","volume-title":"Advances in Neural Information Processing Systems","author":"Singh","year":"2021"},{"key":"2024011907293363300_ocad218-B6","author":"Li","year":"2023"},{"key":"2024011907293363300_ocad218-B7","author":"Arsenyan","year":"2023"},{"issue":"1","key":"2024011907293363300_ocad218-B8","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"npj Digit Med"},{"key":"2024011907293363300_ocad218-B9","author":"Singhal","year":"2022"},{"issue":"2","key":"2024011907293363300_ocad218-B10","doi-asserted-by":"crossref","first-page":"e32875","DOI":"10.2196\/32875","article-title":"Operationalizing and implementing pretrained, large artificial intelligence linguistic models in the US Health Care System: outlook of Generative Pretrained Transformer 3 (GPT-3) as a service model","volume":"10","author":"Sezgin","year":"2022","journal-title":"JMIR Med Inform"},{"issue":"2","key":"2024011907293363300_ocad218-B11","doi-asserted-by":"crossref","first-page":"387","DOI":"10.3390\/biom13020387","article-title":"Using GPT-3 to build a lexicon of drugs of abuse synonyms for social media pharmacovigilance","volume":"13","author":"Carpenter","year":"2023","journal-title":"Biomolecules"},{"key":"2024011907293363300_ocad218-B12","author":"Levine","year":"2023"},{"issue":"2","key":"2024011907293363300_ocad218-B13","doi-asserted-by":"crossref","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","year":"2023","journal-title":"PLoS Digital Health"},{"key":"2024011907293363300_ocad218-B14","author":"Kasai","year":"2023"},{"key":"2024011907293363300_ocad218-B15","author":"Nori","year":"2023"},{"issue":"2","key":"2024011907293363300_ocad218-B16","doi-asserted-by":"crossref","first-page":"605","DOI":"10.12669\/pjms.39.2.7653","article-title":"ChatGPT\u2014reshaping medical education and clinical management","volume":"39","author":"Khan","year":"2023","journal-title":"Pak J Med Sci"},{"key":"2024011907293363300_ocad218-B17","doi-asserted-by":"crossref","first-page":"e46885","DOI":"10.2196\/46885","article-title":"The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with ChatGPT and a call for papers","volume":"9","author":"Eysenbach","year":"2023","journal-title":"JMIR Med Educ"},{"key":"2024011907293363300_ocad218-B18","author":"Li","year":"2023"},{"key":"2024011907293363300_ocad218-B19","author":"Tseo","year":"2020"},{"key":"2024011907293363300_ocad218-B20","author":"Tian","year":"2021"},{"key":"2024011907293363300_ocad218-B21","first-page":"505","volume-title":"IEEE Computer Society","author":"Liu","year":"2021"},{"issue":"7","key":"2024011907293363300_ocad218-B22","doi-asserted-by":"crossref","first-page":"e17832","DOI":"10.2196\/17832","article-title":"An ensemble learning strategy for eligibility criteria text classification for clinical trial recruitment: algorithm development and validation","volume":"8","author":"Zeng","year":"2020","journal-title":"JMIR Med Inform"},{"key":"2024011907293363300_ocad218-B23","first-page":"49","article-title":"Transformer-based named entity recognition for parsing clinical trial eligibility criteria","volume":"2021","author":"Tian","year":"2021","journal-title":"ACM BCB"},{"issue":"1","key":"2024011907293363300_ocad218-B24","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1038\/s41597-020-00620-0","article-title":"Chia, a large annotated corpus of clinical trial eligibility criteria","volume":"7","author":"Kury","year":"2020","journal-title":"Sci Data"},{"issue":"1","key":"2024011907293363300_ocad218-B25","doi-asserted-by":"crossref","first-page":"490","DOI":"10.1038\/s41597-022-01521-0","article-title":"The Leaf Clinical Trials Corpus: a new resource for query generation from clinical trial eligibility criteria","volume":"9","author":"Dobbins","year":"2022","journal-title":"Sci Data"},{"key":"2024011907293363300_ocad218-B26","author":"Wu","year":"2021"},{"key":"2024011907293363300_ocad218-B27","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1186\/s12911-022-01967-7","article-title":"A comparative study of pre-trained language models for named entity recognition in clinical trial eligibility criteria from multiple corpora","volume":"22(Suppl 3)","author":"Li","year":"2022","journal-title":"BMC Med Inform Decis Mak"},{"issue":"4","key":"2024011907293363300_ocad218-B28","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1093\/jamia\/ocy178","article-title":"Criteria2Query: a natural language interface to clinical databases for cohort definition","volume":"26","author":"Yuan","year":"2019","journal-title":"J Am Med Inform Assoc"},{"key":"2024011907293363300_ocad218-B29","author":"Dunn","year":"2022"},{"key":"2024011907293363300_ocad218-B30","author":"Wei","year":"2023"},{"key":"2024011907293363300_ocad218-B31","author":"Agrawal","year":"2022"},{"key":"2024011907293363300_ocad218-B32","author":"Hu","year":"2023"},{"key":"2024011907293363300_ocad218-B33","author":"Peikos","year":"2023"},{"key":"2024011907293363300_ocad218-B34","author":"Wang","year":"2023"},{"key":"2024011907293363300_ocad218-B35","author":"Touvron","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/2\/375\/56209472\/ocad218.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/2\/375\/56209472\/ocad218.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,19]],"date-time":"2024-01-19T07:32:19Z","timestamp":1705649539000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/2\/375\/7413158"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,11]]},"references-count":35,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,11,11]]},"published-print":{"date-parts":[[2024,1,18]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad218","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,2,1]]},"published":{"date-parts":[[2023,11,11]]}}}