{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:47:42Z","timestamp":1782841662610,"version":"3.54.5"},"reference-count":30,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Res Protoc"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Eligibility criteria are essential to clinical trial design, guiding recruitment, and ensuring patient safety and scientific rigor. However, criteria are often lengthy, heterogeneous, and inconsistently formatted, which hinders large-scale interpretation and slows patient-trial matching. Manual review is time-consuming and error-prone. Advances in natural language processing and large language models (LLMs) offer opportunities to standardize and analyze eligibility text at scale.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to develop and evaluate a scalable system that uses LLM-enabled natural language processing and unsupervised learning to identify, normalize, categorize, and visualize clinical trial eligibility criteria, with the goal of improving patient-trial matching and revealing domain-level trends.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We designed a three-part pipeline: (1) representation of eligibility text using embeddings, followed by clustering to group semantically similar criteria; (2) dual-layer zero-shot LLM summarization for concept normalization, refinement, and deduplication of cluster exemplars; and (3) an interactive, web-based visualization interface to explore criteria distributions and trends by disease domain and over time. The pipeline was applied to 53,872 oncology trials (breast, lung, and gastrointestinal cancer) indexed on ClinicalTrials.gov. Outputs include cluster labels, normalized criterion summaries, and per-domain frequency profiles. Feasibility was assessed via successful end-to-end processing and inspection of face validity for cluster coherence and domain-specific patterns.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The system successfully processed all 53,872 trials and generated stable clusters of inclusion and exclusion concepts. The LLM summarization layers produced concise, nonredundant labels that improved the interpretability of clustered criteria. The visualization interface enabled rapid exploration of cross-trial patterns and temporal trends within breast, lung, and gastrointestinal oncology, facilitating identification of common inclusion requirements and potential barriers to enrollment. A public, open-source demonstration instance allows for interactive exploration of these clusters and summaries. Benchmarking through human validation on a random sample of eligibility criteria found the system to be 94% (470\/500) accurate, reflecting its ability to consistently categorize criteria correctly in congruence with human judgment.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n A combined embeddings-clustering-LLM pipeline can standardize heterogeneous eligibility text and surface domain-level patterns at scale. This framework provides a foundation for accelerating patient-trial matching and informing future trial design. While the current implementation was evaluated on ClinicalTrials.gov oncology trials, the approach is readily generalizable to additional diseases and alternative modeling configurations.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/86425","type":"journal-article","created":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T20:21:06Z","timestamp":1778790066000},"page":"e86425-e86425","source":"Crossref","is-referenced-by-count":0,"title":["A Natural Language Processing Framework for Structuring and Visualizing Clinical Trial Eligibility Criteria at Scale: Protocol for a Quantitative Study"],"prefix":"10.2196","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-5674-2494","authenticated-orcid":false,"given":"Justin","family":"Xie","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8706-2962","authenticated-orcid":false,"given":"Jeet","family":"Parikh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2723-345X","authenticated-orcid":false,"given":"Jessica","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-8855-5031","authenticated-orcid":false,"given":"Sameer","family":"Pandya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4972-6705","authenticated-orcid":false,"given":"Guannan","family":"Gong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,5,14]]},"reference":[{"issue":"11","key":"R1","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1001\/jama.297.11.1233","article-title":"Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review","volume":"297","author":"Van Spall","journal-title":"JAMA"},{"issue":"25","key":"R2","doi-asserted-by":"publisher","first-page":"2815","DOI":"10.1200\/JCO.2015.62.1854","article-title":"Modernizing eligibility criteria for molecularly driven trials","volume":"33","author":"Kim","journal-title":"J Clin Oncol"},{"issue":"3","key":"R3","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1016\/j.jbi.2009.12.004","article-title":"Formal representation of eligibility criteria: a literature review","volume":"43","author":"Weng","journal-title":"J Biomed Inform"},{"key":"R4","volume":"2010","author":"Ross","journal-title":"Summit Transl Bioinform"},{"key":"R5","volume":"2005","author":"Embi","journal-title":"AMIA Annu Symp Proc"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.MR000013.pub6","article-title":"Strategies to improve recruitment to randomised trials","volume":"2","author":"Treweek","journal-title":"Cochrane Database Syst Rev"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/j.ijmedinf.2012.11.008","article-title":"Secondary use of routinely collected patient data in a clinical trial: an evaluation of the effects on patient recruitment and data acquisition","volume":"82","author":"K\u00f6pcke","journal-title":"Int J Med Inform"},{"key":"R8","volume":"2011","author":"Weng","journal-title":"AMIA Annu Symp Proc"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-015-0149-3","article-title":"Increasing the efficiency of trial-patient matching: automated clinical trial eligibility pre-screening for pediatric oncology patients","volume":"15","author":"Ni","journal-title":"BMC Med Inform Decis Mak"},{"issue":"5","key":"R10","doi-asserted-by":"publisher","first-page":"824","DOI":"10.1136\/amiajnl-2013-002443","article-title":"Automatic lymphoma classification with sentence subgraph mining from pathology reports","volume":"21","author":"Luo","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1186\/s13326-018-0179-8","article-title":"Clinical natural language processing in languages other than English: opportunities and challenges","volume":"9","author":"N\u00e9v\u00e9ol","journal-title":"J Biomed Semantics"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2021.102086","article-title":"Med7: A transferable clinical natural language processing model for electronic health records","volume":"118","author":"Kormilitzin","journal-title":"Artif Intell Med"},{"issue":"4","key":"R13","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1093\/jamia\/ocy178","article-title":"Criteria2Query: a natural language interface to clinical databases for cohort definition","volume":"26","author":"Yuan","journal-title":"J Am Med Inform Assoc"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Devlin J Chang MW Lee K Toutanova K . BERT: pre-training of deep bidirectional transformers for language understanding. Presented at: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Jun 2-7, 2019; Minneapolis, MN. URL: https:\/\/aclanthology.org\/N19-1423\/ [doi: 10.18653\/v1\/N19-1423]","DOI":"10.18653\/v1\/N19-1423"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","journal-title":"Bioinformatics"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Alsentzer E Murphy J Boag W et al. Publicly available clinical BERT embeddings. Presented at: Proceedings of the 2nd Clinical Natural Language Processing Workshop; Jun 7, 2019; Minneapolis, MN. [doi: 10.18653\/v1\/W19-1909]","DOI":"10.18653\/v1\/W19-1909"},{"issue":"7972","key":"R17","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"issue":"2","key":"R18","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"key":"R19","unstructured":"Nori H King N McKinney SM Carignan D Horvitz E . Capabilities of GPT-4 on medical challenge problems. arXiv. Preprint posted online on Mar 20, 2023. [doi: 10.48550\/arXiv.2303.13375]"},{"issue":"4","key":"R20","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2023.100729","article-title":"Fine-tuning large neural language models for biomedical natural language processing","volume":"4","author":"Tinn","journal-title":"Patterns (N Y)"},{"issue":"6","key":"R21","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","journal-title":"JAMA Intern Med"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-53081-z","article-title":"Matching patients to clinical trials with large language models","volume":"15","author":"Jin","journal-title":"Nat Commun"},{"key":"R23","unstructured":"Bolton E Venigalla A Yasunaga M . Biomedlm: a 2.7B parameter language model trained on biomedical text. arXiv. Preprint posted online on Mar 27, 2024. [doi: 10.48550\/arXiv.2403.18421]"},{"issue":"6","key":"R24","doi-asserted-by":"publisher","first-page":"1062","DOI":"10.1093\/jamia\/ocx019","article-title":"EliIE: An open-source information extraction system for clinical trial eligibility criteria","volume":"24","author":"Kang","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"R25","doi-asserted-by":"publisher","DOI":"10.1093\/jncics\/pkaa034","article-title":"Representativeness of Black patients in cancer clinical trials sponsored by the National Cancer Institute compared with pharmaceutical companies","volume":"4","author":"Unger","journal-title":"JNCI Cancer Spectr"},{"key":"R26","unstructured":"MedViz. URL: https:\/\/medviz.org\/ [Accessed 17-07-2025]"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.conctc.2022.100939","article-title":"Barriers for cancer clinical trial enrollment: a qualitative study of the perspectives of healthcare providers","volume":"28","author":"Kumar","journal-title":"Contemp Clin Trials Commun"},{"issue":"9","key":"R28","doi-asserted-by":"publisher","first-page":"2394","DOI":"10.1158\/1078-0432.CCR-20-3852","article-title":"Continuing to broaden eligibility criteria to make clinical trials more representative and inclusive: ASCO-friends of cancer research joint research statement","volume":"27","author":"Kim","journal-title":"Clin Cancer Res"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.2147\/JHL.S488426","article-title":"Barriers to recruitment and retention among underrepresented populations in cancer clinical trials: a qualitative study of the perspectives of clinical trial research coordinating staff at a cancer center","volume":"16","author":"Yousafi","journal-title":"J Healthc Leadersh"},{"key":"R31","unstructured":"CtrlTrial criteria visualizer. GitHub. URL: https:\/\/github.com\/ctrltrial\/CriteriaVisualizer [Accessed 01-05-2026]"}],"container-title":["JMIR Research Protocols"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e86425","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:14:19Z","timestamp":1782839659000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e86425"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,14]]},"references-count":30,"URL":"https:\/\/doi.org\/10.2196\/86425","relation":{},"ISSN":["1929-0748"],"issn-type":[{"value":"1929-0748","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,14]]},"article-number":"v15i11e86425"}}