{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T04:18:14Z","timestamp":1784607494656,"version":"3.55.0"},"reference-count":33,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2025,8,31]],"date-time":"2025-08-31T00:00:00Z","timestamp":1756598400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U01TR003709"],"award-info":[{"award-number":["U01TR003709"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U24MH136069"],"award-info":[{"award-number":["U24MH136069"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["RF1AG077820"],"award-info":[{"award-number":["RF1AG077820"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01AG073435"],"award-info":[{"award-number":["R01AG073435"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R56AG074604"],"award-info":[{"award-number":["R56AG074604"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01LM013519"],"award-info":[{"award-number":["R01LM013519"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["1R01LM014344"],"award-info":[{"award-number":["1R01LM014344"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01DK128237"],"award-info":[{"award-number":["R01DK128237"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R21AI167418"],"award-info":[{"award-number":["R21AI167418"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R21EY034179"],"award-info":[{"award-number":["R21EY034179"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Systematic reviews in comparative effectiveness research require timely evidence synthesis. With the rapid advancement of medical research, preprint articles play an increasingly important role in accelerating knowledge dissemination. However, as preprint articles are not peer-reviewed before publication, their quality varies significantly, posing challenges for evidence inclusion in systematic reviews.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We developed AutoConfidenceScore (automated confidence score assessment), an advanced framework for predicting preprint publication, which reduces reliance on manual curation and expands the range of predictors, including three key advancements: (1) automated data extraction using natural language processing techniques, (2) semantic embeddings of titles and abstracts, and (3) large language model (LLM)-driven evaluation scores. Additionally, we employed two prediction models: a random forest classifier for binary outcome and a survival cure model that predicts both binary outcome and publication risk over time.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The random forest classifier achieved an area under the receiver operating characteristic curve (AUROC) of 0.747 using all features. The survival cure model achieved an AUROC of 0.731 for binary outcome prediction and a concordance index of 0.667 for time-to-publication risk.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Our study advances the framework for preprint publication prediction through automated data extraction and multiple feature integration. By combining semantic embeddings with LLM-driven evaluations, AutoConfidenceScore significantly enhances predictive performance while reducing manual annotation burden.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>AutoConfidenceScore has the potential to facilitate incorporation of preprint articles during the appraisal phase of systematic reviews, supporting researchers in more effective utilization of preprint resources.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf137","type":"journal-article","created":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T11:42:28Z","timestamp":1754134948000},"page":"1718-1725","source":"Crossref","is-referenced-by-count":4,"title":["Enabling inclusive systematic reviews: incorporating preprint articles with large language model-driven evaluations"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8507-5631","authenticated-orcid":false,"given":"Rui","family":"Yang","sequence":"first","affiliation":[{"name":"Centre for Quantitative Medicine, Duke-NUS Medical School , Singapore, 169857,","place":["Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiayi","family":"Tong","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health , Baltimore, MD, 21205,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haoyuan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Bioinformatics, Duke School of Medicine , Durham, NC, 27710,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hui","family":"Huang","sequence":"additional","affiliation":[{"name":"Children\u2019s Healthcare of Atlanta, Emory University School of Medicine , Atlanta, GA, 30329,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ziyang","family":"Hu","sequence":"additional","affiliation":[{"name":"Department of Genome Sciences, University of Virginia , Charlottesville, VA, 22903,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peiyu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Bioinformatics, Duke School of Medicine , Durham, NC, 27710,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nan","family":"Liu","sequence":"additional","affiliation":[{"name":"Centre for Quantitative Medicine, Duke-NUS Medical School , Singapore, 169857,","place":["Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher J","family":"Lindsell","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Bioinformatics, Duke School of Medicine , Durham, NC, 27710,","place":["United States"]},{"name":"Duke Clinical Research Institute, Duke University , Durham, NC, 27701,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5798-8855","authenticated-orcid":false,"given":"Michael J","family":"Pencina","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Bioinformatics, Duke School of Medicine , Durham, NC, 27710,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0835-0788","authenticated-orcid":false,"given":"Yong","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA, 19104,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7056-9559","authenticated-orcid":false,"given":"Chuan","family":"Hong","sequence":"additional","affiliation":[{"name":"Department of Biostatistics and Bioinformatics, Duke School of Medicine , Durham, NC, 27710,","place":["United States"]},{"name":"Duke Clinical Research Institute, Duke University , Durham, NC, 27701,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,8,31]]},"reference":[{"key":"2025111813570140900_ocaf137-B1","doi-asserted-by":"crossref","first-page":"203","DOI":"10.7326\/0003-4819-151-3-200908040-00125","article-title":"Comparative effectiveness research: a report from the Institute of Medicine","volume":"151","author":"Sox","year":"2009","journal-title":"Ann Intern Med"},{"key":"2025111813570140900_ocaf137-B2","doi-asserted-by":"crossref","first-page":"809","DOI":"10.1093\/jamia\/ocad248","article-title":"Confidence score: a data-driven measure for inclusive systematic reviews considering unpublished preprints","volume":"31","author":"Tong","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025111813570140900_ocaf137-B3","doi-asserted-by":"crossref","first-page":"100968","DOI":"10.1016\/j.patter.2024.100968","article-title":"The landscape of biomedical research","volume":"5","author":"Gonz\u00e1lez-M\u00e1rquez","year":"2024","journal-title":"Patterns (N Y)"},{"key":"2025111813570140900_ocaf137-B4","doi-asserted-by":"crossref","first-page":"e89","DOI":"10.1161\/CIR.0000000000001258","article-title":"Forecasting the economic burden of cardiovascular disease and stroke in the United States through 2050: a presidential advisory from the American Heart Association","volume":"150","author":"Kazi","year":"2024","journal-title":"Circulation"},{"key":"2025111813570140900_ocaf137-B5","article-title":"Global burden of cardiovascular diseases: projections from 2025 to 2050","author":"Chong","year":"2024","journal-title":"Eur J Prev Cardiol"},{"key":"2025111813570140900_ocaf137-B6","doi-asserted-by":"crossref","first-page":"1898","DOI":"10.1016\/j.jacc.2016.07.781","article-title":"The changing landscape of randomized clinical trials in cardiovascular disease","volume":"68","author":"Jones","year":"2016","journal-title":"J Am Coll Cardiol"},{"key":"2025111813570140900_ocaf137-B7","doi-asserted-by":"crossref","first-page":"e3000959","DOI":"10.1371\/journal.pbio.3000959","article-title":"The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape","volume":"19","author":"Fraser","year":"2021","journal-title":"PLoS Biol"},{"key":"2025111813570140900_ocaf137-B8","doi-asserted-by":"crossref","first-page":"e2245847","DOI":"10.1001\/jamanetworkopen.2022.45847","article-title":"Comparison of clinical study results reported in medRxiv preprints vs peer-reviewed journal articles","volume":"5","author":"Janda","year":"2022","journal-title":"JAMA Netw Open"},{"key":"2025111813570140900_ocaf137-B9","doi-asserted-by":"crossref","first-page":"899","DOI":"10.1126\/science.aaf9133","article-title":"SCIENTIFIC COMMUNITY. Preprints for the life sciences","volume":"352","author":"Berg","year":"2016","journal-title":"Science"},{"key":"2025111813570140900_ocaf137-B10","doi-asserted-by":"crossref","first-page":"m3111","DOI":"10.1136\/bmj.m3111","article-title":"The perils of preprints","volume":"370","author":"van Schalkwyk","year":"2020","journal-title":"BMJ"},{"key":"2025111813570140900_ocaf137-B11","doi-asserted-by":"crossref","first-page":"1840","DOI":"10.1001\/jama.2020.20674","article-title":"Preprints involving medical research-do the benefits outweigh the challenges?","volume":"324","author":"Flanagin","year":"2020","journal-title":"JAMA"},{"key":"2025111813570140900_ocaf137-B12","article-title":"Incorporating preprints in systematic reviews: a novel method for rapid evidence synthesis","author":"Tong","journal-title":"J Am Med Inform Assoc"},{"key":"2025111813570140900_ocaf137-B13","doi-asserted-by":"crossref","first-page":"e48330","DOI":"10.2196\/48330","article-title":"Comparing open-access database and traditional intensive care studies using machine learning: bibliometric analysis study","volume":"26","author":"Ke","year":"2024","journal-title":"J Med Internet Res"},{"key":"2025111813570140900_ocaf137-B14","doi-asserted-by":"crossref","first-page":"e60601","DOI":"10.2196\/60601","article-title":"Ascle-A Python natural language processing toolkit for medical text generation: development and evaluation study","volume":"26","author":"Yang","year":"2024","journal-title":"J Med Internet Res"},{"key":"2025111813570140900_ocaf137-B15","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1056\/AIoa2400196","article-title":"Can large language models provide useful feedback on research papers? A large-scale empirical analysis","volume":"1","author":"Liang","year":"2024","journal-title":"NEJM AI"},{"key":"2025111813570140900_ocaf137-B16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1056\/AIoa2400555","article-title":"Autonomous LLM-driven research\u2014from data to human-verifiable research papers","volume":"2","author":"Ifargan","year":"2025","journal-title":"NEJM AI"},{"key":"2025111813570140900_ocaf137-B17","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1002\/hcs2.61","article-title":"Large language models in health care: development, applications, and challenges","volume":"2","author":"Yang","year":"2023","journal-title":"Health Care Sci"},{"key":"2025111813570140900_ocaf137-B18","doi-asserted-by":"crossref","first-page":"e147","DOI":"10.1017\/cts.2024.571","article-title":"Realizing the potential of social determinants data in EHR systems: a scoping review of approaches for screening, linkage, extraction, analysis, and interventions","volume":"8","author":"Li","year":"2024","journal-title":"J Clin Transl Sci"},{"key":"2025111813570140900_ocaf137-B19","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1038\/s41746-025-01622-1","article-title":"An evaluation framework for ambient digital scribing tools in clinical applications","volume":"8","author":"Wang","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2025111813570140900_ocaf137-B20","doi-asserted-by":"crossref","first-page":"36120","DOI":"10.1109\/ACCESS.2023.3266377","article-title":"A survey of text representation and embedding techniques in NLP","volume":"11","author":"Patil","year":"2023","journal-title":"IEEE Access"},{"key":"2025111813570140900_ocaf137-B21"},{"key":"2025111813570140900_ocaf137-B22","year":"2024"},{"key":"2025111813570140900_ocaf137-B23","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1093\/bioinformatics\/btad651","article-title":"MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval","volume":"39","author":"Jin","year":"2023","journal-title":"Bioinformatics"},{"key":"2025111813570140900_ocaf137-B24","author":"McInnes"},{"key":"2025111813570140900_ocaf137-B25"},{"key":"2025111813570140900_ocaf137-B26","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"VaroquauxGa\u00ebl","year":"2021","journal-title":"J Mach Learn Res"},{"key":"2025111813570140900_ocaf137-B27","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1111\/j.2517-6161.1972.tb00899.x","article-title":"Regression models and life-tables","volume":"34","author":"Cox","year":"1972","journal-title":"J R Stat Soc Series B Stat Methodol"},{"key":"2025111813570140900_ocaf137-B28","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1006\/jmps.1999.1279","article-title":"Cross-validation methods","volume":"44","author":"Browne","year":"2000","journal-title":"J Math Psychol"},{"key":"2025111813570140900_ocaf137-B29","doi-asserted-by":"crossref","DOI":"10.1201\/9780429246593","volume-title":"An Introduction to the Bootstrap","author":"Efron","year":"1994"},{"key":"2025111813570140900_ocaf137-B30","author":"Rayyan"},{"key":"2025111813570140900_ocaf137-B31"},{"key":"2025111813570140900_ocaf137-B32","year":"2021"},{"key":"2025111813570140900_ocaf137-B33"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/11\/1718\/64169385\/ocaf137.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/11\/1718\/64169385\/ocaf137.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T18:57:12Z","timestamp":1763492232000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/11\/1718\/8244908"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,31]]},"references-count":33,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,8,31]]},"published-print":{"date-parts":[[2025,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf137","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,11]]},"published":{"date-parts":[[2025,8,31]]}}}