{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T22:22:35Z","timestamp":1787696555929,"version":"build-2784847793"},"posted":{"date-parts":[[2026,7,17]]},"group-title":"Dermatology","reference-count":17,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T00:00:00Z","timestamp":1784246400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.medrxiv.org\/about\/FAQ#license"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"crossref","award":["P50CA236733"],"award-info":[{"award-number":["P50CA236733"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"crossref","award":["U54CA163072"],"award-info":[{"award-number":["U54CA163072"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]},{"name":"VA Tennessee Valley Healthcare System","award":["VUMC120842"],"award-info":[{"award-number":["VUMC120842"]}]},{"name":"Lifespan Rhode Island Hospital","award":["U01CA287008"],"award-info":[{"award-number":["U01CA287008"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,8,22]]},"abstract":"Abstract<\/jats:title>\n \n Background and objective<\/jats:title>\n \n The\n F<\/jats:italic>\n 1<\/jats:sub>\n score and its generalized\n \n F\n \u03b2<\/jats:sub>\n <\/jats:italic>\n score are widely used to evaluate machine learning and artificial intelligence (AI) models in healthcare, particularly for imbalanced clinical datasets. In practice, competing prediction models are commonly evaluated on the same patient cohort, resulting in correlated classifier decisions. However, existing approaches for statistical inference of\n F<\/jats:italic>\n 1<\/jats:sub>\n -related metrics typically assume independent classifier decisions or lack integrated procedures for comparative evaluation, power analysis, and sample size determination in paired validation studies.\n <\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We propose psF1pair, a unified framework for confidence interval estimation, hypothesis testing, and power and sample size calculation for comparative\n F<\/jats:italic>\n 1<\/jats:sub>\n and\n \n F\n \u03b2<\/jats:sub>\n <\/jats:italic>\n scores under paired evaluation designs. Dependence between classifiers is modeled using a four-component multinomial representation of the joint decision process, allowing explicit estimation and incorporation of classifier correlations commonly encountered when AI models are evaluated on the same patient cohort. Exact distributions are used for small sample settings, while asymptotic approximations are employed for computational efficiency in large studies.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Simulation studies demonstrated that the proposed confidence intervals achieved nominal coverage probabilities across a wide range of sample sizes and correlation settings. Estimated power closely agreed with empirical power, with discrepancies generally below 3%. Compared with existing methods, psF1pair showed competitive or superior statistical power while maintaining appropriate type I error rates across a broad range of scenarios. Applications to skin cancer classification and breast cancer screening demonstrated that accounting for classifier correlation produced narrower confidence intervals and improved statistical efficiency.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n \n psF1pair provides a practical and rigorous framework for evaluation and study planning of medical AI systems using\n F<\/jats:italic>\n 1<\/jats:sub>\n and\n \n F\n \u03b2<\/jats:sub>\n <\/jats:italic>\n metrics. The method supports comparative benchmarking, uncertainty quantification, and sample size determination for future validation studies. An open-source R package is freely available.\n <\/jats:p>\n <\/jats:sec>\n \n Highlights<\/jats:title>\n \n \n \n Unified statistical inference and study-design framework for comparative\n F<\/jats:italic>\n 1<\/jats:sub>\n and\n \n F\n \u03b2<\/jats:sub>\n <\/jats:italic>\n scores under paired evaluation designs.\n <\/jats:p>\n <\/jats:list-item>\n \n Explicit modeling of dependence between classifiers evaluated on the same dataset using a multinomial framework.<\/jats:p>\n <\/jats:list-item>\n \n Open-source R package supporting comparative evaluation and validation studies of predictive and AI models.<\/jats:p>\n <\/jats:list-item>\n <\/jats:list>\n <\/jats:sec>","DOI":"10.64898\/2026.07.15.26358166","type":"posted-content","created":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T22:55:12Z","timestamp":1784328912000},"source":"Crossref","is-referenced-by-count":0,"title":["A Unified Framework for Statistical Inference and Study Design of Comparative\n F<\/i>\n 1<\/sub>\n and\n \n F\n \u03b2<\/sub>\n <\/i>\n Scores under Paired Classifier Evaluations"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8325-2112","authenticated-orcid":false,"given":"Chih-Yuan","family":"Hsu","sequence":"first","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37203, USA"},{"name":"Center for Quantitative Sciences, Vanderbilt University Medical Center, Nashville, TN 37203, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qi","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37203, USA"},{"name":"Center for Quantitative Sciences, Vanderbilt University Medical Center, Nashville, TN 37203, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu","family":"Shyr","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37203, USA"},{"name":"Center for Quantitative Sciences, Vanderbilt University Medical Center, Nashville, TN 37203, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026082515000983000_2026.07.15.26358166v2.1","doi-asserted-by":"crossref","first-page":"1047","DOI":"10.1039\/D5SD00146C","article-title":"Artificial intelligence in healthcare diagnosis: evidence-based recent advances and clinical implications","volume":"4","year":"2025","journal-title":"Sensors and Diagnostics"},{"key":"2026082515000983000_2026.07.15.26358166v2.2","doi-asserted-by":"crossref","unstructured":"Asif S , Wenhui Y , Tao Y , et al. An ensemble machine learning method for the prediction of heart disease. 2021 4th International Conference on Artificial Intelligence and Big Data (ICAIBD). 2021;98\u2013103.","DOI":"10.1109\/ICAIBD51990.2021.9459010"},{"key":"2026082515000983000_2026.07.15.26358166v2.3","doi-asserted-by":"crossref","first-page":"80151","DOI":"10.1109\/ACCESS.2022.3165792","article-title":"Efficient medical diagnosis of human heart diseases using machine learning techniques with and without GridSearchCV","volume":"10","year":"2022","journal-title":"IEEE Access"},{"key":"2026082515000983000_2026.07.15.26358166v2.4","doi-asserted-by":"crossref","first-page":"106755","DOI":"10.1016\/j.cmpb.2022.106755","article-title":"Classification of skin cancer using convolutional neural networks analysis of Raman Spectra","volume":"219","year":"2022","journal-title":"Comput Methods Programs Biomed"},{"key":"2026082515000983000_2026.07.15.26358166v2.5","doi-asserted-by":"crossref","first-page":"9579","DOI":"10.1007\/s00521-020-04842-6","article-title":"Automatic lung cancer detection from CT image using improved deep neural network and ensemble classifier","volume":"34","year":"2022","journal-title":"Neural Comput Appl"},{"key":"2026082515000983000_2026.07.15.26358166v2.6","doi-asserted-by":"crossref","first-page":"103182","DOI":"10.1016\/j.bspc.2021.103182","article-title":"A deep learning based approach for automatic detection of COVID-19 cases using chest X-ray images","volume":"71","year":"2022","journal-title":"Biomed Signal Process Control"},{"key":"2026082515000983000_2026.07.15.26358166v2.7","doi-asserted-by":"crossref","first-page":"514","DOI":"10.3390\/bioengineering12050514","article-title":"Machine learning techniques applied to COVID-19 prediction: A systematic literature review","volume":"12","year":"2025","journal-title":"Bioengineering (Basel)"},{"key":"2026082515000983000_2026.07.15.26358166v2.8","doi-asserted-by":"crossref","first-page":"1723","DOI":"10.3390\/brainsci13121723","article-title":"Limitations in evaluating machine learning models for imbalanced binary outcome classification in spine surgery: a systematic review","volume":"13","year":"2023","journal-title":"Brain Sci"},{"key":"2026082515000983000_2026.07.15.26358166v2.9","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa1407349"},{"key":"2026082515000983000_2026.07.15.26358166v2.10","doi-asserted-by":"crossref","first-page":"853","DOI":"10.1007\/s11831-024-10148-w","article-title":"Advancements and prospects of machine learning in medical diagnostics: unveiling the future of diagnostic precision","volume":"32","year":"2025","journal-title":". Archives of Computational Methods in Engineering"},{"key":"2026082515000983000_2026.07.15.26358166v2.11","first-page":"345","article-title":"A probabilistic interpretation of precision, recall and f-score, with implication for evaluation","volume":"3408","year":"2005","journal-title":"Advances in Information Retrieval"},{"key":"2026082515000983000_2026.07.15.26358166v2.12","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.14621"},{"key":"2026082515000983000_2026.07.15.26358166v2.13","doi-asserted-by":"publisher","DOI":"10.1002\/sim.9853"},{"issue":"10-12","key":"2026082515000983000_2026.07.15.26358166v2.14","doi-asserted-by":"crossref","first-page":"e70557","DOI":"10.1002\/sim.70557","article-title":"A unified framework for statistical inference and power analysis of single and comparative F\u03b2 scores","volume":"45","year":"2026","journal-title":"Stat Med"},{"key":"2026082515000983000_2026.07.15.26358166v2.15","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1186\/s12859-022-04739-2","article-title":"An equivalence test between features lists, based on the Sorensen\u2013Dice index and the joint frequencies of GO term enrichment","volume":"23","year":"2022","journal-title":"BMC Bioinformatics"},{"issue":"8","key":"2026082515000983000_2026.07.15.26358166v2.16","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.3390\/biom10081123","article-title":"The development of a skin cancer classification system for pigmented skin lesions using deep learning","volume":"10","year":"2020","journal-title":"Biomolecules"},{"key":"2026082515000983000_2026.07.15.26358166v2.17","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1799-6"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.07.15.26358166","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T22:00:20Z","timestamp":1787695220000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.64898\/2026.07.15.26358166"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,17]]},"references-count":17,"URL":"https:\/\/doi.org\/10.64898\/2026.07.15.26358166","relation":{},"subject":[],"published":{"date-parts":[[2026,7,17]]},"subtype":"preprint"}}