{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T15:27:42Z","timestamp":1788449262684,"version":"build-2803163510"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T00:00:00Z","timestamp":1774137600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T00:00:00Z","timestamp":1774224000000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003945","name":"Link\u00f6ping University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003945","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Deep learning based cervical cancer classification can potentially increase access to screening in low-resource regions. However, deep learning models are often overconfident and do not reliably reflect diagnostic uncertainty. Moreover, they are typically optimized to generate maximum-likelihood predictions, which fail to convey uncertainty or ambiguity in their results. Such challenges can be addressed using conformal prediction, a model-agnostic framework for generating prediction sets that contain likely classes for trained deep-learning models. The size of these prediction sets indicates model uncertainty, contracting as model confidence increases. However, existing validation of conformal prediction primarily focuses on whether the prediction set includes or covers the true class, often overlooking the presence of extraneous classes. We argue that prediction sets should be truthful and valuable to end users, ensuring that the listed likely classes align with human expectations rather than being overly relaxed and including false positives or unlikely classes. In this study, we comprehensively validate conformal prediction sets using expert annotation sets collected from multiple annotators. We evaluate three conformal prediction approaches applied to three deep-learning models trained for cervical atypia classification. Our expert annotation-based analysis reveals that conventional coverage-based validations overestimate performance and that current conformal prediction methods often produce prediction sets that are not well aligned with human labels. Additionally, we explore the capabilities of the conformal prediction methods in identifying ambiguous and out-of-distribution data.<\/jats:p>","DOI":"10.1038\/s41598-026-44850-5","type":"journal-article","created":{"date-parts":[[2026,3,22]],"date-time":"2026-03-22T07:04:46Z","timestamp":1774163086000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Validation of conformal prediction in cervical atypia classification"],"prefix":"10.1038","volume":"16","author":[{"given":"Misgina Tsighe","family":"Hagos","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Antti","family":"Suutala","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dmitrii","family":"Bychkov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hakan","family":"K\u00fcc\u00fckel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joar","family":"von Bahr","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Milda","family":"Poceviciute","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johan","family":"Lundin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nina","family":"Linder","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Claes","family":"Lundstr\u00f6m","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,3,22]]},"reference":[{"key":"44850_CR1","first-page":"209","volume":"71","author":"H Sung","year":"2021","unstructured":"Sung, H. et al. Global cancer statistics Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: Cancer J. Clin. 71, 209\u2013249 (2021).","journal-title":"CA: Cancer J. Clin."},{"key":"44850_CR2","unstructured":"World Health Organization. Cervical cancer. https:\/\/www.who.int\/news-room\/fact-sheets\/detail\/cervical-cancer (2024). Accessed: 2025-03-10."},{"key":"44850_CR3","unstructured":"Cervical Cancer Screening, vol.\u00a018 of IARC Handbooks of Cancer Prevention (International Agency for Research on Cancer, Lyon, France, 2022)."},{"key":"44850_CR4","doi-asserted-by":"publisher","first-page":"e211740","DOI":"10.1001\/jamanetworkopen.2021.1740","volume":"4","author":"O Holmstr\u00f6m","year":"2021","unstructured":"Holmstr\u00f6m, O. et al. Point-of-care digital cytology with artificial intelligence for cervical cancer screening in a resource-limited setting. JAMA Network Open 4, e211740\u2013e211740 (2021).","journal-title":"JAMA Network Open"},{"key":"44850_CR5","doi-asserted-by":"publisher","first-page":"16","DOI":"10.25259\/Cytojournal_24_2021","volume":"18","author":"A Alrajjal","year":"2021","unstructured":"Alrajjal, A., Pansare, V., Choudhury, M. S. R., Khan, M. Y. A. & Shidham, V. B. Squamous intraepithelial lesions (sil: Lsil, hsil, ascus, asc-h, lsil-h) of uterine cervix and bethesda system. Cytojournal 18, 16 (2021).","journal-title":"Cytojournal"},{"key":"44850_CR6","unstructured":"Guo, C., Pleiss, G., Sun, Y. & Weinberger, K.\u00a0Q. On calibration of modern neural networks. In International Conference on Machine Learning, 1321\u20131330 (PMLR, 2017)."},{"key":"44850_CR7","unstructured":"Nixon, J., Dusenberry, M.\u00a0W., Zhang, L., Jerfel, G. & Tran, D. Measuring calibration in deep learning. In CVPR workshops, vol.\u00a02 (2019)."},{"key":"44850_CR8","unstructured":"Ovadia, Y. et al. Can you trust your model\u2019s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing systems 32 (2019)."},{"key":"44850_CR9","unstructured":"Ghoshal, B. & Tucker, A. On calibrated model uncertainty in deep learning. arXiv preprint arXiv:2206.07795 (2022)."},{"key":"44850_CR10","unstructured":"Wang, C. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.01222 (2023)."},{"key":"44850_CR11","unstructured":"Vovk, V., Gammerman, A. & Shafer, G. Algorithmic learning in a random world, vol.\u00a029 (Springer)."},{"key":"44850_CR12","first-page":"3581","volume":"33","author":"Y Romano","year":"2020","unstructured":"Romano, Y., Sesia, M. & Candes, E. Classification with valid and adaptive coverage. Adv. Neural Inf. Process. Syst. 33, 3581\u20133591 (2020).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"44850_CR13","unstructured":"Angelopoulos, A.\u00a0N. & Bates, S. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprintarXiv:2107.07511 (2021)."},{"key":"44850_CR14","doi-asserted-by":"publisher","first-page":"1819","DOI":"10.1109\/TKDE.2013.39","volume":"26","author":"M-L Zhang","year":"2013","unstructured":"Zhang, M.-L. & Zhou, Z.-H. A review on multi-label learning algorithms. IEEE Trans. Knowl. Data Eng. 26, 1819\u20131837 (2013).","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"44850_CR15","doi-asserted-by":"crossref","unstructured":"Lu, C., Lemay, A., Chang, K., H\u00f6bel, K. & Kalpathy-Cramer, J. Fair conformal predictors for applications in medical imaging. In Proceedings of the AAAI Conference on Artificial Intelligence 36, 12008\u201312016 (2022).","DOI":"10.1609\/aaai.v36i11.21459"},{"key":"44850_CR16","doi-asserted-by":"publisher","first-page":"7761","DOI":"10.1038\/s41467-022-34945-8","volume":"13","author":"H Olsson","year":"2022","unstructured":"Olsson, H. et al. Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction. Nature Commun. 13, 7761 (2022).","journal-title":"Nature Commun."},{"key":"44850_CR17","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2024.108231","volume":"253","author":"J Fayyad","year":"2024","unstructured":"Fayyad, J., Alijani, S. & Najjaran, H. Empirical validation of conformal prediction for trustworthy skin lesions classification. Computer Methods Programs Biomed. 253, 108231 (2024).","journal-title":"Computer Methods Programs Biomed."},{"key":"44850_CR18","doi-asserted-by":"crossref","unstructured":"Clark, C. et al. Conformal prediction and monte carlo inference for addressing uncertainty in cervical cancer screening. In International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, 205\u2013214 (Springer, 2024).","DOI":"10.1007\/978-3-031-73158-7_19"},{"key":"44850_CR19","unstructured":"Angelopoulos, A.\u00a0N., Bates, S., Jordan, M. & Malik, J. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations (2021)."},{"key":"44850_CR20","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/S0951-8320(96)00077-4","volume":"54","author":"SC Hora","year":"1996","unstructured":"Hora, S. C. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliabil. Eng. Syst. Safety 54, 217\u2013223 (1996).","journal-title":"Reliabil. Eng. Syst. Safety"},{"key":"44850_CR21","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","volume":"110","author":"E H\u00fcllermeier","year":"2021","unstructured":"H\u00fcllermeier, E. & Waegeman, W. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Mac. Learn. 110, 457\u2013506 (2021).","journal-title":"Mac. Learn."},{"key":"44850_CR22","doi-asserted-by":"crossref","unstructured":"Ivezic, V. et al. Cytofm: The first cytology foundation model. In Proceedings of the Computer Vision and Pattern Recognition Conference, 4749\u20134757 (2025).","DOI":"10.1109\/CVPRW67362.2025.00461"},{"key":"44850_CR23","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770\u2013778 (2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"44850_CR24","unstructured":"Tan, M. & Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, 6105\u20136114 (PMLR, 2019)."},{"key":"44850_CR25","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky, O. et al. Imagenet large scale visual recognition challenge. Int. J. Computer Vision 115, 211\u2013252 (2015).","journal-title":"Int. J. Computer Vision"},{"key":"44850_CR26","unstructured":"Kingma, D.\u00a0P. & Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR) (2015). arXiv1412.6980."},{"key":"44850_CR27","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1080\/01621459.2017.1395341","volume":"114","author":"M Sadinle","year":"2019","unstructured":"Sadinle, M., Lei, J. & Wasserman, L. Least ambiguous set-valued classifiers with bounded error levels. J. Am. Statist. Assoc. 114, 223\u2013234 (2019).","journal-title":"J. Am. Statist. Assoc."},{"key":"44850_CR28","unstructured":"Matek, C., Krappe, S., M\u00fcnzenmayer, C., Haferlach, T. & Marr, C. An expert-annotated dataset of bone marrow cytology in hematologic malignancies, DOIurl10.7937\/TCIA.AXH3-T579 (2021). [Data set.] Accessed: 2025-01-25."},{"key":"44850_CR29","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","volume":"76","author":"JL Fleiss","year":"1971","unstructured":"Fleiss, J. L. Measuring nominal scale agreement among many raters. Psychol. Bull. 76, 378 (1971).","journal-title":"Psychol. Bull."},{"key":"44850_CR30","doi-asserted-by":"crossref","unstructured":"Landis, J.\u00a0R. & Koch, G.\u00a0G. The measurement of observer agreement for categorical data. Biometrics 159\u2013174 (1977).","DOI":"10.2307\/2529310"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-44850-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-44850-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-44850-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T05:04:18Z","timestamp":1774415058000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-026-44850-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,22]]},"references-count":30,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["44850"],"URL":"https:\/\/doi.org\/10.1038\/s41598-026-44850-5","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,22]]},"assertion":[{"value":"1 August 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 March 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 March 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Johan Lundin is a co-founder and co-owner of Aiforia Technologies Plc. Claes Lundstr\u00f6m is an employee of Sectra AB. No other disclosures were reported.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"Ethical approvals for the current study have been issued by the Ethics Review Committee of Pwani University (ERC\/PU-STAFF\/005\/2018) and the Ethical Review Committee of the Technical University of Mombasa (No: TUM ERC EXT\/001\/2022), both accredited by the National Commission for Science, Technology and Innovation (NACOSTI), Kenya. The entire study has been conducted following the Helsinki Declaration and ICH Good Clinical Practice Guidelines.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}],"article-number":"9649"}}