{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,10]],"date-time":"2026-11-10T19:04:17Z","timestamp":1794337457599,"version":"4.9.0"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100009741","name":"David och Astrid Hagel\u00e9ns Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100009741","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007083","name":"Orionin Tutkimuss\u00e4\u00e4ti\u00f6","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100007083","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009450","name":"KAUTE-S\u00e4\u00e4ti\u00f6","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009450","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Oskar Huttunen Foundation"},{"DOI":"10.13039\/501100004359","name":"Vetenskapsr\u00e5det","doi-asserted-by":"publisher","award":["2020-00692"],"award-info":[{"award-number":["2020-00692"]}],"id":[{"id":"10.13039\/501100004359","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002794","name":"Cancerfonden","doi-asserted-by":"publisher","award":["21 1715 Pj"],"award-info":[{"award-number":["21 1715 Pj"]}],"id":[{"id":"10.13039\/501100002794","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100006285","name":"Magnus Bergvalls Stiftelse","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100006285","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Region Stockholm, Svenska Druidorden, \u00c5ke Wibergs Stiftelse, Swedish e-Science Research Center (SeRC), the Swedish Prostate Cancer Foundation","id":[{"id":"https:\/\/ror.org\/01e59yk92","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Commun"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Unreliable predictions can occur when an artificial intelligence (AI) system is presented with data it has not been exposed to during training. We demonstrate the use of conformal prediction to detect unreliable predictions, using histopathological diagnosis and grading of prostate biopsies as example. We digitized 7788 prostate biopsies from 1192 men in the STHLM3 diagnostic study, used for training, and 3059 biopsies from 676 men used for testing. With conformal prediction, 1 in 794 (0.1%) predictions is incorrect for cancer diagnosis (compared to 14 errors [2%] without conformal prediction) while 175 (22%) of the predictions are flagged as unreliable when the AI-system is presented with new data from the same lab and scanner that it was trained on. Conformal prediction could with small samples (\n                    <jats:italic>N<\/jats:italic>\n                    \u2009=\u200949 for external scanner,\n                    <jats:italic>N<\/jats:italic>\n                    \u2009=\u200910 for external lab and scanner, and\n                    <jats:italic>N<\/jats:italic>\n                    \u2009=\u200912 for external lab, scanner and pathology assessment) detect systematic differences in external data leading to worse predictive performance. The AI-system with conformal prediction commits 3 (2%) errors for cancer detection in cases of atypical prostate tissue compared to 44 (25%) without conformal prediction, while the system flags 143 (80%) unreliable predictions. We conclude that conformal prediction can increase patient safety of AI-systems.\n                  <\/jats:p>","DOI":"10.1038\/s41467-022-34945-8","type":"journal-article","created":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T01:03:44Z","timestamp":1671066224000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":125,"title":["Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction"],"prefix":"10.1038","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2270-2017","authenticated-orcid":false,"given":"Henrik","family":"Olsson","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9470-4783","authenticated-orcid":false,"given":"Kimmo","family":"Kartasalo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nita","family":"Mulliqi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Capuccini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pekka","family":"Ruusuvuori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hemamali","family":"Samaratunga","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brett","family":"Delahunt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5611-1015","authenticated-orcid":false,"given":"Cecilia","family":"Lindskog","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emiel A. M.","family":"Janssen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anders","family":"Blilie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"ISUP Prostate Imagebase Expert Panel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lars","family":"Egevad","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8083-2864","authenticated-orcid":false,"given":"Ola","family":"Spjuth","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5032-5266","authenticated-orcid":false,"given":"Martin","family":"Eklund","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,12,15]]},"reference":[{"key":"34945_CR1","doi-asserted-by":"publisher","first-page":"1101","DOI":"10.1001\/jama.2018.11100","volume":"320","author":"G Hinton","year":"2018","unstructured":"Hinton, G. Deep learning-a technology with the potential to transform health care. J. Am. Med. Assoc. 320, 1101\u20131102 (2018).","journal-title":"J. Am. Med. Assoc."},{"key":"34945_CR2","doi-asserted-by":"publisher","first-page":"2353","DOI":"10.1001\/jama.2016.17438","volume":"316","author":"S Jha","year":"2016","unstructured":"Jha, S. & Topol, E. J. Adapting to artificial intelligence: radiologists and pathologists as information specialists. J. Am. Med. Assoc. 316, 2353\u20132354 (2016).","journal-title":"J. Am. Med. Assoc."},{"key":"34945_CR3","doi-asserted-by":"publisher","unstructured":"AI diagnostics need attention. Nature. 555, 285 (2018). https:\/\/doi.org\/10.1038\/d41586-018-03067-x.","DOI":"10.1038\/d41586-018-03067-x"},{"key":"34945_CR4","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1038\/s41746-019-0155-4","volume":"2","author":"T Panch","year":"2019","unstructured":"Panch, T., Mattie, H. & Celi, L. A. The \u201cinconvenient truth\u201d about AI in healthcare. npj Digit. Med. 2, 4\u20136 (2019).","journal-title":"npj Digit. Med."},{"key":"34945_CR5","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1016\/j.euf.2021.07.002","volume":"7","author":"K Kartasalo","year":"2021","unstructured":"Kartasalo, K. et al. Artificial Intelligence for diagnosis and Gleason Grading of Prostate Cancer in Biopsies\u2014current status and next steps. Eur. Urol. Focus 7, 687\u2013691 (2021).","journal-title":"Eur. Urol. Focus"},{"key":"34945_CR6","doi-asserted-by":"publisher","unstructured":"Vovk, V., Gammerman, A. & Shafer, G. Algorithmic Learning in a Random World. https:\/\/doi.org\/10.1007\/b106715 (2005).","DOI":"10.1007\/b106715"},{"key":"34945_CR7","unstructured":"Str\u00f6m, P. et al. Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study. Lancet Oncol. 21, 222\u2013232 (2020)."},{"key":"34945_CR8","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1016\/S1470-2045(19)30739-9","volume":"21","author":"W Bulten","year":"2020","unstructured":"Bulten, W. et al. Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study. Lancet Oncol. 21, 233\u2013241 (2020).","journal-title":"Lancet Oncol."},{"key":"34945_CR9","doi-asserted-by":"publisher","first-page":"1372","DOI":"10.1001\/jamaoncol.2020.2485","volume":"6","author":"K Nagpal","year":"2020","unstructured":"Nagpal, K. et al. Development and validation of a deep learning algorithm for Gleason grading of prostate cancer from biopsy specimens. JAMA Oncol. 6, 1372\u20131380 (2020).","journal-title":"JAMA Oncol."},{"key":"34945_CR10","doi-asserted-by":"publisher","unstructured":"Bulten, W. et al. Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge. Nat. Med. https:\/\/doi.org\/10.1038\/s41591-021-01620-2 (2022).","DOI":"10.1038\/s41591-021-01620-2"},{"key":"34945_CR11","doi-asserted-by":"publisher","first-page":"1667","DOI":"10.1016\/S1470-2045(15)00361-7","volume":"16","author":"H Gr\u00f6nberg","year":"2015","unstructured":"Gr\u00f6nberg, H. et al. Prostate cancer screening in men aged 50\u201369 years (STHLM3): a prospective population-based diagnostic study. Lancet Oncol. 16, 1667\u20131676 (2015).","journal-title":"Lancet Oncol."},{"key":"34945_CR12","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1111\/his.13471","volume":"73","author":"L Egevad","year":"2018","unstructured":"Egevad, L. et al. Utility of pathology Imagebase for standardisation of prostate cancer grading. Histopathology 73, 8\u201318 (2018).","journal-title":"Histopathology"},{"key":"34945_CR13","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1097\/PAS.0000000000001202","volume":"43","author":"L Beltran","year":"2019","unstructured":"Beltran, L. et al. Histopathologic false-positive diagnoses of prostate cancer in the age of immunohistochemistry. Am. J. Surg. Pathol. 43, 361\u2013368 (2019).","journal-title":"Am. J. Surg. Pathol."},{"key":"34945_CR14","doi-asserted-by":"publisher","first-page":"759","DOI":"10.1111\/j.1365-2559.2011.03794.x","volume":"58","author":"JD Oxley","year":"2011","unstructured":"Oxley, J. D. & Sen, C. Error rates in reporting prostatic core biopsies. Histopathology 58, 759\u2013765 (2011).","journal-title":"Histopathology"},{"key":"34945_CR15","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1093\/jcr\/ucz013","volume":"46","author":"C Longoni","year":"2019","unstructured":"Longoni, C., Bonezzi, A. & Morewedge, C. K. Resistance to medical artificial intelligence. J. Consum. Res. 46, 629\u2013650 (2019).","journal-title":"J. Consum. Res."},{"key":"34945_CR16","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1111\/his.12008","volume":"62","author":"L Egevad","year":"2013","unstructured":"Egevad, L. et al. Standardization of Gleason grading among 337 European pathologists. Histopathology 62, 247\u2013256 (2013).","journal-title":"Histopathology"},{"key":"34945_CR17","unstructured":"Fedorova, V., Gammerman, A., Nouretdinov, I. & Vovk, V. Plug-in martingales for testing exchangeability on-line. in Proceedings of the 29th International Conference on Machine Learning, ICML 2012 2 (2012)."},{"key":"34945_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-017-17876-z","volume":"7","author":"C Leibig","year":"2017","unstructured":"Leibig, C., Allken, V., Ayhan, M. S., Berens, P. & Wahl, S. Leveraging uncertainty information from deep neural networks for disease detection. Sci. Rep. 7, 1\u201314 (2017).","journal-title":"Sci. Rep."},{"key":"34945_CR19","first-page":"1651","volume":"3","author":"Y Gal","year":"2016","unstructured":"Gal, Y. & Ghahramani, Z. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. 33rd Int. Conf. Mach. Learn. 3, 1651\u20131660 (2016).","journal-title":"33rd Int. Conf. Mach. Learn."},{"key":"34945_CR20","first-page":"6403","volume":"2017","author":"B Lakshminarayanan","year":"2017","unstructured":"Lakshminarayanan, B., Pritzel, A. & Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Adv. Neural Inf. Process. Syst. 2017, 6403\u20136414 (2017).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"34945_CR21","doi-asserted-by":"publisher","unstructured":"Papadopoulos, H. Inductive conformal prediction: theory and application to neural networks. Tools in Artificial Intelligence https:\/\/doi.org\/10.5772\/6078 (2008).","DOI":"10.5772\/6078"},{"key":"34945_CR22","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1093\/comjnl\/bxl065","volume":"50","author":"A Gammerman","year":"2007","unstructured":"Gammerman, A. & Vovk, V. Hedging predictions in machine learning. Comput. J. 50, 151\u2013163 (2007).","journal-title":"Comput. J."},{"key":"34945_CR23","first-page":"371","volume":"9","author":"G Shafer","year":"2008","unstructured":"Shafer, G. & Vovk, V. A tutorial on conformal prediction. J. Mach. Learn. Res. 9, 371\u2013421 (2008).","journal-title":"J. Mach. Learn. Res."},{"key":"34945_CR24","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1016\/j.xphs.2020.09.055","volume":"110","author":"J Alvarsson","year":"2021","unstructured":"Alvarsson, J., Arvidsson McShane, S., Norinder, U. & Spjuth, O. Predicting with confidence: using conformal prediction in drug discovery. J. Pharm. Sci. 110, 42\u201349 (2021).","journal-title":"J. Pharm. Sci."},{"key":"34945_CR25","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1109\/JBHI.2020.2996300","volume":"25","author":"H Wieslander","year":"2021","unstructured":"Wieslander, H. et al. Deep learning with conformal prediction for hierarchical analysis of large-scale whole-slide tissue images. IEEE J. Biomed. Health Inform. 25, 371\u2013380 (2021).","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"34945_CR26","first-page":"2818","volume":"2016","author":"C Szegedy","year":"2016","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2016, 2818\u20132826 (2016).","journal-title":"Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit."},{"key":"34945_CR27","doi-asserted-by":"publisher","unstructured":"Deng, J. et al. ImageNet: A large-scale hierarchical image database. in 2009 IEEE Conference on Computer Vision and Pattern Recognition 248\u2013255, https:\/\/doi.org\/10.1109\/CVPR.2009.5206848 (IEEE, 2009).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"34945_CR28","doi-asserted-by":"publisher","unstructured":"Chen, T. & Guestrin, C. XGBoost. in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785\u2013794, https:\/\/doi.org\/10.1145\/2939672.2939785 (ACM, 2016).","DOI":"10.1145\/2939672.2939785"},{"key":"34945_CR29","doi-asserted-by":"crossref","unstructured":"Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. M. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med. 13, 1 (2015).","DOI":"10.1186\/s12916-014-0241-z"}],"container-title":["Nature Communications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41467-022-34945-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41467-022-34945-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41467-022-34945-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,18]],"date-time":"2022-12-18T05:03:20Z","timestamp":1671339800000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41467-022-34945-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,15]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["34945"],"URL":"https:\/\/doi.org\/10.1038\/s41467-022-34945-8","relation":{},"ISSN":["2041-1723"],"issn-type":[{"value":"2041-1723","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,15]]},"assertion":[{"value":"2 May 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 November 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 December 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"M.E. has four patents (WO2013EP74259 20131120, WO2013EP74270 20131120, WO2018EP52473 20180201, and WO2013SE50554 20130516) related to prostate cancer diagnostics pending, and has patent applications licensed to Thermo Fisher Scientific, and is named on a pending patent (1900061-1) related to cancer diagnostics quality control. All other authors declare no competing interests. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"7761"}}