{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,26]],"date-time":"2026-04-26T05:33:08Z","timestamp":1777181588805,"version":"3.51.4"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,2,26]],"date-time":"2020-02-26T00:00:00Z","timestamp":1582675200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,2,26]],"date-time":"2020-02-26T00:00:00Z","timestamp":1582675200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Artificial intelligence (AI) algorithms continue to rival human performance on a variety of clinical tasks, while their actual impact on human diagnosticians, when incorporated into clinical workflows, remains relatively unexplored. In this study, we developed a deep learning-based assistant to help pathologists differentiate between two subtypes of primary liver cancer, hepatocellular carcinoma and cholangiocarcinoma, on hematoxylin and eosin-stained whole-slide images (WSI), and evaluated its effect on the diagnostic performance of 11 pathologists with varying levels of expertise. Our model achieved accuracies of 0.885 on a validation set of 26 WSI, and 0.842 on an independent test set of 80 WSI. Although use of the assistant did not change the mean accuracy of the 11 pathologists (<jats:italic>p<\/jats:italic>\u2009=\u20090.184, OR\u2009=\u20091.281), it significantly improved the accuracy (<jats:italic>p<\/jats:italic>\u2009=\u20090.045, OR\u2009=\u20091.499) of a subset of nine pathologists who fell within well-defined experience levels (GI subspecialists, non-GI subspecialists, and trainees). In the assisted state, model accuracy significantly impacted the diagnostic decisions of all 11 pathologists. As expected, when the model\u2019s prediction was correct, assistance significantly improved accuracy (<jats:italic>p<\/jats:italic>\u2009=\u20090.000, OR\u2009=\u20094.289), whereas when the model\u2019s prediction was incorrect, assistance significantly decreased accuracy (<jats:italic>p<\/jats:italic>\u2009=\u20090.000, OR\u2009=\u20090.253), with both effects holding across all pathologist experience levels and case difficulty levels. Our results highlight the challenges of translating AI models into the clinical setting, and emphasize the importance of taking into account potential unintended negative consequences of model assistance when designing and testing medical AI-assistance tools.<\/jats:p>","DOI":"10.1038\/s41746-020-0232-8","type":"journal-article","created":{"date-parts":[[2020,2,26]],"date-time":"2020-02-26T11:03:19Z","timestamp":1582714999000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":269,"title":["Impact of a deep learning assistant on the histopathologic classification of liver cancer"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4471-477X","authenticated-orcid":false,"given":"Amirhossein","family":"Kiani","sequence":"first","affiliation":[]},{"given":"Bora","family":"Uyumazturk","sequence":"additional","affiliation":[]},{"given":"Pranav","family":"Rajpurkar","sequence":"additional","affiliation":[]},{"given":"Alex","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Rebecca","family":"Gao","sequence":"additional","affiliation":[]},{"given":"Erik","family":"Jones","sequence":"additional","affiliation":[]},{"given":"Yifan","family":"Yu","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8972-8051","authenticated-orcid":false,"given":"Curtis P.","family":"Langlotz","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7335-3339","authenticated-orcid":false,"given":"Robyn L.","family":"Ball","sequence":"additional","affiliation":[]},{"given":"Thomas J.","family":"Montine","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8539-9556","authenticated-orcid":false,"given":"Brock A.","family":"Martin","sequence":"additional","affiliation":[]},{"given":"Gerald J.","family":"Berry","sequence":"additional","affiliation":[]},{"given":"Michael G.","family":"Ozawa","sequence":"additional","affiliation":[]},{"given":"Florette K.","family":"Hazard","sequence":"additional","affiliation":[]},{"given":"Ryanne A.","family":"Brown","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0409-5978","authenticated-orcid":false,"given":"Simon B.","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Mona","family":"Wood","sequence":"additional","affiliation":[]},{"given":"Libby S.","family":"Allard","sequence":"additional","affiliation":[]},{"given":"Lourdes","family":"Ylagan","sequence":"additional","affiliation":[]},{"given":"Andrew Y.","family":"Ng","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1519-0308","authenticated-orcid":false,"given":"Jeanne","family":"Shen","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,2,26]]},"reference":[{"key":"232_CR1","doi-asserted-by":"publisher","first-page":"237428951771476","DOI":"10.1177\/2374289517714767","volume":"4","author":"JL Conant","year":"2017","unstructured":"Conant, J. L., Gibson, P. C., Bunn, J. & Ambaye, A. B. Transition to subspecialty sign-out at an academic institution and its advantages. Acad. Pathol. 4, 2374289517714767 (2017).","journal-title":"Acad. Pathol."},{"key":"232_CR2","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/j.anndiagpath.2016.07.007","volume":"24","author":"Y-J Liu","year":"2016","unstructured":"Liu, Y.-J., Kessler, M., Zander, D. S. & Karamchandani, D. M. Trends in extramural consultation: comparison between subspecialized and general surgical pathology service models. Ann. Diagn. Pathol. 24, 20\u201324 (2016).","journal-title":"Ann. Diagn. Pathol."},{"key":"232_CR3","doi-asserted-by":"publisher","first-page":"871","DOI":"10.5858\/arpa.2014-0084-ED","volume":"138","author":"SJ Sarewitz","year":"2014","unstructured":"Sarewitz, S. J. Subspecialization in community pathology practice. Arch. Pathol. Lab. Med. 138, 871\u2013872 (2014).","journal-title":"Arch. Pathol. Lab. Med."},{"key":"232_CR4","first-page":"S74","volume":"106","author":"JC Watts","year":"1996","unstructured":"Watts, J. C. Is there still a need for the general surgical pathologist? Am. J. Clin. Pathol. 106, S74\u2013S76 (1996).","journal-title":"Am. J. Clin. Pathol."},{"key":"232_CR5","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1136\/jcp.53.1.7","volume":"53","author":"N Kirkham","year":"2000","unstructured":"Kirkham, N. The pathologist in the 21st century\u2014generalist or specialist? J. Clin. Pathol. 53, 7\u20139 (2000).","journal-title":"J. Clin. Pathol."},{"key":"232_CR6","doi-asserted-by":"publisher","first-page":"761","DOI":"10.1136\/jcp.2010.079640","volume":"63","author":"M Heatley","year":"2010","unstructured":"Heatley, M. Subspecialisation and despecialisation in anatomical pathology. J. Clin. Pathol. 63, 761\u2013767 (2010).","journal-title":"J. Clin. Pathol."},{"key":"232_CR7","doi-asserted-by":"publisher","first-page":"1156","DOI":"10.1038\/bjc.1997.199","volume":"75","author":"SE Parkes","year":"1997","unstructured":"Parkes, S. E. et al. The need for specialist review of pathology in paediatric cancer. Br. J. Cancer 75, 1156\u20131159 (1997).","journal-title":"Br. J. Cancer"},{"key":"232_CR8","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1136\/bmj.317.7151.93","volume":"317","author":"A Rydholm","year":"1998","unstructured":"Rydholm, A. Improving the management of soft tissue sarcoma. BMJ 317, 93\u201394 (1998).","journal-title":"BMJ"},{"key":"232_CR9","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002686","volume":"15","author":"P Rajpurkar","year":"2018","unstructured":"Rajpurkar, P. et al. Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLOS Med. 15, e1002686 (2018).","journal-title":"PLOS Med."},{"key":"232_CR10","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1038\/s41591-018-0268-3","volume":"25","author":"AY Hannun","year":"2019","unstructured":"Hannun, A. Y. et al. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat. Med. 25, 65 (2019).","journal-title":"Nat. Med."},{"key":"232_CR11","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115\u2013118 (2017).","journal-title":"Nature"},{"key":"232_CR12","doi-asserted-by":"publisher","first-page":"2199","DOI":"10.1001\/jama.2017.14585","volume":"318","author":"BE Bejnordi","year":"2017","unstructured":"Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199\u20132210 (2017).","journal-title":"JAMA"},{"key":"232_CR13","doi-asserted-by":"publisher","first-page":"595","DOI":"10.1007\/s00345-007-0209-0","volume":"25","author":"R Engers","year":"2007","unstructured":"Engers, R. Reproducibility and reliability of tumor grading in urological neoplasms. World J. Urol. 25, 595\u2013605 (2007).","journal-title":"World J. Urol."},{"key":"232_CR14","doi-asserted-by":"publisher","first-page":"1574","DOI":"10.1038\/modpathol.2012.106","volume":"25","author":"E Thunnissen","year":"2012","unstructured":"Thunnissen, E. et al. Reproducibility of histopathological subtypes and invasion in pulmonary adenocarcinoma. An international interobserver study. Mod. Pathol. 25, 1574\u20131583 (2012).","journal-title":"Mod. Pathol."},{"key":"232_CR15","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1186\/s13000-014-0162-3","volume":"9","author":"T Pongpruttipan","year":"2014","unstructured":"Pongpruttipan, T. et al. Interobserver variation in classifying lymphomas among hematopathologists. Diagn. Pathol. 9, 162 (2014).","journal-title":"Diagn. Pathol."},{"key":"232_CR16","doi-asserted-by":"publisher","first-page":"1175","DOI":"10.1053\/hupa.2002.130104","volume":"33","author":"SK Lau","year":"2002","unstructured":"Lau, S. K., Prakash, S., Geller, S. A. & Alsabeh, R. Comparative immunohistochemical profile of hepatocellular carcinoma, cholangiocarcinoma, and metastatic adenocarcinoma. Hum. Pathol. 33, 1175\u20131181 (2002).","journal-title":"Hum. Pathol."},{"key":"232_CR17","doi-asserted-by":"publisher","first-page":"1559","DOI":"10.1038\/s41591-018-0177-5","volume":"24","author":"N Coudray","year":"2018","unstructured":"Coudray, N. et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24, 1559\u20131567 (2018).","journal-title":"Nat. Med."},{"key":"232_CR18","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-019-00182-7","author":"MZ Alom","year":"2019","unstructured":"Alom, M. Z., Yakopcic, C., Nasrin Mst. S., Taha, T. M. & Asari, V. K. Breast cancer classification from histopathological images with inception recurrent residual convolutional neural network. J. Digit. Imaging. https:\/\/doi.org\/10.1007\/s10278-019-00182-7 (2019).","journal-title":"J. Digit. Imaging"},{"key":"232_CR19","doi-asserted-by":"publisher","first-page":"1636","DOI":"10.1097\/PAS.0000000000001151","volume":"42","author":"DF Steiner","year":"2018","unstructured":"Steiner, D. F. et al. Impact of deep learning assistance on the histopathologic review of lymph nodes for metastatic breast cancer. Am. J. Surg. Pathol. 42, 1636\u20131646 (2018).","journal-title":"Am. J. Surg. Pathol."},{"key":"232_CR20","unstructured":"Amin, M. B., Edge, S. B. & American Joint Committee on Cancer. AJCC Cancer Staging Manual (2017)."},{"key":"232_CR21","first-page":"201","volume":"38","author":"SF Altekruse","year":"2011","unstructured":"Altekruse, S. F., Devesa, S. S., Dickie, L. A., McGlynn, K. A. & Kleiner, D. E. Histological classification of liver and intrahepatic bile duct cancers in SEER registries. J. Regist. Manag. 38, 201\u2013205 (2011).","journal-title":"J. Regist. Manag."},{"key":"232_CR22","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1309\/59TNEFALUL5WJ94M","volume":"125","author":"J-Y Lei","year":"2006","unstructured":"Lei, J.-Y., Bourne, P. A., diSant\u2019Agnese, P. A. & Huang, J. Cytoplasmic staining of TTF-1 in the differential diagnosis of hepatocellular carcinoma vs cholangiocarcinoma and metastatic carcinoma of the liver. Am. J. Clin. Pathol. 125, 519\u2013525 (2006).","journal-title":"Am. J. Clin. Pathol."},{"key":"232_CR23","doi-asserted-by":"publisher","first-page":"306","DOI":"10.5009\/gnl17277","volume":"12","author":"HG Hass","year":"2018","unstructured":"Hass, H. G., Vogel, U., Scheurlen, M. & Jobst, J. Subclassification and detection of new markers for the discrimination of primary liver tumors by gene expression analysis using oligonucleotide arrays. Gut Liver 12, 306\u2013315 (2018).","journal-title":"Gut Liver"},{"key":"232_CR24","doi-asserted-by":"publisher","first-page":"1109","DOI":"10.1056\/NEJMp1607591","volume":"375","author":"RL Grossman","year":"2016","unstructured":"Grossman, R. L. et al. Toward a shared vision for cancer genomic data. N. Engl. J. Med. 375, 1109\u20131112 (2016).","journal-title":"N. Engl. J. Med."},{"key":"232_CR25","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"232_CR26","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., van der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2261\u20132269 (2017).","DOI":"10.1109\/CVPR.2017.243"},{"key":"232_CR27","doi-asserted-by":"crossref","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. & Torralba, A. Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2921\u20132929 (2016).","DOI":"10.1109\/CVPR.2016.319"},{"key":"232_CR28","doi-asserted-by":"publisher","first-page":"1710","DOI":"10.5858\/arpa.2013-0093-CP","volume":"137","author":"L Pantanowitz","year":"2013","unstructured":"Pantanowitz, L. et al. Validating whole slide imaging for diagnostic purposes in pathology: guideline from the College of American Pathologists Pathology and Laboratory Quality Center. Arch. Pathol. Lab. Med. 137, 1710\u20131722 (2013).","journal-title":"Arch. Pathol. Lab. Med."},{"key":"232_CR29","doi-asserted-by":"publisher","first-page":"991","DOI":"10.1006\/ijhc.1999.0252","volume":"51","author":"LJ Skitka","year":"1999","unstructured":"Skitka, L. J., Mosier, K. L. & Burdick, M. Does automation bias decision-making? Int. J. Hum. -Comput. Stud. 51, 991\u20131006 (1999).","journal-title":"Int. J. Hum. -Comput. Stud."},{"key":"232_CR30","doi-asserted-by":"publisher","first-page":"1723","DOI":"10.5858\/arpa.2013-0200-OA","volume":"137","author":"SJ Robboy","year":"2013","unstructured":"Robboy, S. J. et al. Pathologist workforce in the United States: I. Development of a predictive model to examine factors influencing supply. Arch. Pathol. Lab. Med. 137, 1723\u20131732 (2013).","journal-title":"Arch. Pathol. Lab. Med."},{"key":"232_CR31","doi-asserted-by":"publisher","first-page":"1927","DOI":"10.1016\/S0140-6736(18)30458-6","volume":"391","author":"ML Wilson","year":"2018","unstructured":"Wilson, M. L. et al. Access to pathology and laboratory medicine services: a crucial gap. Lancet Lond. Engl. 391, 1927\u20131938 (2018).","journal-title":"Lancet Lond. Engl."},{"key":"232_CR32","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1080\/01621459.1927.10502953","volume":"22","author":"EB Wilson","year":"1927","unstructured":"Wilson, E. B. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 22, 209\u2013212 (1927).","journal-title":"J. Am. Stat. Assoc."},{"key":"232_CR33","first-page":"1","volume":"28","author":"M Kuhn","year":"2008","unstructured":"Kuhn, M. Building predictive models in R using the caret package. J. Stat. Softw. Artic. 28, 1\u201326 (2008).","journal-title":"J. Stat. Softw. Artic."},{"key":"232_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v067.i01","volume":"67","author":"D Bates","year":"2015","unstructured":"Bates, D., M\u00e4chler, M., Bolker, B. & Walker, S. Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67, 1\u201348 (2015).","journal-title":"J. Stat. Softw."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0232-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0232-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0232-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T01:54:47Z","timestamp":1670378087000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0232-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,26]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["232"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0232-8","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,26]]},"assertion":[{"value":"3 May 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 February 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 February 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"23"}}