{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,4]],"date-time":"2025-04-04T12:12:13Z","timestamp":1743768733651},"reference-count":12,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,12,20]],"date-time":"2023-12-20T00:00:00Z","timestamp":1703030400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,12,20]],"date-time":"2023-12-20T00:00:00Z","timestamp":1703030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Fraunhofer-Institut f\u00fcr Intelligente Analyse- und Informationssysteme IAIS"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["AI Ethics"],"published-print":{"date-parts":[[2024,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Our Visual Analytics (VA) tool ScrutinAI supports human analysts to investigate interactively model performance and data sets. Model performance depends on labeling quality to a large extent. In particular in medical settings, generation of high quality labels requires in depth expert knowledge and is very costly. Often, data sets are labeled by collecting opinions of groups of experts. We use our VA tool to analyze the influence of label variations between different experts on the model performance. ScrutinAI facilitates to perform a root cause analysis that distinguishes weaknesses of deep neural network (DNN) models caused by varying or missing labeling quality from true weaknesses. We scrutinize the overall detection of intracranial hemorrhages and the more subtle differentiation between subtypes in a publicly available data set.<\/jats:p>","DOI":"10.1007\/s43681-023-00399-x","type":"journal-article","created":{"date-parts":[[2023,12,20]],"date-time":"2023-12-20T12:02:29Z","timestamp":1703073749000},"page":"151-156","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Using ScrutinAI for visual inspection of DNN performance in a medical use case"],"prefix":"10.1007","volume":"4","author":[{"given":"Rebekka","family":"G\u00f6rge","sequence":"first","affiliation":[]},{"given":"Elena","family":"Haedecke","sequence":"additional","affiliation":[]},{"given":"Michael","family":"Mock","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,12,20]]},"reference":[{"issue":"1","key":"399_CR1","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1109\/MCG.2021.3130314","volume":"42","author":"N Andrienko","year":"2022","unstructured":"Andrienko, N., Andrienko, G., Adilova, L., Wrobel, S.: Visual analytics for human-centered machine learning. IEEE Comput. Graph. Appl. 42(1), 123\u2013133 (2022)","journal-title":"IEEE Comput. Graph. Appl."},{"key":"399_CR2","doi-asserted-by":"crossref","unstructured":"Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.\u00a0N.: Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE winter conference on applications of computer vision (WACV), 839\u2013847. IEEE (2018)","DOI":"10.1109\/WACV.2018.00097"},{"key":"399_CR3","doi-asserted-by":"crossref","unstructured":"Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N.\u00a0G., Venugopal, V.\u00a0K., Mahajan, V., Rao, P., Warier, P.: Development and validation of deep learning algorithms for detection of critical findings in head CT scans. arXiv preprint arXiv:1803.05854 (2018)","DOI":"10.1016\/S0140-6736(18)31645-3"},{"issue":"3","key":"399_CR4","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2020190211","volume":"2","author":"AE Flanders","year":"2020","unstructured":"Flanders, A.E., Prevedello, L.M., Shih, G., Halabi, S.S., Kalpathy-Cramer, J., Ball, R., Mongan, J.T., Stein, A., Kitamura, F.C., Lungren, M.P., et al.: Construction of a machine learning dataset through collaboration: the RSNA 2019 brain CT hemorrhage challenge. Radiol. Artif. Intell. 2(3), e190211 (2020)","journal-title":"Radiol. Artif. Intell."},{"key":"399_CR5","unstructured":"G\u00f6rge, R.: Verwendung von Deep Learning zur Erkennung von h\u00e4morrhagischen Schlaganf\u00e4llen in CT-Scans des Sch\u00e4dels. Master\u2019s thesis, University Koblenz-Landau (2021)"},{"key":"399_CR6","unstructured":"Haedecke, E., Mock, M., Akila, M.: ScrutinAI: A Visual Analytics Approach for the Semantic Analysis of Deep Neural Network Predictions. In Bernard, J.; and Angelini, M., eds., EuroVis Workshop on Visual Analytics (EuroVA). The Eurographics Association. ISBN 978-3-03868-183-0 (2022)"},{"key":"399_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101759","volume":"65","author":"D Karimi","year":"2020","unstructured":"Karimi, D., Dou, H., Warfield, S.K., Gholipour, A.: Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. Med. Image Anal. 65, 101759 (2020)","journal-title":"Med. Image Anal."},{"issue":"6","key":"399_CR8","doi-asserted-by":"publisher","first-page":"1868","DOI":"10.1109\/TMI.2019.2959209","volume":"39","author":"Z Liao","year":"2019","unstructured":"Liao, Z., Girgis, H., Abdi, A., Vaseli, H., Hetherington, J., Rohling, R., Gin, K., Tsang, T., Abolmaesumi, P.: On modelling label uncertainty in deep neural networks: automatic estimation of intra-observer variability in 2d echocardiography quality assessment. IEEE Trans. Med. Imaging 39(6), 1868\u20131883 (2019)","journal-title":"IEEE Trans. Med. Imaging"},{"key":"399_CR9","unstructured":"Reis, E.\u00a0P., Nascimento, F., Aranha, M., Secol, F.\u00a0M., Machado, B., Felix, M., Stein, A., Amaro, E.: Brain hemorrhage extended (bhx): Bounding box extrapolation from thick to thin slice ct images. https:\/\/physionet.org\/content\/bhx-brain-bounding-box\/1.1\/. Accessed: 2023-01-13 (2020)"},{"key":"399_CR10","doi-asserted-by":"publisher","first-page":"1604","DOI":"10.1109\/TVCG.2014.2346481","volume":"20","author":"D Sacha","year":"2014","unstructured":"Sacha, D., Stoffel, A., Stoffel, F., Kwon, B.C., Ellis, G.P., Keim, D.: Knowledge generation model for visual analytics. IEEE Trans. Visual Comput. Graph. 20, 1604\u20131613 (2014)","journal-title":"IEEE Trans. Visual Comput. Graph."},{"issue":"11","key":"399_CR11","doi-asserted-by":"publisher","first-page":"4793","DOI":"10.1109\/TNNLS.2020.3027314","volume":"32","author":"E Tjoa","year":"2020","unstructured":"Tjoa, E., Guan, C.: A survey on explainable artificial intelligence (xai): toward medical xai. IEEE Trans. Neural Netw. Learn. Syst. 32(11), 4793\u20134813 (2020)","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"399_CR12","doi-asserted-by":"crossref","unstructured":"Xie, Y., Chen, M., Kao, D., Gao, G., Chen, X.: CheXplain: enabling physicians to explore and understand data-driven, AI-enabled medical imaging analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1\u201313 (2020)","DOI":"10.1145\/3313831.3376807"}],"container-title":["AI and Ethics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43681-023-00399-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s43681-023-00399-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43681-023-00399-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,15]],"date-time":"2024-04-15T09:08:56Z","timestamp":1713172136000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s43681-023-00399-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,20]]},"references-count":12,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,2]]}},"alternative-id":["399"],"URL":"https:\/\/doi.org\/10.1007\/s43681-023-00399-x","relation":{},"ISSN":["2730-5953","2730-5961"],"issn-type":[{"value":"2730-5953","type":"print"},{"value":"2730-5961","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,20]]},"assertion":[{"value":"20 December 2023","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}