{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T17:05:12Z","timestamp":1775581512540,"version":"3.50.1"},"reference-count":70,"publisher":"SAGE Publications","issue":"3","license":[{"start":{"date-parts":[[2020,3,19]],"date-time":"2020-03-19T00:00:00Z","timestamp":1584576000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Information Visualization"],"published-print":{"date-parts":[[2020,7]]},"abstract":"<jats:p> Research in machine learning has become very popular in recent years, with many types of models proposed to comprehend and predict patterns and trends in data originating from different domains. As these models get more and more complex, it also becomes harder for users to assess and trust their results, since their internal operations are mostly hidden in black boxes. The interpretation of machine learning models is currently a hot topic in the information visualization community, with results showing that insights from machine learning models can lead to better predictions and improve the trustworthiness of the results. Due to this, multiple (and extensive) survey articles have been published recently trying to summarize the high number of original research papers published on the topic. But there is not always a clear definition of what these surveys cover, what is the overlap between them, which types of machine learning models they deal with, or what exactly is the scenario that the readers will find in each of them. In this article, we present a meta-analysis (i.e. a \u201csurvey of surveys\u201d) of manually collected survey papers that refer to the visual interpretation of machine learning models, including the papers discussed in the selected surveys. The aim of our article is to serve both as a detailed summary and as a guide through this survey ecosystem by acquiring, cataloging, and presenting fundamental knowledge of the state of the art and research opportunities in the area. Our results confirm the increasing trend of interpreting machine learning with visualizations in the past years, and that visualization can assist in, for example, online training processes of deep learning models and enhancing trust into machine learning. However, the question of exactly how this assistance should take place is still considered as an open challenge of the visualization community. <\/jats:p>","DOI":"10.1177\/1473871620904671","type":"journal-article","created":{"date-parts":[[2020,3,19]],"date-time":"2020-03-19T13:37:41Z","timestamp":1584625061000},"page":"207-233","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":122,"title":["A survey of surveys on the use of visualization for interpreting machine learning models"],"prefix":"10.1177","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9079-2376","authenticated-orcid":false,"given":"Angelos","family":"Chatzimparmpas","sequence":"first","affiliation":[{"name":"Department of Computer Science and Media Technology, Linnaeus University, V\u00e4xj\u00f6, Sweden"}]},{"given":"Rafael M.","family":"Martins","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Media Technology, Linnaeus University, V\u00e4xj\u00f6, Sweden"}]},{"given":"Ilir","family":"Jusufi","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Media Technology, Linnaeus University, V\u00e4xj\u00f6, Sweden"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0519-2537","authenticated-orcid":false,"given":"Andreas","family":"Kerren","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Media Technology, Linnaeus University, V\u00e4xj\u00f6, Sweden"}]}],"member":"179","published-online":{"date-parts":[[2020,3,19]]},"reference":[{"key":"bibr1-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"},{"key":"bibr2-1473871620904671","first-page":"1","volume-title":"Proceedings of the 2015 IEEE conference on computer vision and pattern recognition (CVPR)","author":"Szegedy C"},{"key":"bibr3-1473871620904671","first-page":"1135","volume-title":"Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (KDD \u201916), 2016","author":"Ribeiro MT"},{"key":"bibr4-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2013.120"},{"key":"bibr5-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598831"},{"key":"bibr6-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346578"},{"key":"bibr7-1473871620904671","first-page":"818","volume-title":"Proceedings of the computer vision\u2014ECCV 2014","author":"Zeiler MD"},{"key":"bibr8-1473871620904671","first-page":"1026","volume-title":"Proceedings of the 2015 IEEE international conference on computer vision (ICCV) (ICCV \u201915)","author":"He K"},{"key":"bibr9-1473871620904671","first-page":"3","volume-title":"Proceedings of the computer vision\u2014ECCV 2016","author":"Hendricks LA"},{"key":"bibr10-1473871620904671","first-page":"1701","volume-title":"Proceedings of the 2014 IEEE conference on computer vision and pattern recognition","author":"Taigman Y"},{"key":"bibr11-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.01.105"},{"key":"bibr12-1473871620904671","first-page":"3558","volume-title":"Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems (CHI EA \u201916)","author":"Gillies M"},{"key":"bibr13-1473871620904671","first-page":"80","volume-title":"Proceedings of the 2018 IEEE 5th international conference on data science and advanced analytics (DSAA)","author":"Gilpin LH"},{"key":"bibr14-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13212"},{"key":"bibr15-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1145\/2817552"},{"key":"bibr16-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/MDAT.2017.2709310"},{"key":"bibr17-1473871620904671","first-page":"143","volume-title":"Proceedings of the computer graphics and visual computing (CGVC)","author":"Alharbi M"},{"key":"bibr18-1473871620904671","first-page":"133","volume-title":"Proceedings of the EuroVis 2017\u2014short papers","author":"Alharbi N"},{"key":"bibr19-1473871620904671","first-page":"1","volume-title":"Proceedings of the 18th international conference on evaluation and assessment in software engineering (EASE \u201914)","author":"Wohlin C"},{"key":"bibr20-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2843369"},{"key":"bibr21-1473871620904671","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.2.24831.10403"},{"key":"bibr22-1473871620904671","first-page":"1","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI \u201918)","author":"Abdul A"},{"key":"bibr23-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2870052"},{"key":"bibr24-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1145\/3236009"},{"key":"bibr25-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864838"},{"key":"bibr26-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1007\/s11390-016-1663-1"},{"key":"bibr27-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13092"},{"key":"bibr28-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2640960"},{"key":"bibr29-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"bibr30-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2018.042731661"},{"issue":"2","key":"bibr31-1473871620904671","first-page":"81","volume":"8","author":"Dudley JJ","year":"2018","journal-title":"ACM T Interact Intell Syst"},{"key":"bibr32-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2018.09.018"},{"key":"bibr33-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1016\/j.visinf.2017.01.006"},{"key":"bibr34-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-016-6028-y"},{"key":"bibr35-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13210"},{"key":"bibr36-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598495"},{"key":"bibr37-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-54024-5_6"},{"key":"bibr38-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1016\/j.visinf.2018.09.001"},{"key":"bibr39-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1631\/FITEE.1700808"},{"key":"bibr40-1473871620904671","volume-title":"Proceedings of the ICML visualization for deep learning workshop","author":"Gr\u00fcn F"},{"key":"bibr41-1473871620904671","volume-title":"Proceedings of the 24th European symposium on artificial neural networks, computational intelligence and machine learning","author":"Sacha D"},{"issue":"1","key":"bibr42-1473871620904671","first-page":"39","volume":"1","author":"Samek W","year":"2018","journal-title":"ICT Discov"},{"key":"bibr43-1473871620904671","first-page":"248","volume-title":"Proceedings of the 2009 IEEE conference on computer vision and pattern recognition","author":"Deng J"},{"key":"bibr44-1473871620904671","first-page":"689","volume-title":"Proceedings of the 23rd ACM international conference on multimedia","author":"Vedaldi A"},{"key":"bibr45-1473871620904671","unstructured":"Baehrens D, Schroeter T, Harmeling S, et al. How to explain individual classification decisions. J Mach Learn Res 2010; 11: 1803\u20131831. http:\/\/dl.acm.org\/citation.cfm?id=1756006.1859912"},{"key":"bibr46-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0130140"},{"key":"bibr47-1473871620904671","first-page":"97","volume-title":"Proceedings of the 11th international symposium on visual information communication and interaction (VINCI \u201918)","author":"Kucher K"},{"key":"bibr48-1473871620904671","first-page":"993","volume":"3","author":"Blei DM","year":"2003","journal-title":"J Mach Learn Res"},{"key":"bibr49-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0307752101"},{"key":"bibr50-1473871620904671","doi-asserted-by":"publisher","DOI":"10.3115\/1225403.1225421"},{"key":"bibr51-1473871620904671","first-page":"361","volume-title":"Proceedings of the international AAAI conference on weblogs and social media, 2009","author":"Bastian M"},{"key":"bibr52-1473871620904671","first-page":"2579","volume":"9","author":"van der Maaten L","year":"2008","journal-title":"J Mach Learn Res"},{"key":"bibr53-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1002\/spe.4380211102"},{"key":"bibr54-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.212"},{"key":"bibr55-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2015.2467554"},{"key":"bibr56-1473871620904671","first-page":"183","volume-title":"Proceedings of the 2014 IEEE conference on visual analytics science and technology (VAST)","author":"Liu S"},{"key":"bibr57-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2570755"},{"key":"bibr58-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2515592"},{"key":"bibr59-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346482"},{"key":"bibr60-1473871620904671","unstructured":"Poulin B, Eisner R, Szafron D, et al. Visual explanation of evidence in additive classifiers. In: Proceedings of the 18th conference on innovative applications of artificial intelligence\u2014volume 2 (IAAI \u201906), 2006, pp. 1822\u20131829. AAAI Press. http:\/\/dl.acm.org\/citation.cfm?id=1597122.1597143"},{"key":"bibr61-1473871620904671","unstructured":"Strumbelj E, Kononenko I. An efficient explanation of individual classifications using game theory. J Mach Learn Res 2010; 11: 1\u201318. http:\/\/dl.acm.org\/citation.cfm?id=1756006.1756007"},{"key":"bibr62-1473871620904671","first-page":"13","volume-title":"Proceedings of the 2017 IEEE conference on visual analytics science and technology (VAST)","author":"Ming Y"},{"key":"bibr63-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.125"},{"key":"bibr64-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598828"},{"key":"bibr65-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346321"},{"key":"bibr66-1473871620904671","first-page":"770","volume-title":"Proceedings of the 2016 IEEE conference on computer vision and pattern recognition (CVPR)","author":"He K"},{"key":"bibr67-1473871620904671","first-page":"2261","volume-title":"Proceedings of the 2017 IEEE conference on computer vision and pattern recognition (CVPR)","author":"Huang G"},{"key":"bibr68-1473871620904671","first-page":"51","volume-title":"Proceedings of the 2009 IEEE symposium on visual analytics science and technology","author":"Correa CD"},{"key":"bibr69-1473871620904671","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2010.183"},{"key":"bibr70-1473871620904671","first-page":"111","volume-title":"Proceedings of the 2011 IEEE conference on visual analytics science and technology (VAST)","author":"May T"}],"container-title":["Information Visualization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1473871620904671","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1473871620904671","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1473871620904671","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T23:19:34Z","timestamp":1740871174000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1473871620904671"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,19]]},"references-count":70,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,7]]}},"alternative-id":["10.1177\/1473871620904671"],"URL":"https:\/\/doi.org\/10.1177\/1473871620904671","relation":{},"ISSN":["1473-8716","1473-8724"],"issn-type":[{"value":"1473-8716","type":"print"},{"value":"1473-8724","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,19]]}}}