{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:41:46Z","timestamp":1776112906815,"version":"3.50.1"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2020,5,28]],"date-time":"2020-05-28T00:00:00Z","timestamp":1590624000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,5,28]]},"abstract":"<jats:p>As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.<\/jats:p>","DOI":"10.1145\/3392878","type":"journal-article","created":{"date-parts":[[2020,5,29]],"date-time":"2020-05-29T16:01:06Z","timestamp":1590768066000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":177,"title":["Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs"],"prefix":"10.1145","volume":"4","author":[{"given":"Sungsoo Ray","family":"Hong","sequence":"first","affiliation":[{"name":"New York University, New York, NY, USA"}]},{"given":"Jessica","family":"Hullman","sequence":"additional","affiliation":[{"name":"Northwestern University, Evanston, IL, USA"}]},{"given":"Enrico","family":"Bertini","sequence":"additional","affiliation":[{"name":"New York University, New York, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,5,29]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.","author":"Attenberg Josh M","year":"2011","unstructured":"Josh M Attenberg , Pagagiotis G Ipeirotis , and Foster Provost . 2011 . Beat the machine: Challenging workers to find the unknown unknowns . In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. Josh M Attenberg, Pagagiotis G Ipeirotis, and Foster Provost. 2011. Beat the machine: Challenging workers to find the unknown unknowns. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_1_2_1","volume-title":"The Effect of Heterogeneous Data for Alzheimer's Disease Detection from Speech. arXiv preprint arXiv:1811.12254","author":"Balagopalan Aparna","year":"2018","unstructured":"Aparna Balagopalan , Jekaterina Novikova , Frank Rudzicz , and Marzyeh Ghassemi . 2018. The Effect of Heterogeneous Data for Alzheimer's Disease Detection from Speech. arXiv preprint arXiv:1811.12254 ( 2018 ). Aparna Balagopalan, Jekaterina Novikova, Frank Rudzicz, and Marzyeh Ghassemi. 2018. The Effect of Heterogeneous Data for Alzheimer's Disease Detection from Speech. arXiv preprint arXiv:1811.12254 (2018)."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012429"},{"key":"e_1_2_1_4_1","volume-title":"24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. 77--82","author":"Bibal Adrien","year":"2016","unstructured":"Adrien Bibal and Beno\u00eet Frenay . 2016 . Interpretability of Machine Learning Models and Representations: an Introduction . In 24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. 77--82 . Adrien Bibal and Beno\u00eet Frenay. 2016. Interpretability of Machine Learning Models and Representations: an Introduction. In 24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. 77--82."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-4560.1992.tb01946.x"},{"key":"e_1_2_1_6_1","volume-title":"Classification and Regression Trees","author":"Breiman Leo","unstructured":"Leo Breiman . 2017. Classification and Regression Trees . Routledge . Leo Breiman. 2017. Classification and Regression Trees. Routledge."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICHI.2015.26"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788613"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300460"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359164"},{"key":"e_1_2_1_11_1","unstructured":"Mark Craven and Jude W Shavlik. 1996. Extracting tree-structured representations of trained networks. In Advances in neural information processing systems. 24--30.  Mark Craven and Jude W Shavlik. 1996. Extracting tree-structured representations of trained networks. In Advances in neural information processing systems. 24--30."},{"key":"e_1_2_1_12_1","volume-title":"Qualitative inquiry and research design: Choosing among five approaches","author":"Creswell John W","unstructured":"John W Creswell and Cheryl N Poth . 2016. Qualitative inquiry and research design: Choosing among five approaches . Sage publications. John W Creswell and Cheryl N Poth. 2016. Qualitative inquiry and research design: Choosing among five approaches. Sage publications."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2865043"},{"key":"e_1_2_1_14_1","volume-title":"Towards a Rigorous Science of Interpretable Machine Learning. arXiv preprint arXiv:1702.08608","author":"Doshi-Velez Finale","year":"2017","unstructured":"Finale Doshi-Velez and Been Kim . 2017. Towards a Rigorous Science of Interpretable Machine Learning. arXiv preprint arXiv:1702.08608 ( 2017 ). https:\/\/arxiv.org\/abs\/1702.08608 Finale Doshi-Velez and Been Kim. 2017. Towards a Rigorous Science of Interpretable Machine Learning. arXiv preprint arXiv:1702.08608 (2017). https:\/\/arxiv.org\/abs\/1702.08608"},{"key":"e_1_2_1_15_1","volume-title":"Considerations for Evaluation and Generalization in Interpretable Machine Learning","author":"Doshi-Velez Finale","unstructured":"Finale Doshi-Velez and Been Kim . 2018. Considerations for Evaluation and Generalization in Interpretable Machine Learning . Springer International Publishing , 3--17. https:\/\/doi.org\/10.1007\/978--3--319--98131--4_1 10.1007\/978--3--319--98131--4_1 Finale Doshi-Velez and Been Kim. 2018. Considerations for Evaluation and Generalization in Interpretable Machine Learning. Springer International Publishing, 3--17. https:\/\/doi.org\/10.1007\/978--3--319--98131--4_1"},{"key":"e_1_2_1_16_1","volume-title":"Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639","author":"Esteva Andre","year":"2017","unstructured":"Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 ( 2017 ), 115. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 (2017), 115."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1099-1727(199824)14:4<309::AID-SDR154>3.0.CO;2-5"},{"key":"e_1_2_1_18_1","volume-title":"Greedy function approximation: a gradient boosting machine. Annals of statistics","author":"Friedman Jerome H","year":"2001","unstructured":"Jerome H Friedman . 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics ( 2001 ), 1189--1232. Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189--1232."},{"key":"e_1_2_1_19_1","volume-title":"Explaining Explanations: An Overview of Interpretability of Machine Learning. 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) (2018","author":"Gilpin Leilani H.","year":"2018","unstructured":"Leilani H. Gilpin , David Bau , Ben Z. Yuan , Ayesha Bajwa , Michael Specter , and Lalana Kagal . 2018 . Explaining Explanations: An Overview of Interpretability of Machine Learning. 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) (2018 ). https:\/\/doi.org\/10.1109\/dsaa.2018.00018 10.1109\/dsaa.2018.00018 Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining Explanations: An Overview of Interpretability of Machine Learning. 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) (2018). https:\/\/doi.org\/10.1109\/dsaa.2018.00018"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359152"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236009"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.3007062"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2648584.2648589"},{"key":"e_1_2_1_24_1","first-page":"1","article-title":"Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York","volume":"579","author":"Hohman Fred","year":"2019","unstructured":"Fred Hohman , Andrew Head , Rich Caruana , Robert DeLine , and Steven M. Drucker . 2019 . Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York , NY, USA , 579 : 1 -- 579 :13. https:\/\/doi.org\/10.1145\/3290605.3300809 10.1145\/3290605.3300809 Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M. Drucker. 2019. Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 579:1--579:13. https:\/\/doi.org\/10.1145\/3290605.3300809","journal-title":"NY, USA"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-93843-1_12"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19)","author":"Holstein Kenneth","unstructured":"Kenneth Holstein , Jennifer Wortman Vaughan , Hal Daum\u00e9 , III , Miro Dudik , and Hanna Wallach . [n. d.]. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? . In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19) . ACM, New York, NY, USA, 600:1--600:16. https:\/\/doi.org\/10.1145\/3290605.3300830 10.1145\/3290605.3300830 Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daum\u00e9, III, Miro Dudik, and Hanna Wallach. [n. d.]. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, 600:1--600:16. https:\/\/doi.org\/10.1145\/3290605.3300830"},{"key":"e_1_2_1_27_1","volume-title":"CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM.","author":"Hong Sungsoo Ray","year":"2019","unstructured":"Sungsoo Ray Hong , Jorge Piazentin Ono , Juliana Freire , and Enrico Bertini . 2019 . Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process . In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM. Sungsoo Ray Hong, Jorge Piazentin Ono, Juliana Freire, and Enrico Bertini. 2019. Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process. In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM."},{"key":"e_1_2_1_28_1","volume-title":"Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PloS one 10, 11","author":"Hullman Jessica","year":"2015","unstructured":"Jessica Hullman , Paul Resnick , and Eytan Adar . 2015. Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PloS one 10, 11 ( 2015 ). https:\/\/doi.org\/10.1371\/journal.pone.0142444 10.1371\/journal.pone.0142444 Jessica Hullman, Paul Resnick, and Eytan Adar. 2015. Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PloS one 10, 11 (2015). https:\/\/doi.org\/10.1371\/journal.pone.0142444"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744718"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC4.2009.4909197"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.219"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376219"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858558"},{"key":"e_1_2_1_34_1","unstructured":"Been Kim Rajiv Khanna and Oluwasanmi O Koyejo. 2016. Examples are not enough learn to criticize! criticism for interpretability. In Advances in Neural Information Processing Systems. 2280--2288.  Been Kim Rajiv Khanna and Oluwasanmi O Koyejo. 2016. Examples are not enough learn to criticize! criticism for interpretability. In Advances in Neural Information Processing Systems. 2280--2288."},{"key":"e_1_2_1_35_1","volume-title":"Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation","author":"Kim Yea-Seul","year":"2017","unstructured":"Yea-Seul Kim , Katharina Reinecke , and Jessica Hullman . 2017. Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation . IEEE transactions on visualization and computer graphics 24, 1 ( 2017 ), 760--769. Yea-Seul Kim, Katharina Reinecke, and Jessica Hullman. 2017. Data through others' eyes: The impact of visualizing others' expectations on visualization interpretation. IEEE transactions on visualization and computer graphics 24, 1 (2017), 760--769."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025592"},{"key":"e_1_2_1_37_1","first-page":"3","article-title":"Improving Human-machine Cooperative Visual Search with Soft Highlighting","volume":"15","author":"Kneusel Ronald T","year":"2017","unstructured":"Ronald T Kneusel and Michael C Mozer . 2017 . Improving Human-machine Cooperative Visual Search with Soft Highlighting . ACM Transactions on Applied Perception (TAP) 15 , 1 (2017), 3 . https:\/\/doi.org\/10.1145\/3129669 10.1145\/3129669 Ronald T Kneusel and Michael C Mozer. 2017. Improving Human-machine Cooperative Visual Search with Soft Highlighting. ACM Transactions on Applied Perception (TAP) 15, 1 (2017), 3. https:\/\/doi.org\/10.1145\/3129669","journal-title":"ACM Transactions on Applied Perception (TAP)"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858529"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701399"},{"key":"e_1_2_1_40_1","unstructured":"Isaac Lage Andrew Ross Samuel J Gershman Been Kim and Finale Doshi-Velez. 2018. Human-in-the-loop interpretability prior. In Advances in Neural Information Processing Systems. 10159--10168.  Isaac Lage Andrew Ross Samuel J Gershman Been Kim and Finale Doshi-Velez. 2018. Human-in-the-loop interpretability prior. In Advances in Neural Information Processing Systems. 10159--10168."},{"key":"e_1_2_1_41_1","volume-title":"Sociological practice: Linking theory and social research","author":"Layder Derek","unstructured":"Derek Layder . 1998. Sociological practice: Linking theory and social research . Sage . Derek Layder. 1998. Sociological practice: Linking theory and social research. Sage."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1214\/15-AOAS848"},{"key":"e_1_2_1_43_1","volume-title":"The Mythos of Model Interpretability. CoRR abs\/1606.03490","author":"Lipton Zachary Chase","year":"2016","unstructured":"Zachary Chase Lipton . 2016. The Mythos of Model Interpretability. CoRR abs\/1606.03490 ( 2016 ). http:\/\/arxiv.org\/abs\/1606.03490 Zachary Chase Lipton. 2016. The Mythos of Model Interpretability. CoRR abs\/1606.03490 (2016). http:\/\/arxiv.org\/abs\/1606.03490"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511628122"},{"key":"e_1_2_1_45_1","volume-title":"Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates","author":"Lundberg Scott M","unstructured":"Scott M Lundberg and Su-In Lee . 2017. A Unified Approach to Interpreting Model Predictions . In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates , Inc ., 4765--4774. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates, Inc., 4765--4774."},{"key":"e_1_2_1_46_1","volume-title":"Automatically Explaining Machine Learning Prediction Results: a Demonstration on Type 2 Diabetes Risk Prediction. Health information science and systems 4, 1","author":"Luo Gang","year":"2016","unstructured":"Gang Luo . 2016. Automatically Explaining Machine Learning Prediction Results: a Demonstration on Type 2 Diabetes Risk Prediction. Health information science and systems 4, 1 ( 2016 ), 2. https:\/\/doi.org\/10.1186\/s13755-016-0015--4 10.1186\/s13755-016-0015--4 Gang Luo. 2016. Automatically Explaining Machine Learning Prediction Results: a Demonstration on Type 2 Diabetes Risk Prediction. Health information science and systems 4, 1 (2016), 2. https:\/\/doi.org\/10.1186\/s13755-016-0015--4"},{"key":"e_1_2_1_47_1","volume-title":"BayesDB: A probabilistic programming system for querying the probable implications of data. arXiv preprint arXiv:1512.05006","author":"Mansinghka Vikash","year":"2015","unstructured":"Vikash Mansinghka , Richard Tibbetts , Jay Baxter , Pat Shafto , and Baxter Eaves . 2015. BayesDB: A probabilistic programming system for querying the probable implications of data. arXiv preprint arXiv:1512.05006 ( 2015 ). Vikash Mansinghka, Richard Tibbetts, Jay Baxter, Pat Shafto, and Baxter Eaves. 2015. BayesDB: A probabilistic programming system for querying the probable implications of data. arXiv preprint arXiv:1512.05006 (2015)."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.25300\/MISQ\/2014\/38.1.04"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488200"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1080\/01900692.2019.1575664"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.07.007"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864812"},{"key":"e_1_2_1_53_1","volume-title":"Machine Learning for Science: State of the Art and Future Prospects. Science 293, 5537","author":"Mjolsness Eric","year":"2001","unstructured":"Eric Mjolsness and Dennis DeCoste . 2001. Machine Learning for Science: State of the Art and Future Prospects. Science 293, 5537 ( 2001 ), 2051--2055. https:\/\/doi.org\/10.1126\/science.293.5537.2051 10.1126\/science.293.5537.2051 Eric Mjolsness and Dennis DeCoste. 2001. Machine Learning for Science: State of the Art and Future Prospects. Science 293, 5537 (2001), 2051--2055. https:\/\/doi.org\/10.1126\/science.293.5537.2051"},{"key":"e_1_2_1_54_1","volume-title":"Paisley","author":"Modarres Ceena","year":"2018","unstructured":"Ceena Modarres , Mark Ibrahim , Melissa Louie , and John W . Paisley . 2018 . Towards Explainable Deep Learning for Credit Lending: A Case Study. CoRR abs\/1811.06471 (2018). arXiv:1811.06471 http:\/\/arxiv.org\/abs\/1811.06471 Ceena Modarres, Mark Ibrahim, Melissa Louie, and John W. Paisley. 2018. Towards Explainable Deep Learning for Credit Lending: A Case Study. CoRR abs\/1811.06471 (2018). arXiv:1811.06471 http:\/\/arxiv.org\/abs\/1811.06471"},{"key":"e_1_2_1_55_1","unstructured":"Christoph Molnar. 2019. Interpretable Machine Learning. https:\/\/christophm.github.io\/interpretable-ml-book\/ Accessed: 2019-09--19.  Christoph Molnar. 2019. Interpretable Machine Learning. https:\/\/christophm.github.io\/interpretable-ml-book\/ Accessed: 2019-09--19."},{"key":"e_1_2_1_56_1","volume-title":"How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation. CoRR abs\/1802.00682","author":"Narayanan Menaka","year":"2018","unstructured":"Menaka Narayanan , Emily Chen , Jeffrey He , Been Kim , Sam Gershman , and Finale Doshi-Velez . 2018. How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation. CoRR abs\/1802.00682 ( 2018 ). http:\/\/arxiv.org\/abs\/1802.00682 Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2018. How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation. CoRR abs\/1802.00682 (2018). http:\/\/arxiv.org\/abs\/1802.00682"},{"key":"e_1_2_1_57_1","unstructured":"Luke Oakden-Rayner. 2017. Exploring the ChestXray14 dataset: problems. https:\/\/lukeoakdenrayner.wordpress.com\/2017\/12\/18\/the-chestxray14-dataset-problems\/. Accessed: 2019-09--19.  Luke Oakden-Rayner. 2017. Exploring the ChestXray14 dataset: problems. https:\/\/lukeoakdenrayner.wordpress.com\/2017\/12\/18\/the-chestxray14-dataset-problems\/. Accessed: 2019-09--19."},{"key":"e_1_2_1_58_1","volume-title":"Uncertain judgements: eliciting experts' probabilities","author":"O'Hagan Anthony","unstructured":"Anthony O'Hagan , Caitlin E Buck , Alireza Daneshkhah , J Richard Eiser , Paul H Garthwaite , David J Jenkinson , Jeremy E Oakley , and Tim Rakow . 2006. Uncertain judgements: eliciting experts' probabilities . John Wiley & Sons . Anthony O'Hagan, Caitlin E Buck, Alireza Daneshkhah, J Richard Eiser, Paul H Garthwaite, David J Jenkinson, Jeremy E Oakley, and Tim Rakow. 2006. Uncertain judgements: eliciting experts' probabilities. John Wiley & Sons."},{"key":"e_1_2_1_59_1","volume-title":"TCAV: Interpretability Beyond Feature Attribution. https:\/\/towardsdatascience.com\/tcav-interpretability-beyond-feature-attribution-79b4d3610b4d Accessed: 2019-09--19.","author":"Pandey Parul","year":"2019","unstructured":"Parul Pandey . 2019 . TCAV: Interpretability Beyond Feature Attribution. https:\/\/towardsdatascience.com\/tcav-interpretability-beyond-feature-attribution-79b4d3610b4d Accessed: 2019-09--19. Parul Pandey. 2019. TCAV: Interpretability Beyond Feature Attribution. https:\/\/towardsdatascience.com\/tcav-interpretability-beyond-feature-attribution-79b4d3610b4d Accessed: 2019-09--19."},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI","author":"Patel Kayur","year":"2011","unstructured":"Kayur Patel , Steven Drucker , James Fogarty , Ashish Kapoor , and Desney Tan . 2011 . Using Multiple Models to Understand Data . In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2011). AAAI Press, 1723--1728. Kayur Patel, Steven Drucker, James Fogarty, Ashish Kapoor, and Desney Tan. 2011. Using Multiple Models to Understand Data. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2011). AAAI Press, 1723--1728."},{"key":"e_1_2_1_61_1","volume-title":"Wallach","author":"Poursabzi-Sangdeh Forough","year":"2018","unstructured":"Forough Poursabzi-Sangdeh , Daniel G. Goldstein , Jake M. Hofman , JenniferWortman Vaughan , and Hanna M . Wallach . 2018 . Manipulating and Measuring Model Interpretability. CoRR abs\/1802.07810 (2018). arXiv:1802.07810 http:\/\/arxiv.org\/abs\/1802.07810 Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, JenniferWortman Vaughan, and Hanna M.Wallach. 2018. Manipulating and Measuring Model Interpretability. CoRR abs\/1802.07810 (2018). arXiv:1802.07810 http:\/\/arxiv.org\/abs\/1802.07810"},{"key":"e_1_2_1_62_1","volume-title":"Squares: Supporting interactive performance analysis for multiclass classifiers","author":"Ren Donghao","year":"2016","unstructured":"Donghao Ren , Saleema Amershi , Bongshin Lee , Jina Suh , and Jason D Williams . 2016 . Squares: Supporting interactive performance analysis for multiclass classifiers . IEEE transactions on visualization and computer graphics 23, 1 (2016), 61--70. https:\/\/doi.org\/10.1109\/TVCG.2016.2598828 10.1109\/TVCG.2016.2598828 Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, and Jason D Williams. 2016. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE transactions on visualization and computer graphics 23, 1 (2016), 61--70. https:\/\/doi.org\/10.1109\/TVCG.2016.2598828"},{"key":"e_1_2_1_63_1","volume-title":"Anchors: High-precision Model-agnostic Explanations. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI","author":"Ribeiro Marco","year":"2018","unstructured":"Marco Ribeiro , Tulio, Carlos Sameer Singh , and Carlos Guestrin . [n. d.]. Anchors: High-precision Model-agnostic Explanations. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2018 ). AAAI Press. Marco Ribeiro, Tulio, Carlos Sameer Singh, and Carlos Guestrin. [n. d.]. Anchors: High-precision Model-agnostic Explanations. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2018). AAAI Press."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_2_1_65_1","volume-title":"Detecting dependencies in sparse, multivariate databases using probabilistic programming and non-parametric Bayes. arXiv preprint arXiv:1611.01708","author":"Saad Feras","year":"2016","unstructured":"Feras Saad and Vikash Mansinghka . 2016. Detecting dependencies in sparse, multivariate databases using probabilistic programming and non-parametric Bayes. arXiv preprint arXiv:1611.01708 ( 2016 ). Feras Saad and Vikash Mansinghka. 2016. Detecting dependencies in sparse, multivariate databases using probabilistic programming and non-parametric Bayes. arXiv preprint arXiv:1611.01708 (2016)."},{"key":"e_1_2_1_66_1","volume-title":"The coding manual for qualitative researchers","author":"Salda\u00f1a Johnny","unstructured":"Johnny Salda\u00f1a . 2015. The coding manual for qualitative researchers . Sage . Johnny Salda\u00f1a. 2015. The coding manual for qualitative researchers. Sage."},{"key":"e_1_2_1_67_1","volume-title":"Quantifying Interpretability and Trust in Machine Learning Systems. arXiv preprint arXiv:1901.08558","author":"Schmidt Philipp","year":"2019","unstructured":"Philipp Schmidt and Felix Biessmann . 2019. Quantifying Interpretability and Trust in Machine Learning Systems. arXiv preprint arXiv:1901.08558 ( 2019 ). Philipp Schmidt and Felix Biessmann. 2019. Quantifying Interpretability and Trust in Machine Learning Systems. arXiv preprint arXiv:1901.08558 (2019)."},{"key":"e_1_2_1_68_1","volume-title":"Deep inside Convolutional Networks: Visualising Image Classification models and Saliency Maps. arXiv preprint arXiv:1312.6034","author":"Simonyan Karen","year":"2013","unstructured":"Karen Simonyan , Andrea Vedaldi , and Andrew Zisserman . 2013. Deep inside Convolutional Networks: Visualising Image Classification models and Saliency Maps. arXiv preprint arXiv:1312.6034 ( 2013 ). https:\/\/arxiv.org\/abs\/1312.6034v2 Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside Convolutional Networks: Visualising Image Classification models and Saliency Maps. arXiv preprint arXiv:1312.6034 (2013). https:\/\/arxiv.org\/abs\/1312.6034v2"},{"key":"e_1_2_1_69_1","volume-title":"Direct-Manipulation Visualization of Deep Networks. CoRR abs\/1708.03788","author":"Smilkov Daniel","year":"2017","unstructured":"Daniel Smilkov , Shan Carter , D. Sculley , Fernanda B. Vi\u00e9gas , and Martin Wattenberg . 2017. Direct-Manipulation Visualization of Deep Networks. CoRR abs\/1708.03788 ( 2017 ). http:\/\/arxiv.org\/abs\/1708.03788 Daniel Smilkov, Shan Carter, D. Sculley, Fernanda B. Vi\u00e9gas, and Martin Wattenberg. 2017. Direct-Manipulation Visualization of Deep Networks. CoRR abs\/1708.03788 (2017). http:\/\/arxiv.org\/abs\/1708.03788"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518895"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3176814"},{"key":"e_1_2_1_73_1","volume-title":"Visualizing Dataflow Graphs of Deep learning Models in Tensorflow","author":"Wongsuphasawat Kanit","year":"2017","unstructured":"Kanit Wongsuphasawat , Daniel Smilkov , James Wexler , Jimbo Wilson , Dandelion Mane , Doug Fritz , Dilip Krishnan , Fernanda B Vi\u00e9gas , and MartinWattenberg. 2017. Visualizing Dataflow Graphs of Deep learning Models in Tensorflow . IEEE transactions on visualization and computer graphics 24, 1 ( 2017 ), 1--12. https:\/\/doi.org\/10.1109\/TVCG.2017.2744878 10.1109\/TVCG.2017.2744878 Kanit Wongsuphasawat, Daniel Smilkov, James Wexler, Jimbo Wilson, Dandelion Mane, Doug Fritz, Dilip Krishnan, Fernanda B Vi\u00e9gas, and MartinWattenberg. 2017. Visualizing Dataflow Graphs of Deep learning Models in Tensorflow. IEEE transactions on visualization and computer graphics 24, 1 (2017), 1--12. https:\/\/doi.org\/10.1109\/TVCG.2017.2744878"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196729"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392826"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/tvcg.2018.2864499"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359158"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3392878","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3392878","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:16Z","timestamp":1750200076000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3392878"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,28]]},"references-count":77,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2020,5,28]]}},"alternative-id":["10.1145\/3392878"],"URL":"https:\/\/doi.org\/10.1145\/3392878","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,28]]},"assertion":[{"value":"2020-05-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}