{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:40:08Z","timestamp":1750185608032,"version":"3.41.0"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T00:00:00Z","timestamp":1646352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF and USDA-NIFA under the AI Institute\u2019s Agricultural AI for Transforming Workforce and Decision Support","award":["#2021-67021-3534"],"award-info":[{"award-number":["#2021-67021-3534"]}]},{"DOI":"10.13039\/100000185","name":"DARPA","doi-asserted-by":"crossref","award":["#N66001-17-2-4030"],"award-info":[{"award-number":["#N66001-17-2-4030"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2022,3,31]]},"abstract":"<jats:p>\n            Would you allow an AI agent to make decisions on your behalf? If the answer is \u201cnot always,\u201d the next question becomes \u201cin what circumstances\u201d? Answering this question requires human users to be able to assess an AI agent\u2014and not just with overall pass\/fail assessments or statistics. Here users need to be able to\n            <jats:italic>localize<\/jats:italic>\n            an agent\u2019s bugs so that they can determine when they are willing to rely on the agent and when they are not. After-Action Review for AI (AAR\/AI), a new AI assessment process for integration with Explainable AI systems, aims to support human users in this endeavor, and in this article we empirically investigate AAR\/AI\u2019s effectiveness with domain-knowledgeable users. Our results show that AAR\/AI participants not only located significantly\n            <jats:italic>more<\/jats:italic>\n            bugs than non-AAR\/AI participants did (i.e., showed greater recall) but also located them more\n            <jats:italic>precisely<\/jats:italic>\n            (i.e., with greater precision). In fact, AAR\/AI participants outperformed non-AAR\/AI participants on every bug and were, on average, almost six times as likely as non-AAR\/AI participants to find any particular bug. Finally, evidence suggests that incorporating labeling into the AAR\/AI process may encourage domain-knowledgeable users to abstract above individual instances of bugs; we hypothesize that doing so may have contributed further to AAR\/AI participants\u2019 effectiveness.\n          <\/jats:p>","DOI":"10.1145\/3487065","type":"journal-article","created":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T10:14:01Z","timestamp":1646388841000},"page":"1-33","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Finding AI\u2019s Faults with AAR\/AI: An Empirical Study"],"prefix":"10.1145","volume":"12","author":[{"given":"Roli","family":"Khanna","sequence":"first","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonathan","family":"Dodge","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrew","family":"Anderson","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rupika","family":"Dikkala","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jed","family":"Irvine","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zeyad","family":"Shureih","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kin-Ho","family":"Lam","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Caleb R.","family":"Matthews","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhengxian","family":"Lin","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Minsuk","family":"Kahng","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alan","family":"Fern","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Margaret","family":"Burnett","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,3,4]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174156"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300233"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.5555\/3133461"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366485"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.5555\/3367032.3367221"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2004.2"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1002\/acp.3083"},{"key":"e_1_3_3_9_2","volume-title":"Taxonomy of Educational Objectives","author":"Bloom Benjamin S.","year":"1956","unstructured":"Benjamin S. Bloom, Max D. Engelhart, Edward J. Furst, Walker H. Hill, and David R. Krathwohl. 1956. Taxonomy of Educational Objectives. Longmans, Green & Co. Ltd."},{"key":"e_1_3_3_10_2","first-page":"119","volume-title":"Proceedings of the 2019 International Conference on Human Factors in Robots and Unmanned Systems (AHFE\u201919)","author":"Brewer Ralph","year":"2019","unstructured":"Ralph Brewer, Anthony Walker, E. Ray Pursel, Eduardo Cerame, Anthony Baker, and Kristin Schaefer. 2019. Assessment of manned-unmanned team performance: Comprehensive after-action review technology development. In Proceedings of the 2019 International Conference on Human Factors in Robots and Unmanned Systems (AHFE\u201919). 119\u2013130."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/876"},{"key":"e_1_3_3_12_2","first-page":"853","volume-title":"Proceedings of the ACM Conference on Human Factors in Computing Systems","author":"Castelli Nico","year":"2017","unstructured":"Nico Castelli, Corinna Ogonowski, Timo Jakobi, Martin Stein, Gunnar Stevens, and Volker Wulf. 2017. What happened in my home? An end-user development approach for smart home data visualization. In Proceedings of the ACM Conference on Human Factors in Computing Systems. ACM, New York, NY, 853\u2013866."},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172950"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog1302_1"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397481.3450644"},{"key":"e_1_3_3_16_2","volume-title":"Proceedings of IUI Workshops","author":"Chromik Michael","year":"2019","unstructured":"Michael Chromik, Malin Eiband, Sarah Theres V\u00f6lkel, and Daniel Buschek. 2019. Dark patterns of explainability, transparency, and user control for intelligent systems. In Proceedings of IUI Workshops."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203771587"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053114"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1017\/dmp.2018.82"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453173"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174136"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-018-9482-5"},{"key":"e_1_3_3_23_2","unstructured":"David Freedman Robert Pisani and Roger Purves. 2007. Statistics (4th international student ed.). W. W. Norton New York NY."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.59"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/SYSCON.2018.8369516"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1177\/1541931218621047"},{"key":"e_1_3_3_28_2","article-title":"Metrics for explainable AI: Challenges and prospects","volume":"1812","author":"Hoffman Robert R.","year":"2018","unstructured":"Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. CoRR abs\/1812.04608 (2018). arxiv:1812.04608http:\/\/arxiv.org\/abs\/1812.04608.","journal-title":"CoRR"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2843369"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1177\/1049732305276687"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1177\/1046496417697148"},{"key":"e_1_3_3_32_2","first-page":"223","article-title":"Nouvelles recherches sur la distribution florale","volume":"44","author":"Jaccard Paul","year":"1908","unstructured":"Paul Jaccard. 1908. Nouvelles recherches sur la distribution florale. Bulletin de la Societe Vaudoise des Sciences Naturelles 44, 163 (1908), 223\u2013270.","journal-title":"Bulletin de la Societe Vaudoise des Sciences Naturelles"},{"key":"e_1_3_3_33_2","first-page":"47","volume-title":"Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence","author":"Juozapaitis Zoe","year":"2019","unstructured":"Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. 2019. Explainable reinforcement learning via reward decomposition. In Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence. 47\u201353."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1037\/apl0000821"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2892305"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2892305"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9279-3"},{"key":"e_1_3_3_38_2","article-title":"Can AI be taught to explain itself?","author":"Kuang Cliff","year":"2017","unstructured":"Cliff Kuang. 2017. Can AI be taught to explain itself?New York Times. Retrieved December 26, 2017 from https:\/\/www.nytimes.com\/2017\/11\/21\/magazine\/can-ai-be-taught-to-explain-itself.html.","journal-title":"New York Times"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701399"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2010.15"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2013.6645235"},{"key":"e_1_3_3_42_2","volume-title":"Proceedings of the IJCAI-PRICAI 2020 Workshop on XAI","author":"Lam Kin-Ho","year":"2020","unstructured":"Kin-Ho Lam, Zhengxian Lin, Jed Irvine, Jonathan Dodge, Zeyad T. Shureih, Roli Khanna, Minsuk Kahng, and Alan Fern. 2020. Identifying reasoning flaws in planning-based RL using tree explanations. In Proceedings of the IJCAI-PRICAI 2020 Workshop on XAI. https:\/\/drive.google.com\/file\/d\/1ihT39-S2SFpCsarJfzDsnMOUT85YjCXK\/view?usp=sharing."},{"issue":"5","key":"e_1_3_3_43_2","first-page":"61","article-title":"The art of the after-action review","volume":"171","author":"Lareau Adam","year":"2018","unstructured":"Adam Lareau and Brice Long. 2018. The art of the after-action review. Fire Engineering 171, 5 (May 2018), 61\u201364. http:\/\/search.proquest.com\/docview\/2157468757\/.","journal-title":"Fire Engineering"},{"key":"e_1_3_3_44_2","volume-title":"Improving Understanding and Trust with Intelligibility in Context-Aware Applications","author":"Lim Brian Y.","year":"2012","unstructured":"Brian Y. Lim. 2012. Improving Understanding and Trust with Intelligibility in Context-Aware Applications. Ph.D. Dissertation. Carnegie Mellon University."},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/1620545.1620576"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0090-2616(99)90027-X"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377325.3377525"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2009.11.006"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2017.00021"},{"issue":"112","key":"e_1_3_3_50_2","first-page":"7","article-title":"Some observations on mental models","volume":"7","author":"Norman Donald A.","year":"1983","unstructured":"Donald A. Norman. 1983. Some observations on mental models. Mental Models 7, 112 (1983), 7\u201314.","journal-title":"Mental Models"},{"key":"e_1_3_3_51_2","unstructured":"European Group on Ethics in Science and New Technologies. 2018. Statement on artificial intelligence robotics and \u201cAutonomous\u201d Systems. Retrieved Novembewr 17 2021 from https:\/\/ec.europa.eu\/info\/news\/ethics-artificial-intelligence-statement-ege-released-2018-apr-24_en."},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2013.2286295"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3396047"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172946"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10055-013-0229-6"},{"key":"e_1_3_3_56_2","volume-title":"The Statistical Sleuth: A Course in Methods of Data Analysis","author":"Ramsey Fred","year":"2012","unstructured":"Fred Ramsey and Daniel Schafer. 2012. The Statistical Sleuth: A Course in Methods of Data Analysis. Cengage Learning."},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1006\/ceps.1997.0959"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X00075506"},{"key":"e_1_3_3_59_2","volume-title":"Artificial Intelligence: A Modern Approach","author":"Russell Stuart J.","year":"2016","unstructured":"Stuart J. Russell and Peter Norvig. 2016. Artificial Intelligence: A Modern Approach. Pearson Education Ltd., Malaysia."},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1097\/SIH.0b013e31829ac85c"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302308"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2865044"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1023\/a:1022193728205"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240630"},{"key":"e_1_3_3_65_2","volume-title":"Training Circular 25-20: A Leader\u2019s Guide to After-Action Reviews","author":"Army U.S.","year":"1993","unstructured":"U.S. Army. 1993. Training Circular 25-20: A Leader\u2019s Guide to After-Action Reviews. Technical Report. Department of the Army, Washington, DC."},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300831"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864504"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1073"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376301"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300468"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2013.39"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487065","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3487065","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3487065","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:10:09Z","timestamp":1750183809000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487065"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,4]]},"references-count":71,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,3,31]]}},"alternative-id":["10.1145\/3487065"],"URL":"https:\/\/doi.org\/10.1145\/3487065","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"type":"print","value":"2160-6455"},{"type":"electronic","value":"2160-6463"}],"subject":[],"published":{"date-parts":[[2022,3,4]]},"assertion":[{"value":"2021-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}