{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:45:41Z","timestamp":1776109541165,"version":"3.50.1"},"reference-count":58,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2020,10,14]],"date-time":"2020-10-14T00:00:00Z","timestamp":1602633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Amazon Research Award"},{"name":"CMU Block Center for Technology and Society grant"},{"DOI":"10.13039\/100000001","name":"National Science Fundation","doi-asserted-by":"crossref","award":["DGE1745016, IIS-2000782, IIS-2001851, IIS-1939606"],"award-info":[{"award-number":["DGE1745016, IIS-2000782, IIS-2001851, IIS-1939606"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,10,14]]},"abstract":"<jats:p>Ensuring effective public understanding of algorithmic decisions that are powered by machine learning techniques has become an urgent task with the increasing deployment of AI systems into our society. In this work, we present a concrete step toward this goal by redesigning confusion matrices for binary classification to support non-experts in understanding the performance of machine learning models. Through interviews (n=7) and a survey (n=102), we mapped out two major sets of challenges lay people have in understanding standard confusion matrices: the general terminologies and the matrix design. We further identified three sub-challenges regarding the matrix design, namely, confusion about the direction of reading the data, layered relations and quantities involved. We then conducted an online experiment with 483 participants to evaluate how effective a series of alternative representations target each of those challenges in the context of an algorithm for making recidivism predictions. We developed three levels of questions to evaluate users' objective understanding. We assessed the effectiveness of our alternatives for accuracy in answering those questions, completion time, and subjective understanding. Our results suggest that (1) only by contextualizing terminologies can we significantly improve users' understanding and (2) flow charts, which help point out the direction of reading the data, were most useful in improving objective understanding. Our findings set the stage for developing more intuitive and generally understandable representations of the performance of machine learning models.<\/jats:p>","DOI":"10.1145\/3415224","type":"journal-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T22:27:49Z","timestamp":1602800869000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":41,"title":["Designing Alternative Representations of Confusion Matrices to Support Non-Expert Public Understanding of Algorithm Performance"],"prefix":"10.1145","volume":"4","author":[{"given":"Hong","family":"Shen","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Haojian","family":"Jin","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"\u00c1ngel Alexander","family":"Cabrera","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Adam","family":"Perer","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Haiyi","family":"Zhu","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Jason I.","family":"Hong","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,10,15]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174156"},{"key":"e_1_2_1_2_1","volume-title":"Introduction to machine learning","author":"Alpaydin Ethem"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264590"},{"key":"e_1_2_1_4_1","volume-title":"Machine bias. ProPublica (May","author":"Angwin Julia","year":"2016"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1037\/0278-7393.14.4.579"},{"key":"e_1_2_1_6_1","first-page":"671","article-title":"Big data?s disparate impact","volume":"104","author":"Barocas Solon","year":"2016","journal-title":"Calif. L. Rev."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3232078.3232096"},{"key":"e_1_2_1_8_1","unstructured":"Louis H Berry. 1991. The interaction of color realism and pictorial recall memory. ERIC.  Louis H Berry. 1991. The interaction of color realism and pictorial recall memory. ERIC."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173951"},{"key":"e_1_2_1_10_1","volume-title":"IJCAI-17 workshop on explainable AI (XAI)","volume":"8","author":"Biran Or","year":"2017"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300271"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300789"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0047"},{"key":"e_1_2_1_15_1","volume-title":"Amazon scraps secret AI recruiting tool that showed bias against women","author":"Dastin Jeffrey","year":"2018"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.42"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1515\/popets-2015-0007"},{"key":"e_1_2_1_18_1","unstructured":"Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile police and punish the poor. St. Martin?s Press.  Virginia Eubanks. 2018. Automating inequality: How high-tech tools profile police and punish the poor. St. Martin?s Press."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","volume-title":"The rise of big data policing: Surveillance, race, and the future of law enforcement","author":"Ferguson Andrew G","DOI":"10.2307\/j.ctt1pwtb27"},{"key":"e_1_2_1_20_1","volume-title":"Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education","author":"Friel Susan N","year":"2001"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6053.2008.00033.x"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359152"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 3, CSCW","author":"Nina","year":"2019"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 2018 World Wide Web Conference. 903--912","author":"Nina","year":"2018"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1179\/000870403235002042"},{"key":"e_1_2_1_27_1","volume-title":"International journal of data mining & knowledge management process 5, 2","author":"Hossin Mohammad","year":"2015"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2015.07.001"},{"key":"e_1_2_1_29_1","volume-title":"Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807","author":"Kleinberg Jon","year":"2016"},{"key":"e_1_2_1_30_1","volume-title":"Using visual analytics to interpret predictive machine learning models. arXiv preprint arXiv:1606.05685","author":"Krause Josua","year":"2016"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858529"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939874"},{"key":"e_1_2_1_33_1","volume-title":"How we analyzed the COMPAS recidivism algorithm. ProPublica (May","author":"Larson Jeff","year":"2016"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951718756684"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359284"},{"key":"e_1_2_1_36_1","unstructured":"William Lidwell Kritina Holden and Jill Butler. 2010. Universal principles of design revised and updated: 125 ways to enhance usability influence perception increase appeal make better design decisions and teach through design. Rockport Pub.  William Lidwell Kritina Holden and Jill Butler. 2010. Universal principles of design revised and updated: 125 ways to enhance usability influence perception increase appeal make better design decisions and teach through design. Rockport Pub."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0028085"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.199"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.07.007"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT*).","author":"Narayanan Arvind","year":"2018"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-005-4609-5"},{"key":"e_1_2_1_42_1","volume-title":"Weapons of math destruction: How big data increases inequality and threatens democracy","author":"O'Neil Cathy"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025727"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/3241189.3241263"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfh008"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598828"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314248"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518895"},{"key":"e_1_2_1_49_1","volume-title":"Designing interfaces: Patterns for effective interaction design","author":"Tidwell Jenifer"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359130"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1080\/00335557143000068"},{"key":"e_1_2_1_52_1","first-page":"56","article-title":"The What-If tool: Interactive probing of machine learning models","volume":"26","author":"Wexler James","year":"2019","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_2_1_53_1","volume-title":"Cognitive interviewing: A tool for improving questionnaire design","author":"Willis Gordon B"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174230"},{"key":"e_1_2_1_55_1","volume-title":"AI helps auto-loan company handle industry's trickiest turn. The Wall Street Journal (Jan","author":"Yerak Becky","year":"2019"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357236.3395528"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1631\/FITEE.1700808"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274463"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415224","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415224","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:11Z","timestamp":1750197791000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415224"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,14]]},"references-count":58,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2020,10,14]]}},"alternative-id":["10.1145\/3415224"],"URL":"https:\/\/doi.org\/10.1145\/3415224","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,14]]},"assertion":[{"value":"2020-10-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}