{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T23:40:01Z","timestamp":1755906001298,"version":"3.44.0"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"EICS","license":[{"start":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T00:00:00Z","timestamp":1718582400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"New Zealand Royal Society Marsden Grant","award":["M1218"],"award-info":[{"award-number":["M1218"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,6,17]]},"abstract":"<jats:p>The implicit-association test (IAT) is a method for measuring subconscious associations between concepts in memory. It is widely used in social psychology research for assessing associations that people may be unable or unwilling to articulate, including those relating to race, gender, self harm, and risk-taking behaviour. We describe the motivation for adapting the IAT to user interface evaluation, including its potential to support rapid A\/B testing that is amenable to online crowd-source dissemination, while also potentially reducing the validity risks caused by biases such as the good subject effect. We present a method (the UI-IAT) for conducting implicit association tests for A\/B user interface evaluation, and we present results of two experiments demonstrating that, although the method can successfully discriminate between 'good' and 'bad' interfaces, its sensitivity is low. We discuss implications for practical use of the UI-IAT and for further work.<\/jats:p>","DOI":"10.1145\/3664636","type":"journal-article","created":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T18:25:37Z","timestamp":1718648737000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["User Interface Evaluation Through Implicit-Association Tests"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1074-4268","authenticated-orcid":false,"given":"Andy","family":"Cockburn","sequence":"first","affiliation":[{"name":"University of Canterbury, Christchurch, Canterbury, New Zealand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-2641-6715","authenticated-orcid":false,"given":"Declan","family":"Hills","sequence":"additional","affiliation":[{"name":"University of Saskatchewan, Saskatoon, Saskatchewan, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3539-4049","authenticated-orcid":false,"given":"Zhe","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Canterbury, Christchurch, Canterbury, New Zealand"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4470-9495","authenticated-orcid":false,"given":"Carl","family":"Gutwin","sequence":"additional","affiliation":[{"name":"University of Saskatchewan, Saskatoon, Saskatchewan, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,6,17]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Tetlock","author":"Arkes Hal R.","year":"2004","unstructured":"Hal R. Arkes and Philip E. Tetlock. 2004. Attributions of Implicit Prejudice, or \"Would Jesse Jackson 'Fail' the Implicit Association Test?\". Psychological inquiry 15, 4 (2004), 257--278."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610661.3617513"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.275.5304.1293"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3451739"},{"key":"e_1_2_1_5_1","volume-title":"SUS: A 'Quick' and 'Dirty' Usability Scale. In Usability Evaluation in Industry, Patrick W","author":"Brooke John","year":"1996","unstructured":"John Brooke. 1996. SUS: A 'Quick' and 'Dirty' Usability Scale. In Usability Evaluation in Industry, Patrick W. Jordan, Bruce Thomas, Bernard A. Weerdmeester, and Ian Lyall McClelland (Eds.). Taylor and Francis, Chapter 21, 189--194."},{"volume-title":"Sketching user experiences: getting the design right and the right design","author":"Buxton William","key":"e_1_2_1_6_1","unstructured":"William Buxton. 2007;2010;. Sketching user experiences: getting the design right and the right design. Elsevier\/Morgan Kaufmann, Amsterdam;Boston;."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/57167.57203"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240723"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1177\/002224377200900122"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Nicola Dell Vidya Vaidyanathan Indrani Medhi Edward Cutrell and William Thies. 2012. \"Yours is better!\": participant response bias in HCI. ACM 1321--1330.","DOI":"10.1145\/2207676.2208589"},{"volume-title":"Sociological methods: a sourcebook. Aldine Pub","author":"Denzin Norman K.","key":"e_1_2_1_12_1","unstructured":"Norman K. Denzin. 1970. Sociological methods: a sourcebook. Aldine Pub. Co, Chicago."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0271748"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517570"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979046"},{"key":"e_1_2_1_16_1","volume-title":"Simon","author":"Ericsson K. A.","year":"1984","unstructured":"K. A. Ericsson and Herbert A. Simon. 1984. Protocol analysis: verbal reports as data. MIT Press, Cambridge, Mass."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP58684.2023.00033"},{"key":"e_1_2_1_18_1","volume-title":"Freeman and Nalini Ambady","author":"Jonathan","year":"2010","unstructured":"Jonathan B. Freeman and Nalini Ambady. 2010. MouseTracker: Software for studying real-time mental processing using a computer mouse-tracking method. Behavior research methods 42, 1 (2010), 226--241."},{"key":"e_1_2_1_19_1","volume-title":"Farmer","author":"Freeman Jonathan B.","year":"2011","unstructured":"Jonathan B. Freeman, Rick Dale, and Thomas A. Farmer. 2011. Hand in motion reveals mind in motion. Frontiers in psychology 2 (2011), 59--59."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/332040.332455"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445772"},{"key":"e_1_2_1_22_1","unstructured":"S. Greenberg S. Carpendale N. Marquardt and B. Buxton. 2011. Sketching User Experiences: The Workbook. Elsevier Science. https:\/\/books.google.co.nz\/books?id=4i6XLHE7wpkC"},{"key":"e_1_2_1_23_1","volume-title":"Banaji","author":"Greenwald Anthony G.","year":"1995","unstructured":"Anthony G. Greenwald and Mahzarin R. Banaji. 1995. Implicit Social Cognition: Attitudes, Self-Esteem, and Stereotypes. Psychological review 102, 1 (1995), 4--27."},{"key":"e_1_2_1_24_1","volume-title":"Wiers","author":"Greenwald Anthony G.","year":"2022","unstructured":"Anthony G. Greenwald, Miguel Brendl, Huajian Cai, Dario Cvencek, John F. Dovidio, Malte Friese, Adam Hahn, Eric Hehman, Wilhelm Hofmann, Sean Hughes, Ian Hussey, Christian Jordan, Teri A. Kirby, Calvin K. Lai, Jonas W. B. Lang, Kristen P. Lindgren, Dominika Maison, Brian D. Ostafin, James R. Rae, Kate A. Ratliff, Adriaan Spruyt, and Reinout W. Wiers. 2022. Best research practices for using the Implicit Association Test. Behavior research methods 54, 3 (2022), 1161--1180."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.74.6.1464"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0015575"},{"key":"e_1_2_1_27_1","volume-title":"Lie detection and the polygraph: A historical review. The journal of forensic psychiatry & psychology 16, 2","author":"Grubin Don","year":"2005","unstructured":"Don Grubin and Lars Madsen. 2005. Lie detection and the polygraph: A historical review. The journal of forensic psychiatry & psychology 16, 2 (2005), 357--369."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1294211.1294231"},{"key":"e_1_2_1_29_1","volume-title":"Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human mental workload 1, 3","author":"Hart Sandra G","year":"1988","unstructured":"Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human mental workload 1, 3 (1988), 139--183."},{"key":"e_1_2_1_30_1","volume-title":"Bohil","author":"Hill Audrey P.","year":"2016","unstructured":"Audrey P. Hill and Corey J. Bohil. 2016. Applications of Optical Neuroimaging in Usability Research. Ergonomics in design 24, 2 (2016), 4--9."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2005.06.002"},{"key":"e_1_2_1_32_1","volume-title":"The surprising power of online experiments: getting the most out of A\/B and other controlled tests. Harvard business review 95, 5","author":"Kohavi Ron","year":"2017","unstructured":"Ron Kohavi and Stefan Thomke. 2017. The surprising power of online experiments: getting the most out of A\/B and other controlled tests. Harvard business review 95, 5 (2017), 74."},{"key":"e_1_2_1_33_1","volume-title":"List","author":"Levitt Steven D.","year":"2011","unstructured":"Steven D. Levitt and John A. List. 2011. Was There Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments. American economic journal. Applied economics 3, 1 (2011), 224--238."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300787"},{"key":"e_1_2_1_35_1","volume-title":"Applying Implicit Association Test Techniques and Facial Expression Analyses in the Comparative Evaluation of Website User Experience. Frontiers in psychology 12","author":"Mauri Maurizio","year":"2021","unstructured":"Maurizio Mauri, Gaia Rancati, Andrea Gaggioli, and Giuseppe Riva. 2021. Applying Implicit Association Test Techniques and Facial Expression Analyses in the Comparative Evaluation of Website User Experience. Frontiers in psychology 12 (2021), 674159--674159."},{"key":"e_1_2_1_36_1","volume-title":"Molesworth and Betty Chang","author":"Brett R.","year":"2009","unstructured":"Brett R. C. Molesworth and Betty Chang. 2009. Predicting Pilots' Risk-Taking Behavior Through an Implicit Association Test. Human factors 51, 6 (2009), 845--857."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858064"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.3200\/GENP.135.2.151-166"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00027"},{"volume-title":"Evaluating the Thinking-Aloud Technique for Use by Computer Scientists","author":"Nielsen Jakob","key":"e_1_2_1_40_1","unstructured":"Jakob Nielsen. 1993. Evaluating the Thinking-Aloud Technique for Use by Computer Scientists. Ablex Publishing Corp., USA, 69--82."},{"key":"e_1_2_1_41_1","unstructured":"Jakob Nielsen. 2001. First Rule of Usability? Don't Listen to Users. https:\/\/www.nngroup.com\/articles\/first-rule-of-usability-dont-listen-to-users\/. Accessed: 2023-07-24."},{"key":"e_1_2_1_42_1","volume-title":"Banaji","author":"Nock Matthew K.","year":"2007","unstructured":"Matthew K. Nock and Mahzarin R. Banaji. 2007. Assessment of Self-Injurious Thoughts Using a Behavioral Test. The American journal of psychiatry 164, 5 (2007), 820--823."},{"key":"e_1_2_1_43_1","volume-title":"On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. The American psychologist 17, 11","author":"Orne Martin T.","year":"1962","unstructured":"Martin T. Orne. 1962. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. The American psychologist 17, 11 (1962), 776--783."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0032734"},{"key":"e_1_2_1_45_1","volume-title":"Oakes","author":"Ottaway Scott A.","year":"2001","unstructured":"Scott A. Ottaway, Davis C. Hayden, and Mark A. Oakes. 2001. Implicit attitudes and racism: Effects of word familiarity and frequency on the implicit association test. Social cognition 19, 2 (2001), 97--144."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2024.112011"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/175276.175288"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3410404.3414227"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194760.3194766"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"volume-title":"Applied Affective Computing (1 ed.)","author":"Tian Leimin","key":"e_1_2_1_51_1","unstructured":"Leimin Tian, Sharon Oviatt, Michal Muszynski, Brent C. Chamberlain, Jennifer Healey, and Akane Sano. 2022. Applied Affective Computing (1 ed.). Vol. 41. Association for Computing Machinery, New York, NY, USA."},{"key":"e_1_2_1_52_1","volume-title":"Strategic brand association maps: developing brand insight. The journal of product & brand management 20, 2","author":"Till Brian D.","year":"2011","unstructured":"Brian D. Till, Daniel Baack, and Brian Waterman. 2011. Strategic brand association maps: developing brand insight. The journal of product & brand management 20, 2 (2011), 92--100."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450741.3465257"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3358143"},{"volume-title":"Implicit Gender Biases in Professional Software Development: An Empirical Study","author":"Wang Yi","key":"e_1_2_1_55_1","unstructured":"Yi Wang and David Redmiles. 2019. Implicit Gender Biases in Professional Software Development: An Empirical Study. IEEE, 1--10."},{"key":"e_1_2_1_56_1","volume-title":"Beyond Reaction Times: Incorporating Mouse-Tracking Measures into the Implicit Association Test to Examine its Underlying Process. Social cognition 30, 3","author":"Yu Zonghuo","year":"2012","unstructured":"Zonghuo Yu, Fei Wang, Dengfeng Wang, and Mike Bastin. 2012. Beyond Reaction Times: Incorporating Mouse-Tracking Measures into the Implicit Association Test to Examine its Underlying Process. Social cognition 30, 3 (2012), 289--306."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3664636","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3664636","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T23:20:06Z","timestamp":1755904806000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3664636"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,17]]},"references-count":56,"journal-issue":{"issue":"EICS","published-print":{"date-parts":[[2024,6,17]]}},"alternative-id":["10.1145\/3664636"],"URL":"https:\/\/doi.org\/10.1145\/3664636","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2024,6,17]]},"assertion":[{"value":"2024-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}