{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T23:17:17Z","timestamp":1776122237669,"version":"3.50.1"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2020,10,14]],"date-time":"2020-10-14T00:00:00Z","timestamp":1602633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,10,14]]},"abstract":"<jats:p>Human-AI interaction is pervasive across many areas of our day to day lives. In this paper, we investigate human-AI collaboration in the context of a collaborative AI-driven word association game with partially observable information. In our experiments, we test various dimensions of subjective social perceptions (rapport, intelligence, creativity and likeability) of participants towards their partners when participants believe they are playing with an AI or with a human. We also test subjective social perceptions of participants towards their partners when participants are presented with a variety of confidence levels. We ran a large scale study on Mechanical Turk (n=164) of this collaborative game. Our results show that when participants believe their partners were human, they found their partners to be more likeable, intelligent, creative and having more rapport and use more positive words to describe their partner's attributes than when they believed they were interacting with an AI partner. We also found no differences in game outcome including win rate and turns to completion. Drawing on both quantitative and qualitative findings, we discuss AI agent transparency, include design implications for tools incorporating or supporting human-AI collaboration, and lay out directions for future research. Our findings lead to implications for other forms of human-AI interaction and communication.<\/jats:p>","DOI":"10.1145\/3415167","type":"journal-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T22:28:12Z","timestamp":1602800892000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":75,"title":["Human-AI Collaboration in a Cooperative Game Setting"],"prefix":"10.1145","volume":"4","author":[{"given":"Zahra","family":"Ashktorab","sequence":"first","affiliation":[{"name":"IBM Research AI, Yorktown, NY, USA"}]},{"given":"Q. Vera","family":"Liao","sequence":"additional","affiliation":[{"name":"IBM Research AI, Yorktown, NY, USA"}]},{"given":"Casey","family":"Dugan","sequence":"additional","affiliation":[{"name":"IBM Research AI, Cambridge, MA, USA"}]},{"given":"James","family":"Johnson","sequence":"additional","affiliation":[{"name":"IBM Research AI, Cambridge, MA, USA"}]},{"given":"Qian","family":"Pan","sequence":"additional","affiliation":[{"name":"IBM Research AI, Cambridge, MA, USA"}]},{"given":"Wei","family":"Zhang","sequence":"additional","affiliation":[{"name":"IBM Research AI, Yorktown, NY, USA"}]},{"given":"Sadhana","family":"Kumaravel","sequence":"additional","affiliation":[{"name":"IBM Research AI, Yorktown, NY, USA"}]},{"given":"Murray","family":"Campbell","sequence":"additional","affiliation":[{"name":"IBM Research AI, Yorktown, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,10,15]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Paul Van Eecke, et al","author":"Adrian Kemo","year":"2016"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380254"},{"key":"e_1_2_1_3_1","volume-title":"Antecedents of team creativity: An examination of team emotional intelligence, team trust and collaborative culture. Creativity and innovation management 19, 4","author":"Barczak Gloria","year":"2010"},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Nolan Bard Jakob N Foerster Sarath Chandar Neil Burch Marc Lanctot H Francis Song Emilio Parisotto Vincent Dumoulin Subhodeep Moitra Edward Hughes etal 2019. The Hanabi Challenge: A New Frontier for AI Research. arXiv preprint arXiv:1902.00506 (2019).  Nolan Bard Jakob N Foerster Sarath Chandar Neil Burch Marc Lanctot H Francis Song Emilio Parisotto Vincent Dumoulin Subhodeep Moitra Edward Hughes et al. 2019. The Hanabi Challenge: A New Frontier for AI Research. arXiv preprint arXiv:1902.00506 (2019).","DOI":"10.1016\/j.artint.2019.103216"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-008-0001-3"},{"key":"e_1_2_1_6_1","volume-title":"A Bot and a Smile: Interpersonal Impressions of Chatbots and Humans Using Emoji in Computer-mediated Communication. Communication Studies","author":"Beattie Austin","year":"2020"},{"key":"e_1_2_1_7_1","volume-title":"Understanding information systems continuance: an expectation-confirmation model. MIS quarterly","author":"Bhattacherjee Anol","year":"2001"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 1265--1274. Proc. ACM Hum.-Comput. Interact.","volume":"4","year":"2009"},{"key":"e_1_2_1_9_1","volume-title":"Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science 359, 6374","author":"Brown Noam","year":"2018"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302289"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359206"},{"key":"e_1_2_1_12_1","unstructured":"California Governor. 2018. California new Autobot Law. https:\/\/leginfo.legislature.ca.gov\/faces\/billTextClient.xhtml?bill_id=201720180SB1001.  California Governor. 2018. California new Autobot Law. https:\/\/leginfo.legislature.ca.gov\/faces\/billTextClient.xhtml?bill_id=201720180SB1001."},{"key":"e_1_2_1_13_1","volume-title":"A Joseph Hoane Jr, and Feng-hsiung Hsu","author":"Campbell Murray","year":"2002"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020208"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300789"},{"key":"e_1_2_1_16_1","doi-asserted-by":"crossref","volume-title":"Artificial Psychology","author":"Crowder James A","DOI":"10.1007\/978-3-030-17081-3"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.321"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174136"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Nicole B Ellison and Danah M Boyd. 2013. Sociality through social network sites. In The Oxford handbook of internet studies.  Nicole B Ellison and Danah M Boyd. 2013. Sociality through social network sites. In The Oxford handbook of internet studies.","DOI":"10.1093\/oxfordhb\/9780199589074.013.0008"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bushor.2011.11.004"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1349822.1349842"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM.","author":"Gero Katy"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1038\/529445a"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TE.2005.856149"},{"key":"e_1_2_1_25_1","volume-title":"Perspectives on Human-Computer Interaction Research with Older People","author":"Gutierrez Francisco J"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025198"},{"key":"e_1_2_1_27_1","unstructured":"Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Advances in neural information processing systems. 2149--2159.  Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Advances in neural information processing systems. 2149--2159."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0113-5"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300469"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287590"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240695"},{"key":"e_1_2_1_32_1","volume-title":"Proc. ACM Hum.-Comput. Interact.","volume":"4","author":"Lanctot Marc","year":"2017"},{"key":"e_1_2_1_33_1","unstructured":"Adam Lerer Hengyuan Hu Jakob Foerster and Noam Brown. [n. d.]. Search in Cooperative Partially-Observable Games. ([n. d.]).  Adam Lerer Hengyuan Hu Jakob Foerster and Noam Brown. [n. d.]. Search in Cooperative Partially-Observable Games. ([n. d.])."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2015.04.005"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025206"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300325"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1080\/13528165.2019.1594267"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359297"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3311957.3359440"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's Mechanical Turk. Association for Computational Linguistics, 122--130","author":"Munro Robert","year":"2010"},{"key":"e_1_2_1_41_1","first-page":"396","article-title":"Relationship between color and emotion: A study of college students","volume":"38","author":"Az KAYA","year":"2004","journal-title":"College Student J"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03195588"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07230-2_45"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.2753\/MIS0742-1222280102"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174223"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1186\/s40327-019-0068-y"},{"key":"e_1_2_1_47_1","volume-title":"Demonstration papers at HLT-NAACL","author":"Pedersen Ted","year":"2004"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v39i1.2779"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCT.2019.8869102"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173965"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376843"},{"key":"e_1_2_1_52_1","volume-title":"Jingwen Zhang, Saurav Sahay, and Zhou Yu.","author":"Shi Weiyan","year":"2020"},{"key":"e_1_2_1_53_1","doi-asserted-by":"crossref","unstructured":"David Silver Thomas Hubert Julian Schrittwieser Ioannis Antonoglou Matthew Lai Arthur Guez Marc Lanctot Laurent Sifre Dharshan Kumaran Thore Graepel etal 2018. A general reinforcement learning algorithm that masters chess shogi and Go through self-play. Science 362 6419 (2018) 1140--1144.  David Silver Thomas Hubert Julian Schrittwieser Ioannis Antonoglou Matthew Lai Arthur Guez Marc Lanctot Laurent Sifre Dharshan Kumaran Thore Graepel et al. 2018. A general reinforcement learning algorithm that masters chess shogi and Go through self-play. Science 362 6419 (2018) 1140--1144.","DOI":"10.1126\/science.aar6404"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1108\/ER-03-2018-0093"},{"key":"e_1_2_1_55_1","volume-title":"Michelle Yeo, Alireza Makhzani, Heinrich K\u00fcttler, John Agapiou, Julian Schrittwieser, et al.","author":"Vinyals Oriol","year":"2017"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/191666.191708"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2006.05.002"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300831"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3312817"},{"key":"e_1_2_1_62_1","volume-title":"Philosophical investigations","author":"Wittgenstein Ludwig"},{"key":"e_1_2_1_63_1","unstructured":"Yang Xu and Charles Kemp. 2010. Inference and communication in the game of Password. In Advances in neural information processing systems. 2514--2522.  Yang Xu and Charles Kemp. 2010. Inference and communication in the game of Password. In Advances in neural information processing systems. 2514--2522."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240626"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372852"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415167","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415167","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:10Z","timestamp":1750197790000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415167"}},"subtitle":["Measuring Social Perception and Outcomes"],"short-title":[],"issued":{"date-parts":[[2020,10,14]]},"references-count":64,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2020,10,14]]}},"alternative-id":["10.1145\/3415167"],"URL":"https:\/\/doi.org\/10.1145\/3415167","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,14]]},"assertion":[{"value":"2020-10-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}