{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:38:10Z","timestamp":1776109090240,"version":"3.50.1"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:p>Traditional crowdsourcing has mostly been viewed as requester-worker interaction where requesters publish tasks to solicit input from human crowdworkers. While most of this research area is catered towards the interest of requesters, we view this workflow as a teacher-learner interaction scenario where one or more human-teachers solve Human Intelligence Tasks to train machine learners. In this work, we explore how teachable machine learners can impact their human-teachers, and whether they form a trustable relation that can be relied upon for task delegation in the context of crowdsourcing. Specifically, we focus our work on teachable agents that learn to classify news articles while also guiding the teaching process through conversational interventions. In a two-part study, where several crowd workers individually teach the agent, we investigate whether this learning by teaching approach benefits human-machine collaboration, and whether it leads to trustworthy AI agents that crowd workers would delegate tasks to. Results demonstrate the benefits of the learning by teaching approach, in terms of perceived usefulness for crowdworkers, and the dynamics of trust built through the teacher-learner interaction.<\/jats:p>","DOI":"10.1145\/3555223","type":"journal-article","created":{"date-parts":[[2022,11,11]],"date-time":"2022-11-11T22:59:06Z","timestamp":1668207546000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Teachable Conversational Agents for Crowdwork: Effects on Performance and Trust"],"prefix":"10.1145","volume":"6","author":[{"given":"Nalin","family":"Chhibber","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Joslin","family":"Goh","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Edith","family":"Law","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]}],"member":"320","published-online":{"date-parts":[[2022,11,11]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Ernest Adams. 2014. Fundamentals of game design. Pearson Education.  Ernest Adams. 2014. Fundamentals of game design. Pearson Education."},{"key":"e_1_2_2_2_1","volume-title":"Categorical data analysis","author":"Agresti Alan","unstructured":"Alan Agresti . 2003. Categorical data analysis . Vol. 482 . John Wiley & Sons . Alan Agresti. 2003. Categorical data analysis. Vol. 482. John Wiley & Sons."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-00982-2_1"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.72.5.593"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1080\/08839510590910200"},{"key":"e_1_2_2_7_1","volume-title":"Artificial intelligence in education","author":"Brophy Sean","unstructured":"Sean Brophy , Gautam Biswas , Thomas Katzlberger , John Bransford , and Daniel Schwartz . 1999. Teachable agents: Combining insights from learning theory and computer science . In Artificial intelligence in education , Vol. 50 . Citeseer , 21--28. Sean Brophy, Gautam Biswas, Thomas Katzlberger, John Bransford, and Daniel Schwartz. 1999. Teachable agents: Combining insights from learning theory and computer science. In Artificial intelligence in education, Vol. 50. Citeseer, 21--28."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAMD.2010.2051030"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/DEVLRN.2010.5578865"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2014.08.005"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30391-4_3"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/332051.332075"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1111\/poms.13259"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10956-009-9180-4"},{"key":"e_1_2_2_15_1","volume-title":"Virtual shopping agents. Journal of Research in Interactive Marketing","author":"Chattaraman Veena","year":"2014","unstructured":"Veena Chattaraman , Wi-Suk Kwon , Juan E Gilbert , and Yishuang Li. 2014. Virtual shopping agents. Journal of Research in Interactive Marketing ( 2014 ). Veena Chattaraman, Wi-Suk Kwon, Juan E Gilbert, and Yishuang Li. 2014. Virtual shopping agents. Journal of Research in Interactive Marketing (2014)."},{"key":"e_1_2_2_16_1","volume-title":"Towards the Learning, Perception, and Effectiveness of Teachable Conversational Agents. Master's thesis","author":"Chhibber Nalin","unstructured":"Nalin Chhibber . 2019. Towards the Learning, Perception, and Effectiveness of Teachable Conversational Agents. Master's thesis . University of Waterloo . Nalin Chhibber. 2019. Towards the Learning, Perception, and Effectiveness of Teachable Conversational Agents. Master's thesis. University of Waterloo."},{"key":"e_1_2_2_17_1","volume-title":"Using conversational agents to support learning by teaching. arXiv preprint arXiv:1909.13443","author":"Chhibber Nalin","year":"2019","unstructured":"Nalin Chhibber and Edith Law . 2019. Using conversational agents to support learning by teaching. arXiv preprint arXiv:1909.13443 ( 2019 ). Nalin Chhibber and Edith Law. 2019. Using conversational agents to support learning by teaching. arXiv preprint arXiv:1909.13443 (2019)."},{"key":"e_1_2_2_18_1","volume-title":"Towards Teachable Conversational Agents. arXiv preprint arXiv:2102.10387","author":"Chhibber Nalin","year":"2021","unstructured":"Nalin Chhibber and Edith Law . 2021. Towards Teachable Conversational Agents. arXiv preprint arXiv:2102.10387 ( 2021 ). Nalin Chhibber and Edith Law. 2021. Towards Teachable Conversational Agents. arXiv preprint arXiv:2102.10387 (2021)."},{"key":"e_1_2_2_19_1","volume-title":"Mei-Hung Chiu, and Christian LaVancher.","author":"Chi Michelene TH","year":"1994","unstructured":"Michelene TH Chi , Nicholas De Leeuw , Mei-Hung Chiu, and Christian LaVancher. 1994 . Eliciting self-explanations improves understanding. Cognitive science 18, 3 (1994), 439--477. Michelene TH Chi, Nicholas De Leeuw, Mei-Hung Chiu, and Christian LaVancher. 1994. Eliciting self-explanations improves understanding. Cognitive science 18, 3 (1994), 439--477."},{"key":"e_1_2_2_20_1","first-page":"29","article-title":"A multimodal approach to assessing user experiences with agent helpers","volume":"6","author":"Clark Leigh","year":"2016","unstructured":"Leigh Clark , Abdulmalik Ofemile , Svenja Adolphs , and Tom Rodden . 2016 . A multimodal approach to assessing user experiences with agent helpers . ACM Transactions on Interactive Intelligent Systems (TiiS) 6 , 4 (2016), 29 . Leigh Clark, Abdulmalik Ofemile, Svenja Adolphs, and Tom Rodden. 2016. A multimodal approach to assessing user experiences with agent helpers. ACM Transactions on Interactive Intelligent Systems (TiiS) 6, 4 (2016), 29.","journal-title":"ACM Transactions on Interactive Intelligent Systems (TiiS)"},{"key":"e_1_2_2_21_1","volume-title":"Educational outcomes of tutoring: A meta-analysis of findings. American educational research journal 19, 2","author":"Cohen Peter A","year":"1982","unstructured":"Peter A Cohen , James A Kulik , and Chen-Lin C Kulik . 1982. Educational outcomes of tutoring: A meta-analysis of findings. American educational research journal 19, 2 ( 1982 ), 237--248. Peter A Cohen, James A Kulik, and Chen-Lin C Kulik. 1982. Educational outcomes of tutoring: A meta-analysis of findings. American educational research journal 19, 2 (1982), 237--248."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3098279.3098539"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1027\/1866-5888\/a000075"},{"key":"e_1_2_2_24_1","doi-asserted-by":"crossref","unstructured":"Edward L Deci and Richard M Ryan. 2012. Self-determination theory. (2012).  Edward L Deci and Richard M Ryan. 2012. Self-determination theory. (2012).","DOI":"10.4135\/9781446249215.n21"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.intcom.2010.07.001"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604056"},{"key":"e_1_2_2_29_1","volume-title":"Meet EDGAR, a tutoring agent at MONSERRATE. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 61--66","author":"Fialho Pedro","year":"2013","unstructured":"Pedro Fialho , Lu\u00edsa Coheur , S\u00e9rgio Curto , Pedro Cl\u00e1udio , \u00c2ngela Costa , Alberto Abad , Hugo Meinedo , and Isabel Trancoso . 2013 . Meet EDGAR, a tutoring agent at MONSERRATE. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 61--66 . Pedro Fialho, Lu\u00edsa Coheur, S\u00e9rgio Curto, Pedro Cl\u00e1udio, \u00c2ngela Costa, Alberto Abad, Hugo Meinedo, and Isabel Trancoso. 2013. Meet EDGAR, a tutoring agent at MONSERRATE. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. 61--66."},{"key":"e_1_2_2_30_1","volume-title":"Design for Teaching and Learning in a Networked World","author":"Gadiraju Ujwal","unstructured":"Ujwal Gadiraju , Besnik Fetahu , and Ricardo Kawase . 2015. Training workers for improving performance in crowd-sourcing microtasks . In Design for Teaching and Learning in a Networked World . Springer , 100--114. Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase. 2015. Training workers for improving performance in crowd-sourcing microtasks. In Design for Teaching and Learning in a Networked World. Springer, 100--114."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-011-0194-3"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1006\/jcss.1995.1003"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2928269"},{"key":"e_1_2_2_34_1","unstructured":"Andrew Guillory and Jeff A Bilmes. 2011. Simultaneous learning and covering with adversarial noise. (2011).  Andrew Guillory and Jeff A Bilmes. 2011. Simultaneous learning and covering with adversarial noise. (2011)."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741102"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1177\/0018720814547570"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2696454.2696479"},{"key":"e_1_2_2_38_1","unstructured":"Matthias Jerusalem and Ralf Schwarzer. 1979. The general self-efficacy scale (GSE).[Updated 2006 Oct 7].  Matthias Jerusalem and Ralf Schwarzer. 1979. The general self-efficacy scale (GSE).[Updated 2006 Oct 7]."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327566IJCE0401_04"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3334480.3382783"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858144"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2007.07.002"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1519023"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2021.1876355"},{"key":"e_1_2_2_45_1","volume-title":"Talk to me: Exploring user interactions with the Amazon Alexa. Journal of Librarianship and Information Science","author":"Lopatovska Irene","year":"2018","unstructured":"Irene Lopatovska , Katrina Rink , Ian Knight , Kieran Raines , Kevin Cosenza , Harriet Williams , Perachya Sorsche , David Hirsch , Qi Li , and Adrianna Martinez . 2018. Talk to me: Exploring user interactions with the Amazon Alexa. Journal of Librarianship and Information Science ( 2018 ), 0961000618759414. Irene Lopatovska, Katrina Rink, Ian Knight, Kieran Raines, Kevin Cosenza, Harriet Williams, Perachya Sorsche, David Hirsch, Qi Li, and Adrianna Martinez. 2018. Talk to me: Exploring user interactions with the Amazon Alexa. Journal of Librarianship and Information Science (2018), 0961000618759414."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITST.2017.7972192"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858288"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(88)90065-3"},{"key":"e_1_2_2_49_1","volume-title":"11th australasian conference on information systems","author":"Madsen Maria","unstructured":"Maria Madsen and Shirley Gregor . 2000. Measuring human-computer trust . In 11th australasian conference on information systems , Vol. 53 . Citeseer , 6--8. Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In 11th australasian conference on information systems, Vol. 53. Citeseer, 6--8."},{"key":"e_1_2_2_50_1","volume-title":"Human performance and ergonomics","author":"Massaro Dominic W","unstructured":"Dominic W Massaro , Michael M Cohen , Sharon Daniel , and Ronald A Cole . 1999. Developing and evaluating conversational agents . In Human performance and ergonomics . Elsevier , 173--194. Dominic W Massaro, Michael M Cohen, Sharon Daniel, and Ronald A Cole. 1999. Developing and evaluating conversational agents. In Human performance and ergonomics. Elsevier, 173--194."},{"key":"e_1_2_2_51_1","volume-title":"A model of interactive teaching. journal of computer and system sciences 54, 3","author":"Mathias H David","year":"1997","unstructured":"H David Mathias . 1997. A model of interactive teaching. journal of computer and system sciences 54, 3 ( 1997 ), 487--501. H David Mathias. 1997. A model of interactive teaching. journal of computer and system sciences 54, 3 (1997), 487--501."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479519"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3320435.3320439"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.2307\/256727"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.68"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1037\/edu0000071"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13421-014-0416-z"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357204"},{"key":"e_1_2_2_60_1","volume-title":"Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and instruction 1, 2","author":"Palinscar Aannemarie Sullivan","year":"1984","unstructured":"Aannemarie Sullivan Palinscar and Ann L Brown . 1984. Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and instruction 1, 2 ( 1984 ), 117--175. Aannemarie Sullivan Palinscar and Ann L Brown. 1984. Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and instruction 1, 2 (1984), 117--175."},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518709"},{"key":"e_1_2_2_62_1","volume-title":"International journal of electronic commerce 7, 3","author":"Pavlou Paul A","year":"2003","unstructured":"Paul A Pavlou . 2003. Consumer acceptance of electronic commerce: Integrating trust and risk with the technology acceptance model . International journal of electronic commerce 7, 3 ( 2003 ), 101--134. Paul A Pavlou. 2003. Consumer acceptance of electronic commerce: Integrating trust and risk with the technology acceptance model. International journal of electronic commerce 7, 3 (2003), 101--134."},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2013.74"},{"key":"e_1_2_2_64_1","volume-title":"Not so different after all: A cross-discipline view of trust","author":"Rousseau Denise M","year":"1998","unstructured":"Denise M Rousseau , Sim B Sitkin , Ronald S Burt , and Colin Camerer . 1998. Not so different after all: A cross-discipline view of trust . Academy of management review 23, 3 ( 1998 ), 393--404. Denise M Rousseau, Sim B Sitkin, Ronald S Burt, and Colin Camerer. 1998. Not so different after all: A cross-discipline view of trust. Academy of management review 23, 3 (1998), 393--404."},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.43.3.450"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2014.11.004"},{"key":"e_1_2_2_67_1","doi-asserted-by":"crossref","unstructured":"F David Schoorman Roger C Mayer and James H Davis. 2007. An integrative model of organizational trust: Past present and future.  F David Schoorman Roger C Mayer and James H Davis. 2007. An integrative model of organizational trust: Past present and future.","DOI":"10.5465\/amr.2007.24348410"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196772"},{"key":"e_1_2_2_69_1","doi-asserted-by":"crossref","unstructured":"John R Searle Ferenc Kiefer Manfred Bierwisch etal 1980. Speech act theory and pragmatics. Vol. 10. Springer.  John R Searle Ferenc Kiefer Manfred Bierwisch et al. 1980. Speech act theory and pragmatics. Vol. 10. Springer.","DOI":"10.1007\/978-94-009-8964-1"},{"key":"e_1_2_2_71_1","volume-title":"Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, et al .","author":"Simard Patrice Y","year":"2017","unstructured":"Patrice Y Simard , Saleema Amershi , David M Chickering , Alicia Edelman Pelton , Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, et al . 2017 . Machine teaching: A new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742 (2017). Patrice Y Simard, Saleema Amershi, David M Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, et al . 2017. Machine teaching: A new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742 (2017)."},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33197-8_25"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2016.03.007"},{"key":"e_1_2_2_74_1","volume-title":"Predicting learning from student interaction: Defining the interaction variables. Educational psychologist 18, 1","author":"Webb Noreen M","year":"1983","unstructured":"Noreen M Webb . 1983. Predicting learning from student interaction: Defining the interaction variables. Educational psychologist 18, 1 ( 1983 ), 33--41. Noreen M Webb. 1983. Predicting learning from student interaction: Defining the interaction variables. Educational psychologist 18, 1 (1983), 33--41."},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359126"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v11i1.14876"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v27i1.8461"},{"key":"e_1_2_2_78_1","unstructured":"Xiang Zhang Junbo Zhao and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems. 649--657.  Xiang Zhang Junbo Zhao and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems. 649--657."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555223","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555223","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:14Z","timestamp":1750182554000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555223"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":76,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2022,11,7]]}},"alternative-id":["10.1145\/3555223"],"URL":"https:\/\/doi.org\/10.1145\/3555223","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}