{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T05:56:37Z","timestamp":1782280597074,"version":"3.54.5"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2021,4,13]],"date-time":"2021-04-13T00:00:00Z","timestamp":1618272000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["#1822861"],"award-info":[{"award-number":["#1822861"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"crossref","award":["N00014-18-1-2755"],"award-info":[{"award-number":["N00014-18-1-2755"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,4,13]]},"abstract":"<jats:p>AI-based educational technologies may be most welcome in classrooms when they align with teachers' goals, preferences, and instructional practices. Teachers, however, have scarce time to make such customizations themselves. How might the crowd be leveraged to help time-strapped teachers? Crowdsourcing pipelines have traditionally focused on content generation. It is an open question how a pipeline might be designed so the crowd can succeed in a revision\/customization task. In this paper, we explore an initial version of a teacher-guided crowdsourcing pipeline designed to improve the adaptive math hints of an AI-based tutoring system so they fit teachers' preferences, while requiring minimal expert guidance. In two experiments involving 144 math teachers and 481 crowdworkers, we found that such an expert-guided revision pipeline could save experts' time and produce better crowd-revised hints (in terms of teacher satisfaction) than two comparison conditions. The revised hints however, did not improve on the existing hints in the AI tutor, which were carefully-written but still have room for improvement and customization. Further analysis revealed that the main challenge for crowdworkers may lie in understanding teachers' brief written comments and implementing them in the form of effective edits, without introducing new problems. We also found that teachers preferred their own revisions over other sources of hints, and exhibited varying preferences for hints. Overall, the results confirm that there is a clear need for customizing hints to individual teachers' preferences. They also highlight the need for more elaborate scaffolds so the crowd can have specific knowledge of the requirements that teachers have for hints. The study represents a first exploration in the literature of how to support crowds with minimal expert guidance in revising and customizing instructional materials.<\/jats:p>","DOI":"10.1145\/3449193","type":"journal-article","created":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T17:47:59Z","timestamp":1619113679000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Can Crowds Customize Instructional Materials with Minimal Expert Guidance?"],"prefix":"10.1145","volume":"5","author":[{"given":"Kexin Bella","family":"Yang","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tomohiro","family":"Nagashima","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junhui","family":"Yao","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph Jay","family":"Williams","sequence":"additional","affiliation":[{"name":"University of Toronto, Toronto, ON, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kenneth","family":"Holstein","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vincent","family":"Aleven","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,4,22]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2009.8"},{"key":"e_1_2_2_2_1","volume-title":"International handbook of metacognition and learning technologies","author":"Aleven Vincent","unstructured":"Vincent Aleven. 2013. Help seeking and intelligent tutoring systems: Theoretical perspectives and a step towards theoretical integration. In International handbook of metacognition and learning technologies. Springer, 311--335."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45108-0_33"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-015-0089-1"},{"key":"e_1_2_2_5_1","volume-title":"Rules of the mind Hillsdale. New Jersey","author":"Anderson JR","year":"1993","unstructured":"JR Anderson. 1993. Rules of the mind Hillsdale. New Jersey, Laurence Erlbaum Associates,, 319p (1993)."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39112-5_44"},{"key":"e_1_2_2_7_1","volume-title":"Creating usable innovations in systemic reform: Scaling up technology-embedded project-based science in urban schools. Educational psychologist","author":"Blumenfeld Phyllis","year":"2000","unstructured":"Phyllis Blumenfeld, Barry J Fishman, Joseph Krajcik, Ronald W Marx, and Elliot Soloway. 2000. Creating usable innovations in systemic reform: Scaling up technology-embedded project-based science in urban schools. Educational psychologist, Vol. 35, 3 (2000), 149--164."},{"key":"e_1_2_2_8_1","doi-asserted-by":"crossref","unstructured":"Daren C Brabham. 2013. Crowdsourcing. Mit Press.","DOI":"10.7551\/mitpress\/9693.001.0001"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820023"},{"key":"e_1_2_2_10_1","volume-title":"Fourth AAAI Conference on Human Computation and Crowdsourcing.","author":"Chen Yvonne","year":"2016","unstructured":"Yvonne Chen, Travis Mandel, Yun-En Liu, and Zoran Popovi\u0107. 2016. Crowdsourcing accurate and creative word problems and hints. In Fourth AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2591062.2591153"},{"key":"e_1_2_2_12_1","unstructured":"Ahmad Chettih David Gross-Amblard David Guyon Erwann Legeay and Zolt\u00e1n Mikl\u00f3s. 2014. Crowd a platform for the crowdsourcing of complex tasks."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208620"},{"key":"e_1_2_2_14_1","volume-title":"Reducing confusion about grounded theory and qualitative content analysis: Similarities and differences. The qualitative report","author":"Cho Ji Young","year":"2014","unstructured":"Ji Young Cho and Eun-Hee Lee. 2014. Reducing confusion about grounded theory and qualitative content analysis: Similarities and differences. The qualitative report, Vol. 19, 32 (2014), 1."},{"key":"e_1_2_2_15_1","volume-title":"Joseph Jay Williams, and Juho Kim","author":"Young Chung John Joon","year":"2018","unstructured":"John Joon Young Chung, Joseph Jay Williams, and Juho Kim. 2018. Collaborative Crowdsourcing Between Experts and Crowds for Chronological Ordering of Narrative Events. (2018), 621--626."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531670"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557217"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_2_19_1","volume-title":"Crowdsourcing and Education: Towards a Theory and Praxis of Learnersourcing","author":"Doroudi Shayan","unstructured":"Shayan Doroudi, Joseph Williams, Juho Kim, Thanaporn Patikorn, Korinn Ostrow, Douglas Selent, Neil T Heffernan, Thomas Hills, and Carolyn Ros\u00e9. 2018. Crowdsourcing and Education: Towards a Theory and Praxis of Learnersourcing. International Society of the Learning Sciences, Inc.[ISLS]."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470686"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13270"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39112-5_78"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820011"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2874319"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753357"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-016-0094-z"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12528-015-9089-2"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027385.3027451"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-93843-1_12"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2019.62.3"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-23204-7_14"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173596"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723731"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-2720"},{"key":"e_1_2_2_37_1","unstructured":"Yuchao Jiang Daniel Schlagwein and Boualem Benatallah. 2018. A Review on Crowdsourcing for Education: State of the Art of Literature and Practice.. In PACIS. 180."},{"key":"e_1_2_2_38_1","volume-title":"Do we need teachers as designers of technology enhanced learning? Instructional science","author":"Kirschner Paul A","year":"2015","unstructured":"Paul A Kirschner. 2015. Do we need teachers as designers of technology enhanced learning? Instructional science, Vol. 43, 2 (2015), 309--322."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.3102\/0034654315581420"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.72"},{"key":"e_1_2_2_44_1","volume-title":"Second AAAI Conference on Human Computation and Crowdsourcing.","author":"Kulkarni Anand","year":"2014","unstructured":"Anand Kulkarni, Prayag Narula, David Rolnitzky, and Nathan Kontny. 2014. Wish: Amplifying creative ability with expert crowds. In Second AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702565"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/VIS47514.2020.00045"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866040"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39112-5_23"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675283"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0037123"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10972-016-9459-2"},{"key":"e_1_2_2_52_1","unstructured":"Barbara Means William R Penuel and Christine Padilla. 2001. The connected school: Technology and learning in high school. ERIC."},{"key":"e_1_2_2_53_1","unstructured":"Tomohiro Nagashima Kexin Yang Anna Bartel Elena Silla Nicholas Vest Martha Alibali and Vincent Aleven. 2020. Pedagogical Affordance Analysis: Leveraging Teachers'? Pedagogical Knowledge to Elicit Pedagogical Affordances and Constraints of Instructional Tools. (2020)."},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07221-0_74"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10833-006-9006-6"},{"key":"e_1_2_2_56_1","volume-title":"Enhancing the quality and credibility of qualitative analysis. Health services research","author":"Patton Michael Quinn","year":"1999","unstructured":"Michael Quinn Patton. 1999. Enhancing the quality and credibility of qualitative analysis. Health services research, Vol. 34, 5 Pt 2 (1999), 1189."},{"key":"e_1_2_2_57_1","volume-title":"Designing formative assessment software with teachers: An analysis of the co-design process. Research and practice in technology enhanced learning","author":"Penuel William R","year":"2007","unstructured":"William R Penuel, Jeremy Roschelle, and Nicole Shechtman. 2007. Designing formative assessment software with teachers: An analysis of the co-design process. Research and practice in technology enhanced learning, Vol. 2, 01 (2007), 51--74."},{"key":"e_1_2_2_58_1","volume-title":"How teachers are using technology at home and in their classrooms","author":"Purcell Kristen","year":"2013","unstructured":"Kristen Purcell, Alan Heaps, Judy Buchanan, and Linda Friedrich. 2013. How teachers are using technology at home and in their classrooms. Washington, DC: Pew Research Center?-\u00f4s Internet & American Life Project (2013)."},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767866"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13388-6_39"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647409"},{"key":"e_1_2_2_62_1","volume-title":"Cognitive Tutor: Applied research in mathematics education. Psychonomic bulletin & review","author":"Ritter Steven","year":"2007","unstructured":"Steven Ritter, John R Anderson, Kenneth R Koedinger, and Albert Corbett. 2007. Cognitive Tutor: Applied research in mathematics education. Psychonomic bulletin & review, Vol. 14, 2 (2007), 249--255."},{"key":"e_1_2_2_63_1","volume-title":"Formative assessment and the design of instructional systems. Instructional science","author":"Sadler D Royce","year":"1989","unstructured":"D Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional science, Vol. 18, 2 (1989), 119--144."},{"key":"e_1_2_2_64_1","volume-title":"Focus on formative feedback. Review of educational research","author":"Shute Valerie J","year":"2008","unstructured":"Valerie J Shute. 2008. Focus on formative feedback. Review of educational research, Vol. 78, 1 (2008), 153--189."},{"key":"e_1_2_2_65_1","first-page":"2066","article-title":"A review of the uses of crowdsourcing in higher education","volume":"3","author":"Solemon Badariah","year":"2013","unstructured":"Badariah Solemon, Izyana Ariffin, Marina Md Din, Rina Md Anwar, et almbox. 2013. A review of the uses of crowdsourcing in higher education. International Journal of Asian Social Science, Vol. 3, 9 (2013), 2066--2073.","journal-title":"International Journal of Asian Social Science"},{"key":"e_1_2_2_66_1","volume-title":"Cidr","volume":"2013","author":"Stonebraker Michael","year":"2013","unstructured":"Michael Stonebraker, Daniel Bruckner, Ihab F Ilyas, George Beskales, Mitch Cherniack, Stanley B Zdonik, Alexander Pagan, and Shan Xu. 2013. Data curation at scale: the data tamer system.. In Cidr, Vol. 2013."},{"key":"e_1_2_2_67_1","unstructured":"James Surowiecki. 2005. The wisdom of crowds. Anchor."},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858121"},{"key":"e_1_2_2_69_1","volume-title":"Oncology nursing forum","author":"Triangulation Data Source","unstructured":"Data Source Triangulation. 2014. The use of triangulation in qualitative research. In Oncology nursing forum, Vol. 41. 545."},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2012.07.016"},{"key":"e_1_2_2_71_1","volume-title":"Carolyn Penstein Ros\u00e9, and Justin Reich","author":"Wang Xu","year":"2020","unstructured":"Xu Wang, Meredith Thompson, Kexin Yang, Dan Roy, Kenneth Koedinger, Carolyn Penstein Ros\u00e9, and Justin Reich. 2020. Practice-Based Teacher Education with ELK: A Role-Playing Simulation for Eliciting Learner Knowledge. (2020)."},{"key":"e_1_2_2_72_1","volume-title":"Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.","author":"Weld Daniel S","year":"2012","unstructured":"Daniel S Weld, Eytan Adar, Lydia Chilton, Raphael Hoffmann, Eric Horvitz, Mitchell Koch, James Landay, Christopher H Lin, and Mausam Mausam. 2012. Personalized online education?-- crowdsourcing challenge. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3051457.3053973"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2876034.2876042"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449193","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449193","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449193","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:17Z","timestamp":1750191497000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449193"}},"subtitle":["Exploring Teacher-guided Crowdsourcing for Improving Hints in an AI-based Tutor"],"short-title":[],"issued":{"date-parts":[[2021,4,13]]},"references-count":74,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2021,4,13]]}},"alternative-id":["10.1145\/3449193"],"URL":"https:\/\/doi.org\/10.1145\/3449193","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,13]]},"assertion":[{"value":"2021-04-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}