{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T04:59:07Z","timestamp":1755838747303,"version":"3.41.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Crowdsourcing has emerged as an effective method of scaling-up tasks previously reserved for a small set of experts. Accordingly, researchers in the large-scale online learning space have begun to employ crowdworkers to conduct research about large-scale, open online learning. We here report results from a crowdsourcing study (N=135) to evaluate the extent to which crowdworkers and MOOC learners behave comparably on lecture viewing and quiz tasks---the most utilized learning activities in MOOCs. This serves to (i) validate the assumption of previous research that crowdworkers are indeed reliable proxies of online learners and (ii) address the potential of employing crowdworkers as a means of online learning environment testing. Overall, we observe mixed results---in certain contexts (quiz performance and video watching behavior) crowdworkers appear to behave comparably to MOOC learners, and in other situations (interactions with in-video quizzes), their behaviors appear to be disparate. We conclude that future research should be cautious if employing crowdworkers to carry out learning tasks, as the two populations do not behave comparably on all learning-related activities.<\/jats:p>","DOI":"10.1145\/3274311","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Evaluating Crowdworkers as a Proxy for Online Learners in Video-Based Learning Contexts"],"prefix":"10.1145","volume":"2","author":[{"given":"Dan","family":"Davis","sequence":"first","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}]},{"given":"Claudia","family":"Hauff","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}]},{"given":"Geert-Jan","family":"Houben","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_2_1_1_1","DOI":"10.1016\/j.ipm.2012.01.004"},{"doi-asserted-by":"publisher","key":"e_1_2_1_2_1","DOI":"10.1145\/1480506.1480508"},{"doi-asserted-by":"publisher","key":"e_1_2_1_3_1","DOI":"10.1016\/j.compedu.2015.08.011"},{"doi-asserted-by":"publisher","key":"e_1_2_1_4_1","DOI":"10.1016\/j.compedu.2014.12.012"},{"doi-asserted-by":"crossref","unstructured":"John Bohannon. 2011. Social science for pennies. (2011).","key":"e_1_2_1_5_1","DOI":"10.1126\/science.334.6054.307"},{"doi-asserted-by":"publisher","key":"e_1_2_1_6_1","DOI":"10.1016\/j.jbi.2017.04.003"},{"doi-asserted-by":"publisher","key":"e_1_2_1_7_1","DOI":"10.1145\/2675133.2675251"},{"doi-asserted-by":"publisher","key":"e_1_2_1_8_1","DOI":"10.1145\/2998181.2998268"},{"key":"e_1_2_1_9_1","volume-title":"EDM '16","author":"Cutumisu Maria","year":"2016","unstructured":"Maria Cutumisu and Daniel L Schwartz. 2016. Choosing versus Receiving Feedback: The Impact of Feedback Valence on Learning in an Assessment Game. In EDM '16. 341--346."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 9th International Conference on Educational Data Mining. 54--61","author":"Davis Dan","year":"2016","unstructured":"Dan Davis, Guanliang Chen, Claudia Hauff, and Geert-Jan Houben. 2016. Gauging MOOC Learners' Adherence to the Designed Learning Path. In Proceedings of the 9th International Conference on Educational Data Mining. 54--61."},{"doi-asserted-by":"publisher","key":"e_1_2_1_11_1","DOI":"10.1016\/j.compedu.2018.05.019"},{"doi-asserted-by":"publisher","key":"e_1_2_1_12_1","DOI":"10.1145\/2858036.2858268"},{"doi-asserted-by":"publisher","key":"e_1_2_1_13_1","DOI":"10.1145\/3027385.3027402"},{"volume-title":"Design for Teaching and Learning in a Networked World","author":"Gadiraju Ujwal","unstructured":"Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase. 2015. Training workers for improving performance in crowdsourcing microtasks. In Design for Teaching and Learning in a Networked World. Springer, 100--114.","key":"e_1_2_1_14_1"},{"volume-title":"Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments","author":"Gadiraju Ujwal","unstructured":"Ujwal Gadiraju, Sebastian M\u00f6ller, Martin N\u00f6llenburg, Dietmar Saupe, Sebastian Egger-Lampl, Daniel Archambault, and Brian Fisher. 2017. Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd. In Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Springer, 6--26.","key":"e_1_2_1_15_1"},{"key":"e_1_2_1_16_1","first-page":"1","article-title":"Modeling Student Behavior With Two-Layer Hidden Markov Models","volume":"9","author":"Geigle Chase","year":"2017","unstructured":"Chase Geigle and ChengXiang Zhai. 2017. Modeling Student Behavior With Two-Layer Hidden Markov Models. Journal of Educational Data Mining, Vol. 9, 1 (2017), 1--24.","journal-title":"Journal of Educational Data Mining"},{"doi-asserted-by":"publisher","key":"e_1_2_1_17_1","DOI":"10.5555\/1866696.1866723"},{"doi-asserted-by":"publisher","key":"e_1_2_1_18_1","DOI":"10.1145\/2556325.2566239"},{"volume-title":"Experiences and best practices in and around MOOCs","author":"Halawa Sherif","unstructured":"Sherif Halawa, Daniel Greene, and John Mitchell. 2014. Dropout prediction in MOOCs using learner activity features. In Experiences and best practices in and around MOOCs, Vol. 7. 3--12.","key":"e_1_2_1_19_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_20_1","DOI":"10.1007\/BF01063612"},{"key":"e_1_2_1_21_1","volume-title":"Tommy Mullaney, Jim Waldo, and Isaac Chuang.","author":"Ho Andrew Dean","year":"2014","unstructured":"Andrew Dean Ho, Justin Reich, Sergiy O Nesterko, Daniel Thomas Seaton, Tommy Mullaney, Jim Waldo, and Isaac Chuang. 2014. HarvardX and MITx: The first year of open online courses, fall 2012-summer 2013. Technical Report. Harvard University and Massachusetts Institute of Technology."},{"key":"e_1_2_1_22_1","volume-title":"Crowdsourcing: How the power of the crowd is driving the future of business ,Random House.","author":"Howe Jeff","year":"2008","unstructured":"Jeff Howe. 2008. Crowdsourcing: How the power of the crowd is driving the future of business .Random House."},{"key":"e_1_2_1_23_1","volume-title":"SIGIR 2009 Workshop on the Future of IR Evaluation. 21--22","author":"Kazai Gabriella","year":"2009","unstructured":"Gabriella Kazai and Natasa Milic-Frayling. 2009. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In SIGIR 2009 Workshop on the Future of IR Evaluation. 21--22."},{"doi-asserted-by":"publisher","key":"e_1_2_1_24_1","DOI":"10.1145\/2556325.2566237"},{"doi-asserted-by":"publisher","key":"e_1_2_1_25_1","DOI":"10.1145\/3025453.3025592"},{"doi-asserted-by":"publisher","key":"e_1_2_1_26_1","DOI":"10.1145\/1357054.1357127"},{"doi-asserted-by":"publisher","key":"e_1_2_1_27_1","DOI":"10.1016\/j.compedu.2016.10.001"},{"doi-asserted-by":"publisher","key":"e_1_2_1_28_1","DOI":"10.1145\/2724660.2724681"},{"doi-asserted-by":"publisher","key":"e_1_2_1_29_1","DOI":"10.1145\/2876034.2876041"},{"doi-asserted-by":"publisher","key":"e_1_2_1_30_1","DOI":"10.1145\/2858036.2858101"},{"doi-asserted-by":"publisher","key":"e_1_2_1_31_1","DOI":"10.1017\/S0043887110000195"},{"doi-asserted-by":"publisher","key":"e_1_2_1_32_1","DOI":"10.1145\/3178876.3186034"},{"key":"e_1_2_1_33_1","volume-title":"EDM '16","author":"Maass Jaclyn K","year":"2016","unstructured":"Jaclyn K Maass and Philip I Pavlik Jr. 2016. Modeling the Influence of Format and Depth during Effortful Retrieval Practice. In EDM '16. 143--150."},{"doi-asserted-by":"publisher","key":"e_1_2_1_34_1","DOI":"10.1145\/2998181.2998319"},{"doi-asserted-by":"publisher","key":"e_1_2_1_35_1","DOI":"10.1145\/3027385.3027430"},{"doi-asserted-by":"publisher","key":"e_1_2_1_36_1","DOI":"10.1145\/3170358.3170376"},{"key":"e_1_2_1_37_1","volume-title":"Using Inverse Planning for Personalized Feedback. In EDM '16","author":"Rafferty Anna N","year":"2016","unstructured":"Anna N Rafferty, Rachel A Jansen, and Thomas L Griffiths. 2016. Using Inverse Planning for Personalized Feedback. In EDM '16. 472--477."},{"unstructured":"Daniel T Seaton Sergiy Nesterko Tommy Mullaney Justin Reich Andrew Ho and Isaac Chuang. 2014. Characterizing video use in the catalogue of MITx MOOCs. In European MOOC Stakeholders Summit Lausanne. 140--146.","key":"e_1_2_1_38_1"},{"doi-asserted-by":"crossref","unstructured":"Abdulhadi Shoufan. 2018. Estimating the cognitive value of YouTube's educational videos: A learning analytics approach. Computers in Human Behavior Vol. - - (2018) --.","key":"e_1_2_1_39_1","DOI":"10.1016\/j.chb.2018.03.036"},{"doi-asserted-by":"publisher","key":"e_1_2_1_40_1","DOI":"10.3115\/v1\/W14-4102"},{"doi-asserted-by":"publisher","key":"e_1_2_1_41_1","DOI":"10.1177\/1745691614528518"},{"key":"e_1_2_1_42_1","volume-title":"Coding psychological constructs in text using Mechanical Turk: A reliable, accurate, and efficient alternative. Frontiers in psychology","author":"Tosti-Kharas Jennifer","year":"2016","unstructured":"Jennifer Tosti-Kharas and Caryn Conley. 2016. Coding psychological constructs in text using Mechanical Turk: A reliable, accurate, and efficient alternative. Frontiers in psychology, Vol. 7 (2016), 741."},{"doi-asserted-by":"publisher","key":"e_1_2_1_43_1","DOI":"10.1016\/j.compedu.2016.03.004"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the iConference. 1--17","author":"Vakharia Donna","year":"2015","unstructured":"Donna Vakharia and Matthew Lease. 2015. Beyond Mechanical Turk: An analysis of paid crowd work platforms. In Proceedings of the iConference. 1--17."},{"doi-asserted-by":"publisher","key":"e_1_2_1_45_1","DOI":"10.1145\/2876034.2876051"},{"doi-asserted-by":"publisher","key":"e_1_2_1_46_1","DOI":"10.1027\/1864-1105\/a000208"},{"doi-asserted-by":"publisher","key":"e_1_2_1_47_1","DOI":"10.1145\/2661829.2662033"},{"doi-asserted-by":"publisher","key":"e_1_2_1_49_1","DOI":"10.1145\/2876034.2876042"},{"doi-asserted-by":"publisher","key":"e_1_2_1_50_1","DOI":"10.1145\/2858036.2858361"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274311","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274311","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:06Z","timestamp":1750207446000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274311"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":49,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274311"],"URL":"https:\/\/doi.org\/10.1145\/3274311","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}