{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T03:46:07Z","timestamp":1774669567253,"version":"3.50.1"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2020,1,14]],"date-time":"2020-01-14T00:00:00Z","timestamp":1578960000000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"},{"start":{"date-parts":[[2020,1,14]],"date-time":"2020-01-14T00:00:00Z","timestamp":1578960000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Educ Inf Technol"],"published-print":{"date-parts":[[2020,7]]},"DOI":"10.1007\/s10639-020-10102-w","type":"journal-article","created":{"date-parts":[[2020,1,14]],"date-time":"2020-01-14T01:02:10Z","timestamp":1578963730000},"page":"2971-2986","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":63,"title":["Utilizing crowdsourcing and machine learning in education: Literature review"],"prefix":"10.1007","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3774-368X","authenticated-orcid":false,"given":"Hadeel S.","family":"Alenezi","sequence":"first","affiliation":[]},{"given":"Maha H.","family":"Faisal","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,1,14]]},"reference":[{"key":"10102_CR1","doi-asserted-by":"publisher","unstructured":"Alghamdi, E. A., Aljohani, N. R., Alsaleh, A. N., Bedewi, W., & Basheri, M. (2015). CrowdyQ: A virtual crowdsourcing platform for question items development in higher education. Proceedings of the 17th International Conference on Information Integration and Web-Based Applications &Services - IiWAS \u201815, 1\u20134. https:\/\/doi.org\/10.1145\/2837185.2843852.","DOI":"10.1145\/2837185.2843852"},{"key":"10102_CR2","doi-asserted-by":"publisher","unstructured":"Barbosa, C. E., Epelbaum, V. J., Antelio, M., Oliveira, J., & Moreira de Souza, J. (2013). Crowdsourcing environments in E-learning scenario: A classification based on educational and collaboration criteria. 2013 IEEE International Conference on Systems, Man, and Cybernetics, 687\u2013692. https:\/\/doi.org\/10.1109\/SMC.2013.122.","DOI":"10.1109\/SMC.2013.122"},{"key":"10102_CR3","doi-asserted-by":"publisher","unstructured":"Choi, J.-R., Kim, S., & Lim, S.-B. (2017). The feedback block model for an adaptive E-book. Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST \u201817, 127\u2013128. https:\/\/doi.org\/10.1145\/3131785.3131834.","DOI":"10.1145\/3131785.3131834"},{"issue":"6","key":"10102_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2810239","volume":"22","author":"H Corrigan-Gibbs","year":"2015","unstructured":"Corrigan-Gibbs, H., Gupta, N., Northcutt, C., Cutrell, E., & Thies, W. (2015). Deterring cheating in online environments. ACM Transactions on Computer-Human Interaction, 22(6), 1\u201323. https:\/\/doi.org\/10.1145\/2810239.","journal-title":"ACM Transactions on Computer-Human Interaction"},{"issue":"4","key":"10102_CR5","doi-asserted-by":"publisher","first-page":"438","DOI":"10.3102\/00346543058004438","volume":"58","author":"TJ Crooks","year":"1988","unstructured":"Crooks, T. J. (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58(4), 438\u2013481. https:\/\/doi.org\/10.3102\/00346543058004438.","journal-title":"Review of Educational Research"},{"key":"10102_CR6","doi-asserted-by":"publisher","unstructured":"de Alfaro, L., & Shavlovsky, M. (2014). CrowdGrader: A tool for crowdsourcing the evaluation of homework assignments. Proceedings of the 45th ACM Technical Symposium on Computer Science Education - SIGCSE \u201814, 415\u2013420. https:\/\/doi.org\/10.1145\/2538862.2538900.","DOI":"10.1145\/2538862.2538900"},{"key":"10102_CR7","doi-asserted-by":"publisher","unstructured":"Dow, S., Gerber, E., & Wong, A. (2013). A pilot study of using crowds in the classroom. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI \u201813, 227. https:\/\/doi.org\/10.1145\/2470654.2470686.","DOI":"10.1145\/2470654.2470686"},{"key":"10102_CR8","doi-asserted-by":"publisher","unstructured":"Faisal, M. H., AlAmeeri, A. W., & Alsumait, A. A. (2015). An adaptive e-learning framework: Crowdsourcing approach. Proceedings of the 17th International Conference on Information Integration and Web-Based Applications &Services - IiWAS \u201815, 1\u20135. https:\/\/doi.org\/10.1145\/2837185.2837249.","DOI":"10.1145\/2837185.2837249"},{"key":"10102_CR9","doi-asserted-by":"publisher","unstructured":"Farasat, A., Nikolaev, A., Miller, S., & Gopalsamy, R. (2017). CROWDLEARNING: Towards collaborative problem-posing at scale. Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale - L@S \u201817, 221\u2013224. https:\/\/doi.org\/10.1145\/3051457.3053990.","DOI":"10.1145\/3051457.3053990"},{"key":"10102_CR10","doi-asserted-by":"publisher","unstructured":"Gutierrez, F., Dou, D., Martini, A., Fickas, S., & Zong, H. (2013). Hybrid ontology-based information extraction for automated text grading. 2013 12th International Conference on Machine Learning and Applications, 359\u2013364. https:\/\/doi.org\/10.1109\/ICMLA.2013.73.","DOI":"10.1109\/ICMLA.2013.73"},{"key":"10102_CR11","unstructured":"Harlen, W., Crick, R. D., Broadfoot, P., Daugherty, R., Gardner, J., James, M., & Stobart, G. (2002). A systematic review of the impact of summative assessment and tests on students\u2019 motivation for learning. EPPI-Centre, University of London, 151."},{"key":"10102_CR12","doi-asserted-by":"publisher","unstructured":"Huang, S.-W., Tu, P.-F., Fu, W.-T., & Amamzadeh, M. (2013). Leveraging the crowd to improve feature-sentiment analysis of user reviews. Proceedings of the 2013 International Conference on Intelligent User Interfaces - IUI \u201813, 3. https:\/\/doi.org\/10.1145\/2449396.2449400.","DOI":"10.1145\/2449396.2449400"},{"key":"10102_CR13","unstructured":"Kamar, E. (2016). Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence. In IJCAI, 4070-4073."},{"key":"10102_CR14","doi-asserted-by":"publisher","unstructured":"Kamath, A., Biswas, A., & Balasubramanian, V. (2016). A crowdsourced approach to student engagement recognition in e-learning environments. 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), 1\u20139. https:\/\/doi.org\/10.1109\/WACV.2016.7477618.","DOI":"10.1109\/WACV.2016.7477618"},{"key":"10102_CR15","doi-asserted-by":"publisher","unstructured":"Kashi, A., Shastri, S., Deshpande, A. R., Doreswamy, J., & Srinivasa, G. (2016). A score recommendation system towards automating assessment in professional courses. 2016 IEEE Eighth International Conference on Technology for Education (T4E), 140\u2013143. https:\/\/doi.org\/10.1109\/T4E.2016.036.","DOI":"10.1109\/T4E.2016.036"},{"key":"10102_CR16","doi-asserted-by":"publisher","unstructured":"Kim, J., Sterman, S., Cohen, A. A. B., & Bernstein, M. S. (2017). Mechanical novel: Crowdsourcing complex work through reflection and revision. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing - CSCW \u201817, 233\u2013245. https:\/\/doi.org\/10.1145\/2998181.2998196.","DOI":"10.1145\/2998181.2998196"},{"key":"10102_CR17","doi-asserted-by":"publisher","unstructured":"Kotturi, Y., Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. (2015). Structure and messaging techniques for online peer learning systems that increase stickiness. Proceedings of the Second (2015) ACM Conference on Learning @ Scale - L@S \u201815, 31\u201338. https:\/\/doi.org\/10.1145\/2724660.2724676.","DOI":"10.1145\/2724660.2724676"},{"key":"10102_CR18","doi-asserted-by":"publisher","unstructured":"Krause, M., Garncarz, T., Song, J., Gerber, E. M., Bailey, B. P., & Dow, S. P. (2017). Critique style guide: Improving Crowdsourced design feedback with a natural language model. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI \u201817, 4627\u20134639. https:\/\/doi.org\/10.1145\/3025453.3025883.","DOI":"10.1145\/3025453.3025883"},{"key":"10102_CR19","doi-asserted-by":"publisher","unstructured":"Kulkarni, C., Cambre, J., Kotturi, Y., Bernstein, M. S., & Klemmer, S. R. (2015a). Talkabout: Making Distance matter with small groups in massive classes. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW \u201815, 1116\u20131128. https:\/\/doi.org\/10.1145\/2675133.2675166.","DOI":"10.1145\/2675133.2675166"},{"key":"10102_CR20","doi-asserted-by":"publisher","unstructured":"Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015b). PeerStudio: Rapid peer feedback emphasizes revision and improves performance. Proceedings of the Second (2015) ACM Conference on Learning @ Scale - L@S \u201815, 75\u201384. https:\/\/doi.org\/10.1145\/2724660.2724670.","DOI":"10.1145\/2724660.2724670"},{"key":"10102_CR21","doi-asserted-by":"publisher","unstructured":"Le, L. T., Shah, C., & Choi, E. (2016). Evaluating the quality of educational answers in community question-answering. Proceedings of the 16th ACM\/IEEE-CS on Joint Conference on Digital Libraries - JCDL \u201816, 129\u2013138. https:\/\/doi.org\/10.1145\/2910896.2910900.","DOI":"10.1145\/2910896.2910900"},{"key":"10102_CR22","unstructured":"Lemley, D., Sudweeks, R., Howell, S., Laws, R. D., & Sawyer, O. (2007). The effects of immediate and delayed feedback on secondary distance learners. Quarterly Review of Distance Education, 8(3), 251."},{"key":"10102_CR23","doi-asserted-by":"publisher","unstructured":"Li, Y. (2018). An application of EDM: Design of a new Online System for correcting exam paper. 2018 13th International Conference on Computer Science & Education (ICCSE), 1\u20136. https:\/\/doi.org\/10.1109\/ICCSE.2018.8468821.","DOI":"10.1109\/ICCSE.2018.8468821"},{"key":"10102_CR24","doi-asserted-by":"publisher","unstructured":"Li, X., Chang, K., Yuan, Y., & Hauptmann, A. (2015). Massive open online proctor: Protecting the credibility of MOOCs certificates. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing - CSCW \u201815, 1129\u20131137. https:\/\/doi.org\/10.1145\/2675133.2675245.","DOI":"10.1145\/2675133.2675245"},{"key":"10102_CR25","doi-asserted-by":"publisher","unstructured":"Luger, S. K. K., & Bowles, J. (2013). An analysis of question quality and user performance in crowdsourced exams. Proceedings of the 2103 Workshop on Data-Driven User Behavioral Modelling and Mining from Social Media - DUBMOD \u201813, 29\u201332. https:\/\/doi.org\/10.1145\/2513577.2513584.","DOI":"10.1145\/2513577.2513584"},{"key":"10102_CR26","doi-asserted-by":"publisher","unstructured":"Nkoana, T. H. (2016). E-learning: Crowdsourcing as an alternative model to traditional learning. 2016 International Conference on Advances in Computing and Communication Engineering (ICACCE), 423\u2013428. https:\/\/doi.org\/10.1109\/ICACCE.2016.8073786.","DOI":"10.1109\/ICACCE.2016.8073786"},{"key":"10102_CR27","unstructured":"Oxford Dictionary [Dictionary]. (2020). Retrieved January 1, 2020, from Lexico website: https:\/\/www.lexico.com\/en\/definition\/machine_learning. Accessed 1 Jan 2020"},{"key":"10102_CR28","doi-asserted-by":"publisher","unstructured":"Pirttinen, N., Kangas, V., Nikkarinen, I., Nygren, H., Leinonen, J., & Hellas, A. (2018). Crowdsourcing programming assignments with CrowdSorcerer. Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education - ITiCSE 2018, 326\u2013331. https:\/\/doi.org\/10.1145\/3197091.3197117.","DOI":"10.1145\/3197091.3197117"},{"issue":"2","key":"10102_CR29","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1111\/j.1540-5885.2011.00893.x","volume":"29","author":"MK Poetz","year":"2012","unstructured":"Poetz, M. K., & Schreier, M. (2012). The value of crowdsourcing: Can users really compete with professionals in generating new product ideas?: The value of crowdsourcing. Journal of Product Innovation Management, 29(2), 245\u2013256. https:\/\/doi.org\/10.1111\/j.1540-5885.2011.00893.x.","journal-title":"Journal of Product Innovation Management"},{"key":"10102_CR30","unstructured":"Schenk, E., & Guittard, C. (2009). Crowdsourcing: What can be Outsourced to the Crowd, and Why. In Workshop on open source innovation, Strasbourg, France 72, 3."},{"key":"10102_CR31","unstructured":"Shetty, B. (2018). Natural Language Processing(NLP) for machine learning. Retrieved November 24, 2018, from towards data science website: towardsdatascience.com\/natural-language-processing-nlp-for-machine-learning-d44498845d5b. Accessed 24 Sept 2019"},{"key":"10102_CR32","doi-asserted-by":"publisher","unstructured":"Shongwe, T., & Zuva, T. (2018). Benefits of crowdsourcing and its growth in the education sector. 2018 International Conference on Advances in Big Data, Computing and Data Communication Systems (IcABCD), 1\u20136. https:\/\/doi.org\/10.1109\/ICABCD.2018.8465436.","DOI":"10.1109\/ICABCD.2018.8465436"},{"key":"10102_CR33","doi-asserted-by":"publisher","unstructured":"Tang, T., Smith, R., Rixner, S., & Warren, J. (2016). Data-driven test case generation for automated programming assessment. Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education - ITiCSE \u201816, 260\u2013265. https:\/\/doi.org\/10.1145\/2899415.2899423.","DOI":"10.1145\/2899415.2899423"},{"key":"10102_CR34","doi-asserted-by":"publisher","unstructured":"Taniguchi, A., & Inoue, S. (2015). A method for automatic assessment of user-generated tests and its evaluation. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers - UbiComp \u201815, 225\u2013228. https:\/\/doi.org\/10.1145\/2800835.2800931.","DOI":"10.1145\/2800835.2800931"},{"key":"10102_CR35","doi-asserted-by":"publisher","unstructured":"Vaish, R., Goel, S., Davis, J., Bernstein, M. S., Gaikwad, S. (Neil) S., Kovacs, G., \u2026 Belongie, S. (2017). Crowd research: Open and scalable university laboratories. Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST \u201817, 829\u2013843. https:\/\/doi.org\/10.1145\/3126594.3126648.","DOI":"10.1145\/3126594.3126648"},{"key":"10102_CR36","unstructured":"Vaughan, J. W. (2017). Making better use of the crowd: How crowdsourcing can advance machine learning research. Journal of Machine Learning Research, 18, 193-1."},{"key":"10102_CR37","doi-asserted-by":"publisher","unstructured":"Watson, P., Ma, T., Tejwani, R., Chang, M., Ahn, J., & Sundararajan, S. (2018). Human-level multiple choice question guessing without domain knowledge: Machine-learning of framing Effects. Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW \u201818, 299\u2013303. https:\/\/doi.org\/10.1145\/3184558.3186340.","DOI":"10.1145\/3184558.3186340"},{"key":"10102_CR38","unstructured":"Whitehill, J., & Seltzer, M. (2016). A crowdsourcing approach to collecting tutorial videos\u2014Toward personalized learning-at-scale. ArXiv:1606.09610 [Cs]. Retrieved from http:\/\/arxiv.org\/abs\/1606.09610. Accessed 15 Aug 2019"},{"key":"10102_CR39","doi-asserted-by":"publisher","unstructured":"Williams, J. J., Kim, J., Rafferty, A., Maldonado, S., Gajos, K. Z., Lasecki, W. S., & Heffernan, N. (2016). AXIS: Generating explanations at scale with Learnersourcing and machine learning. Proceedings of the Third (2016) ACM Conference on Learning @ Scale - L@S \u201816, 379\u2013388. https:\/\/doi.org\/10.1145\/2876034.2876042.","DOI":"10.1145\/2876034.2876042"}],"container-title":["Education and Information Technologies"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10639-020-10102-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10639-020-10102-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10639-020-10102-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,13]],"date-time":"2021-01-13T20:54:05Z","timestamp":1610571245000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10639-020-10102-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,14]]},"references-count":39,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,7]]}},"alternative-id":["10102"],"URL":"https:\/\/doi.org\/10.1007\/s10639-020-10102-w","relation":{},"ISSN":["1360-2357","1573-7608"],"issn-type":[{"value":"1360-2357","type":"print"},{"value":"1573-7608","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,14]]},"assertion":[{"value":"22 October 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 January 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 January 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}