{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T02:40:43Z","timestamp":1760668843721,"version":"build-2065373602"},"reference-count":110,"publisher":"Association for Computing Machinery (ACM)","issue":"7","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,10,18]]},"abstract":"<jats:p>As algorithmic systems increasingly mediate human activities across diverse domains, they shift more responsibility for data collection onto users, fundamentally altering the nature of data work. This paper examines the implications of this shift by investigating student-led data collection and automated feedback interpretation using a mobile, multimodal learning analytics (MMLA) tool designed to coach oral presentation skills. Our findings reveal that while this user-controlled data collection provides greater flexibility, it also imposes speculative labor, compelling students to adjust their behavior to align with perceived standards of good data even when such changes are unwarranted. The study highlights the often-overlooked informal labor involved in managing the socio-material conditions of data collection, emphasizing the need for MMLA tools that offer adaptive support and guidance. These insights extend to algorithmic system design in educational and professional contexts, advocating for systems that balance user autonomy with workload-minimizing guidance to achieve equitable accountability.<\/jats:p>","DOI":"10.1145\/3757672","type":"journal-article","created":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:06:01Z","timestamp":1760634361000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["The Hidden Workload: Student Data Work in Multimodal Algorithmic Evaluations"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3440-1115","authenticated-orcid":false,"given":"Gonzalo Gabriel","family":"M\u00e9ndez","sequence":"first","affiliation":[{"name":"Universitat Polit\u00e8cnica de Val\u00e8ncia, Valencia, Spain and Inria, Rennes, France"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6143-9007","authenticated-orcid":false,"given":"Jhonston Hernan","family":"Benjumea","sequence":"additional","affiliation":[{"name":"Escuela Superior Polit\u00e9cnica del Litoral, Guayaquil, Ecuador"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3594-9289","authenticated-orcid":false,"given":"Leonardo David","family":"Eras","sequence":"additional","affiliation":[{"name":"Escuela Superior Polit\u00e9cnica del Litoral, Guayaquil, Ecuador"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3655-2179","authenticated-orcid":false,"given":"Federico","family":"Dom\u00ednguez","sequence":"additional","affiliation":[{"name":"Escuela Superior Polit\u00e9cnica del Litoral, Guayaquil, Ecuador"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8071-6716","authenticated-orcid":false,"given":"Marisol","family":"Wong-Villacres","sequence":"additional","affiliation":[{"name":"Escuela Superior Polit\u00e9cnica del Litoral, Guayaquil, Ecuador"}]}],"member":"320","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","unstructured":"Haifa Alwahaby and Mutlu Cukurova. 2022. The ethical implications of using Multimodal Learning Analytics: Towards an ethical research and practice framework. https:\/\/doi.org\/10.35542\/osf.io\/4znby","DOI":"10.35542\/osf.io\/4znby"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1080\/17439884.2020.1686014"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300534"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.3390\/educsci12090581"},{"key":"e_1_2_1_5_1","volume-title":"Beebe","author":"Beebe Steven A.","year":"2024","unstructured":"Steven A. Beebe and Susan J. Beebe. 2024. Public Speaking: An Audience-Centered Approach (12th ed.). Pearson Education, Boston, MA."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460296.2460316"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025694"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/1460458219864730"},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Shawn Bullock. 2016. Multimodal Teaching and Learning: The Rhetorics of the Science Classroom by G. Kress C. Jewitt J. Ogborn & C. Tsatsarelis (review). 152-153 pages.","DOI":"10.3138\/cmlr.72.1.152"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.20884\/1.jli.2021.12.2.4002"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3441331"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.17705\/1CAIS.05221"},{"key":"e_1_2_1_13_1","volume-title":"Luis Gal\u00e1rraga, Gonzalo Gabriel M\u00e9ndez, Margarita Ortiz-Rojas, and Alberto Jim\u00e9nez.","author":"Castells Jaime","year":"2020","unstructured":"Jaime Castells, Mohammad Poul Doust, Luis Gal\u00e1rraga, Gonzalo Gabriel M\u00e9ndez, Margarita Ortiz-Rojas, and Alberto Jim\u00e9nez. 2020. A Student-oriented Tool to Support Course Selection in Academic Counseling Sessions. In Proceedings of the Workshop on Adoption, Adaptation and Pilots of Learning Analytics in Under-represented Regions, co-located with the 15th European Conference on Technology Enhanced Learning 2020 (ECTEL 2020). CEUR-WS, Aachen, Germany. http:\/\/ceur-ws.org\/Vol-2704\/paper4.pdf"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0010-0277(02)00017-3"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1080\/17439760.2016.1262613"},{"key":"e_1_2_1_16_1","volume-title":"Research Design: Qualitative, Quantitative, and Mixed Methods Approaches","author":"Creswell John W.","year":"2022","unstructured":"John W. Creswell and J. David Creswell. 2022. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (6th ed.). SAGE Publications, Inc., Thousand Oaks, California. 320 pages.","edition":"6"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.15700\/201412120933"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702314"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2844110"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550328"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637433"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2021.3100778"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449284"},{"key":"e_1_2_1_24_1","first-page":"1564","article-title":"How the design of presentation slides affects audience comprehension: A case for the assertion-evidence approach","volume":"29","author":"Garner Joanna","year":"2013","unstructured":"Joanna Garner and Michael Alley. 2013. How the design of presentation slides affects audience comprehension: A case for the assertion-evidence approach. International Journal of Engineering Education, Vol. 29, 6 (2013), 1564-1579.","journal-title":"International Journal of Engineering Education"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11528-014-0822-x"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.61468\/jofdl.v25i1.447"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0136905"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11423-020-09788-z"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274345"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610203"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1080\/17439884.2019.1573833"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1177\/016146811511700405"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3641898"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359286"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998303"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/0001-6918(67)90005-4"},{"key":"e_1_2_1_37_1","volume-title":"Horgan","author":"Knapp Mark L.","year":"2013","unstructured":"Mark L. Knapp, Judith A. Hall, and Terrence G. Horgan. 2013. Nonverbal Communication in Human Interaction (8th ed.). Cengage Learning, Belmont, CA. 528 pages."},{"volume-title":"Clear and to the Point: 8 Psychological Principles for Compelling PowerPoint Presentations","author":"Kosslyn Stephen M.","key":"e_1_2_1_38_1","unstructured":"Stephen M. Kosslyn. 2007. Clear and to the Point: 8 Psychological Principles for Compelling PowerPoint Presentations. Oxford University Press, Oxford, UK. 144 pages."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1186\/s40594-020-00219-2"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580882"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702548"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3636555.3636872"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3516427"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479596"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461778.3462084"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1186\/s41239-017-0073-y"},{"key":"e_1_2_1_47_1","first-page":"1","article-title":"Building educators' data literacy: Differing perspectives","volume":"13","author":"Mandinach Ellen B","year":"2013","unstructured":"Ellen B Mandinach and Edith S Gummer. 2013. Building educators' data literacy: Differing perspectives. The Journal of Educational Research & Policy Studies, Vol. 13, 2 (2013), 1-5.","journal-title":"The Journal of Educational Research & Policy Studies"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1111\/jcal.12542"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1111\/jcal.12590"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594009"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376148"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3622784"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567574.2567591"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2014.13.6"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445718"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581575"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359144"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.5937\/ekoizazov2323082m"},{"key":"e_1_2_1_59_1","first-page":"1","volume-title":"Proceedings of the 17th Conference on Artificial Intelligence in Medicine (AIME 2019). AIME 2019 Doctoral Consortium","author":"Mitri Daniele Di","year":"2019","unstructured":"Daniele Di Mitri. 2019. Detecting Medical Simulation Errors with Machine Learning and Multimodal Data. In Proceedings of the 17th Conference on Artificial Intelligence in Medicine (AIME 2019). AIME 2019 Doctoral Consortium, Poznan, Poland, 1-6. https:\/\/aime19.aimedicine.info"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386389"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458026.3462161"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.18608\/hla22.006"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.12987"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170358.3170406"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2016.32.10"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3070294"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2023.100136"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.7189\/jogh.09.020318"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2020.3020499"},{"volume-title":"The Definitive Book of Body Language: Why What People Say Is Very Different from What They Think or Feel","author":"Pease Allan","key":"e_1_2_1_70_1","unstructured":"Allan Pease and Barbara Pease. 2006. The Definitive Book of Body Language: Why What People Say Is Very Different from What They Think or Feel. Bantam Books, New York. 386 pages."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3503724"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272973.3273017"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.13373"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372828"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3008327"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10758-017-9316-1"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445748"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1080\/10749039409524673"},{"key":"e_1_2_1_79_1","volume-title":"Proceedings of the First International Workshop on Multimodal Artificial Intelligence in Education (MAIED 2021) (CEUR Workshop Proceedings","volume":"60","author":"Ronda-Carracao Miguel A.","year":"2021","unstructured":"Miguel A. Ronda-Carracao, Olga C. Santos, Gloria Fernandez-Nieto, and Roberto Martinez-Maldonado. 2021. Towards Exploring Stress Reactions in Teamwork using Multimodal Physiological Data. In Proceedings of the First International Workshop on Multimodal Artificial Intelligence in Education (MAIED 2021) (CEUR Workshop Proceedings, Vol. 2902), Daniele Di Mitri, Roberto Mart\u00ednez-Maldonado, Olga C. Santos, Jan Schneider, Khaleel Asyraaf Mat Sanusi, Mutlu Cukurova, Daniel Spikol, Inge Molenaar, Michail Giannakos, Roland Klemke, and Roger Azevedo (Eds.). CEUR-WS, Aachen, Germany, 49-60. http:\/\/ceur-ws.org\/Vol-2902\/paper5.pdf"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.18502\/kss.v8i4.12979"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555198"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3161172"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376229"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.60.3.362"},{"key":"e_1_2_1_85_1","first-page":"117","article-title":"Quality Indicators for Learning Analytics","volume":"17","author":"Scheffel Maren","year":"2014","unstructured":"Maren Scheffel, Hendrik Drachsler, Slavi Stoyanov, and Marcus Specht. 2014. Quality Indicators for Learning Analytics. Journal of Educational Technology & Society, Vol. 17, 4 (2014), 117-132. http:\/\/www.jstor.org\/stable\/jeductechsoci.17.4.117","journal-title":"Journal of Educational Technology & Society"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3012009"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-45153-4_20"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2016.31.3"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1080\/17439884.2014.921628"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/1620545.1620548"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2021.104192"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533177"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1008651105359"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1533-8525.1985.tb00212.x"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/223248.223263"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858110"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501868"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780190691288.001.0001"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9566.13114"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745499918761509"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027385.3027400"},{"key":"e_1_2_1_102_1","volume-title":"The Cognitive Style of PowerPoint: Pitching Out Corrupts Within","author":"Tufte Edward R.","unstructured":"Edward R. Tufte. 2006. The Cognitive Style of PowerPoint: Pitching Out Corrupts Within (2nd ed.). Graphics Press, Cheshire, CT. 31 pages.","edition":"2"},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781003161875-17"},{"key":"e_1_2_1_104_1","article-title":"Artificial Intelligence within Recruitment: Eliminating Biases in Human Resource Management","volume":"8","author":"Veluchamy Ramar","year":"2021","unstructured":"Ramar Veluchamy, Sanchari Chattaraj, and Surbhi Kumari Gupta. 2021. Artificial Intelligence within Recruitment: Eliminating Biases in Human Resource Management. Journal of Critical Reviews, Vol. 8, 3 (2021).","journal-title":"Journal of Critical Reviews"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781351252805-14"},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1080\/13562517.2020.1748811"},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2019.1676396"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501866"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1002\/1097-4571(2000)9999:9999::AID-ASI10393.0.CO;2-O"},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1145\/3636555.3636855"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3757672","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T02:01:28Z","timestamp":1760666488000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3757672"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":110,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,10,18]]}},"alternative-id":["10.1145\/3757672"],"URL":"https:\/\/doi.org\/10.1145\/3757672","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}