{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T18:15:42Z","timestamp":1778609742675,"version":"3.51.4"},"reference-count":30,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,1,12]],"date-time":"2022-01-12T00:00:00Z","timestamp":1641945600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>While the majority of social scientists still rely on traditional research instruments (e.g., surveys, self-reports, qualitative observations), multimodal sensing is becoming an emerging methodology for capturing human behaviors. Sensing technology has the potential to complement and enrich traditional measures by providing high frequency data on people\u2019s behavior, cognition and affects. However, there is currently no easy-to-use toolkit for recording multimodal data streams. Existing methodologies rely on the use of physical sensors and custom-written code for accessing sensor data. In this paper, we present the EZ-MMLA toolkit. This toolkit was implemented as a website and provides easy access to multimodal data collection algorithms. One can collect a variety of data modalities: data on users\u2019 attention (eye-tracking), physiological states (heart rate), body posture (skeletal data), gestures (from hand motion), emotions (from facial expressions and speech) and lower-level computer vision algorithms (e.g., fiducial\/color tracking). This toolkit can run from any browser and does not require dedicated hardware or programming experience. We compare this toolkit with traditional methods and describe a case study where the EZ-MMLA toolkit was used by aspiring educational researchers in a classroom context. We conclude by discussing future work and other applications of this toolkit, potential limitations and implications.<\/jats:p>","DOI":"10.3390\/s22020568","type":"journal-article","created":{"date-parts":[[2022,1,12]],"date-time":"2022-01-12T09:10:36Z","timestamp":1641978636000},"page":"568","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Augmenting Social Science Research with Multimodal Data Collection: The EZ-MMLA Toolkit"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0922-2593","authenticated-orcid":false,"given":"Bertrand","family":"Schneider","sequence":"first","affiliation":[{"name":"Harvard Graduate School of Education, Harvard University, Cambridge, MA 02138, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Javaria","family":"Hassan","sequence":"additional","affiliation":[{"name":"Harvard Graduate School of Education, Harvard University, Cambridge, MA 02138, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gahyun","family":"Sung","sequence":"additional","affiliation":[{"name":"Harvard Graduate School of Education, Harvard University, Cambridge, MA 02138, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1016\/j.ppnp.2012.03.001","article-title":"The Large Hadron Collider","volume":"67","author":"Burkhardt","year":"2012","journal-title":"Prog. Part. Nucl. Phys."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1007\/s11214-006-8315-7","article-title":"The James Webb Space Telescope","volume":"4","author":"Gardner","year":"2006","journal-title":"Space Sci. Rev."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Blikstein, P. (2013, January 8\u201313). Multimodal Learning Analytics. Proceedings of the Third International Conference on Learning Analytics and Knowledge, New York, NY, USA.","DOI":"10.1145\/2460296.2460316"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1441","DOI":"10.1111\/bjet.13015","article-title":"The Promise and Challenges of Multimodal Learning Analytics","volume":"51","author":"Cukurova","year":"2020","journal-title":"Br. J. Educ. Technol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"4097","DOI":"10.3390\/s150204097","article-title":"Augmenting the Senses: A Review on Sensor-Based Learning Support","volume":"15","author":"Schneider","year":"2015","journal-title":"Sensors"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1450","DOI":"10.1111\/bjet.12993","article-title":"Multimodal Data Capabilities for Learning: What Can Multimodal Data Tell Us about Learning?","volume":"51","author":"Sharma","year":"2020","journal-title":"Br. J. Educ. Technol."},{"key":"ref_7","first-page":"89","article-title":"Unraveling Students\u2019 Interaction around a Tangible Interface Using Multimodal Learning Analytics","volume":"7","author":"Schneider","year":"2015","journal-title":"J. Educ. Data Min."},{"key":"ref_8","unstructured":"Worsley, M., and Blikstein, P. (2011, January 6\u20138). What\u2019s an Expert? Using Learning Analytics to Identify Emergent Markers of Expertise through Automated Speech, Sentiment and Sketch Analysis. Proceedings of the 4th International Conference on Educational Data Mining (EDM \u201911), Eindhoven, The Netherlands."},{"key":"ref_9","first-page":"1370","article-title":"Quantitative Field Observation (QFOs) Baker-Rodrigo Observation Method Protocol (BROMP) 1.0 Training Manual Version 1.0 2012","volume":"91","author":"Ocumpaugh","year":"2003","journal-title":"Proc. IEEE"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1207\/s15516709cog2601_3","article-title":"Spanning Seven Orders of Magnitude: A Challenge for Cognitive Modeling","volume":"26","author":"Anderson","year":"2002","journal-title":"Cogn. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1007\/s41979-020-00027-x","article-title":"Lowering Barriers for Accessing Sensor Data in Education: Lessons Learned from Teaching Multimodal Learning Analytics to Educators","volume":"3","author":"Schneider","year":"2020","journal-title":"J. STEM Educ. Res."},{"key":"ref_12","first-page":"1","article-title":"Teacher Tracking with Integrity: What Indoor Positioning Can Reveal About Instructional Proxemics","volume":"4","author":"Mangaroska","year":"2020","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3161172","article-title":"Sensei: Sensing Educational Interaction","volume":"1","author":"Saquib","year":"2018","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3351229","article-title":"EduSense: Practical Classroom Sensing at Scale","volume":"3","author":"Ahuja","year":"2019","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_15","unstructured":"iMotions (2019, August 21). iMotions Biometric Research Platform. Available online: https:\/\/imotions.com\/."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., and Andr\u00e9, E. (2013, January 21\u201325). The Social Signal Interpretation (SSI) Framework: Multimodal Signal Processing and Recognition in Real-Time. Proceedings of the 21st ACM international conference on Multimedia, Barcelona, Spain.","DOI":"10.1145\/2502081.2502223"},{"key":"ref_17","unstructured":"Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M.G., and Lee, J. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"220","DOI":"10.18608\/jla.2016.32.11","article-title":"Multimodal Learning Analytics and Education Data Mining: Using Computational Technologies to Measure Complex Learning Tasks","volume":"3","author":"Blikstein","year":"2016","journal-title":"J. Learn. Anal."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1145\/1073204.1073223","article-title":"Motion Magnification","volume":"24","author":"Liu","year":"2005","journal-title":"ACM Trans. Graph."},{"key":"ref_20","first-page":"39","article-title":"The Quantified Self (QS) Movement and Some Emerging Opportunities for the Educational Technology Field","volume":"53","author":"Lee","year":"2013","journal-title":"Educ. Technol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1111\/jcal.12288","article-title":"From Signals to Knowledge: A Conceptual Model for Multimodal Learning Analytics","volume":"34","author":"Mitri","year":"2018","journal-title":"J. Comput. Assist. Learn."},{"key":"ref_22","unstructured":"Brooke, J. (1996). SUS: A \u201cQuick and Dirty\u201d Usability Scale. Usability Evaluation in Industry, CRC Press."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1191\/1478088706qp063oa","article-title":"Using Thematic Analysis in Psychology","volume":"3","author":"Braun","year":"2006","journal-title":"Qual. Res. Psychol."},{"key":"ref_24","first-page":"114","article-title":"Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale","volume":"4","author":"Bangor","year":"2009","journal-title":"J. Usability Stud."},{"key":"ref_25","unstructured":"Newman, F. (1996). Authentic Achievement: Restructuring Schools for Intellectual Quality, Jossey-Bass."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1002\/tea.21465","article-title":"The Effect of Authentic Project-Based Learning on Attitudes and Career Aspirations in STEM","volume":"56","author":"Beier","year":"2019","journal-title":"J. Res. Sci. Teach."},{"key":"ref_27","unstructured":"Papoutsaki, A., Sangkloy, P., Laskey, J., Daskalova, N., Huang, J., and Hays, J. (2016, January 9\u201315). WebGazer: Scalable Webcam Eye Tracking Using User Interactions. Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI), New York, NY, USA."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1007\/s11412-018-9281-2","article-title":"Leveraging Mobile Eye-Trackers to Capture Joint Visual Attention in Co-Located Collaborative Learning Groups","volume":"13","author":"Schneider","year":"2018","journal-title":"Int. J. Comput.-Support. Collab. Learn."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"893","DOI":"10.1037\/0022-3514.76.6.893","article-title":"The Chameleon Effect: The Perception\u2013Behavior Link and Social Interaction","volume":"76","author":"Chartrand","year":"1999","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1109\/MC.2018.3620965","article-title":"Toward Human-Understandable, Explainable AI","volume":"51","author":"Hagras","year":"2018","journal-title":"Computer"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/2\/568\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T14:14:58Z","timestamp":1760364898000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/2\/568"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,12]]},"references-count":30,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22020568"],"URL":"https:\/\/doi.org\/10.3390\/s22020568","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,12]]}}}