{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T20:36:27Z","timestamp":1768595787001,"version":"3.49.0"},"reference-count":43,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2019,7,13]],"date-time":"2019-07-13T00:00:00Z","timestamp":1562976000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100013276","name":"Interreg","doi-asserted-by":"publisher","award":["SafePAT project INTERREG V-A"],"award-info":[{"award-number":["SafePAT project INTERREG V-A"]}],"id":[{"id":"10.13039\/100013276","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This study investigated to what extent multimodal data can be used to detect mistakes during Cardiopulmonary Resuscitation (CPR) training. We complemented the Laerdal QCPR ResusciAnne manikin with the Multimodal Tutor for CPR, a multi-sensor system consisting of a Microsoft Kinect for tracking body position and a Myo armband for collecting electromyogram information. We collected multimodal data from 11 medical students, each of them performing two sessions of two-minute chest compressions (CCs). We gathered in total 5254 CCs that were all labelled according to five performance indicators, corresponding to common CPR training mistakes. Three out of five indicators, CC rate, CC depth and CC release, were assessed automatically by the ReusciAnne manikin. The remaining two, related to arms and body position, were annotated manually by the research team. We trained five neural networks for classifying each of the five indicators. The results of the experiment show that multimodal data can provide accurate mistake detection as compared to the ResusciAnne manikin baseline. We also show that the Multimodal Tutor for CPR can detect additional CPR training mistakes such as the correct use of arms and body weight. Thus far, these mistakes were identified only by human instructors. Finally, to investigate user feedback in the future implementations of the Multimodal Tutor for CPR, we conducted a questionnaire to collect valuable feedback aspects of CPR training.<\/jats:p>","DOI":"10.3390\/s19143099","type":"journal-article","created":{"date-parts":[[2019,7,15]],"date-time":"2019-07-15T04:55:27Z","timestamp":1563166527000},"page":"3099","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":34,"title":["Detecting Mistakes in CPR Training with Multimodal Data and Neural Networks"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9331-6893","authenticated-orcid":false,"given":"Daniele","family":"Di Mitri","sequence":"first","affiliation":[{"name":"Welten Institute, Research Centre for Learning, Teaching and Technology Open University of the Netherlands, Valkenburgerweg, 177 6401 AT Heerlen, The Netherlands"}]},{"given":"Jan","family":"Schneider","sequence":"additional","affiliation":[{"name":"DIPF - Leibniz Institute for Research and Information in Education, Rostocker Stra\u00dfe 6, 60323 Frankfurt, Germany"}]},{"given":"Marcus","family":"Specht","sequence":"additional","affiliation":[{"name":"Welten Institute, Research Centre for Learning, Teaching and Technology Open University of the Netherlands, Valkenburgerweg, 177 6401 AT Heerlen, The Netherlands"}]},{"given":"Hendrik","family":"Drachsler","sequence":"additional","affiliation":[{"name":"Welten Institute, Research Centre for Learning, Teaching and Technology Open University of the Netherlands, Valkenburgerweg, 177 6401 AT Heerlen, The Netherlands"},{"name":"DIPF - Leibniz Institute for Research and Information in Education, Rostocker Stra\u00dfe 6, 60323 Frankfurt, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2019,7,13]]},"reference":[{"key":"ref_1","unstructured":"Schon, D. (1983). The Reflective Practitioner, Basic Book Inc."},{"key":"ref_2","unstructured":"Polson, M.C., Richardson, J.J., and Soloway, E. (1988). Foundations of Intelligent Tutoring Systems, Erlbaum Associates Inc."},{"key":"ref_3","first-page":"7","article-title":"A review about Smart Objects, Sensors, and Actuators","volume":"4","year":"2017","journal-title":"Int. J. Interact. Multimed. Artif. Intell."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Oviatt, S., Schuller, B., Cohen, P.R., Sonntag, D., Potamianos, G., and Kr\u00fcger, A. (2018). The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations, Morgan & Claypool.","DOI":"10.1145\/3107990"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1339","DOI":"10.1016\/j.inffus.2016.09.005","article-title":"Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges","volume":"35","author":"Gravina","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Vinciarelli, A., Pantic, M., Bourlard, H., and Pentland, A. (2008, January 26\u201331). Social signal processing: State-of-the-art and future perspectives of an emerging domain. Proceedings of the 16th ACM International Conference on Multimedia, Vancouver, BC, Canada.","DOI":"10.1145\/1459359.1459573"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal Machine Learning: A Survey and Taxonomy","volume":"41","author":"Baltrusaitis","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1449","DOI":"10.1109\/JPROC.2015.2460697","article-title":"Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects","volume":"103","author":"Lahat","year":"2015","journal-title":"Proc. IEEE"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Blikstein, P. (2013, January 8\u201313). Multimodal learning analytics. Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK \u201913), Leuven, Belgium.","DOI":"10.1145\/2460296.2460316"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"213","DOI":"10.18608\/jla.2016.32.10","article-title":"Augmenting Learning Analytics with Multimodal Sensory Data","volume":"3","author":"Ochoa","year":"2016","journal-title":"J. Lear. Anal."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Worsley, M. (2014, January 12). Multimodal Learning Analytics as a Tool for Bridging Learning Theory and Complex Learning Behaviors. Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA \u201914), Istanbul, Turkey.","DOI":"10.1145\/2666633.2666634"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Martinez-Maldonado, R., Echeverria, V., Santos, O.C., Santos, A.D.P.D., and Yacef, K. (2018, January 7\u20139). Physical learning analytics: A multimodal perspective. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK \u201918), Sydney, Australia.","DOI":"10.1145\/3170358.3170379"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Echeverria, V., Martinez-Maldonado, R., Granda, R., Chiluiza, K., Conati, C., and Shum, S.B. (2018, January 7\u20139). Driving data storytelling from learning design. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK \u201918), Sydney, Australia.","DOI":"10.1145\/3170358.3170380"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Cukurova, M., Kent, C., and Luckin, R. (2019). Artificial intelligence and multimodal data in the service of human decision-making: A case study in debate tutoring. Br. J. Educ. Technol.","DOI":"10.1111\/bjet.12829"},{"key":"ref_15","unstructured":"Martinez-Maldonado, R. (2018). The Big Five: Addressing Recurrent Multimodal Learning Data Challenges. Proceedings of the Second Multimodal Learning Analytics Across (Physical and Digital) Spaces (CrossMMLA), Sydney, Australia, 6 March 2018, CEUR Workshop Proceedings."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.resuscitation.2015.07.015","article-title":"European Resuscitation Council Guidelines for Resuscitation 2015: Section 2. Adult basic life support and automated external defibrillation","volume":"95","author":"Perkins","year":"2015","journal-title":"Resuscitation"},{"key":"ref_17","first-page":"30","article-title":"Intelligent Tutoring Goes To School in the Big City","volume":"8","author":"Koedinger","year":"1997","journal-title":"Int. J. Artif. Intell. Educ. (IJAIED)"},{"key":"ref_18","first-page":"152","article-title":"ALEKS: A Web-based intelligent tutoring system","volume":"35","author":"Canfield","year":"2001","journal-title":"Math. Comput. Educ."},{"key":"ref_19","first-page":"173","article-title":"An Intelligent SQL Tutor on the Web","volume":"13","author":"Mitrovic","year":"2003","journal-title":"Int. J. Artif. Intell. Educ."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1080\/00461520.2011.611369","article-title":"The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems","volume":"46","author":"VanLehn","year":"2011","journal-title":"Educ. Psychol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1037\/a0034752","article-title":"A meta-analysis of the effectiveness of intelligent tutoring systems on college students\u2019 academic learning","volume":"106","author":"Cooper","year":"2014","journal-title":"J. Educ. Psychol."},{"key":"ref_22","unstructured":"Eliot, C., and Woolf, B.P. An intelligent learning environment for advanced cardiac life support. Proceedings of the AMIA Annual Fall Symposium."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1109\/MIS.2007.66","article-title":"COMET: A Collaborative for Medical Problem-Based Learning","volume":"22","author":"Suebnukarn","year":"2007","journal-title":"IEEE Intell. Syst."},{"key":"ref_24","first-page":"612","article-title":"AutoTutor detects and responds to learners affective and cognitive states","volume":"48","author":"Jackson","year":"2008","journal-title":"IEEE Trans. Educ."},{"key":"ref_25","first-page":"28","article-title":"Affective learning companions","volume":"47","author":"Burleson","year":"2007","journal-title":"Educ. Technol."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Schneider, J., B\u00f6rner, D., van Rosmalen, P., and Specht, M. (2015, January 9\u201313). Presentation Trainer, your Public Speaking Multimodal Coach. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI \u201915), Seattle, WA, USA.","DOI":"10.1145\/2818346.2830603"},{"key":"ref_27","unstructured":"Limbu, B., Schneider, J., Klemke, R., and Specht, M. (2018, January 23\u201325). Augmentation of practice with expert performance data: Presenting a calligraphy use case. Proceedings of the 3rd International Conference on Smart Learning Ecosystem and Regional Development\u2014The Interplay of Data, Technology, Place and People, Aalborg, Denmark."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"220","DOI":"10.18608\/jla.2016.32.11","article-title":"Multimodal Learning Analytics and Education Data Mining: Using Computational Technologies to Measure Complex Learning Tasks","volume":"3","author":"Blikstein","year":"2016","journal-title":"J. Learn. Anal."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"4097","DOI":"10.3390\/s150204097","article-title":"Augmenting the Senses: A Review on Sensor-Based Learning Support","volume":"15","author":"Schneider","year":"2015","journal-title":"Sensors"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Raca, M., and Dillenbourg, P. (2014, January 12). Holistic Analysis of the Classroom. Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA \u201914), Istanbul, Turkey.","DOI":"10.1145\/2666633.2666636"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"D\u2019Mello, S., Olney, A., Blanchard, N., Sun, X., Ward, B., Samei, B., and Kelly, S. (2015, January 9\u201313). Multimodal Capture of Teacher-Student Interactions for Automated Dialogic Analysis in Live Classrooms. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830602"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Dom\u00ednguez, F., Echeverr\u00eda, V., Chiluiza, K., and Ochoa, X. (2015, January 9\u201313). Multimodal Selfies: Designing a Multimodal Recording Device for Students in Traditional Classrooms. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.","DOI":"10.1145\/2818346.2830606"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ochoa, X., Chiluiza, K., M\u00e9ndez, G., Luzardo, G., Guam\u00e1n, B., and Castells, J. (2013, January 12\u201316). Expertise estimation based on simple multimodal features. Proceedings of the 15th ACM International Conference on Multimodal Iteraction (ICMI \u201913), Istanbul, Turkey.","DOI":"10.1145\/2522848.2533789"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Prieto, L., Sharma, K., Kidzinski, \u0141., Rodr\u00edguez-Triana, M., and Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. J. Comput. Assist. Learn.","DOI":"10.1111\/jcal.12232"},{"key":"ref_35","first-page":"513","article-title":"Multimodal tutor for CPR","volume":"Volume 10948 LNAI","year":"2018","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Di Mitri, D., Schneider, J., Klemke, R., Specht, M., and Drachsler, H. (2019, January 4\u20138). Read Between the Lines: An Annotation Tool for Multimodal Data for Learning. Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), Tempe, AZ, USA.","DOI":"10.1145\/3303772.3303776"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e20","DOI":"10.1016\/j.resuscitation.2012.08.052","article-title":"A new Kinect-based system for the analysis of performances in cardiopulmonary resuscitation (CPR) training","volume":"83","author":"Semeraro","year":"2012","journal-title":"Resuscitation"},{"key":"ref_38","first-page":"51","article-title":"A kinect-based system for cardiopulmonary resuscitation simulation: A pilot study","volume":"Volume 8101 LNCS","author":"Wattanasoontorn","year":"2013","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"577","DOI":"10.1016\/j.ajem.2017.09.022","article-title":"Kinect-based real-time audiovisual feedback device improves CPR quality of lower-body-weight rescuers","volume":"36","author":"Wang","year":"2018","journal-title":"Am. J. Emerg. Med."},{"key":"ref_40","first-page":"45","article-title":"Multimodal Learning Hub: A Tool for Capturing Customizable Multimodal Learning Experiences","volume":"Volume 11082 LNCS","author":"Schneider","year":"2018","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_41","unstructured":"Schneider, J. (2019, July 12). Learning Hub. In GitHub Repository. Available online: https:\/\/github.com\/janschneiderou\/LearningHub."},{"key":"ref_42","unstructured":"Di Mitri, D. (2019, July 12). Visual Inspection Tool: Annotation and visualization tool for Multimodal Data. In GitHub Repository. Available online: https:\/\/github.com\/dimstudio\/visual-inspection-tool."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/14\/3099\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:05:19Z","timestamp":1760187919000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/19\/14\/3099"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,13]]},"references-count":43,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2019,7]]}},"alternative-id":["s19143099"],"URL":"https:\/\/doi.org\/10.3390\/s19143099","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7,13]]}}}