{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T03:36:04Z","timestamp":1774064164470,"version":"3.50.1"},"reference-count":37,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T00:00:00Z","timestamp":1679961600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["1936970"],"award-info":[{"award-number":["1936970"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["2033413"],"award-info":[{"award-number":["2033413"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["DGE 19-22697"],"award-info":[{"award-number":["DGE 19-22697"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF NRT","doi-asserted-by":"publisher","award":["1936970"],"award-info":[{"award-number":["1936970"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF NRT","doi-asserted-by":"publisher","award":["2033413"],"award-info":[{"award-number":["2033413"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF NRT","doi-asserted-by":"publisher","award":["DGE 19-22697"],"award-info":[{"award-number":["DGE 19-22697"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Adaptive human\u2013computer systems require the recognition of human behavior states to provide real-time feedback to scaffold skill learning. These systems are being researched extensively for intervention and training in individuals with autism spectrum disorder (ASD). Autistic individuals are prone to social communication and behavioral differences that contribute to their high rate of unemployment. Teamwork training, which is beneficial for all people, can be a pivotal step in securing employment for these individuals. To broaden the reach of the training, virtual reality is a good option. However, adaptive virtual reality systems require real-time detection of behavior. Manual labeling of data is time-consuming and resource-intensive, making automated data annotation essential. In this paper, we propose a semi-supervised machine learning method to supplement manual data labeling of multimodal data in a collaborative virtual environment (CVE) used to train teamwork skills. With as little as 2.5% of the data manually labeled, the proposed semi-supervised learning model predicted labels for the remaining unlabeled data with an average accuracy of 81.3%, validating the use of semi-supervised learning to predict human behavior.<\/jats:p>","DOI":"10.3390\/s23073524","type":"journal-article","created":{"date-parts":[[2023,3,28]],"date-time":"2023-03-28T02:58:21Z","timestamp":1679972301000},"page":"3524","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Semi-Supervised Behavior Labeling Using Multimodal Data during Virtual Teamwork-Based Collaborative Activities"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2373-9337","authenticated-orcid":false,"given":"Abigale","family":"Plunk","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37240, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ashwaq Zaini","family":"Amat","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37240, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8703-0402","authenticated-orcid":false,"given":"Mahrukh","family":"Tauseef","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37240, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Richard Alan","family":"Peters","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37240, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nilanjan","family":"Sarkar","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37240, USA"},{"name":"Department of Mechanical Engineering, Vanderbilt University, Nashville, TN 37240, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,28]]},"reference":[{"key":"ref_1","unstructured":"Unbehauen, H.D. (2009). Control Systems, Robotics and Automation\u2014Volume XXI: Elements of Automation, EOLSS Publications."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1007\/s43494-020-00020-4","article-title":"Teaching Vocational and Social Skills to Adolescents with Autism Using Video Modeling","volume":"43","author":"Stauch","year":"2020","journal-title":"Educ. Treat. Child."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"e2020049437N","DOI":"10.1542\/peds.2020-049437N","article-title":"First Do No Harm: Suggestions Regarding Respectful Autism Language","volume":"149","author":"Dwyer","year":"2022","journal-title":"Pediatrics"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1177\/1362361315588200","article-title":"Which terms should be used to describe autism? Perspectives from the UK autism community","volume":"20","author":"Kenny","year":"2016","journal-title":"Autism Int. J. Res. Pract."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1016\/j.ijhsc.2008.04.003","article-title":"Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder","volume":"66","author":"Liu","year":"2008","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1016\/j.compedu.2016.03.018","article-title":"Design and application of an immersive virtual reality system to enhance emotional skills for children with autism spectrum disorders","volume":"98","author":"Lorenzo","year":"2016","journal-title":"Comput. Educ."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"O\u2019Hara, S., Lui, Y.M., and Draper, B.A. (2011, January 21\u201325). Unsupervised Learning of Human Expressions, Gestures, and Actions. Proceedings of 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa, Barbara, CA, USA.","DOI":"10.1109\/FG.2011.5771473"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1007\/s10994-019-05855-6","article-title":"A survey on semi-supervised learning","volume":"109","author":"Hoos","year":"2020","journal-title":"Mach. Learn."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Rosenberg, C., Hebert, M., and Schneiderman, H. (2005). Semi-Supervised Self-Training of Object Detection Models, Carnegie Mellon University.","DOI":"10.1109\/ACVMOT.2005.107"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Yin, H., Camacho, D., Tino, P., Allmendinger, R., Tall\u00f3n-Ballesteros, A.J., Tang, K., Cho, S.-B., Novais, P., and Nascimento, S. (2021). 22nd International Conference, Proceedings of the Intelligent Data Engineering and Automated Learning\u2014IDEAL 2021, Manchester, UK, November 25\u201327, Springer International Publishing.","DOI":"10.1007\/978-3-030-91608-4"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1108","DOI":"10.1109\/TITS.2015.2496157","article-title":"Driver Distraction Detection Using Semi-Supervised Machine Learning","volume":"17","author":"Liu","year":"2016","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Ringeval, F., Dong, B., Coutinho, E., Marchi, E., and Sch\u00fcller, B. (2016, January 20\u201325). Enhanced semi-supervised learning for multimodal emotion recognition. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.","DOI":"10.1109\/ICASSP.2016.7472666"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"S55","DOI":"10.21037\/tp.2019.09.09","article-title":"Autism spectrum disorder: Definition, epidemiology, causes, and clinical evaluation","volume":"9","author":"Hodges","year":"2020","journal-title":"Transl. Pediatr."},{"key":"ref_14","unstructured":"(2022, November 03). Autism Facts & Statistics\u2014Autism Society Greater Cincinnati. Available online: https:\/\/www.autismcincy.org\/autism-facts-statistics\/."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Litchfield, P., Cooper, C., Hancock, C., and Watt, P. (2016). Work and Wellbeing in the 21st Century. Int. J. Environ. Res. Public Health, 13.","DOI":"10.3390\/ijerph13111065"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1037\/1076-8998.6.4.324","article-title":"The relative effects of deprivation of the latent and manifest benefits of employment on the well-being of unemployed people","volume":"6","author":"Creed","year":"2001","journal-title":"J. Occup. Health Psychol."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Scott, M., Falkmer, M., Girdler, S., and Falkmer, T. (2015). Viewpoints on Factors for Successful Employment for Adults with Autism Spectrum Disorder. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0143674"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5607\/en.2016.25.1.1","article-title":"A Short Review on the Current Understanding of Autism Spectrum Disorders","volume":"25","author":"Park","year":"2016","journal-title":"Exp. Neurobiol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"e028280","DOI":"10.1136\/bmjopen-2018-028280","article-title":"How effective is teamwork really? The relationship between teamwork and performance in healthcare teams: A systematic review and meta-analysis","volume":"9","author":"Schmutz","year":"2019","journal-title":"BMJ Open"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1727","DOI":"10.1007\/s10803-017-3441-5","article-title":"An Evaluation of a Social Skills Intervention for Adults with Autism Spectrum Disorder and Intellectual Disabilities preparing for Employment in Ireland: A Pilot Study","volume":"48","author":"Walsh","year":"2018","journal-title":"J. Autism Dev. Disord."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1016\/j.procs.2012.10.023","article-title":"Multitouch Tabletop Technology for People with Autism Spectrum Disorder: A Review of the Literature","volume":"14","author":"Chen","year":"2012","journal-title":"Procedia Comput. Sci."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1023\/A:1010660502130","article-title":"Enhancing social problem solving in children with autism and normal children through computer-assisted instruction","volume":"31","author":"Sriram","year":"2001","journal-title":"J. Autism Dev. Disord."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1542","DOI":"10.1177\/1362361318801345","article-title":"Development, feasibility, and preliminary efficacy of an employment-related social skills intervention for young adults with high-functioning autism","volume":"23","author":"Sung","year":"2019","journal-title":"Autism Int. J. Res. Pract."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Cordell, A.T. (Interv. Sch. Clin., 2023). Creating a Vocational Training Site for Individuals With Autism Spectrum Disorder, Interv. Sch. Clin., online first.","DOI":"10.1177\/10534512231156882"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mesa-Gresa, P., Gil-G\u00f3mez, H., Lozano-Quilis, J.-A., and Gil-G\u00f3mez, J.-A. (2018). Effectiveness of Virtual Reality for Children and Adolescents with Autism Spectrum Disorder: An Evidence-Based Systematic Review. Sensors, 18.","DOI":"10.3390\/s18082486"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Chi\u021bu, I.B., Tec\u0103u, A.S., Constantin, C.P., Tesca\u0219iu, B., Br\u0103tucu, T.-O., Br\u0103tucu, G., and Purcaru, I.-M. (2023). Exploring the Opportunity to Use Virtual Reality for the Education of Children with Disabilities. Children, 10.","DOI":"10.3390\/children10030436"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Wu, X., Liu, W., Jia, J., Zhang, X., Leifer, L., and Hu, S. (2023). Prototyping an Online Virtual Simulation Course Platform for College Students to Learn Creative Thinking. Systems, 11.","DOI":"10.3390\/systems11020089"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Thordarson, A., and Vilhj\u00e1lmsson, H.H. SoCueVR: Virtual Reality Game for Social Cue Detection Training. Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, Paris, France, 2\u20135 July 2019.","DOI":"10.1145\/3308532.3329440"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Amat, A., Breen, M., Hunt, S., Wilson, D., Khaliq, Y., Byrnes, N., Cox, D., Czarnecki, S., Justice, C., and Kennedy, D. (2021). Collaborative Virtual Environment to Encourage Teamwork in Autistic Adults in Workplace Settings, Springer International Publishing.","DOI":"10.1007\/978-3-030-78092-0_22"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Plunk, A., Amat, A.Z., and Wilkes, D.M. (2022, January 21\u201325). Automated Behavior Labeling During Team-Based Activities Involving Neurodiverse and Neurotypical Partners Using Multimodal Data. Proceedings of the Human Behavior Understanding Workshop in conjunction with the International Conference on Pattern Recognition, Montreal, QC, Canada.","DOI":"10.1007\/978-3-031-37660-3_14"},{"key":"ref_31","unstructured":"Technologies, U. (2022, September 28). Unity Real-Time Development Platform|3D, 2D VR & AR Engine. Available online: https:\/\/unity.com\/."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1007\/s11412-006-9005-x","article-title":"A rating scheme for assessing the quality of computer-supported collaboration processes","volume":"2","author":"Meier","year":"2007","journal-title":"Int. J. Comput.-Support. Collab. Learn."},{"key":"ref_33","unstructured":"(2022, September 28). MathWorks\u2014Makers of MATLAB and Simulink. Available online: https:\/\/www.mathworks.com\/."},{"key":"ref_34","unstructured":"(2023, March 09). Support Vector Machines for Binary Classification\u2014MATLAB & Simulink. Available online: https:\/\/www.mathworks.com\/help\/stats\/support-vector-machines-for-binary-classification.html#bsr5b42."},{"key":"ref_35","unstructured":"Gandhi, R. (2023, March 09). Support Vector Machine\u2014Introduction to Machine Learning Algorithms. Available online: https:\/\/towardsdatascience.com\/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1822","DOI":"10.1016\/j.eswa.2011.08.068","article-title":"An autonomous labeling approach to support vector machines algorithms for network traffic anomaly detection","volume":"39","author":"Catania","year":"2012","journal-title":"Expert Syst. Appl."},{"key":"ref_37","unstructured":"(2023, March 09). Classify Observations Using Support Vector Machine (SVM) Classifier\u2014MATLAB Predict. Available online: https:\/\/www.mathworks.com\/help\/stats\/classreg.learning.classif.compactclassificationsvm.predict.html."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/7\/3524\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:04:39Z","timestamp":1760123079000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/7\/3524"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,28]]},"references-count":37,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["s23073524"],"URL":"https:\/\/doi.org\/10.3390\/s23073524","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,28]]}}}