{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,19]],"date-time":"2026-07-19T20:10:37Z","timestamp":1784491837669,"version":"3.55.0"},"reference-count":48,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2023,7,21]],"date-time":"2023-07-21T00:00:00Z","timestamp":1689897600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2114808"],"award-info":[{"award-number":["2114808"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["H327S210005"],"award-info":[{"award-number":["H327S210005"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["H327S200009"],"award-info":[{"award-number":["H327S200009"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Education","award":["2114808"],"award-info":[{"award-number":["2114808"]}]},{"name":"U.S. Department of Education","award":["H327S210005"],"award-info":[{"award-number":["H327S210005"]}]},{"name":"U.S. Department of Education","award":["H327S200009"],"award-info":[{"award-number":["H327S200009"]}]},{"name":"University of Central Florida\u2019s Preeminent Postdoctoral Program (P3)","award":["2114808"],"award-info":[{"award-number":["2114808"]}]},{"name":"University of Central Florida\u2019s Preeminent Postdoctoral Program (P3)","award":["H327S210005"],"award-info":[{"award-number":["H327S210005"]}]},{"name":"University of Central Florida\u2019s Preeminent Postdoctoral Program (P3)","award":["H327S200009"],"award-info":[{"award-number":["H327S200009"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Recognizing the affective state of children with autism spectrum disorder (ASD) in real-world settings poses challenges due to the varying head poses, illumination levels, occlusion and a lack of datasets annotated with emotions in in-the-wild scenarios. Understanding the emotional state of children with ASD is crucial for providing personalized interventions and support. Existing methods often rely on controlled lab environments, limiting their applicability to real-world scenarios. Hence, a framework that enables the recognition of affective states in children with ASD in uncontrolled settings is needed. This paper presents a framework for recognizing the affective state of children with ASD in an in-the-wild setting using heart rate (HR) information. More specifically, an algorithm is developed that can classify a participant\u2019s emotion as positive, negative, or neutral by analyzing the heart rate signal acquired from a smartwatch. The heart rate data are obtained in real time using a smartwatch application while the child learns to code a robot and interacts with an avatar. The avatar assists the child in developing communication skills and programming the robot. In this paper, we also present a semi-automated annotation technique based on facial expression recognition for the heart rate data. The HR signal is analyzed to extract features that capture the emotional state of the child. Additionally, in this paper, the performance of a raw HR-signal-based emotion classification algorithm is compared with a classification approach based on features extracted from HR signals using discrete wavelet transform (DWT). The experimental results demonstrate that the proposed method achieves comparable performance to state-of-the-art HR-based emotion recognition techniques, despite being conducted in an uncontrolled setting rather than a controlled lab environment. The framework presented in this paper contributes to the real-world affect analysis of children with ASD using HR information. By enabling emotion recognition in uncontrolled settings, this approach has the potential to improve the monitoring and understanding of the emotional well-being of children with ASD in their daily lives.<\/jats:p>","DOI":"10.3390\/s23146572","type":"journal-article","created":{"date-parts":[[2023,7,24]],"date-time":"2023-07-24T03:03:25Z","timestamp":1690167805000},"page":"6572","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["In-the-Wild Affect Analysis of Children with ASD Using Heart Rate"],"prefix":"10.3390","volume":"23","author":[{"given":"Kamran","family":"Ali","sequence":"first","affiliation":[{"name":"Synthetic Reality Lab, Department of Computer Science, University of Central Florida, Orlando, FL 32816, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4494-3629","authenticated-orcid":false,"given":"Sachin","family":"Shah","sequence":"additional","affiliation":[{"name":"Synthetic Reality Lab, Department of Computer Science, University of Central Florida, Orlando, FL 32816, USA"},{"name":"Department of Computer Science, University of Maryland, College Park, MD 20742, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2528-3380","authenticated-orcid":false,"given":"Charles E.","family":"Hughes","sequence":"additional","affiliation":[{"name":"Synthetic Reality Lab, Department of Computer Science, University of Central Florida, Orlando, FL 32816, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,7,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"18659","DOI":"10.1038\/s41598-020-75768-1","article-title":"A wearable heart rate measurement device for children with autism spectrum disorder","volume":"10","author":"Fioriello","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1080\/02699939208411068","article-title":"An argument for basic emotions","volume":"6","author":"Ekman","year":"1992","journal-title":"Cogn. Emot."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1016\/j.amjcard.2007.03.056","article-title":"Changes in Heart Rate and Heart Rate Variability Over Time in Middle-Aged Men and Women in the General Population (from the Whitehall II Cohort Study)","volume":"100","author":"Britton","year":"2007","journal-title":"Am. J. Cardiol."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Valderas, M.T., Bolea, J., Laguna, P., Vallverd\u00fa, M., and Bail\u00f3n, R. (2015, January 25\u201329). Human emotion recognition using heart rate variability analysis with spectral bands based on respiration. Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society, Milan, Italy.","DOI":"10.1109\/EMBC.2015.7319792"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1109\/MTS.2018.2795096","article-title":"Robot enhanced therapy for children with Autism (DREAM): A social model of autism","volume":"37","author":"Richardson","year":"2018","journal-title":"IEEE Technol. Soc. Mag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1002\/aur.1527","article-title":"Autism and social robotics: A systematic review","volume":"9","author":"Pennisi","year":"2016","journal-title":"Autism Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1146\/annurev-bioeng-071811-150036","article-title":"Robots for use in autism research","volume":"14","author":"Scassellati","year":"2012","journal-title":"Ann. Rev. Biomed. Eng."},{"key":"ref_8","unstructured":"Ferari, E., Robins, B., and Dautenhahn, K. (2009, January 3\u20135). Robot as a social mediator\u2014A play scenario implementation with children with autism. Proceedings of the International Conference on Interaction Design and Children, Como, Italy."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1177\/0022466918761120","article-title":"Computer programming with preK1st grade students with intellectual disabilities","volume":"52","author":"Taylor","year":"2018","journal-title":"J. Spec. Educ."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1177\/0162643417704439","article-title":"Computer programming with early elementary students with Down syndrome","volume":"32","author":"Taylor","year":"2017","journal-title":"J. Spec. Educ. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Fadhil, T.Z., and Mandeel, A.R. (2018, January 9\u201311). Live Monitoring System for Recognizing Varied Emotions of Autistic Children. Proceedings of the 2018 International Conference on Advanced Science and Engineering (ICOASE), Duhok, Iraq.","DOI":"10.1109\/ICOASE.2018.8548931"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/j.compbiomed.2016.04.001","article-title":"Monitoring of autonomic response to sociocognitive tasks during treatment in children with Autism Spectrum Disorders by wearable technologies: A feasibility study","volume":"85","author":"Tonacci","year":"2017","journal-title":"Comput. Biol. Med."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"662","DOI":"10.1016\/j.ijhsc.2008.04.003","article-title":"Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder","volume":"66","author":"Liu","year":"2008","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Nguyen, N.T., Nguyen, N.V., Tran, M.H.T., and Nguyen, B.T. (2017, January 19\u201321). A potential approach for emotion prediction using heart rate signals. Proceedings of the International Conference on Knowledge and Systems Engineering, Hue, Vietnam.","DOI":"10.1109\/KSE.2017.8119462"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Shu, L., Yu, Y., Chen, W., Hua, H., Li, Q., Jin, J., and Xu, X. (2020). Wearable emotion recognition using heart rate data from a smart bracelet. Sensors, 20.","DOI":"10.3390\/s20030718"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1186\/s40537-020-00401-x","article-title":"Multiclass emotion prediction using heart rate and virtual reality stimuli","volume":"8","author":"Bulagang","year":"2021","journal-title":"J. Big Data"},{"key":"ref_17","first-page":"53","article-title":"Identifying correlation between facial expression and heart rate and skin conductance with iMotions biometric platform","volume":"2","author":"Lei","year":"2017","journal-title":"J. Emerg. Forensic Sci. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1649","DOI":"10.3390\/s22041649","article-title":"Automatic emotion recognition in children with autism: A systematic literature review","volume":"22","author":"Lowska","year":"2022","journal-title":"Sensors"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Liu, C., Conn, K., Sarkar, N., and Stone, W. (2007, January 26\u201329). Online Affect Detection and Adaptation in Robot Assisted Rehabilitation for Children with Autism. Proceedings of the RO-MAN 2007\u2014The 16th IEEE International Symposium on Robot and Human Interactive Communication, Jeju, Republic of Korea.","DOI":"10.1109\/ROMAN.2007.4415153"},{"key":"ref_20","unstructured":"Ali, K., and Hughes, C.E. (2020, January 10\u201315). Facial Expression Recognition By Using a Disentangled Identity-Invariant Expression Representation. Proceedings of the International Conference on Pattern Recognition, Online."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kim, J.H., Poulose, A., and Han, D.S. (2021). The extensive usage of the facial image threshing machine for facial emotion recognition performance. Sensors, 21.","DOI":"10.3390\/s21062026"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1016\/j.ins.2021.10.005","article-title":"A survey on facial emotion recognition techniques: A state-of-the-art literature review","volume":"582","author":"Canal","year":"2022","journal-title":"Inf. Sci."},{"key":"ref_23","first-page":"5006631","article-title":"Understanding deep learning techniques for recognition of human emotions using facial expressions: A comprehensive survey","volume":"7","author":"Karnati","year":"2023","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"125538","DOI":"10.1109\/ACCESS.2022.3225684","article-title":"Deep Learning-Based Speech Emotion Recognition Using Multi-Level Fusion of Concurrent Features","volume":"10","author":"Kakuba","year":"2022","journal-title":"IEEE Access"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yan, Y., and Shen, X. (2022). Research on speech emotion recognition based on AA-CBGRU network. Electronics, 11.","DOI":"10.3390\/electronics11091409"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Lin, W., and Li, C. (2023). Review of Studies on Emotion Recognition and Judgment Based on Physiological Signals. Appl. Sci., 13.","DOI":"10.3390\/app13042573"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Pollreisz, D., and TaheriNejad, N. (2017, January 11\u201315). A simple algorithm for emotion recognition, using physiological signals of a smart watch. Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society, Jeju, Republic of Korea.","DOI":"10.1109\/EMBC.2017.8037328"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Liu, C., Conn, K., Sarkar, N., and Stone, W. (2007, January 10\u201314). Affect recognition in robot-assisted rehabilitation of children with autism spectrum disorder. Proceedings of the International Conference on Robotics and Automation, Rome, Italy.","DOI":"10.1109\/ROBOT.2007.363576"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Javed, H., Jeon, M., and Park, C.H. (2018, January 26\u201330). Adaptive framework for emotional engagement in child-robot interactions for autism interventions. Proceedings of the International Conference on Ubiquitous Robots, Honolulu, HI, USA.","DOI":"10.1109\/URAI.2018.8441775"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"eaao6760","DOI":"10.1126\/scirobotics.aao6760","article-title":"Personalized machine learning for robot perception of affect and engagement in autism therapy","volume":"3","author":"Rudovic","year":"2018","journal-title":"Sci. Robot."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1007\/s12369-017-0461-4","article-title":"Human\u2013Robot Facial Expression Reciprocal Interaction Platform: Case Studies on Children with Autism","volume":"10","author":"Pour","year":"2018","journal-title":"Soc. Robot."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Del Coco, M., Leo, M., Carcagn\u00ec, P., Spagnolo, P., Mazzeo, P.L., Bernava, M., Marino, F., Pioggia, G., and Distante, C. (2017, January 22\u201329). A Computer Vision Based Approach for Understanding Emotional Involvements in Children with Autism Spectrum Disorders. Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy.","DOI":"10.1109\/ICCVW.2017.166"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Leo, M., Del Coco, M., Carcagni, P., Distante, C., Bernava, M., Pioggia, G., and Palestra, G. (2015, January 7\u201315). Automatic Emotion Recognition in Robot-Children Interaction for ASD Treatment. Proceedings of the IEEE International Conference on Computer Vision Workshops, Santiago, Chile.","DOI":"10.1109\/ICCVW.2015.76"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Silva, V., Soares, F., and Esteves, J. (2017, January 16\u201318). Mirroring and recognizing emotions through facial expressions for a RoboKind platform. Proceedings of the IEEE 5th Portuguese Meeting on Bioengineering, Coimbra, Portugal.","DOI":"10.1109\/ENBENG.2017.7889480"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Guo, C., Zhang, K., Chen, J., Xu, R., and Gao, L. (2021, January 2\u20135). Design and application of facial expression analysis system in empathy ability of children with autism spectrum disorder. Proceedings of the Conference on Computer Science and Intelligence Systems, Online.","DOI":"10.15439\/2021F91"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Silva, V., Soares, F., Esteves, J.S., Santos, C.P., and Pereira, A.P. (2021). Fostering Emotion Recognition in Children with Autism Spectrum Disorder. Multimodal Technol. Interact., 5.","DOI":"10.3390\/mti5100057"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Barolli, L., Amato, F., Moscato, F., Enokido, T., and Takizawa, M. (2020). Web, Artificial Intelligence and Network Applications, Springer International Publishing.","DOI":"10.1007\/978-3-030-44038-1"},{"key":"ref_38","unstructured":"Li, J., Bhat, A., and Barmaki, R. (2021). A Two-stage Multi-Modal Affect Analysis Framework for Children with Autism Spectrum Disorder. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1186\/s13229-020-0312-2","article-title":"Children with autism spectrum disorder produce more ambiguous and less socially meaningful facial expressions: An experimental study using random forest classifiers","volume":"11","author":"Grossard","year":"2020","journal-title":"Mol. Autism"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Marinoiu, E., Zanfir, M., Olaru, V., and Sminchisescu, C. (2018, January 18\u201322). 3D Human Sensing, Action and Emotion Recognition in Robot Assisted Therapy of Children with Autism. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00230"},{"key":"ref_41","first-page":"159","article-title":"Emotion Recognition System for Autism Children using Non-verbal Communication","volume":"8","author":"Santhoshkumar","year":"2019","journal-title":"Innov. Technol. Explor. Eng"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"588","DOI":"10.1109\/TAFFC.2018.2820049","article-title":"Physiological Detection of Affective States in Children with Autism Spectrum Disorder","volume":"11","author":"Sarabadani","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"120818","DOI":"10.1109\/ACCESS.2020.3006004","article-title":"Implementation of Wavelet Analysis on Thermal Images for Affective States Recognition of Children With Autism Spectrum Disorder","volume":"8","author":"Rusli","year":"2020","journal-title":"IEEE Access"},{"key":"ref_44","first-page":"18","article-title":"Affectnet: A database for facial expression, valence, and arousal computing in the wild","volume":"37","author":"Mollahosseini","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1016\/j.jneumeth.2006.05.033","article-title":"Recovering EEG brain signals: Artifact suppression with wavelet enhanced independent component analysis","volume":"158","author":"Castellanos","year":"2006","journal-title":"J. Neurosci. Methods"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1016\/j.compbiomed.2006.08.013","article-title":"Long-term signal detection, segmentation and summarization using wavelets and fractal dimension: A bioacoustics application in gastrointestinal-motility monitoring","volume":"37","author":"Dimoulas","year":"2007","journal-title":"Comput. Biol. Med."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.21105\/joss.01237","article-title":"PyWavelets: A Python package for wavelet analysis","volume":"36","author":"Lee","year":"2019","journal-title":"Open Source Softw."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/14\/6572\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:16:25Z","timestamp":1760127385000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/14\/6572"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,21]]},"references-count":48,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["s23146572"],"URL":"https:\/\/doi.org\/10.3390\/s23146572","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,21]]}}}