{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:19:58Z","timestamp":1760235598303,"version":"build-2065373602"},"reference-count":57,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2021,8,30]],"date-time":"2021-08-30T00:00:00Z","timestamp":1630281600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004663","name":"Ministry of Science and Technology, Taiwan","doi-asserted-by":"publisher","award":["MOST 110-2634-F-002-040 & MOST 110-2634-F-002-042"],"award-info":[{"award-number":["MOST 110-2634-F-002-040 & MOST 110-2634-F-002-042"]}],"id":[{"id":"10.13039\/501100004663","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Taiwan University","award":["NTU-CC-110L9A00703 & NTU-CC-110L9A00704"],"award-info":[{"award-number":["NTU-CC-110L9A00703 & NTU-CC-110L9A00704"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Mental health is as crucial as physical health, but it is underappreciated by mainstream biomedical research and the public. Compared to the use of AI or robots in physical healthcare, the use of AI or robots in mental healthcare is much more limited in number and scope. To date, psychological resilience\u2014the ability to cope with a crisis and quickly return to the pre-crisis state\u2014has been identified as an important predictor of psychological well-being but has not been commonly considered by AI systems (e.g., smart wearable devices) or social robots to personalize services such as emotion coaching. To address the dearth of investigations, the present study explores the possibility of estimating personal resilience using physiological and speech signals measured during human\u2013robot conversations. Specifically, the physiological and speech signals of 32 research participants were recorded while the participants answered a humanoid social robot\u2019s questions about their positive and negative memories about three periods of their lives. The results from machine learning models showed that heart rate variability and paralinguistic features were the overall best predictors of personal resilience. Such predictability of personal resilience can be leveraged by AI and social robots to improve user understanding and has great potential for various mental healthcare applications in the future.<\/jats:p>","DOI":"10.3390\/s21175844","type":"journal-article","created":{"date-parts":[[2021,8,31]],"date-time":"2021-08-31T22:58:15Z","timestamp":1630450695000},"page":"5844","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Personal Resilience Can Be Well Estimated from Heart Rate Variability and Paralinguistic Features during Human\u2013Robot Conversations"],"prefix":"10.3390","volume":"21","author":[{"given":"Shin-Min","family":"Hsu","sequence":"first","affiliation":[{"name":"Department of Psychology, National Taiwan University, Taipei 10617, Taiwan"},{"name":"MOST Joint Research Center for AI Technology and All Vista Healthcare, Taipei 10617, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sue-Huei","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Psychology, National Taiwan University, Taipei 10617, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4396-7943","authenticated-orcid":false,"given":"Tsung-Ren","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Psychology, National Taiwan University, Taipei 10617, Taiwan"},{"name":"MOST Joint Research Center for AI Technology and All Vista Healthcare, Taipei 10617, Taiwan"},{"name":"MOST AI Biomedical Research Center, Tainan 70101, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1036","DOI":"10.1176\/ps.50.8.1036","article-title":"Life expectancy and causes of death in a population treated for serious mental illness","volume":"50","author":"Dembling","year":"1999","journal-title":"Psychiatr. Serv."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"2110","DOI":"10.1016\/S0140-6736(01)07216-6","article-title":"Death by homicide, suicide, and other unnatural causes in people with mental illness: A population-based study","volume":"358","author":"Hiroeh","year":"2001","journal-title":"Lancet"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1016\/j.paid.2014.11.039","article-title":"A meta-analysis of the trait resilience and mental health","volume":"76","author":"Hu","year":"2015","journal-title":"Personal. Individ. Differ."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1023","DOI":"10.1016\/S0006-3223(01)01157-X","article-title":"The role of childhood trauma in the neurobiology of mood and anxiety disorders: Preclinical and clinical studies","volume":"49","author":"Heim","year":"2001","journal-title":"Biol. Psychiatry"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1111\/j.1600-0447.2009.01491.x","article-title":"Childhood life events and childhood trauma in adult patients with depressive, anxiety and comorbid disorders vs. controls","volume":"122","author":"Hovens","year":"2010","journal-title":"Acta Psychiatr. Scand."},{"key":"ref_6","first-page":"1","article-title":"Risk and resilience factors of post-traumatic stress disorder: A review of current research","volume":"2","author":"Lauth","year":"2016","journal-title":"Clin. Exp. Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1037\/0003-066X.56.3.227","article-title":"Ordinary magic: Resilience processes in development","volume":"56","author":"Masten","year":"2001","journal-title":"Am. Psychol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1037\/0003-066X.59.1.20","article-title":"Loss, trauma, and human resilience: Have we underestimated the human capacity to thrive after extremely aversive events?","volume":"59","author":"Bonanno","year":"2004","journal-title":"Am. Psychol."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Fletcher, D., and Sarkar, M. (2013). Psychological resilience. Eur. Psychol., 18.","DOI":"10.1027\/1016-9040\/a000124"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Nishimi, K., Choi, K.W., Cerutti, J., Powers, A., Bradley, B., and Dunn, E.C. (2020). Measures of adult psychological resilience following early-life adversity: How congruent are different measures?. Psychol. Med., 1\u201310.","DOI":"10.1017\/S0033291720001191"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1016\/j.cpr.2010.03.003","article-title":"Resilience and mental health","volume":"30","author":"Davydov","year":"2010","journal-title":"Clin. Psychol. Rev."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1770","DOI":"10.1038\/s41380-019-0457-6","article-title":"Genomics and psychological resilience: A research agenda","volume":"24","author":"Choi","year":"2019","journal-title":"Mol. Psychiatry"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1109\/TAFFC.2014.2330816","article-title":"A Survey of Personality Computing","volume":"5","author":"Vinciarelli","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_14","unstructured":"Silveira Jacques Junior, J.C., Gucluturk, Y., Perez, M., Guclu, U., Andujar, C., Baro, X., Escalante, H.J., Guyon, I., Van Gerven, M.A.J., and Van Lier, R. (2019). First Impressions: A Survey on Vision-based Apparent Personality Trait Analysis. IEEE Trans. Affect. Comput., 1\u201320."},{"key":"ref_15","first-page":"1","article-title":"AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups","volume":"12","author":"Abadi","year":"2021","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1109\/TAFFC.2016.2625250","article-title":"ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors","volume":"9","author":"Subramanian","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_17","unstructured":"Scherer, K.R., and Giles, H. (1979). Social Markers in Speech, Cambridge University Press."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1613\/jair.2349","article-title":"Using linguistic cues for the automatic recognition of personality in conversation and text","volume":"30","author":"Mairesse","year":"2007","journal-title":"J. Artif. Intell. Res."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1146\/annurev.psych.54.101601.145041","article-title":"Psychological aspects of natural language use: Our words, our selves","volume":"54","author":"Pennebaker","year":"2003","journal-title":"Annu. Rev. Psychol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1016\/j.jrp.2010.04.001","article-title":"Personality in 100,000 Words: A large-scale analysis of personality and word use among bloggers","volume":"44","author":"Yarkoni","year":"2010","journal-title":"J. Res. Personal."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1016\/j.neubiorev.2016.05.003","article-title":"In the search for integrative biomarker of resilience to psychological stress","volume":"74","author":"Walker","year":"2017","journal-title":"Neurosci. Biobehav. Rev."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"104506","DOI":"10.1016\/j.childyouth.2019.104506","article-title":"Identifying psychological resilience in Chinese migrant youth through multidisciplinary language pattern decoding","volume":"107","author":"Li","year":"2019","journal-title":"Child. Youth Serv. Rev."},{"key":"ref_23","first-page":"e199","article-title":"Houston Strong: Linguistic markers of resilience after Hurricane Harvey","volume":"9","author":"Marshall","year":"2020","journal-title":"J. Trauma. Stress Disord. Treat."},{"key":"ref_24","first-page":"32","article-title":"Personality and heart rate variability: Exploring pathways from personality to cardiac coherence and health","volume":"1","author":"Zohar","year":"2013","journal-title":"Open J. Soc. Sci."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1177\/1524838005277438","article-title":"Trauma, PTSD, and resilience: A review of the literature","volume":"6","author":"Agaibi","year":"2005","journal-title":"Trauma Violence Abus."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.jpsychores.2005.06.002","article-title":"Preliminary evidence of parasympathetic influence on basal heart rate in posttraumatic stress disorder","volume":"60","author":"Hopper","year":"2006","journal-title":"J. Psychosom. Res."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.ijpsycho.2010.04.006","article-title":"Depression, heart rate related variables and cardiovascular disease","volume":"78","author":"Taylor","year":"2010","journal-title":"Int. J. Psychophysiol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1016\/S0193-953X(02)00019-9","article-title":"Rationale for a posttraumatic stress spectrum disorder","volume":"25","author":"Moreau","year":"2002","journal-title":"Psychiat Clin. N. Am."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1111\/j.1460-2466.2002.tb02559.x","article-title":"Research on the relationship between verbal and nonverbal communication: Emerging integrations","volume":"52","author":"Jones","year":"2002","journal-title":"J. Commun."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Johar, S. (2015). Emotion, Affect and Personality in Speech: The Bias of Language and Paralanguage, Springer.","DOI":"10.1007\/978-3-319-28047-9"},{"key":"ref_31","unstructured":"Zhang, T., and Kuo, C.C.J. (2013). Content-Based Audio Classification and Retrieval for Audiovisual Data Parsing, Kluwer Academic."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1296","DOI":"10.1037\/0022-3514.77.6.1296","article-title":"Linguistic styles: Language use as an individual difference","volume":"77","author":"Pennebaker","year":"1999","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_33","first-page":"343","article-title":"The psychological functions of function words","volume":"1","author":"Chung","year":"2007","journal-title":"Soc. Commun."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1177\/002383098202500201","article-title":"On the differences between spoken and written language","volume":"25","author":"Akinnaso","year":"1982","journal-title":"Lang. Speech"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1080\/10705500802222972","article-title":"The brief resilience scale: Assessing the ability to bounce back","volume":"15","author":"Smith","year":"2008","journal-title":"Int J. Behav. Med."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1002\/da.10113","article-title":"Development of a new resilience scale: The Connor-Davidson Resilience Scale (CD-RISC)","volume":"18","author":"Connor","year":"2003","journal-title":"Depress. Anxiety"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1002\/mpr.15","article-title":"Resilience in relation to personality and intelligence","volume":"14","author":"Friborg","year":"2005","journal-title":"Int. J. Methods Psychiatr. Res."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1186\/1477-7525-9-8","article-title":"A methodological review of resilience measurement scales","volume":"9","author":"Windle","year":"2011","journal-title":"Health Qual. Life Outcomes"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"793","DOI":"10.1111\/j.1467-6494.2005.00329.x","article-title":"Continuity and correlates of emotions and motives in self-defining memories","volume":"73","author":"Sutin","year":"2005","journal-title":"J. Personal."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1689","DOI":"10.3758\/s13428-020-01516-y","article-title":"NeuroKit2: A python toolbox for neurophysiological signal processing","volume":"53","author":"Makowski","year":"2021","journal-title":"Behav. Res. Methods"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Gabrieli, G., Azhari, A., and Esposito, G. (2020). PySiology: A python package for physiological feature extraction. Neural Approaches to Dynamics of Signal Exchanges, Springer.","DOI":"10.1007\/978-981-13-8950-4_35"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Schuller, B., Steidl, S., Batliner, A., Vinciarelli, A., Scherer, K., Ringeval, F., Chetouani, M., Weninger, F., Eyben, F., and Marchi, E. (2013, January 25\u201329). The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism. Proceedings of the 14th Annual Conference of the International Speech Communication Association, Lyon, France.","DOI":"10.21437\/Interspeech.2013-56"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Eyben, F., W\u00f6llmer, M., and Schuller, B. (2010, January 25\u201329). Opensmile: The munich versatile and fast open-source audio feature extractor. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.","DOI":"10.1145\/1873951.1874246"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1109\/TAFFC.2015.2457417","article-title":"The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing","volume":"7","author":"Eyben","year":"2015","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Li, P.-H., Fu, T.-J., and Ma, W.-Y. (2020, January 7\u201312). Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i05.6338"},{"key":"ref_46","unstructured":"Kutner, M.H., Nachtsheim, C.J., Neter, J., and Li, W. (2005). Applied Linear Statistical Models, McGraw-Hill."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.paid.2018.01.048","article-title":"Resilience and Big Five personality traits: A meta-analysis","volume":"127","author":"Oshio","year":"2018","journal-title":"Personal. Individ. Differ."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1037\/a0013342","article-title":"Chronic psychosocial factors and acute physiological responses to laboratory-induced stress in healthy populations: A quantitative review of 30 years of investigations","volume":"134","author":"Chida","year":"2008","journal-title":"Psychol. Bull."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1000","DOI":"10.1016\/j.jelectrocard.2015.08.005","article-title":"ECG signatures of psychological stress","volume":"48","author":"Lampert","year":"2015","journal-title":"J. Electrocardiol."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1111\/j.1469-8986.2007.00551.x","article-title":"Neuroticism is associated with larger and more prolonged electrodermal responses to emotionally evocative pictures","volume":"44","author":"Norris","year":"2007","journal-title":"Psychophysiology"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1016\/j.jvoice.2006.03.002","article-title":"Acoustical correlates of affective prosody","volume":"21","author":"Harnmerschmidt","year":"2007","journal-title":"J. Voice"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1037\/0022-3514.84.2.365","article-title":"What good are positive emotions in crises? A prospective study of resilience and emotions following the terrorist attacks on the United States on September 11th, 2001","volume":"84","author":"Fredrickson","year":"2003","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_53","unstructured":"Pennebaker, J.W., Boyd, R.L., Jordan, K., and Blackburn, K. (2015). The Development and Psychometric Properties of LIWC2015, The University of Texas at Austin."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1177\/0261927X09351676","article-title":"The psychological meaning of words: LIWC and computerized text analysis methods","volume":"29","author":"Tausczik","year":"2010","journal-title":"J. Lang. Soc. Psychol."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1037\/0022-3514.86.2.320","article-title":"Resilient individuals use positive emotions to bounce back from negative emotional experiences","volume":"86","author":"Tugade","year":"2004","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1037\/a0021786","article-title":"Flexible Emotional Responsiveness in Trait Resilience","volume":"11","author":"Waugh","year":"2011","journal-title":"Emotion"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1016\/j.cpr.2010.12.007","article-title":"Resilience to suicidality: The buffering hypothesis","volume":"31","author":"Johnson","year":"2011","journal-title":"Clin. Psychol. Rev."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/17\/5844\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:55:56Z","timestamp":1760165756000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/17\/5844"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,30]]},"references-count":57,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["s21175844"],"URL":"https:\/\/doi.org\/10.3390\/s21175844","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,8,30]]}}}