{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T07:26:50Z","timestamp":1782286010746,"version":"3.54.5"},"publisher-location":"New York, NY, USA","reference-count":48,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T00:00:00Z","timestamp":1603065600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,20]]},"DOI":"10.1145\/3383652.3423916","type":"proceedings-article","created":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T22:10:31Z","timestamp":1603145431000},"page":"1-8","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Toward Automated Evaluation of Empathetic Responses in Virtual Human Interaction Systems for Mental Health Scenarios"],"prefix":"10.1145","author":[{"given":"Heng","family":"Yao","sequence":"first","affiliation":[{"name":"University of Florida, Gainesville, Florida"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandre Gomes","family":"de Siqueira","sequence":"additional","affiliation":[{"name":"University of Florida, Gainesville, Florida"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adriana","family":"Foster","sequence":"additional","affiliation":[{"name":"Florida International University, Miami, Florida"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Igor","family":"Galynker","sequence":"additional","affiliation":[{"name":"Mount Sinai Beth Israel, New York"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Lok","sequence":"additional","affiliation":[{"name":"University of Florida, Gainesville, Florida"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,10,19]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Emotional responses to suicidal patients: Factor structure, construct, and predictive validity of the Therapist Response Questionnaire-Suicide Form. Frontiers in psychiatry 9","author":"Barzilay Shira","year":"2018","unstructured":"Shira Barzilay , Zimri S Yaseen , Mariah Hawes , Bernard Gorman , Rachel Altman , Adriana Foster , Alan Apter , Paul Rosenfield , and Igor Galynker . 2018. Emotional responses to suicidal patients: Factor structure, construct, and predictive validity of the Therapist Response Questionnaire-Suicide Form. Frontiers in psychiatry 9 ( 2018 ), 104. Shira Barzilay, Zimri S Yaseen, Mariah Hawes, Bernard Gorman, Rachel Altman, Adriana Foster, Alan Apter, Paul Rosenfield, and Igor Galynker. 2018. Emotional responses to suicidal patients: Factor structure, construct, and predictive validity of the Therapist Response Questionnaire-Suicide Form. Frontiers in psychiatry 9 (2018), 104."},{"key":"e_1_3_2_1_2_1","volume-title":"KEER2014. Proceedings of the 5th Kanesi Engineering and Emotion Research","author":"Borish Michael","unstructured":"Michael Borish , Andrew Cordar , Adriana Foster , Thomas Kim , James Murphy , Neelam Chaudhary , and Benjamin Lok . 2014. Utilizing real-time human-assisted virtual humans to increase real-world interaction empathy . In KEER2014. Proceedings of the 5th Kanesi Engineering and Emotion Research ; International Conference; Link\u00f6ping; Sweden; June 11-13. Link\u00f6ping University Electronic Press , 441--455. Michael Borish, Andrew Cordar, Adriana Foster, Thomas Kim, James Murphy, Neelam Chaudhary, and Benjamin Lok. 2014. Utilizing real-time human-assisted virtual humans to increase real-world interaction empathy. In KEER2014. Proceedings of the 5th Kanesi Engineering and Emotion Research; International Conference; Link\u00f6ping; Sweden; June 11-13. Link\u00f6ping University Electronic Press, 441--455."},{"key":"e_1_3_2_1_3_1","volume-title":"Weka manual for version 3-6-0","author":"Bouckaert Remco R","year":"2008","unstructured":"Remco R Bouckaert , Eibe Frank , Mark Hall , Richard Kirkby , Peter Reutemann , Alex Seewald , and David Scuse . 2008. Weka manual for version 3-6-0 . University of Waikato , Hamilton, New Zealand ( 2008 ), 63. Remco R Bouckaert, Eibe Frank, Mark Hall, Richard Kirkby, Peter Reutemann, Alex Seewald, and David Scuse. 2008. Weka manual for version 3-6-0. University of Waikato, Hamilton, New Zealand (2008), 63."},{"key":"e_1_3_2_1_4_1","volume-title":"The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30, 7","author":"Bradley Andrew P","year":"1997","unstructured":"Andrew P Bradley . 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30, 7 ( 1997 ), 1145--1159. Andrew P Bradley. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30, 7 (1997), 1145--1159."},{"key":"e_1_3_2_1_5_1","volume-title":"Empathic communication and gender in the physician-patient encounter. Patient education and counseling 48, 3","author":"Bylund Carma L","year":"2002","unstructured":"Carma L Bylund and Gregory Makoul . 2002. Empathic communication and gender in the physician-patient encounter. Patient education and counseling 48, 3 ( 2002 ), 207--216. Carma L Bylund and Gregory Makoul. 2002. Empathic communication and gender in the physician-patient encounter. Patient education and counseling 48, 3 (2002), 207--216."},{"key":"e_1_3_2_1_6_1","volume-title":"Examining empathy in medical encounters: an observational study using the empathic communication coding system. Health communication 18, 2","author":"Bylund Carma L","year":"2005","unstructured":"Carma L Bylund and Gregory Makoul . 2005. Examining empathy in medical encounters: an observational study using the empathic communication coding system. Health communication 18, 2 ( 2005 ), 123--140. Carma L Bylund and Gregory Makoul. 2005. Examining empathy in medical encounters: an observational study using the empathic communication coding system. Health communication 18, 2 (2005), 123--140."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622407.1622416"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143874"},{"key":"e_1_3_2_1_9_1","volume-title":"Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. 1061--1068","author":"DeVault David","year":"2014","unstructured":"David DeVault , Ron Artstein , Grace Benn , Teresa Dey , Ed Fast , Alesia Gainer , Kallirroi Georgila , Jon Gratch , Arno Hartholt , Margaux Lhommet , 2014 . SimSensei Kiosk: A virtual human interviewer for healthcare decision support . In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. 1061--1068 . David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014. SimSensei Kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. 1061--1068."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/347090.347126"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1097\/SIH.0000000000000142"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1001\/archpsyc.1990.01810150028006"},{"key":"e_1_3_2_1_14_1","volume-title":"The suicidal crisis: clinical guide to the assessment of imminent suicide risk","author":"Galynker Igor","unstructured":"Igor Galynker and Igor I Galynker . 2017. The suicidal crisis: clinical guide to the assessment of imminent suicide risk . Oxford University Press . Igor Galynker and Igor I Galynker. 2017. The suicidal crisis: clinical guide to the assessment of imminent suicide risk. Oxford University Press."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/3110856"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/3110856"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-19721-5"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1656274.1656278"},{"key":"e_1_3_2_1_19_1","volume-title":"A comparison of methods for multiclass support vector machines","author":"Hsu Chih-Wei","year":"2002","unstructured":"Chih-Wei Hsu and Chih-Jen Lin . 2002. A comparison of methods for multiclass support vector machines . IEEE transactions on Neural Networks 13, 2 ( 2002 ), 415--425. Chih-Wei Hsu and Chih-Jen Lin. 2002. A comparison of methods for multiclass support vector machines. IEEE transactions on Neural Networks 13, 2 (2002), 415--425."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.50"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.5555\/645326.649721"},{"key":"e_1_3_2_1_22_1","unstructured":"Sascha Kaufmann. 1969. CUBA: Artificial conviviality and user-behaviour analysis in web-feeds. Ph.D. Dissertation. Citeseer.  Sascha Kaufmann. 1969. CUBA: Artificial conviviality and user-behaviour analysis in web-feeds. Ph.D. Dissertation. Citeseer."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.hpe.2017.03.006"},{"key":"e_1_3_2_1_24_1","volume-title":"The effects of physician empathy on patient satisfaction and compliance. Evaluation & the health professions 27, 3","author":"Kim Sung Soo","year":"2004","unstructured":"Sung Soo Kim , Stan Kaplowitz , and Mark V Johnston . 2004. The effects of physician empathy on patient satisfaction and compliance. Evaluation & the health professions 27, 3 ( 2004 ), 237--251. Sung Soo Kim, Stan Kaplowitz, and Mark V Johnston. 2004. The effects of physician empathy on patient satisfaction and compliance. Evaluation & the health professions 27, 3 (2004), 237--251."},{"key":"e_1_3_2_1_25_1","volume-title":"Understanding empathy training with virtual patients. Computers in human behavior 52","author":"Kleinsmith Andrea","year":"2015","unstructured":"Andrea Kleinsmith , Diego Rivera-Gutierrez , Glen Finney , Juan Cendan , and Benjamin Lok . 2015. Understanding empathy training with virtual patients. Computers in human behavior 52 ( 2015 ), 151--158. Andrea Kleinsmith, Diego Rivera-Gutierrez, Glen Finney, Juan Cendan, and Benjamin Lok. 2015. Understanding empathy training with virtual patients. Computers in human behavior 52 (2015), 151--158."},{"key":"e_1_3_2_1_26_1","volume-title":"Supervised machine learning: A review of classification techniques. Emerging artificial intelligence applications in computer engineering 160","author":"Kotsiantis Sotiris B","year":"2007","unstructured":"Sotiris B Kotsiantis , I Zaharakis , and P Pintelas . 2007. Supervised machine learning: A review of classification techniques. Emerging artificial intelligence applications in computer engineering 160 ( 2007 ), 3--24. Sotiris B Kotsiantis, I Zaharakis, and P Pintelas. 2007. Supervised machine learning: A review of classification techniques. Emerging artificial intelligence applications in computer engineering 160 (2007), 3--24."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.3390\/info10040150"},{"key":"e_1_3_2_1_28_1","volume-title":"Proceedings of the 20th International Conference on Machine Learning (ICML-03)","author":"Lachiche Nicolas","year":"2003","unstructured":"Nicolas Lachiche and Peter A Flach . 2003 . Improving accuracy and cost of two-class and multi-class probabilistic classifiers using ROC curves . In Proceedings of the 20th International Conference on Machine Learning (ICML-03) . 416--423. Nicolas Lachiche and Peter A Flach. 2003. Improving accuracy and cost of two-class and multi-class probabilistic classifiers using ROC curves. In Proceedings of the 20th International Conference on Machine Learning (ICML-03). 416--423."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1136\/jme.14.3.129"},{"key":"e_1_3_2_1_30_1","volume-title":"More than reflections: empathy in motivational interviewing includes language style synchrony between therapist and client. Behavior therapy 46, 3","author":"Lord Sarah Peregrine","year":"2015","unstructured":"Sarah Peregrine Lord , Elisa Sheng , Zac E Imel , John Baer , and David C Atkins . 2015. More than reflections: empathy in motivational interviewing includes language style synchrony between therapist and client. Behavior therapy 46, 3 ( 2015 ), 296--303. Sarah Peregrine Lord, Elisa Sheng, Zac E Imel, John Baer, and David C Atkins. 2015. More than reflections: empathy in motivational interviewing includes language style synchrony between therapist and client. Behavior therapy 46, 3 (2015), 296--303."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368567.3368584"},{"key":"e_1_3_2_1_32_1","unstructured":"Constituency Parsing. 2009. Speech and language processing. (2009).  Constituency Parsing. 2009. Speech and language processing. (2009)."},{"key":"e_1_3_2_1_33_1","volume-title":"Assessing the Critical Parameters of eSMART-MH: A Promising Avatar-Based Digital Therapeutic Intervention to Reduce Depressive Symptoms. Perspectives in psychiatric care 52, 3","author":"Pinto Melissa D","year":"2016","unstructured":"Melissa D Pinto , Amy M Greenblatt , Ronald L Hickman , Heather M Rice , Tami L Thomas , and John M Clochesy . 2016. Assessing the Critical Parameters of eSMART-MH: A Promising Avatar-Based Digital Therapeutic Intervention to Reduce Depressive Symptoms. Perspectives in psychiatric care 52, 3 ( 2016 ), 157--168. Melissa D Pinto, Amy M Greenblatt, Ronald L Hickman, Heather M Rice, Tami L Thomas, and John M Clochesy. 2016. Assessing the Critical Parameters of eSMART-MH: A Promising Avatar-Based Digital Therapeutic Intervention to Reduce Depressive Symptoms. Perspectives in psychiatric care 52, 3 (2016), 157--168."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb046814"},{"key":"e_1_3_2_1_35_1","volume-title":"The media equation: How people treat computers, television, and new media like real people and places","author":"Reeves Byron","unstructured":"Byron Reeves and Clifford Ivar Nass . 1996. The media equation: How people treat computers, television, and new media like real people and places . Cambridge university press . Byron Reeves and Clifford Ivar Nass. 1996. The media equation: How people treat computers, television, and new media like real people and places. Cambridge university press."},{"key":"e_1_3_2_1_36_1","volume-title":"Data mining with decision trees: theory and applications","author":"Rokach Lior","unstructured":"Lior Rokach and Oded Z Maimon . 2008. Data mining with decision trees: theory and applications . Vol. 69 . World scientific. Lior Rokach and Oded Z Maimon. 2008. Data mining with decision trees: theory and applications. Vol. 69. World scientific."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-006X.52.4.679"},{"key":"e_1_3_2_1_38_1","volume-title":"Use of empathy in psychiatric practice: constructivist grounded theory study. BJPsych open 3, 1","author":"Ross James","year":"2017","unstructured":"James Ross and Chris Watling . 2017. Use of empathy in psychiatric practice: constructivist grounded theory study. BJPsych open 3, 1 ( 2017 ), 26--33. James Ross and Chris Watling. 2017. Use of empathy in psychiatric practice: constructivist grounded theory study. BJPsych open 3, 1 (2017), 26--33."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2011.11.004"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1108\/14777260510615341"},{"key":"e_1_3_2_1_41_1","volume-title":"A survey of decision tree classifier methodology","author":"Rasoul Safavian S","year":"1991","unstructured":"S Rasoul Safavian and David Landgrebe . 1991. A survey of decision tree classifier methodology . IEEE transactions on systems, man, and cybernetics 21, 3 ( 1991 ), 660--674. S Rasoul Safavian and David Landgrebe. 1991. A survey of decision tree classifier methodology. IEEE transactions on systems, man, and cybernetics 21, 3 (1991), 660--674."},{"key":"e_1_3_2_1_42_1","unstructured":"Simran Sandhu Shira Barzilay Heng Yao Aniuska Luna Carmen Rodriguez Benjamin Lok Adriana Foster and Igor Galynker. 2019. Using Virtual Patient Interaction to Connect with Patients at Risk of Suicide https:\/\/doi.org\/10.13140\/ RG.2.2.36066.68803  Simran Sandhu Shira Barzilay Heng Yao Aniuska Luna Carmen Rodriguez Benjamin Lok Adriana Foster and Igor Galynker. 2019. Using Virtual Patient Interaction to Connect with Patients at Risk of Suicide https:\/\/doi.org\/10.13140\/ RG.2.2.36066.68803"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1097\/NMD.0000000000000187"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjsurg.2006.03.002"},{"key":"e_1_3_2_1_45_1","volume-title":"Multimodal Learning for Identifying Opportunities for Empathetic Responses. In 2019 International Conference on Multimodal Interaction. 95--104","author":"Tavabi Leili","year":"2019","unstructured":"Leili Tavabi , Kalin Stefanov , Setareh Nasihati Gilani , David Traum , and Mohammad Soleymani . 2019 . Multimodal Learning for Identifying Opportunities for Empathetic Responses. In 2019 International Conference on Multimodal Interaction. 95--104 . Leili Tavabi, Kalin Stefanov, Setareh Nasihati Gilani, David Traum, and Mohammad Soleymani. 2019. Multimodal Learning for Identifying Opportunities for Empathetic Responses. In 2019 International Conference on Multimodal Interaction. 95--104."},{"key":"e_1_3_2_1_46_1","volume-title":"Defining an optimal cut-point value in ROC analysis: an alternative approach. Computational and mathematical methods in medicine 2017","author":"Unal Ilker","year":"2017","unstructured":"Ilker Unal . 2017. Defining an optimal cut-point value in ROC analysis: an alternative approach. Computational and mathematical methods in medicine 2017 ( 2017 ). Ilker Unal. 2017. Defining an optimal cut-point value in ROC analysis: an alternative approach. Computational and mathematical methods in medicine 2017 (2017)."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2019.8673177"},{"key":"e_1_3_2_1_48_1","volume-title":"Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 1--4.","author":"Xiao Bo","year":"2012","unstructured":"Bo Xiao , Dogan Can , Panayiotis G Georgiou , David Atkins , and Shrikanth S Narayanan . 2012 . Analyzing the language of therapist empathy in motivational interview based psychotherapy . In Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 1--4. Bo Xiao, Dogan Can, Panayiotis G Georgiou, David Atkins, and Shrikanth S Narayanan. 2012. Analyzing the language of therapist empathy in motivational interview based psychotherapy. In Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 1--4."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0143055"}],"event":{"name":"IVA '20: ACM International Conference on Intelligent Virtual Agents","location":"Virtual Event Scotland UK","acronym":"IVA '20","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence"]},"container-title":["Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383652.3423916","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3383652.3423916","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:57Z","timestamp":1750195917000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383652.3423916"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,19]]},"references-count":48,"alternative-id":["10.1145\/3383652.3423916","10.1145\/3383652"],"URL":"https:\/\/doi.org\/10.1145\/3383652.3423916","relation":{},"subject":[],"published":{"date-parts":[[2020,10,19]]},"assertion":[{"value":"2020-10-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}