{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T12:43:20Z","timestamp":1775047400930,"version":"3.50.1"},"reference-count":38,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2013,8,13]],"date-time":"2013-08-13T00:00:00Z","timestamp":1376352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>We describe the work on infusion of emotion into a limited-task autonomous spoken conversational agent situated in the domestic environment, using a need-inspired task-independent emotion model (NEMO). In order to demonstrate the generation of affect through the use of the model, we describe the work of integrating it with a natural-language mixed-initiative HiFi-control spoken conversational agent (SCA). NEMO and the host system communicate externally, removing the need for the Dialog Manager to be modified, as is done in most existing dialog systems, in order to be adaptive. The first part of the paper concerns the integration between NEMO and the host agent. The second part summarizes the work on automatic affect prediction, namely, frustration and contentment, from dialog features, a non-conventional source, in the attempt of moving towards a more user-centric approach. The final part reports the evaluation results obtained from a user study, in which both versions of the agent (non-adaptive and emotionally-adaptive) were compared. The results provide substantial evidences with respect to the benefits of adding emotion in a spoken conversational agent, especially in mitigating users\u2019 frustrations and, ultimately, improving their satisfaction.<\/jats:p>","DOI":"10.3390\/s130810519","type":"journal-article","created":{"date-parts":[[2013,8,13]],"date-time":"2013-08-13T11:55:07Z","timestamp":1376394907000},"page":"10519-10538","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["I Feel You: The Design and Evaluation of a Domotic Affect-Sensitive Spoken Conversational Agent"],"prefix":"10.3390","volume":"13","author":[{"given":"Syaheerah","family":"Lutfi","sequence":"first","affiliation":[{"name":"School of Computer Sciences, University Science of Malaysia, Penang 11800, Malaysia"}]},{"given":"Fernando","family":"Fern\u00e1ndez-Mart\u00ednez","sequence":"additional","affiliation":[{"name":"Departamento de Teor\u00eda de la Se\u00f1al y Comunicaciones, Universidad Carlos III de Madrid, Madrid 28911, Spain"}]},{"given":"Jaime","family":"Lorenzo-Trueba","sequence":"additional","affiliation":[{"name":"Grupo de Tecnolog\u00eda del Habla, Universidad Polit\u00e9cnica de Madrid, Madrid 28040, Spain"}]},{"given":"Roberto","family":"Barra-Chicote","sequence":"additional","affiliation":[{"name":"Grupo de Tecnolog\u00eda del Habla, Universidad Polit\u00e9cnica de Madrid, Madrid 28040, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7908-5400","authenticated-orcid":false,"given":"Juan","family":"Montero","sequence":"additional","affiliation":[{"name":"Grupo de Tecnolog\u00eda del Habla, Universidad Polit\u00e9cnica de Madrid, Madrid 28040, Spain"}]}],"member":"1968","published-online":{"date-parts":[[2013,8,13]]},"reference":[{"key":"ref_1","unstructured":"Lutfi, S., Barra-Chicote, R., Lucas-Cuesta, J., and Montero, J. (July, January 14\u2013accessed). NEMO: Need-Inspired Emotional Expressions within a Task-Independent Framework. Madrid, Spain."},{"key":"ref_2","first-page":"1069","article-title":"On the dynamic adaptation of language models based on dialogue information","volume":"40","author":"Ferreiros","year":"2012","journal-title":"Expect Syst. Appl."},{"key":"ref_3","first-page":"1539","article-title":"A context model based on ontological languages: A proposal for information visualization. J","volume":"16","author":"Bravo","year":"2010","journal-title":"Univ. Comput. Sci."},{"key":"ref_4","first-page":"1500","article-title":"An agent-based architecture for developing activity-aware systems for assisting elderly","volume":"16","author":"Tentori","year":"2010","journal-title":"J. Univ. Comput. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"6802","DOI":"10.3390\/s120606802","article-title":"Meeting people's needs in a fully interoperable domotic environment","volume":"12","author":"Miori","year":"2012","journal-title":"Sensors"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"9807","DOI":"10.1016\/j.eswa.2012.02.173","article-title":"Text categorization methods for automatic estimation of verbal intelligence","volume":"39","author":"Zablotskaya","year":"2012","journal-title":"Expect Syst. Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1007\/s11257-007-9037-6","article-title":"Automatic detection of learner's affect from conversational cues","volume":"18","author":"Craig","year":"2008","journal-title":"User Model User Adap. Interact."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.csl.2009.12.002","article-title":"Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system","volume":"25","author":"Litman","year":"2011","journal-title":"Comput. Speech Lang."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1115","DOI":"10.1016\/j.specom.2011.02.006","article-title":"Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor","volume":"53","author":"Litman","year":"2011","journal-title":"Speech Commun."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"490","DOI":"10.1007\/978-3-540-69132-7_52","article-title":"Affective transitions in narrative-centered learning environments","volume":"5091","author":"Mcquiggan","year":"2008","journal-title":"Lect. Note. Comput. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1007\/978-3-642-04380-2_36","article-title":"Predicting user psychological characteristics from interactions with empathetic virtual agents","volume":"5773","author":"Robison","year":"2009","journal-title":"Lect. Note. Comput. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1007\/978-3-540-74889-2_10","article-title":"Does body movement engage you more in digital game play? and why?","volume":"4738","author":"Kim","year":"2007","journal-title":"Lect. Note. Comput. Sci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1016\/j.csl.2009.12.004","article-title":"Detecting emotional state of a child in a conversational computer game","volume":"25","author":"Yildirim","year":"2011","journal-title":"Comput. Speech Lang."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1016\/j.aei.2006.11.009","article-title":"Anxiety-based affective communication for implicit human-machine interaction","volume":"21","author":"Rani","year":"2007","journal-title":"Adv. Eng. Inform."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/S0953-5438(01)00053-4","article-title":"This computer responds to user frustration: Theory, design and results","volume":"14","author":"Klein","year":"2002","journal-title":"Interact. Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.csl.2010.03.004","article-title":"Expression of affect in spontaneous speech: Acoustic correlates and automatic detection of irritation and resignation","volume":"25","author":"Laukka","year":"2011","journal-title":"Comput. Speech Lang."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1016\/j.specom.2008.01.001","article-title":"Influence of contextual information in emotion annotation for spoken dialogue systems","volume":"50","author":"Callejas","year":"2008","journal-title":"Speech Commun."},{"key":"ref_18","unstructured":"L\u00f3pez-C\u00f3zar, R., Silovsky, J., and Griol, D. (2010, January 24\u201325). New Technique for Recognition of User Emotional States in Spoken Dialog Systems. Tokyo, Japan."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Payr, S., and Trappl, R. (2004). Agent Culture: Human-Agent Interaction in a Multicultural World, Lawrence Erlbaum Associates.","DOI":"10.1201\/b12476"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1007\/978-3-540-74853-3_9","article-title":"My roomba is rambo: Intimate home appliances","volume":"4717","author":"Sung","year":"2007","journal-title":"Lect. Note. Comput. Sci."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"630","DOI":"10.1016\/j.specom.2008.04.002","article-title":"Towards human-like spoken dialogue systems","volume":"50","author":"Edlund","year":"2008","journal-title":"Speech Commun."},{"key":"ref_22","unstructured":"Reeves, B., and Nass, C. (1996). The Media Equation: How People Treat Computers, Television and New Media Like Real People and Places, CSLI Publications."},{"key":"ref_23","unstructured":"Fern\u00e1ndez-Mart\u00ednez, F., Ferreiros, J., Lucas-Cuesta, J.M., Echeverry, J.D., San-Segundo, R., and C\u00f3rdoba, R. (September, January 30). Flexible, Robust and Dynamic Dialogue Modeling with a Speech Dialogue Interface for Controlling a Hi-Fi Audio System. Bilbao, Spain,."},{"key":"ref_24","unstructured":"Sanz-Moreno, C., Lutfi, S., Barra-Chicote, R., Lucas-Cuesta, J., and Montero, J. (2009, January 24\u201326). Desarrollo de un asistente dom\u00f3tico emocional inteligente. Madrid, Spain."},{"key":"ref_25","unstructured":"Lorenzo-Trueba, J., Watts, O., Barra-Chicote, R., Yamagishi, J., King, S., and Montero, J.M. (2012, January 21\u201323). Simple4All proposals for the Albayzin Evaluations in Speech Synthesis. Madrid, Spain."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ekman, P., and Friesen, W. (1978). The Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press.","DOI":"10.1037\/t27734-000"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"825","DOI":"10.1016\/j.specom.2013.04.005","article-title":"A satisfaction-based model for affect recognition from conversational features in spoken dialog systems","volume":"55","author":"Lutfi","year":"2013","journal-title":"Speech Commun."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1287\/mnsc.29.5.530","article-title":"Development of a tool for measuring and analyzing computer user satisfaction","volume":"24","author":"Bailey","year":"1983","journal-title":"Manag. Sci."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"259","DOI":"10.2307\/248851","article-title":"The measurement of end-user computing satisfaction","volume":"12","author":"Doll","year":"1988","journal-title":"MIS Q."},{"key":"ref_30","unstructured":"Locke, E.A. (1976). The Nature and Causes of Job Satisfaction, Consulting Psychologists Press."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1007\/BF03396779","article-title":"Beyond just being dissatisfied: How angry and helpless customers react to failures when using self-service technologies","volume":"61","author":"Gelbrich","year":"2009","journal-title":"Schmalenbach Bus. Rev."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1108\/08876040510625945","article-title":"The impact of service provider emotional intelligence on customer satisfaction","volume":"19","author":"Kernbach","year":"2005","journal-title":"J. Serv. Market."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez-Mart\u00ednez, F., Bl\u00e1zquez, J., Ferreiros, J., Barra-Chicote, R., Macias-Guarasa, J., and Lucas-Cuesta, J.M. (2008., January 15\u201319). Evaluation of a Spoken Dialog System for controlling a HiFi Audio System. Goa, India.","DOI":"10.1109\/SLT.2008.4777859"},{"key":"ref_34","unstructured":"Fern\u00e1ndez-Mart\u00ednez, F., Lucas-Cuesta, J.M., Chicote, R.B., Ferreiros, J., and Mac\u00edas-Guarasa, J. (2010, January 17\u201323). HIFI-AV: An Audio-Visual Corpus for Spoken Language Human-Machine Dialogue Research in Spanish. Valletta, Malta."},{"key":"ref_35","unstructured":"Witten, I.H., and Frank, E. (2005). Data Mining: Practical Machine Learning Tools and Techniques, Morgan-Kaufmann."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1007\/978-3-540-69369-7_25","article-title":"On the use of aappa coefficients to measure the reliability of the annotation of non-acted emotions","volume":"5078","author":"Callejas","year":"2008","journal-title":"Lect. Note. Comput. Sci."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Field, A. (2005). Discovering Statistics Using SPSS, SAGE. [2nd ed.].","DOI":"10.53841\/bpspag.2005.1.56.31"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1016\/j.specom.2010.11.006","article-title":"Achieving rapport with turn-by-turn, user-responsive emotional colouring","volume":"53","author":"Acosta","year":"2011","journal-title":"Speech Commun."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/13\/8\/10519\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:48:36Z","timestamp":1760219316000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/13\/8\/10519"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,8,13]]},"references-count":38,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2013,8]]}},"alternative-id":["s130810519"],"URL":"https:\/\/doi.org\/10.3390\/s130810519","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,8,13]]}}}