{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T04:18:33Z","timestamp":1777522713009,"version":"3.51.4"},"reference-count":70,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2014,9,19]],"date-time":"2014-09-19T00:00:00Z","timestamp":1411084800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Adaptive Behavior"],"published-print":{"date-parts":[[2014,10]]},"abstract":"<jats:p>In this paper, we investigate the use of emotional information in the learning process of autonomous agents. Inspired by four dimensions that are commonly postulated by appraisal theories of emotions, we construct a set of reward features to guide the learning process and behaviour of a reinforcement learning (RL) agent that inhabits an environment of which it has only limited perception. Much like what occurs in biological agents, each reward feature evaluates a particular aspect of the (history of) interaction of the agent history with the environment, thereby, in a sense, replicating some aspects of appraisal processes observed in humans and other animals. Our experiments in several foraging scenarios demonstrate that by optimising the relative contributions of each reward feature, the resulting \u201cemotional\u201d RL agents perform better than standard goal-oriented agents, particularly in consideration of their inherent perceptual limitations. Our results support the claim that biological evolutionary adaptive mechanisms such as emotions can provide crucial clues in creating robust, general-purpose reward mechanisms for autonomous artificial agents, thereby allowing them to overcome some of the challenges imposed by their inherent limitations.<\/jats:p>","DOI":"10.1177\/1059712314543837","type":"journal-article","created":{"date-parts":[[2014,9,19]],"date-time":"2014-09-19T22:48:25Z","timestamp":1411166905000},"page":"330-349","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":22,"title":["Learning by appraising: an emotion-based approach to intrinsic reward design"],"prefix":"10.1177","volume":"22","author":[{"given":"Pedro","family":"Sequeira","sequence":"first","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa, Portugal"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Francisco S","family":"Melo","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa, Portugal"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ana","family":"Paiva","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa, Portugal"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2014,9,19]]},"reference":[{"key":"bibr1-1059712314543837","volume-title":"A (revised) survey of approximate methods for solving partially observable Markov decision processes","author":"Aberdeen D.","year":"2003"},{"key":"bibr2-1059712314543837","first-page":"1","volume-title":"Proceedings of the 18th European meeting on cybernetics and systems research","author":"Ahn H.","year":"2006"},{"key":"bibr3-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/S1364-6613(97)01007-3"},{"key":"bibr4-1059712314543837","volume-title":"Emotion and personality","author":"Arnold M.","year":"1960"},{"key":"bibr5-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1023\/A:1013689704352"},{"key":"bibr6-1059712314543837","unstructured":"Bobrow G. (1964). Natural language input for a computer problem solving system. PhD Thesis, Massachusetts Institute of Technology, MA."},{"key":"bibr7-1059712314543837","first-page":"213","volume":"3","author":"Brafman R.","year":"2003","journal-title":"Journal of Machine Learning Research"},{"key":"bibr8-1059712314543837","first-page":"407","volume-title":"11th international joint conference on autonomous agents and multiagent systems","author":"Bratman J.","year":"2012"},{"key":"bibr9-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-73053-8_36"},{"key":"bibr10-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1145\/267658.267688"},{"key":"bibr11-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/S0149-7634(02)00007-6"},{"key":"bibr12-1059712314543837","unstructured":"Cassandra A. (1998). Exact and approximate algorithms for partially observable Markov decision processes. PhD Thesis, Brown University, RI."},{"key":"bibr13-1059712314543837","volume-title":"Descartes\u2019 error: Emotion, reason, and the human brain","author":"Damasio A.","year":"1994"},{"key":"bibr14-1059712314543837","first-page":"883","volume":"40","author":"Dawkins M.","year":"2000","journal-title":"American Zoologist"},{"key":"bibr15-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010030809960"},{"key":"bibr16-1059712314543837","volume-title":"Handbook of the affective sciences","author":"Ellsworth P.","year":"2003"},{"key":"bibr17-1059712314543837","first-page":"503","volume":"6","author":"Ernst D.","year":"2005","journal-title":"Journal of Machine Learning Research"},{"key":"bibr18-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-1939-7_11"},{"key":"bibr19-1059712314543837","first-page":"385","volume":"4","author":"Gadanho S.","year":"2003","journal-title":"Journal of Machine Learning Research"},{"key":"bibr20-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-73055-2_52"},{"key":"bibr21-1059712314543837","first-page":"345","volume-title":"Advances in neural information systems","volume":"7","author":"Jaakkola T.","year":"1995"},{"key":"bibr22-1059712314543837","author":"Jacobs E.","year":"2014","journal-title":"AAMAS workshop on adaptive learning agents"},{"key":"bibr23-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(98)00023-X"},{"key":"bibr24-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1613\/jair.301"},{"key":"bibr25-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1023\/A:1017984413808"},{"key":"bibr26-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780195130072.003.0003"},{"key":"bibr27-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.neuro.23.1.155"},{"key":"bibr28-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2007.08.005"},{"key":"bibr29-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1080\/02699938708408361"},{"key":"bibr30-1059712314543837","first-page":"238","volume-title":"Proceedings of the 3rd international conference on simulation of adaptive behavior \u2013 from animals to animats","author":"Littman M.","year":"1994"},{"key":"bibr31-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-377-6.50052-9"},{"key":"bibr32-1059712314543837","first-page":"206","volume-title":"Advances in neural information systems","author":"Lopes M.","year":"2012"},{"key":"bibr33-1059712314543837","first-page":"541","volume-title":"Proceedings of the 16th AAAI conference on artificial intelligence","author":"Madani O.","year":"1999"},{"key":"bibr34-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogsys.2008.03.004"},{"key":"bibr35-1059712314543837","first-page":"21","volume-title":"Blueprint for affective computing","author":"Marsella S.","year":"2010"},{"key":"bibr36-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-335-6.50030-1"},{"key":"bibr37-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-377-6.50055-4"},{"key":"bibr38-1059712314543837","first-page":"349","volume-title":"Proceedings of the 19th European conference on artificial intelligence","author":"Melo F.","year":"2010"},{"key":"bibr39-1059712314543837","volume-title":"The society of mind","author":"Minsky M.","year":"1986"},{"key":"bibr40-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1007\/BF00993104"},{"key":"bibr41-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1109\/DevLrn.2013.6652535"},{"key":"bibr42-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-009-5110-1"},{"key":"bibr43-1059712314543837","volume-title":"The logic theory machine: A complex information processing system","author":"Newell A.","year":"1956"},{"key":"bibr44-1059712314543837","first-page":"278","volume-title":"Proceedings of the 16th international conference on machine learning","author":"Ng A.","year":"1999"},{"key":"bibr45-1059712314543837","first-page":"663","volume-title":"Proceedings of the 17th international conference on machine learning","author":"Ng A.","year":"2000"},{"key":"bibr46-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1109\/TAMD.2010.2051436"},{"key":"bibr47-1059712314543837","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1140.003.0008"},{"key":"bibr48-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316887"},{"key":"bibr49-1059712314543837","first-page":"2586","volume-title":"Proceedings of the 20th international joint conference on artificial intelligence","author":"Ramachandran D.","year":"2007"},{"key":"bibr50-1059712314543837","volume-title":"Proceedings of the 15th international conference on machine learning","author":"Randl\u00f8v J.","year":"1998"},{"key":"bibr51-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogsys.2008.03.001"},{"key":"bibr52-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780195130072.003.0004"},{"key":"bibr53-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780195130072.003.0001"},{"key":"bibr54-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1006\/ceps.1999.1020"},{"key":"bibr55-1059712314543837","first-page":"157","volume-title":"Proceedings of the annual convention on ambient intelligence and simulated behavior","author":"Salichs M.","year":"2006"},{"key":"bibr56-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780195130072.003.0005"},{"key":"bibr57-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24600-5_36"},{"key":"bibr58-1059712314543837","author":"Sequeira P.","year":"2014","journal-title":"Autonomous Agents and Multiagent Systems"},{"key":"bibr59-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1109\/DEVLRN.2011.6037325"},{"key":"bibr60-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1037\/h0024127"},{"key":"bibr61-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-335-6.50042-8"},{"key":"bibr62-1059712314543837","first-page":"2601","volume-title":"Proceedings of the annual conference of the Cognitive Science Society","author":"Singh S.","year":"2009"},{"key":"bibr63-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1109\/TAMD.2010.2051031"},{"key":"bibr64-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1080\/02699930902860386"},{"key":"bibr65-1059712314543837","first-page":"1007","volume-title":"Proceedings of the 27th international conference on machine learning","author":"Sorg J.","year":"2010"},{"key":"bibr66-1059712314543837","first-page":"1","volume-title":"Advances in neural information systems","volume":"23","author":"Sorg J.","year":"2010"},{"key":"bibr67-1059712314543837","volume-title":"Reinforcement learning: An introduction","author":"Sutton R.","year":"1998"},{"key":"bibr68-1059712314543837","unstructured":"Watkins C. (1989). Learning from delayed rewards. PhD Thesis, Cambridge University, UK."},{"key":"bibr69-1059712314543837","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1190"},{"key":"bibr70-1059712314543837","volume-title":"Procedures as a representation for data in a computer program for understanding natural language","author":"Winograd T.","year":"1971"}],"container-title":["Adaptive Behavior"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1059712314543837","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1059712314543837","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1059712314543837","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T16:18:28Z","timestamp":1777393108000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1059712314543837"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,9,19]]},"references-count":70,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2014,10]]}},"alternative-id":["10.1177\/1059712314543837"],"URL":"https:\/\/doi.org\/10.1177\/1059712314543837","relation":{},"ISSN":["1059-7123","1741-2633"],"issn-type":[{"value":"1059-7123","type":"print"},{"value":"1741-2633","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,9,19]]}}}