{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T15:50:25Z","timestamp":1753890625945,"version":"3.41.2"},"reference-count":53,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,1,11]],"date-time":"2024-01-11T00:00:00Z","timestamp":1704931200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Sci."],"abstract":"<jats:p>This paper addresses the need for collecting and labeling affect-related data in ecological settings. Collecting the annotations in the wild is a very challenging task, which, however, is crucial for the creation of datasets and emotion recognition models. We propose a novel solution to collect and annotate such data: a questionnaire based on the appraisal theory, that is accessible through an open-source mobile application. Our approach exploits a commercially available wearable physiological sensor connected to a smartphone. The app detects potentially relevant events from the physiological data, and prompts the users to report their emotions using a novel questionnaire based on the Ortony, Clore, and Collins (OCC) Model. The questionnaire is designed to gather information about the appraisal process concerning the significant event. The app guides a user through the reporting process by posing a series of questions related to the event. As a result, the annotated data can be used, e.g., to develop emotion recognition models. In the paper, we analyze users' reports. To validate the questionnaire, we asked 22 individuals to use the app and the sensor for a week. The analysis of the collected annotations shed new light on self-assessment in terms of appraisals. We compared a proposed method with two commonly used methods for reporting affect-related events: (1) a two-dimensional model of valence and arousal, and (2) a forced-choice list of 22 labels. According to the results, appraisal-based reports largely corresponded to the self-reported values of arousal and valence, but they differed substantially from the labels provided with a forced-choice list. In the latter case, when using the forced-choice list, individuals primarily selected labels of basic emotions such as anger or joy. However, they reported a greater variety of emotional states when using appraisal theory for self-assessment of the same events. Thus, proposed approach aids participants to focus on potential causes of their states, facilitating more precise reporting. We also found that regardless of the reporting mode (mandatory vs. voluntary reporting), the ratio between positive and negative reports remained stable. The paper concludes with a list of guidelines to consider in future data collections using self-assessment.<\/jats:p>","DOI":"10.3389\/fcomp.2023.1285690","type":"journal-article","created":{"date-parts":[[2024,1,11]],"date-time":"2024-01-11T04:26:49Z","timestamp":1704947209000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Self-assessment of affect-related events for physiological data collection in the wild based on appraisal theories"],"prefix":"10.3389","volume":"5","author":[{"given":"Radoslaw","family":"Niewiadomski","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fanny","family":"Larradet","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giacinto","family":"Barresi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Leonardo S.","family":"Mattos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,1,11]]},"reference":[{"key":"B1","first-page":"39","article-title":"\u201cIntegrating the OCC model of emotions in embodied characters,\u201d","volume-title":"Workshop on Virtual Conversational Characters","author":"Bartneck","year":"2002"},{"key":"B2","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1145\/3359566.3364695","article-title":"\u201cA VR game-based system for multimodal emotion data collection,\u201d","volume-title":"Motion, Interaction and Games","author":"Bassano","year":"2019"},{"key":"B3","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","article-title":"Measuring emotion: the self-assessment manikin and the semantic differential","volume":"25","author":"Bradley","year":"1994","journal-title":"J. Behav. Ther. Exp. Psychiatry"},{"key":"B4","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1109\/ACII.2013.48","article-title":"\u201cFood and mood: just-in-time support for emotional eating,\u201d","author":"Carroll","year":"2013","journal-title":"2013 Humaine Association Conference on Affective Computing and Intelligent Interaction"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2017.02351","article-title":"Designing awe in virtual reality: an experimental study","author":"Chirico","year":"2018","journal-title":"Front. Psychol"},{"key":"B6","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1177\/1754073913489751","article-title":"Psychological construction in the OCC model of emotion","volume":"5","author":"Clore","year":"2013","journal-title":"Emot. Rev"},{"key":"B7","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1080\/08839510290030390","article-title":"Probabilistic assessment of user's emotions in educational games","volume":"16","author":"Conati","year":"2002","journal-title":"Appl. Artif. Intell"},{"key":"B8","doi-asserted-by":"crossref","first-page":"944","DOI":"10.1007\/3-540-47987-2_94","article-title":"\u201cModeling students' emotions from cognitive appraisal in educational games,\u201d","volume-title":"Intelligent Tutoring Systems","author":"Conati","year":"2002"},{"key":"B9","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1016\/j.neunet.2005.03.007","article-title":"Challenges in real-life emotion annotation and machine learning based detection","volume":"18","author":"Devillers","year":"2005","journal-title":"Neural Netw"},{"key":"B10","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1145\/2522848.2531739","article-title":"\u201cEmotion recognition in the wild challenge 2013,\u201d","volume-title":"Proceedings of the 15th ACM on International Conference on Multimodal Interaction","author":"Dhall","year":"2013"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2022.102791","article-title":"A design methodology for affective virtual reality","author":"Dozio","year":"2022","journal-title":"Int. J. Hum. Comput. Stud"},{"key":"B12","doi-asserted-by":"publisher","first-page":"572","DOI":"10.1016\/j.patcog.2010.09.020","article-title":"Survey on speech emotion recognition: features, classification schemes, and databases","volume":"44","author":"El Ayadi","year":"2011","journal-title":"Pattern Recognit"},{"key":"B13","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1016\/S0031-3203(02)00052-3","article-title":"Automatic facial expression analysis: a survey","volume":"36","author":"Fasel","year":"2003","journal-title":"Pattern Recognit"},{"key":"B14","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1037\/0003-066X.56.3.218","article-title":"The role of positive emotions in positive psychology. the broaden-and-build theory of positive emotions","volume":"56","author":"Fredrickson","year":"2001","journal-title":"Am. Psychol"},{"key":"B15","doi-asserted-by":"publisher","first-page":"678","DOI":"10.1037\/0003-066X.60.7.678","article-title":"The positive affect and the complex dynamics of human flourishing","volume":"60","author":"Fredrickson","year":"2005","journal-title":"Am. Psychol"},{"key":"B16","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/j.jbi.2017.08.006","article-title":"Monitoring stress with a wrist device using context","volume":"73","author":"Gjoreski","year":"2017","journal-title":"J. Biomed. Inform"},{"key":"B17","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1007\/978-3-642-12654-3_10","article-title":"\u201cOut of the lab and into the fray: towards modeling emotion in everyday life,\u201d","volume-title":"International Conference on Pervasive Computing","author":"Healey","year":"2010"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2011.00266","article-title":"Conceptualizing emotions along the dimensions of valence, arousal, and communicative frequency \u2013 implications for social-cognitive tests and training tools","author":"Hepach","year":"2011","journal-title":"Front. Psychol"},{"key":"B19","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1145\/2750858.2807526","article-title":"\u201ccstress: towards a gold standard for continuous stress assessment in the mobile environment,\u201d","volume-title":"Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing","author":"Hovsepian","year":"2015"},{"key":"B20","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1016\/j.ijhcs.2006.11.007","article-title":"Experimental evaluation of five methods for collecting emotions in field settings with mobile applications","volume":"65","author":"Isomursu","year":"2007","journal-title":"Int. J. Hum. Comput. Stud"},{"key":"B21","doi-asserted-by":"crossref","first-page":"410","DOI":"10.1109\/CSPA.2011.5759912","article-title":"\u201cPhysiological signals based human emotion recognition: a review,\u201d","volume-title":"2011 IEEE 7th International Colloquium on Signal Processing and its Applications","author":"Jerritta","year":"2011"},{"key":"B22","doi-asserted-by":"publisher","first-page":"1985","DOI":"10.1109\/ICSLP.1996.608026","article-title":"\u201cEmotional speech elicited using computer games,\u201d","author":"Johnstone","year":"1996","journal-title":"Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP'96"},{"key":"B23","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1109\/T-AFFC.2012.16","article-title":"Affective body expression perception and recognition: a survey","volume":"4","author":"Kleinsmith","year":"2013","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B24","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/CBMS.2013.6627764","article-title":"\u201cSmart technologies for long-term stress monitoring at work,\u201d","volume-title":"Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems","author":"Kocielnik","year":"2013"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2020.01111","article-title":"Toward emotion recognition from physiological signals in the wild: approaching the methodological issues in real-life data collection","author":"Larradet","year":"2020","journal-title":"Front. Psychol"},{"key":"B26","doi-asserted-by":"publisher","first-page":"5163","DOI":"10.3390\/s20185163","article-title":"Emotion recognition in immersive virtual reality: from statistics to affective computing","volume":"20","author":"Mar\u00edn-Morales","year":"2020","journal-title":"Sensors"},{"key":"B27","first-page":"3361","article-title":"Utilizing emoticons on mobile devices within esm studies to measure emotions in the field","volume":"9","author":"Meschtscherjakov","year":"2009","journal-title":"Proc. MME Conjunct. MobileHCI"},{"key":"B28","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1109\/TAFFC.2018.2864730","article-title":"Induction and profiling of strong multi-componential emotions in virtual reality","volume":"12","author":"Meuleman","year":"2021","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B29","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1109\/TAFFC.2018.2884461","article-title":"Amigos: a dataset for affect, personality and mood research on individuals and groups","volume":"12","author":"Miranda-Correa","year":"2021","journal-title":"IEEE Trans. Affect. Comput"},{"key":"B30","doi-asserted-by":"publisher","first-page":"18","DOI":"10.4018\/jse.2012010102","article-title":"Advocating a componential appraisal model to guide emotion recognition","volume":"3","author":"Mortillaro","year":"2012","journal-title":"Int. J. Synth. Emot"},{"key":"B31","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1177\/1754073920950811","article-title":"Measuring positive emotion outcomes in positive psychology interventions: a literature review","volume":"13","author":"Moskowitz","year":"2021","journal-title":"Emot. Rev"},{"key":"B32","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1007\/s12668-013-0089-2","article-title":"Towards measuring stress with smartphones and wearable devices during workday and sleep","volume":"3","author":"Muaremi","year":"2013","journal-title":"Bionanoscience"},{"key":"B33","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.biopsycho.2004.06.001","article-title":"Emotions in everyday life: an ambulatory monitoring study with female students","volume":"68","author":"Myrtek","year":"2005","journal-title":"Biol. Psychol"},{"key":"B34","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/0301-0511(95)05152-X","article-title":"Perception of emotions in everyday life: studies with patients and normals","volume":"42","author":"Myrtek","year":"1996","journal-title":"Biol. Psychol"},{"key":"B35","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1007\/s10111-003-0143-x","article-title":"Emotion recognition from physiological signals using wireless sensors for presence technologies","volume":"6","author":"Nasoz","year":"2004","journal-title":"Cogn. Technol. Work"},{"key":"B36","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1109\/TOH.2022.3230643","article-title":"Affect recognition in hand-object interaction using object-sensed tactile and kinematic data","volume":"16","author":"Niewiadomski","year":"2022","journal-title":"IEEE Trans. Haptics"},{"key":"B37","first-page":"184","article-title":"\u201cMMLI: multimodal multiperson corpus of laughter in interaction,\u201d","volume-title":"Human Behavior Understanding, Volume 8212 of Lecture Notes in Computer Science","author":"Niewiadomski","year":"2013"},{"volume-title":"The Cognitive Structure of Emotions","year":"1990","author":"Ortony","key":"B38"},{"key":"B39","first-page":"97","article-title":"\u201cContinuous inference of psychological stress from sensory measurements collected in the natural environment,\u201d","author":"Plarre","year":"2011","journal-title":"Proceedings of the 10th ACM\/IEEE International Conference on Information Processing in Sensor Networks"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-023-01026-9","article-title":"Comfortability analysis under a human\u2013robot interaction perspective","author":"Redondo","year":"2023","journal-title":"Int. J. Soc. Robot"},{"key":"B41","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1080\/026999396380240","article-title":"Appraisal determinants of emotions: Constructing a more accurate and comprehensive theory","volume":"10","author":"Roseman","year":"1996","journal-title":"Cogn. Emot"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","author":"Russell","year":"1980","journal-title":"J. Pers. Soc. Psychol"},{"key":"B43","doi-asserted-by":"publisher","first-page":"695","DOI":"10.1177\/0539018405058216","article-title":"What are emotions? and how can they be measured?","volume":"44","author":"Scherer","year":"2005","journal-title":"Soc. Sci. Inf"},{"key":"B44","doi-asserted-by":"publisher","first-page":"1307","DOI":"10.1080\/02699930902928969","article-title":"The dynamic architecture of emotion: evidence for the component process model","volume":"23","author":"Scherer","year":"2009","journal-title":"Cogn. Emot"},{"key":"B45","first-page":"654","article-title":"\u201cLabelling affective states in the wild: Practical guidelines and lessons learned,\u201d","volume-title":"Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers","author":"Schmidt","year":"2018"},{"key":"B46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3390\/s18072074","article-title":"A review of emotion recognition using physiological signals","volume":"18","author":"Shu","year":"2018","journal-title":"Sensors"},{"key":"B47","doi-asserted-by":"publisher","first-page":"339","DOI":"10.1037\/0022-3514.56.3.339","article-title":"Dimensions of appraisal and physiological response in emotion","author":"Smith","year":"1989","journal-title":"J. Pers. Soc. Psychol"},{"key":"B48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0145450","article-title":"Emotions in everyday life","volume":"10","author":"Trampe","year":"2015","journal-title":"PLoS ONE"},{"key":"B49","doi-asserted-by":"publisher","first-page":"663","DOI":"10.1080\/02699930341000167","article-title":"Psychophysiological responses to appraisal dimensions in a computer game","volume":"18","author":"van Reekum","year":"2004","journal-title":"Cogn. Emot"},{"key":"B50","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1016\/B978-0-12-558704-4.50011-6","article-title":"\u201cChapter 5 - The dictionary of affect in language,\u201d","author":"Whissell","year":"1989","journal-title":"The Measurement of Emotions"},{"key":"B51","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1016\/j.biopsycho.2010.01.017","article-title":"Emotions beyond the laboratory: theoretical fundaments, study design, and analytic strategies for advanced ambulatory assessment","volume":"84","author":"Wilhelm","year":"2010","journal-title":"Biol. Psychol"},{"key":"B52","first-page":"345","article-title":"\u201cFrom the lab to the real-world: an investigation on the influence of human movement on emotion recognition using physiological signals,\u201d","author":"Xu","year":"2017","journal-title":"2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops)"},{"key":"B53","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/PERCOMW.2016.7457166","article-title":"\u201cHealthyoffice: mood recognition at work using smartphones and wearable sensors,\u201d","volume-title":"2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops)","author":"Zenonos","year":"2016"}],"container-title":["Frontiers in Computer Science"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2023.1285690\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,11]],"date-time":"2024-01-11T04:27:02Z","timestamp":1704947222000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fcomp.2023.1285690\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,11]]},"references-count":53,"alternative-id":["10.3389\/fcomp.2023.1285690"],"URL":"https:\/\/doi.org\/10.3389\/fcomp.2023.1285690","relation":{},"ISSN":["2624-9898"],"issn-type":[{"type":"electronic","value":"2624-9898"}],"subject":[],"published":{"date-parts":[[2024,1,11]]},"article-number":"1285690"}}