{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T14:09:17Z","timestamp":1787062157324,"version":"3.56.0"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"20","license":[{"start":{"date-parts":[[2023,3,22]],"date-time":"2023-03-22T00:00:00Z","timestamp":1679443200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,3,22]],"date-time":"2023-03-22T00:00:00Z","timestamp":1679443200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004252","name":"Qatar University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004252","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Eating is experienced as an emotional social activity in any culture. There are factors that influence the emotions felt during food consumption. The emotion felt while eating has a significant impact on our lives and affects different health conditions such as obesity. In addition, investigating the emotion during food consumption is considered a multidisciplinary problem ranging from neuroscience to anatomy. In this study, we focus on evaluating the emotional experience of different participants during eating activities and aim to analyze them automatically using deep learning models. We propose a facial expression-based prediction model to eliminate user bias in questionnaire-based assessment systems and to minimize false entries to the system. We measured the neural, behavioral, and physical manifestations of emotions with a mobile app and recognize emotional experiences from facial expressions. In this research, we used three different situations to test whether there could be any factor other than the food that could affect a person\u2019s mood. We asked users to watch videos, listen to music or do nothing while eating. This way we found out that not only food but also external factors play a role in emotional change. We employed three Convolutional Neural Network (CNN) architectures, fine-tuned VGG16, and Deepface to recognize emotional responses during eating. The experimental results demonstrated that the fine-tuned VGG16 provides remarkable results with an overall accuracy of 77.68% for recognizing the four emotions. This system is an alternative to today\u2019s survey-based restaurant and food evaluation systems.<\/jats:p>","DOI":"10.1007\/s11042-023-15008-6","type":"journal-article","created":{"date-parts":[[2023,3,22]],"date-time":"2023-03-22T04:02:36Z","timestamp":1679457756000},"page":"31659-31671","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Analysis of facial emotion expression in eating occasions using deep learning"],"prefix":"10.1007","volume":"82","author":[{"given":"Elif","family":"Yildirim","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fatma Patlar","family":"Akbulut","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0959-2930","authenticated-orcid":false,"given":"Cagatay","family":"Catal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,3,22]]},"reference":[{"key":"15008_CR1","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.measurement.2018.06.050","volume":"128","author":"FP Akbulut","year":"2018","unstructured":"Akbulut F P, Akan A (2018) A smart wearable system for short-term cardiovascular risk assessment with emotional dynamics. Measurement 128:237\u2013246","journal-title":"Measurement"},{"key":"15008_CR2","doi-asserted-by":"publisher","first-page":"101824","DOI":"10.1016\/j.artmed.2020.101824","volume":"104","author":"FP Akbulut","year":"2020","unstructured":"Akbulut F P, Ikitimur B, Akan A (2020) Wearable sensor-based evaluation of psychosocial stress in patients with metabolic syndrome. Artif Intell Med 104:101824","journal-title":"Artif Intell Med"},{"key":"15008_CR3","doi-asserted-by":"publisher","first-page":"105571","DOI":"10.1016\/j.cmpb.2020.105571","volume":"195","author":"FP Akbulut","year":"2020","unstructured":"Akbulut F P, Perros H G, Shahzad M (2020) Bimodal affect recognition based on autoregressive hidden markov models from physiological signals. Comput Methods Programs Biomed 195:105571","journal-title":"Comput Methods Programs Biomed"},{"key":"15008_CR4","doi-asserted-by":"publisher","unstructured":"Altheimer G, Giles G E, Remedios J D, Kanarek R B, Urry H L Do emotions predict eating? The role of previous experiences in emotional eating in the lab and in daily life. Appetite, 158. https:\/\/doi.org\/10.1016\/j.appet.2020.105016","DOI":"10.1016\/j.appet.2020.105016"},{"issue":"3","key":"15008_CR5","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1111\/1467-839X.00084","volume":"4","author":"JR Averill","year":"2001","unstructured":"Averill J R, Chon K K, Hahn D W (2001) Emotions and creativity, east and west. Asian J Soc Psychol 4(3):165\u2013183","journal-title":"Asian J Soc Psychol"},{"key":"15008_CR6","doi-asserted-by":"publisher","first-page":"140990","DOI":"10.1109\/ACCESS.2019.2944001","volume":"7","author":"PJ Bota","year":"2019","unstructured":"Bota P J, Wang C, Fred Ana LN, Da Silva H P (2019) A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals. IEEE Access 7:140990\u2013141020","journal-title":"IEEE Access"},{"issue":"1","key":"15008_CR7","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","volume":"25","author":"MM Bradley","year":"1994","unstructured":"Bradley M M, Lang P J (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Therapy Exper Psych 25(1):49\u201359","journal-title":"J Behav Therapy Exper Psych"},{"key":"15008_CR8","first-page":"1","volume":"1","author":"M Butnariu","year":"2019","unstructured":"Butnariu M, Sarac I, Chandel A (2019) Biochemistry of hormones that influences feelings. Pharmacoepidemiol Drug Saf 1:1\u20136","journal-title":"Pharmacoepidemiol Drug Saf"},{"key":"15008_CR9","doi-asserted-by":"publisher","first-page":"400","DOI":"10.1016\/j.appet.2018.12.001","volume":"133","author":"V Cardi","year":"2019","unstructured":"Cardi V, Leppanen J, Leslie M, Esposito M, Treasure J (2019) The use of a positive mood induction video-clip to target eating behaviour in people with bulimia nervosa or binge eating disorder: an experimental study. Appetite 133:400\u2013404","journal-title":"Appetite"},{"key":"15008_CR10","doi-asserted-by":"crossref","unstructured":"Carroll E A, Czerwinski M, Roseway A, Kapoor A, Johns P, Rowan K, Schraefel MC (2013) Food and mood: just-in-time support for emotional eating. In: 2013 Humaine association conference on affective computing and intelligent interaction, pp 252\u2013257","DOI":"10.1109\/ACII.2013.48"},{"key":"15008_CR11","unstructured":"D RETP (2003) Calm energy: How people regulate mood with food and exercise. Oxford University Press. https:\/\/books.google.com.tr\/books?id=D9HQCwAAQBAJ. Accessed 15 Aug 2022"},{"issue":"1","key":"15008_CR12","first-page":"32","volume":"4","author":"P Desmet","year":"2001","unstructured":"Desmet P, Overbeeke K, Tax S (2001) Designing products with added emotional value: Development and application of an approach for research through design. Des J 4(1):32\u201347","journal-title":"Des J"},{"key":"15008_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.appet.2013.04.007","volume":"68","author":"C Evers","year":"2013","unstructured":"Evers C, Adriaanse M, de Ridder DTD, de Witt Huberts J C (2013) Good mood food. positive emotion as a neglected trigger for food intake. Appetite 68:1\u20137","journal-title":"Appetite"},{"issue":"2","key":"15008_CR14","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1080\/02699939008407142","volume":"4","author":"KW Fischer","year":"1990","unstructured":"Fischer K W, Shaver P R, Carnochan P (1990) How emotions develop and how they organise development. Cogn Emot 4(2):81\u2013127","journal-title":"Cogn Emot"},{"key":"15008_CR15","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1016\/j.appet.2016.02.008","volume":"100","author":"M Frayn","year":"2016","unstructured":"Frayn M, Sears C R, von Ranson K M (2016) A sad mood increases attention to unhealthy food images in women with food addiction. Appetite 100:55\u201363","journal-title":"Appetite"},{"key":"15008_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.neuroscience.2017.12.036","volume":"372","author":"BM Fritz","year":"2018","unstructured":"Fritz B M, Mu\u00f1oz B, Yin F, Bauchle C, Atwood B K (2018) A high-fat, high-sugar \u2018western\u2019 diet alters dorsal striatal glutamate, opioid, and dopamine transmission in mice. Neuroscience 372:1\u201315","journal-title":"Neuroscience"},{"issue":"3","key":"15008_CR17","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1002\/1098-108X(198905)8:3<343::AID-EAT2260080310>3.0.CO;2-C","volume":"8","author":"RM Ganley","year":"1989","unstructured":"Ganley R M (1989) Emotion and eating in obesity: a review of the literature. Int J Eat Disorders 8(3):343\u2013361","journal-title":"Int J Eat Disorders"},{"key":"15008_CR18","doi-asserted-by":"crossref","unstructured":"Goodfellow I J, Erhan D, Carrier P L, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D-H et al (2013) Challenges in representation learning: a report on three machine learning contests. In: International conference on neural information processing. Springer, pp 117\u2013124","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"15008_CR19","doi-asserted-by":"publisher","first-page":"101500","DOI":"10.1016\/j.eatbeh.2021.101500","volume":"41","author":"OM Herren","year":"2021","unstructured":"Herren O M, Agurs-Collins T, Dwyer L A, Perna F M, Ferrer R (2021) Emotion suppression, coping strategies, dietary patterns, and bmi. Eat Behav 41:101500. https:\/\/doi.org\/10.1016\/j.eatbeh.2021.101500","journal-title":"Eat Behav"},{"key":"15008_CR20","doi-asserted-by":"publisher","first-page":"2","DOI":"10.5502\/ijw.v5i2.3","volume":"5","author":"R Hurling","year":"2015","unstructured":"Hurling R, Linley P A, Dovey H, Maltby J, Wilkinson J (2015) Everyday happiness: gifting and eating as everyday activities that influence general positive affect and discrete positive emotions. Int J Wellbeing 5:2","journal-title":"Int J Wellbeing"},{"key":"15008_CR21","doi-asserted-by":"crossref","unstructured":"Jaiswal A, Krishnama Raju A, Deb S (2020) Facial emotion detection using deep learning. In: 2020 International Conference for Emerging Technology (INCET), pp 1\u20135","DOI":"10.1109\/INCET49848.2020.9154121"},{"issue":"11","key":"15008_CR22","doi-asserted-by":"publisher","first-page":"3627","DOI":"10.1002\/jsfa.7717","volume":"96","author":"E Kenney","year":"2016","unstructured":"Kenney E, Adhikari K (2016) Recent developments in identifying and quantifying emotions during food consumption. J Sci Food Agric 96(11):3627\u20133630","journal-title":"J Sci Food Agric"},{"issue":"2","key":"15008_CR23","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1016\/j.foodqual.2009.02.005","volume":"21","author":"SC King","year":"2010","unstructured":"King S C, Meiselman H L (2010) Development of a method to measure consumer emotions associated with foods. Food Qual Preference 21(2):168\u2013177","journal-title":"Food Qual Preference"},{"key":"15008_CR24","doi-asserted-by":"crossref","unstructured":"Larson R, Csikszentmihalyi M (2014) The experience sampling method. In: Flow and the foundations of positive psychology. Springer, pp 21\u201334","DOI":"10.1007\/978-94-017-9088-8_2"},{"issue":"1","key":"15008_CR25","doi-asserted-by":"publisher","first-page":"799","DOI":"10.1146\/annurev-psych-010213-115043","volume":"66","author":"JS Lerner","year":"2015","unstructured":"Lerner J S, Li Y, Valdesolo P, Kassam K S (2015) Emotion and decision making. Annu Rev Psychol 66(1):799\u2013823. https:\/\/doi.org\/10.1146\/annurev-psych-010213-115043. PMID: 25251484","journal-title":"Annu Rev Psychol"},{"key":"15008_CR26","doi-asserted-by":"crossref","unstructured":"Lucey P, Cohn J F, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94\u2013101","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"15008_CR27","unstructured":"Native R (2020) React native. l\u00ednea]. Disponible en: https:\/\/reactnative. dev\/.[\u00daltimo acceso: 2 de noviembre 2019]"},{"issue":"4","key":"15008_CR28","first-page":"405","volume":"27","author":"Y Ouyang","year":"2018","unstructured":"Ouyang Y, Behnke C, Almanza B, Ghiselli R (2018) The influence of food aromas on restaurant consumer emotions, perceptions, and purchases. J Hosp Market Manag 27(4):405\u2013423","journal-title":"J Hosp Market Manag"},{"key":"15008_CR29","doi-asserted-by":"crossref","unstructured":"Patlar Akbulut F (2022) Hybrid deep convolutional model-based emotion recognition using multiple physiological signals. Comput Methods Biomech Biomed Engin, 1\u201313","DOI":"10.1080\/10255842.2022.2032682"},{"key":"15008_CR30","doi-asserted-by":"publisher","first-page":"1517","DOI":"10.1590\/S0100-879X1998001200002","volume":"31","author":"C Prasad","year":"1998","unstructured":"Prasad C (1998) Food, mood and health: a neurobiologic outlook. Braz J Med Biol Res 31:1517\u20131527","journal-title":"Braz J Med Biol Res"},{"issue":"2","key":"15008_CR31","first-page":"77","volume":"38","author":"MB Ripski","year":"2011","unstructured":"Ripski M B, LoCasale-Crouch J, Decker L (2011) Pre-service teachers: dispositional traits, emotional states, and quality of teacher-student interactions. Teacher Educ Quart 38(2):77\u201396","journal-title":"Teacher Educ Quart"},{"key":"15008_CR32","unstructured":"Safta I, Grigore O, C\u0103runtu C (2011) Emotion detection using psycho-physiological signal processing. In: 2011 7th International Symposium on Advanced Topics in Electrical Engineering (ATEE). IEEE, pp 1\u20134"},{"issue":"5","key":"15008_CR33","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1037\/h0046234","volume":"69","author":"S Schachter","year":"1962","unstructured":"Schachter S, Singer J (1962) Cognitive, social, and physiological determinants of emotional state. Psychol Rev 69(5):379","journal-title":"Psychol Rev"},{"issue":"1","key":"15008_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13411-015-0041-1","volume":"4","author":"C Spence","year":"2015","unstructured":"Spence C (2015) Leading the consumer by the nose: on the commercialization of olfactory design for the food and beverage sector. Flavour 4(1):1\u201315","journal-title":"Flavour"},{"key":"15008_CR35","unstructured":"T\u00e4hti M, Niemel\u00e4 M (2006) 3e\u2013expressing emotions and experiences. In: WP9 Workshop on innovative approaches for evaluating affective systems. Citeseer, pp 15\u201319"},{"key":"15008_CR36","doi-asserted-by":"crossref","unstructured":"Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1701\u20131708","DOI":"10.1109\/CVPR.2014.220"},{"key":"15008_CR37","unstructured":"The TripAdvisor influences on diner decision-making. https:\/\/www.tripadvisor.com\/ForRestaurants\/r3227. Accessed 26 Feb 2022"},{"key":"15008_CR38","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/j.appet.2013.02.016","volume":"66","author":"T van Strien","year":"2013","unstructured":"van Strien T, Cebolla A, Etchemendy E, Gutierrez-Maldonado J, Ferrer-Garcia M, Botella C, Ba\u00f1os R (2013) Emotional eating and food intake after sadness and joy. Appetite 66:20\u201325","journal-title":"Appetite"},{"issue":"2","key":"15008_CR39","first-page":"11","volume":"16","author":"A Vatcharaphrueksadee","year":"2020","unstructured":"Vatcharaphrueksadee A, Viboonpanich R, Sakul-ang P, Maliyaem M (2020) Vgg-16 and optimized cnn for emotion classification. Inf Technol J 16 (2):11\u201315","journal-title":"Inf Technol J"},{"issue":"10","key":"15008_CR40","doi-asserted-by":"publisher","first-page":"1499","DOI":"10.1109\/LSP.2016.2603342","volume":"23","author":"K Zhang","year":"2016","unstructured":"Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499\u20131503","journal-title":"IEEE Signal Process Lett"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-023-15008-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-023-15008-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-023-15008-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T09:04:43Z","timestamp":1690362283000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-023-15008-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,22]]},"references-count":40,"journal-issue":{"issue":"20","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["15008"],"URL":"https:\/\/doi.org\/10.1007\/s11042-023-15008-6","relation":{},"ISSN":["1380-7501","1573-7721"],"issn-type":[{"value":"1380-7501","type":"print"},{"value":"1573-7721","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,22]]},"assertion":[{"value":"8 May 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 August 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 February 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 March 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}