{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T16:03:20Z","timestamp":1780070600953,"version":"3.54.0"},"reference-count":40,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T00:00:00Z","timestamp":1780012800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Automated emotion recognition systems often rely on acted datasets and categorical models that miss the nuance of spontaneous affect.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>This work assembled a large corpus of authentic facial emotion expressions from naturalistic outpatient psychotherapy sessions, annotated with free-text descriptions by human labelers. These descriptions were embedded in a 768-dimensional semantic space using a fine-tuned German Sentence-BERT model. Transformer, BILSTM, and deep neural network architectures were trained to map facial landmark features to continuous emotion embeddings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Leave-one-out cross-validation showed model predictions closely matched human annotations with a mean z-score of 1.97. External evaluation against acted datasets (RAVDESS) confirmed strong recognition of joy, sadness, and fear.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>To enhance interpretability, a back-translation mechanism using cosine similarity was implemented and visualized with radar charts. All components were integrated into AFFECT, an open-source pipeline for analyzing emotional expressions in everyday video recordings.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fdgth.2026.1808411","type":"journal-article","created":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T15:10:01Z","timestamp":1780067401000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Automated emotion recognition via video-based semantic embeddings"],"prefix":"10.3389","volume":"8","author":[{"given":"Hannes","family":"Diemerling","sequence":"first","affiliation":[{"name":"Faculty of Health, Health and Medical University","place":["Erfurt, Germany"]},{"name":"Department of Psychology, Humboldt-Universit\u00e4t zu Berlin","place":["Germany"]},{"name":"Thomas Bayes Institute","place":["Berlin, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patricia","family":"Kulla","sequence":"additional","affiliation":[{"name":"Department of Psychology, University of the Bundeswehr M\u00fcnchen","place":["Munich, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joachim","family":"Kruse","sequence":"additional","affiliation":[{"name":"Department of Psychology, University of the Bundeswehr M\u00fcnchen","place":["Munich, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Timo","family":"von Oertzen","sequence":"additional","affiliation":[{"name":"Thomas Bayes Institute","place":["Berlin, Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,5,29]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","DOI":"10.4324\/9780203582015","volume-title":"The Great Psychotherapy Debate: The Evidence for What Makes Psychotherapy Work","author":"Wampold","year":"2015"},{"key":"B2","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1017\/CBO9780511527883.005","article-title":"Emotions as episodes of subsystems synchronization driven by nonlinear appraisal processes","author":"Scherer","year":"2000","journal-title":"The Neuropsychology of Emotion"},{"key":"B3","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1038\/nrn1056","article-title":"Cognitive neuroscience of human social behaviour","volume":"4","author":"Adolphs","year":"2003","journal-title":"Nat Rev Neurosci"},{"key":"B4","doi-asserted-by":"crossref","DOI":"10.1037\/10001-000","volume-title":"The Expression of the Emotions in Man and Animals","author":"Darwin","year":"1872"},{"key":"B5","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1037\/h0030377","article-title":"Constants across cultures in the face and emotion","volume":"17","author":"Ekman","year":"1971","journal-title":"J Pers Soc Psychol"},{"key":"B6","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1111\/1467-8721.01252","article-title":"Universals and cultural differences in recognizing emotions","volume":"12","author":"Elfenbein","year":"2003","journal-title":"Curr Dir Psychol Sci"},{"key":"B7","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1037\/1528-3542.8.1.81","article-title":"The automaticity of emotion recognition","volume":"8","author":"Tracy","year":"2008","journal-title":"Emotion"},{"key":"B8","volume-title":"Facial Action Coding System","author":"Ekman","year":"1978"},{"key":"B9","doi-asserted-by":"publisher","first-page":"286","DOI":"10.1177\/0963721411422522","article-title":"Context in emotion perception","volume":"20","author":"Barrett","year":"2011","journal-title":"Curr Dir Psychol Sci"},{"key":"B10","first-page":"359","article-title":"In-group advantage and other-group bias in facial emotion recognition","volume":"45","author":"Elfenbein","year":"2014","journal-title":"J Cross Cult Psychol"},{"key":"B11","doi-asserted-by":"publisher","first-page":"636","DOI":"10.1177\/175407391989729","article-title":"Cross-cultural emotion recognition and in-group advantage in vocal expression: a meta-analysis","volume":"21","author":"Laukka","year":"2021","journal-title":"Emotion"},{"key":"B12","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell","year":"1980","journal-title":"J Pers Soc Psychol"},{"key":"B13","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1111\/j.1745-6916.2006.00003.x","article-title":"Are emotions natural kinds?","volume":"1","author":"Barrett","year":"2006","journal-title":"Perspect Psychol Sci"},{"key":"B14","doi-asserted-by":"publisher","first-page":"1050","DOI":"10.1111\/j.1467-9280.2007.02024.x","article-title":"The world of emotions is not two-dimensional","volume":"18","author":"Fontaine","year":"2007","journal-title":"Psychol Sci"},{"key":"B15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-020-00289-7","article-title":"A comparative analysis of machine learning methods for emotion recognition using EEG and peripheral physiological signals","volume":"7","author":"Doma","year":"2020","journal-title":"J Big Data"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1249","DOI":"10.3390\/s21041249","article-title":"Deep learning techniques for speech emotion recognition, from databases to models","volume":"21","author":"Abbaschian","year":"2021","journal-title":"Sensors"},{"key":"B17","doi-asserted-by":"publisher","first-page":"e2065","DOI":"10.7717\/peerj-cs.2065","article-title":"A comprehensive review of deep learning in eeg-based emotion recognition: classifications, trends, and practical implications","volume":"10","author":"Ma","year":"2024","journal-title":"Front Neurosci"},{"key":"B18","first-page":"61","article-title":"Language as a tool for emotion concept construction","volume":"3","author":"Lindquist","year":"2015","journal-title":"Curr Opin Psychol"},{"key":"B19","author":"Saxen","year":""},{"key":"B20","doi-asserted-by":"publisher","first-page":"572","DOI":"10.1016\/j.patcog.2010.09.020","article-title":"Survey on speech emotion recognition: features, classification schemes, and databases","volume":"44","author":"El Ayadi","year":"2011","journal-title":"Pattern Recognit"},{"key":"B21","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/T-AFFC.2010.1","article-title":"Affect detection: an interdisciplinary review of models, methods, and their applications","volume":"1","author":"Calvo","year":"2010","journal-title":"IEEE Trans Affect Comput"},{"key":"B22","doi-asserted-by":"publisher","first-page":"8325","DOI":"10.3390\/e25101440","article-title":"A survey of deep learning-based multimodal emotion recognition: speech, text, and face","volume":"23","author":"Lian","year":"2023","journal-title":"Sensors"},{"key":"B23","doi-asserted-by":"publisher","first-page":"104556","DOI":"10.1016\/j.jrp.2024.104556","article-title":"A contextualized emotion perception assessment relates to personal and social well-being","volume":"114","author":"Kafetsios","year":"2025","journal-title":"J Res Pers"},{"key":"B24","author":"Mikolov","year":""},{"key":"B25","author":"Pennington","year":""},{"key":"B26","author":"Devlin","year":""},{"key":"B27","doi-asserted-by":"publisher","first-page":"e0196391","DOI":"10.1371\/journal.pone.0196391","article-title":"The Ryerson audio-visual database of emotional speech and song (RAVDESS): a dynamic, multimodal set of facial and vocal expressions in North American English","volume":"13","author":"Livingstone","year":"2018","journal-title":"PLoS ONE"},{"key":"B28","author":"Burkhardt","year":""},{"key":"B29","doi-asserted-by":"publisher","first-page":"613","DOI":"10.1145\/361219.361220","article-title":"A vector space model for automatic indexing","volume":"18","author":"Salton","year":"1975","journal-title":"Commun ACM"},{"key":"B30","author":"Grishchenko","year":""},{"key":"B31","author":"MacQueen","year":""},{"key":"B32","doi-asserted-by":"publisher","first-page":"2825","DOI":"10.5555\/1953048.2078195","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"B33","year":""},{"key":"B34","author":"Diemerling","year":""},{"key":"B35","year":""},{"key":"B36","author":"Reimers","year":""},{"key":"B37","author":"Vaswani","year":""},{"key":"B38","author":"Chang","year":""},{"key":"B39","doi-asserted-by":"crossref","DOI":"10.1201\/9781351072304","volume-title":"Graphical Methods for Data Analysis","author":"Chambers","year":"2018"},{"key":"B40","volume-title":"Graphical Methods for Data Analysis","author":"Chambers","year":"1983"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2026.1808411\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T15:10:03Z","timestamp":1780067403000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2026.1808411\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,29]]},"references-count":40,"alternative-id":["10.3389\/fdgth.2026.1808411"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2026.1808411","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,29]]},"article-number":"1808411"}}