{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T21:20:26Z","timestamp":1793654426089,"version":"4.5.1"},"reference-count":62,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T00:00:00Z","timestamp":1711497600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Pain Res."],"abstract":"<jats:p>Accurate and objective pain evaluation is crucial in developing effective pain management protocols, aiming to alleviate distress and prevent patients from experiencing decreased functionality. A multimodal automatic assessment framework for acute pain utilizing video and heart rate signals is introduced in this study. The proposed framework comprises four pivotal modules: the <jats:italic>Spatial Module<\/jats:italic>, responsible for extracting embeddings from videos; the <jats:italic>Heart Rate Encoder<\/jats:italic>, tasked with mapping heart rate signals into a higher dimensional space; the <jats:italic>AugmNet<\/jats:italic>, designed to create learning-based augmentations in the latent space; and the <jats:italic>Temporal Module<\/jats:italic>, which utilizes the extracted video and heart rate embeddings for the final assessment. The <jats:italic>Spatial-Module<\/jats:italic> undergoes pre-training on a two-stage strategy: first, with a face recognition objective learning universal facial features, and second, with an emotion recognition objective in a multitask learning approach, enabling the extraction of high-quality embeddings for the automatic pain assessment. Experiments with the facial videos and heart rate extracted from electrocardiograms of the <jats:italic>BioVid<\/jats:italic> database, along with a direct comparison to 29 studies, demonstrate state-of-the-art performances in unimodal and multimodal settings, maintaining high efficiency. Within the multimodal context, 82.74% and 39.77% accuracy were achieved for the binary and multi-level pain classification task, respectively, utilizing <jats:inline-formula><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" id=\"IM1\"><mml:mn>9.62<\/mml:mn><\/mml:math><\/jats:inline-formula> million parameters for the entire framework.<\/jats:p>","DOI":"10.3389\/fpain.2024.1372814","type":"journal-article","created":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T04:55:20Z","timestamp":1711515320000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":59,"title":["Multimodal automatic assessment of acute pain through facial videos and heart rate signals utilizing transformer-based architectures"],"prefix":"10.3389","volume":"5","author":[{"given":"Stefanos","family":"Gkikas","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikolaos S.","family":"Tachos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stelios","family":"Andreadis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vasileios C.","family":"Pezoulas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dimitrios","family":"Zaridis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"George","family":"Gkois","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anastasia","family":"Matonaki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thanos G.","family":"Stavropoulos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dimitrios I.","family":"Fotiadis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2024,3,27]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"2420","DOI":"10.1097\/j.pain.0000000000000613","article-title":"Updating the definition of pain","volume":"157","author":"Williams","year":"2016","journal-title":"Pain"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.7759\/CUREUS.1754","article-title":"Neuroanatomy, neuropsychology of pain","volume":"9","author":"Khalid","year":"2017","journal-title":"Cureus"},{"key":"B3","author":"Turk","year":""},{"key":"B4","doi-asserted-by":"publisher","first-page":"1859","DOI":"10.1111\/j.1526-4637.2010.00983.x","article-title":"Causes, consequences of inadequate management of acute pain","volume":"11","author":"Sinatra","year":"2010","journal-title":"Pain Med"},{"key":"B5","volume-title":"Facing Others in Pain: Why Context Matters","author":"De Ruddere","year":"2018"},{"key":"B6","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1213\/ANE.0000000000001389","article-title":"A systematic review, meta-analysis of the global burden of chronic pain without clear etiology in low- and middle-income countries: trends in heterogeneous data and a proposal for new assessment methods","volume":"123","author":"Jackson","year":"2016","journal-title":"Anesth Analg"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1789","DOI":"10.1016\/S0140-6736(18)32279-7","article-title":"Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990\u20132017: a systematic analysis for the global burden of disease study 2017","volume":"392","year":"2018","journal-title":"Lancet"},{"key":"B8","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1016\/J.SPEN.2016.10.003","article-title":"Pathogenesis of pain","volume":"23","author":"Dinakar","year":"2016","journal-title":"Semin Pediatr Neurol"},{"key":"B9","doi-asserted-by":"publisher","first-page":"107365","DOI":"10.1016\/j.cmpb.2023.107365","article-title":"Automatic assessment of pain based on deep learning methods: a systematic review","volume":"231","author":"Gkikas","year":"2023","journal-title":"Comput Methods Programs Biomed"},{"key":"B10","first-page":"9","article-title":"The fifth vital sign: pain","volume":"99","author":"Joel","year":"1999","journal-title":"Am J Nurs"},{"key":"B11","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1038\/s41746-023-00810-1","article-title":"A systematic review of neurophysiological sensing for the assessment of acute pain","volume":"6","author":"Rojas","year":"2023","journal-title":"NPJ Digit Med"},{"key":"B12","doi-asserted-by":"publisher","first-page":"e25258","DOI":"10.2196\/25258","article-title":"Pain assessment tool with electrodermal activity for postoperative patients: method validation study","volume":"9","author":"Hossein Aqajari","year":"2021","journal-title":"JMIR mHealth uHealth"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.3389\/fpain.2023.1150264","article-title":"Multimodal physiological sensing for the assessment of acute pain","volume":"4","author":"Rojas","year":"2023","journal-title":"Front Pain Res"},{"key":"B14","doi-asserted-by":"publisher","first-page":"P279","DOI":"10.1093\/geronb\/56.5.p279","article-title":"Development of a pain attitudes questionnaire to assess stoicism, cautiousness for possible age differences","volume":"56","author":"Yong","year":"2001","journal-title":"J Gerontol B Psychol Sci Soc Sci"},{"key":"B15","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1093\/bja\/aet127","article-title":"Sex differences in pain: a brief review of clinical and experimental findings","volume":"111","author":"Bartley","year":"2013","journal-title":"Br J Anaesth"},{"key":"B16","doi-asserted-by":"publisher","first-page":"e039292","DOI":"10.1136\/bmjopen-2020-039292","article-title":"Using sensor-fusion and machine-learning algorithms to assess acute pain in non-verbal infants: a study protocol","volume":"11","author":"Rou\u00e9","year":"2021","journal-title":"BMJ Open"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1155\/2016\/9267536","article-title":"Medical evidence influence on inpatients and nurses pain ratings agreement","volume":"2016","author":"Dekel","year":"2016","journal-title":"Pain Res Manage"},{"key":"B18","doi-asserted-by":"publisher","first-page":"4296","DOI":"10.1073\/PNAS.1516047113","article-title":"Racial bias in pain assessment and treatment recommendations, and false beliefs about biological differences between blacks and whites","volume":"113","author":"Hoffman","year":"2016","journal-title":"Proc Natl Acad Sci"},{"key":"B19","author":"Keefe","year":""},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.3389\/fpain.2022.895443","article-title":"Beyond pain scales: a critical phenomenology of the expression of pain","volume":"3","author":"Miglio","year":"2022","journal-title":"Front Pain Res"},{"key":"B21","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1159\/000515576","article-title":"Wearable devices: current status, opportunities in pain assessment, management","volume":"5","author":"Leroux","year":"2021","journal-title":"Digit Biomarkers"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2019.2946774","article-title":"Automatic recognition methods supporting pain assessment: a survey","author":"Werner","year":"2019","journal-title":"IEEE Trans Affect Comput"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.3390\/bioengineering10050548","article-title":"Using ai to detect pain through facial expressions: a review","volume":"10","author":"De Sario","year":"2023","journal-title":"Bioengineering"},{"key":"B24","doi-asserted-by":"publisher","first-page":"2393","DOI":"10.1007\/s10439-023-03341-8","article-title":"Automatic classification of emotions based on cardiac signals: a systematic literature review","volume":"51","author":"Claret","year":"2023","journal-title":"Ann Biomed Eng"},{"key":"B25","doi-asserted-by":"publisher","first-page":"4503","DOI":"10.3390\/s19204503","article-title":"Exploring deep physiological models for nociceptive pain recognition","volume":"19","author":"Thiam","year":"2019","journal-title":"Sensors"},{"key":"B26","author":"Zhao","year":""},{"key":"B27","doi-asserted-by":"publisher","first-page":"5624","DOI":"10.1109\/EMBC.2018.8513575","article-title":"Continuous pain intensity estimation from autonomic signals with recurrent neural networks","volume":"2018","author":"Lopez-Martinez","year":"2018","journal-title":"Annu Int Conf IEEE Eng Med Biol Soc"},{"key":"B28","doi-asserted-by":"crossref","DOI":"10.5220\/0010971700003188","volume-title":"Automatic Pain Intensity Estimation Based on Electrocardiogram, Demographic Factors","author":"Gkikas","year":"2022"},{"key":"B29","volume-title":"The Biovid Heat Pain Database: Data for the Advancement, Systematic Validation of an Automated Pain Recognition","author":"Walter","year":"2013"},{"key":"B30","author":"Werner","year":""},{"key":"B31","author":"Zhi","year":""},{"key":"B32","doi-asserted-by":"publisher","first-page":"1413","DOI":"10.1007\/s11263-019-01191-3","article-title":"A spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics","volume":"127","author":"Tavakolian","year":"2019","journal-title":"Int J Comput Vis"},{"key":"B33","doi-asserted-by":"publisher","first-page":"3300","DOI":"10.1109\/TMM.2021.3096080","article-title":"Spatio-temporal pain estimation network with measuring pseudo heart rate gain","volume":"24","author":"Huang","year":"2022","journal-title":"IEEE Trans Multimed"},{"key":"B34","doi-asserted-by":"publisher","first-page":"839","DOI":"10.3390\/s20030839","article-title":"Two-stream attention network for pain recognition from video sequences","volume":"20","author":"Thiam","year":"2020","journal-title":"Sensors"},{"key":"B35","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1016\/j.patrec.2020.09.012","article-title":"Self-supervised pain intensity estimation from facial videos via statistical spatiotemporal distillation","volume":"140","author":"Tavakolian","year":"2020","journal-title":"Pattern Recognit Lett"},{"key":"B36","doi-asserted-by":"publisher","first-page":"286","DOI":"10.1109\/TAFFC.2016.2537327","article-title":"Automatic pain assessment with facial activity descriptors","volume":"8","author":"Werner","year":"2016","journal-title":"IEEE Trans Affect Comput"},{"key":"B37","author":"Patania","year":""},{"key":"B38","doi-asserted-by":"publisher","first-page":"2948","DOI":"10.1049\/ipr2.12282","article-title":"Pain expression assessment based on a locality and identity aware network","volume":"15","author":"Xin","year":"2021","journal-title":"IET Image Process"},{"key":"B39","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1117\/1.jei.28.4.043008","article-title":"Pain-awareness multistream convolutional neural network for pain estimation","volume":"28","author":"Huang","year":"2019","journal-title":"J Electron Imaging"},{"key":"B40","doi-asserted-by":"publisher","first-page":"28329","DOI":"10.1007\/s11042-020-09397-1","article-title":"Pain-attentive network: a deep spatio-temporal attention model for pain estimation","volume":"79","author":"Huang","year":"2020","journal-title":"Multimed Tools Appl"},{"key":"B41","author":"Gkikas","year":""},{"key":"B42","doi-asserted-by":"crossref","DOI":"10.1109\/ICPR.2014.784","volume-title":"Automatic Pain Recognition From Video and Biomedical Signals","author":"Werner","year":"2014"},{"key":"B43","author":"Gkikas","year":""},{"key":"B44","author":"K\u00e4chele","year":""},{"key":"B45","doi-asserted-by":"publisher","first-page":"854","DOI":"10.1109\/JSTSP.2016.2535962","article-title":"Methods for person-centered continuous pain intensity assessment from bio-physiological channels","volume":"10","author":"K\u00e4chele","year":"2016","journal-title":"IEEE J Sel Top Signal Process"},{"key":"B46","author":"Wang","year":""},{"key":"B47","author":"Thiam","year":""},{"key":"B48","doi-asserted-by":"publisher","first-page":"1499","DOI":"10.1109\/LSP.2016.2603342","article-title":"Joint face detection and alignment using multitask cascaded convolutional networks","volume":"23","author":"Zhang","year":"2016","journal-title":"IEEE Signal Process Lett"},{"key":"B49","author":"Pan","year":""},{"key":"B50","author":"Han","year":""},{"key":"B51","author":"Vaswani","year":""},{"key":"B52","author":"Jaegle","year":""},{"key":"B53","author":"Cheung","year":""},{"key":"B54","author":"Cao","year":""},{"key":"B55","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/TAFFC.2017.2740923","article-title":"Affectnet: a database for facial expression, valence, and arousal computing in the wild","volume":"10","author":"Mollahosseini","year":"2019","journal-title":"IEEE Trans Affect Comput"},{"key":"B56","doi-asserted-by":"publisher","first-page":"E1454","DOI":"10.1073\/pnas.1322355111","article-title":"Compound facial expressions of emotion","volume":"111","author":"Du","year":"2014","journal-title":"Proc Natl Acad Sci"},{"key":"B57","author":"Li","year":""},{"key":"B58","author":"Cipolla","year":""},{"key":"B59","author":"Kachuee","year":""},{"key":"B60","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1109\/51.932724","article-title":"The impact of the mit-bih arrhythmia database","volume":"20","author":"Moody","year":"2001","journal-title":"IEEE Eng Med Biol Mag"},{"key":"B61","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1515\/bmte.1995.40.s1.317","article-title":"Nutzung der ekg-signaldatenbank cardiodat der ptb \u00fcber das internet","volume":"40","author":"Bousseljot","year":"1995","journal-title":"Biomed Eng\/Biomed Tech"},{"key":"B62","doi-asserted-by":"publisher","first-page":"E215","DOI":"10.1161\/01.cir.101.23.e215","article-title":"Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals","volume":"101","author":"Goldberger","year":"2000","journal-title":"Circulation"}],"container-title":["Frontiers in Pain Research"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpain.2024.1372814\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T04:55:25Z","timestamp":1711515325000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpain.2024.1372814\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,27]]},"references-count":62,"alternative-id":["10.3389\/fpain.2024.1372814"],"URL":"https:\/\/doi.org\/10.3389\/fpain.2024.1372814","relation":{},"ISSN":["2673-561X"],"issn-type":[{"value":"2673-561X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,27]]},"article-number":"1372814"}}