{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T03:49:29Z","timestamp":1777348169270,"version":"3.51.4"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T00:00:00Z","timestamp":1774310400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T00:00:00Z","timestamp":1774310400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100012320","name":"Otto-von-Guericke-Universit\u00e4t Magdeburg","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100012320","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Previous approaches to pain intensity classification have typically relied on small sets of top-performing features to maximize accuracy. While effective in constrained scenarios, such strategies neglect the diverse range of modalities available in modern pain databases. In this work, we introduce a multimodal feature set (MMFS) that integrates heterogeneous features from each biosignal modality in the\n                    <jats:italic>BioVid<\/jats:italic>\n                    and\n                    <jats:italic>X-ITE<\/jats:italic>\n                    databases. Our approach captures a broad spectrum of complementary information, maintaining robustness even when individual modalities are unavailable. Experimental results show consistent performance improvements, with classification accuracy increasing by up to 8% overall and by 6% when evaluating all pain levels. Through detailed analyses of individual modalities, confusion matrices, and a modality ablation study, we demonstrate that the combined effect of multimodality and balanced information distribution drives these gains. Furthermore, feature importance analysis reveals which inputs contribute most to final predictions and which features are most beneficial for the more challenging pain classes.\n                  <\/jats:p>","DOI":"10.1007\/s00521-026-11962-y","type":"journal-article","created":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T06:49:41Z","timestamp":1774334981000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Constructing a multimodal feature set for pain intensity classification"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-0883-6029","authenticated-orcid":false,"given":"S\u00f6ren","family":"Nienaber","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9213-0749","authenticated-orcid":false,"given":"Huibin","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0364-8067","authenticated-orcid":false,"given":"Laslo","family":"Dinges","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3632-2402","authenticated-orcid":false,"given":"Ayoub","family":"Al-Hamadi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,3,24]]},"reference":[{"key":"11962_CR1","doi-asserted-by":"publisher","first-page":"437","DOI":"10.3389\/fnins.2019.00437","volume":"13","author":"E Campbell","year":"2019","unstructured":"Campbell E, Phinyomark A, Scheme E (2019) Feature extraction and selection for pain recognition using peripheral physiological signals. Front Neurosci 13:437. https:\/\/doi.org\/10.3389\/fnins.2019.00437","journal-title":"Front Neurosci"},{"issue":"10","key":"11962_CR2","doi-asserted-by":"publisher","first-page":"0140330","DOI":"10.1371\/journal.pone.0140330","volume":"10","author":"S Gruss","year":"2015","unstructured":"Gruss S, Treister R, Werner P, Traue HC, Crawcour S, Andrade A, Walter S (2015) Pain intensity recognition rates via biopotential feature patterns with support vector machines. PLoS ONE 10(10):0140330. https:\/\/doi.org\/10.1371\/journal.pone.0140330","journal-title":"PLoS ONE"},{"issue":"13","key":"11962_CR3","doi-asserted-by":"publisher","first-page":"4992","DOI":"10.3390\/s22134992","volume":"22","author":"E Othman","year":"2022","unstructured":"Othman E, Werner P, Saxen F, Fiedler M-A, Al-Hamadi A (2022) An automatic system for continuous pain intensity monitoring based on analyzing data from uni-, bi-, and multi-modality. Sensors 22(13):4992. https:\/\/doi.org\/10.3390\/s22134992","journal-title":"Sensors"},{"issue":"1","key":"11962_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0304-3959(85)90145-9","volume":"22","author":"CR Chapman","year":"1985","unstructured":"Chapman CR, Casey KL, Dubner R, Foley KM, Gracely RH, Reading AE (1985) Pain measurement: an overview. Pain 22(1):1\u201331. https:\/\/doi.org\/10.1016\/0304-3959(85)90145-9","journal-title":"Pain"},{"key":"11962_CR5","doi-asserted-by":"publisher","first-page":"2258","DOI":"10.1097\/00003246-200112000-00004","volume":"12","author":"J-F Payen","year":"2001","unstructured":"Payen J-F, Bru O, Bosson J-L, Lagrasta A, Novel E, Deschaux I, Lavagne P, Jacquot C (2001) Assessing pain in critically ill sedated patients by using a behavioral pain scale. Crit Care Med 12:2258\u20132263","journal-title":"Crit Care Med"},{"key":"11962_CR6","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1097\/00002508-199703000-00010","volume":"13","author":"ME Robinson","year":"1997","unstructured":"Robinson ME, Myers CD, Sadler IJ, Riley JLI, Kvaal SA, Geisser ME (1997) Bias effects in three common self-report pain assessment measures. Clin J Pain 13:74\u201381","journal-title":"Clin J Pain"},{"issue":"10","key":"11962_CR7","doi-asserted-by":"publisher","first-page":"1779","DOI":"10.1016\/j.jpain.2022.06.002","volume":"23","author":"BL Boring","year":"2022","unstructured":"Boring BL, Ng BW, Nanavaty N, Mathur VA (2022) Over-rating pain is overrated: a fundamental self-other bias in pain reporting behavior. J Pain 23(10):1779\u20131789. https:\/\/doi.org\/10.1016\/j.jpain.2022.06.002","journal-title":"J Pain"},{"issue":"1","key":"11962_CR8","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1054\/jpai.2003.1","volume":"4","author":"MP Jensen","year":"2003","unstructured":"Jensen MP (2003) The validity and reliability of pain measures in adults with cancer. J Pain 4(1):2\u201321. https:\/\/doi.org\/10.1054\/jpai.2003.1","journal-title":"J Pain"},{"issue":"5","key":"11962_CR9","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1177\/0885066618757450","volume":"35","author":"RH Bouajram","year":"2020","unstructured":"Bouajram RH, Sebat CM, Love D, Louie EL, Wilson MD, Duby JJ (2020) Comparison of self-reported and behavioral pain assessment tools in critically ill patients. J Intensive Care Med 35(5):453\u2013460. https:\/\/doi.org\/10.1177\/0885066618757450. (PMID: 29448873)","journal-title":"J Intensive Care Med"},{"key":"11962_CR10","doi-asserted-by":"publisher","unstructured":"Walter S, Gruss S, Ehleiter H, Tan J, Traue HC, Werner P, Al-Hamadi A, Crawcour S, Andrade AO, Silva G (2013) The biovid heat pain database data for the advancement and systematic validation of an automated pain recognition system. In: 2013 IEEE International Conference on Cybernetics (CYBCO), pp. 128\u2013131 . https:\/\/doi.org\/10.1109\/CYBConf.2013.6617456","DOI":"10.1109\/CYBConf.2013.6617456"},{"key":"11962_CR11","doi-asserted-by":"publisher","unstructured":"Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I (2011) Painful data: The unbc-mcmaster shoulder pain expression archive database. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp. 57\u201364 . https:\/\/doi.org\/10.1109\/FG.2011.5771462","DOI":"10.1109\/FG.2011.5771462"},{"issue":"1","key":"11962_CR12","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1109\/TAFFC.2018.2851236","volume":"12","author":"T Nezam","year":"2021","unstructured":"Nezam T, Boostani R, Abootalebi V, Rastegar K (2021) A novel classification strategy to distinguish five levels of pain using the eeg signal features. IEEE Trans Affect Comput 12(1):131\u2013140. https:\/\/doi.org\/10.1109\/TAFFC.2018.2851236","journal-title":"IEEE Trans Affect Comput"},{"key":"11962_CR13","doi-asserted-by":"publisher","first-page":"1372814","DOI":"10.3389\/fpain.2024.1372814","volume":"5","author":"S Gkikas","year":"2024","unstructured":"Gkikas S, Tachos NS, Andreadis S, Pezoulas VC, Zaridis D, Gkois G, Matonaki A, Stavropoulos TG, Fotiadis DI (2024) Multimodal automatic assessment of acute pain through facial videos and heart rate signals utilizing transformer-based architectures. Front Pain Res 5:1372814. https:\/\/doi.org\/10.3389\/fpain.2024.1372814","journal-title":"Front Pain Res"},{"issue":"7","key":"11962_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0254108","volume":"16","author":"F Pouromran","year":"2021","unstructured":"Pouromran F, Radhakrishnan S, Kamarthi S (2021) Exploration of physiological sensors, features, and machine learning models for pain intensity estimation. PLoS ONE 16(7):1\u201317. https:\/\/doi.org\/10.1371\/journal.pone.0254108","journal-title":"PLoS ONE"},{"key":"11962_CR15","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2023.107365","volume":"231","author":"S Gkikas","year":"2023","unstructured":"Gkikas S, Tsiknakis M (2023) Automatic assessment of pain based on deep learning methods: a systematic review. Comput Methods Programs Biomed 231:107365. https:\/\/doi.org\/10.1016\/j.cmpb.2023.107365","journal-title":"Comput Methods Programs Biomed"},{"issue":"3","key":"11962_CR16","doi-asserted-by":"publisher","first-page":"363","DOI":"10.3922\/j.psns.2014.041","volume":"7","author":"S Walter","year":"2014","unstructured":"Walter S, Gruss S, Limbrecht-Ecklundt K, Traue HC, Werner P, Al-Hamadi A, Diniz N, Silva GM, Andrade AO (2014) Automatic pain quantification using autonomic parameters. Psychol& Neurosci 7(3):363\u2013380. https:\/\/doi.org\/10.3922\/j.psns.2014.041 (https:\/\/www.redalyc.org\/pdf\/2070\/207032650014.pdf)","journal-title":"Psychol& Neurosci"},{"key":"11962_CR17","doi-asserted-by":"publisher","unstructured":"Werner P, Al-Hamadi A, Niese R, Walter S, Gruss S, Traue H (2014) Automatic pain recognition from video and biomedical signals, pp. 4582\u20134587 . https:\/\/doi.org\/10.1109\/ICPR.2014.784","DOI":"10.1109\/ICPR.2014.784"},{"key":"11962_CR18","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1007\/978-3-319-20248-8_19","volume-title":"Multiple Classifier Systems","author":"M K\u00e4chele","year":"2015","unstructured":"K\u00e4chele M, Werner P, Al-Hamadi A, Palm G, Walter S, Schwenker F (2015) Bio-visual fusion for person-independent recognition of pain intensity. In: Schwenker F, Roli F, Kittler J (eds) Multiple Classifier Systems. Springer, Cham, pp 220\u2013230"},{"key":"11962_CR19","doi-asserted-by":"crossref","unstructured":"Lopez-Martinez D, Picard R (2017) Multi-task Neural Networks for Personalized Pain Recognition from Physiological Signals . arXiv:https:\/\/arxiv.org\/abs\/1708.08755","DOI":"10.1109\/ACIIW.2017.8272611"},{"key":"11962_CR20","doi-asserted-by":"publisher","first-page":"1294577","DOI":"10.3389\/fphys.2023.1294577","volume":"14","author":"Z Lu","year":"2023","unstructured":"Lu Z, Ozek B, Kamarthi S (2023) Transformer encoder with multiscale deep learning for pain classification using physiological signals. Front Physiol 14:1294577. https:\/\/doi.org\/10.3389\/fphys.2023.1294577","journal-title":"Front Physiol"},{"key":"11962_CR21","doi-asserted-by":"publisher","unstructured":"Thiam P, Kestler H, Schwenker F (2020). Multimodal deep denoising convolutional autoencoders for pain intensity classification based on physiological signals, pp. 289\u2013296. https:\/\/doi.org\/10.5220\/0008896102890296","DOI":"10.5220\/0008896102890296"},{"key":"11962_CR22","doi-asserted-by":"publisher","unstructured":"Wang R, Xu K, Feng H, Chen W (2020) Hybrid rnn-ann based deep physiological network for pain recognition. In: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 5584\u20135587 . https:\/\/doi.org\/10.1109\/EMBC44109.2020.9175247","DOI":"10.1109\/EMBC44109.2020.9175247"},{"key":"11962_CR23","doi-asserted-by":"publisher","DOI":"10.3791\/59057","volume":"146","author":"S Gruss","year":"2019","unstructured":"Gruss S, Geiger M, Werner P, Wilhelm O, Traue HC, Al-Hamadi A, Walter S (2019) Multi-modal signals for analyzing pain responses to thermal and electrical stimuli. JoVE 146:e59057. https:\/\/doi.org\/10.3791\/59057","journal-title":"JoVE"},{"issue":"9","key":"11962_CR24","doi-asserted-by":"publisher","first-page":"1828","DOI":"10.3390\/life13091828","volume":"13","author":"E Othman","year":"2023","unstructured":"Othman E, Werner P, Saxen F, Al-Hamadi A, Gruss S, Walter S (2023) Automated electrodermal activity and facial expression analysis for continuous pain intensity monitoring on the x-ite pain database. Life 13(9):1828. https:\/\/doi.org\/10.3390\/life13091828","journal-title":"Life"},{"key":"11962_CR25","doi-asserted-by":"publisher","unstructured":"Wang H, Nienaber S, Dinges L, Jung M, Al-Hamadi A (2026) Towards a reliable multimodal ai monitoring system for pain detection and quantification. In: R\u00f6ning, J., Filipe, J. (eds.) Robotics, Computer Vision and Intelligent Systems, pp. 164\u2013181. Springer, Cham . https:\/\/doi.org\/10.1007\/978-3-032-00986-9_12","DOI":"10.1007\/978-3-032-00986-9_12"},{"key":"11962_CR26","doi-asserted-by":"publisher","unstructured":"Othman E, Werner P, Saxen F, Al-Hamadi A, Walter S (2019) Cross-database evaluation of pain recognition from facial video. In: 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), pp. 181\u2013186 . https:\/\/doi.org\/10.1109\/ISPA.2019.8868562","DOI":"10.1109\/ISPA.2019.8868562"},{"key":"11962_CR27","doi-asserted-by":"publisher","unstructured":"Benitez-Quiroz CF, Srinivasan R, Martinez AM (2016) Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5562\u20135570 . https:\/\/doi.org\/10.1109\/CVPR.2016.600","DOI":"10.1109\/CVPR.2016.600"},{"issue":"6","key":"11962_CR28","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1109\/TPAMI.2014.2366127","volume":"37","author":"E Sariyanidi","year":"2015","unstructured":"Sariyanidi E, Gunes H, Cavallaro A (2015) Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans Pattern Anal Mach Intell 37(6):1113\u20131133. https:\/\/doi.org\/10.1109\/TPAMI.2014.2366127","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"11962_CR29","doi-asserted-by":"publisher","unstructured":"Werner P, Al-Hamadi A, Gruss S, Walter S (2019) Twofold-multimodal pain recognition with the x-ite pain database. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 290\u2013296 . https:\/\/doi.org\/10.1109\/ACIIW.2019.8925061","DOI":"10.1109\/ACIIW.2019.8925061"},{"key":"11962_CR30","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2023) Attention Is All You Need . arXiv:https:\/\/arxiv.org\/abs\/1706.03762"},{"key":"11962_CR31","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1016\/j.neucom.2018.07.028","volume":"317","author":"Z Yu","year":"2018","unstructured":"Yu Z, Liu G, Liu Q, Deng J (2018) Spatio-temporal convolutional features with nested lstm for facial expression recognition. Neurocomputing 317:50\u201357. https:\/\/doi.org\/10.1016\/j.neucom.2018.07.028","journal-title":"Neurocomputing"},{"key":"11962_CR32","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1016\/j.neucom.2019.11.127","volume":"444","author":"X Sun","year":"2021","unstructured":"Sun X, Xia P, Ren F (2021) Multi-attention based deep neural network with hybrid features for dynamic sequential facial expression recognition. Neurocomputing 444:378\u2013389. https:\/\/doi.org\/10.1016\/j.neucom.2019.11.127","journal-title":"Neurocomputing"},{"key":"11962_CR33","doi-asserted-by":"crossref","unstructured":"Ridouan A, Bohi A, Mourchid Y (2025) Improving Pain Classification using Spatio-Temporal Deep Learning Approaches with Facial Expressions . arXiv:https:\/\/arxiv.org\/abs\/2501.06787","DOI":"10.1117\/12.3055496"},{"key":"11962_CR34","doi-asserted-by":"publisher","unstructured":"Bargshady G, Joseph C, Hirachan N, Goecke R, Rojas RF (2024) Acute pain recognition from facial expression videos using vision transformers. In: 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1\u20134 . https:\/\/doi.org\/10.1109\/EMBC53108.2024.10781616","DOI":"10.1109\/EMBC53108.2024.10781616"},{"issue":"9","key":"11962_CR35","doi-asserted-by":"publisher","first-page":"5151","DOI":"10.3390\/app15095151","volume":"15","author":"J Vianto","year":"2025","unstructured":"Vianto J, Divakaran A, Yang H, Yeom S, Kim S, Kim S, Shin J (2025) Multimodal model for automated pain assessment: leveraging video and fnirs. Appl Sci 15(9):5151. https:\/\/doi.org\/10.3390\/app15095151","journal-title":"Appl Sci"},{"key":"11962_CR36","doi-asserted-by":"publisher","unstructured":"Werner P, Al-Hamadi A, Walter S (2017) Analysis of facial expressiveness during experimentally induced heat pain. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 176\u2013180. https:\/\/doi.org\/10.1109\/ACIIW.2017.8272610","DOI":"10.1109\/ACIIW.2017.8272610"},{"key":"11962_CR37","unstructured":"Singh G, M\u00e9moli F, Carlsson G, Botsch M (2007) Eurographics symposium on point-based graphics. The Eurographics Association Prague"},{"key":"11962_CR38","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2025.111260","volume":"198","author":"H Wang","year":"2025","unstructured":"Wang H, Nienaber S, Dinges L, Al-Hamadi A (2025) Ai-based bi-modal fusion system for automated clinical pain monitoring. Comput Biol Med 198:111260. https:\/\/doi.org\/10.1016\/j.compbiomed.2025.111260","journal-title":"Comput Biol Med"},{"key":"11962_CR39","unstructured":"Lundberg SM, Lee S-I (2017) A unified approach to interpreting model predictions. Curran Associates, Inc. . http:\/\/papers.nips.cc\/paper\/7062-a-unified-approach-to-interpreting-model-predictions.pdf"},{"key":"11962_CR40","unstructured":"Hinton G, Roweis S (2002) Stochastic neighbor embedding. In: Proceedings of the 16th International Conference on Neural Information Processing Systems. NIPS\u201902, pp. 857\u2013864. MIT Press, Cambridge, MA, USA"},{"issue":"86","key":"11962_CR41","first-page":"2579","volume":"9","author":"L Maaten","year":"2008","unstructured":"Maaten L, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(86):2579\u20132605","journal-title":"J Mach Learn Res"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-026-11962-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-026-11962-y","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-026-11962-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T03:26:07Z","timestamp":1777346767000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-026-11962-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,24]]},"references-count":41,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["11962"],"URL":"https:\/\/doi.org\/10.1007\/s00521-026-11962-y","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,24]]},"assertion":[{"value":"7 November 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 February 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 March 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"208"}}