{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,12]],"date-time":"2026-06-12T00:27:06Z","timestamp":1781224026797,"version":"3.54.1"},"reference-count":48,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2021,5,10]],"date-time":"2021-05-10T00:00:00Z","timestamp":1620604800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"German Academic Exchange Service (DAAD)","award":["91525164"],"award-info":[{"award-number":["91525164"]}]},{"name":"Federal Ministry of Education and Research of Germany (BMBF)","award":["; projects HuBA no. 03ZZ0470, 457 RoboAssist no. 03ZZ0448L, and Robo-Lab no. 03ZZ04X02B within the Zwanzig20 Alliance 458 3Dsensation"],"award-info":[{"award-number":["; projects HuBA no. 03ZZ0470, 457 RoboAssist no. 03ZZ0448L, and Robo-Lab no. 03ZZ04X02B within the Zwanzig20 Alliance 458 3Dsensation"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Prior work on automated methods demonstrated that it is possible to recognize pain intensity from frontal faces in videos, while there is an assumption that humans are very adept at this task compared to machines. In this paper, we investigate whether such an assumption is correct by comparing the results achieved by two human observers with the results achieved by a Random Forest classifier (RFc) baseline model (called RFc-BL) and by three proposed automated models. The first proposed model is a Random Forest classifying descriptors of Action Unit (AU) time series; the second is a modified MobileNetV2 CNN classifying face images that combine three points in time; and the third is a custom deep network combining two CNN branches using the same input as for MobileNetV2 plus knowledge of the RFc. We conduct experiments with X-ITE phasic pain database, which comprises videotaped responses to heat and electrical pain stimuli, each of three intensities. Distinguishing these six stimulation types plus no stimulation was the main 7-class classification task for the human observers and automated approaches. Further, we conducted reduced 5-class and 3-class classification experiments, applied Multi-task learning, and a newly suggested sample weighting method. Experimental results show that the pain assessments of the human observers are significantly better than guessing and perform better than the automatic baseline approach (RFc-BL) by about 1%; however, the human performance is quite poor due to the challenge that pain that is ethically allowed to be induced in experimental studies often does not show up in facial reaction. We discovered that downweighting those samples during training improves the performance for all samples. The proposed RFc and two-CNNs models (using the proposed sample weighting) significantly outperformed the human observer by about 6% and 7%, respectively.<\/jats:p>","DOI":"10.3390\/s21093273","type":"journal-article","created":{"date-parts":[[2021,5,10]],"date-time":"2021-05-10T02:54:58Z","timestamp":1620615298000},"page":"3273","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":33,"title":["Automatic vs. Human Recognition of Pain Intensity from Facial Expression on the X-ITE Pain Database"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4738-1897","authenticated-orcid":false,"given":"Ehsan","family":"Othman","sequence":"first","affiliation":[{"name":"Department of Neuro-Information Technology, Institute for Information Technology and Communications, Otto-von-Guericke University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4762-2926","authenticated-orcid":false,"given":"Philipp","family":"Werner","sequence":"additional","affiliation":[{"name":"Department of Neuro-Information Technology, Institute for Information Technology and Communications, Otto-von-Guericke University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3303-3154","authenticated-orcid":false,"given":"Frerk","family":"Saxen","sequence":"additional","affiliation":[{"name":"Department of Neuro-Information Technology, Institute for Information Technology and Communications, Otto-von-Guericke University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3632-2402","authenticated-orcid":false,"given":"Ayoub","family":"Al-Hamadi","sequence":"additional","affiliation":[{"name":"Department of Neuro-Information Technology, Institute for Information Technology and Communications, Otto-von-Guericke University Magdeburg, 39106 Magdeburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sascha","family":"Gruss","sequence":"additional","affiliation":[{"name":"Department of Medical Psychology, Ulm University, 89081 Ulm, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steffen","family":"Walter","sequence":"additional","affiliation":[{"name":"Department of Medical Psychology, Ulm University, 89081 Ulm, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1016\/j.pmn.2011.10.002","article-title":"Pain assessment in the patient unable to self-report: Position statement with clinical practice recommendations","volume":"12","author":"Herr","year":"2011","journal-title":"Pain Manag. Nurs. Off. J. Am. Soc. Pain Manag. Nurses"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Mieronkoski, R., Syrj\u00e4l\u00e4, E., Jiang, M., Rahmani, A., Pahikkala, T., Liljeberg, P., and Salanter\u00e4, S. (2020). Developing a pain intensity prediction model using facial expression: A feasibility study with electromyography. PLoS ONE, 15.","DOI":"10.1371\/journal.pone.0235545"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1037\/a0014772","article-title":"The social communication model of pain","volume":"50","author":"Craig","year":"2009","journal-title":"Can. Psychol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/1058-9139(92)90001-S","article-title":"The facial expression of pain Better than a thousand words?","volume":"1","author":"Craig","year":"1992","journal-title":"APS J."},{"key":"ref_5","unstructured":"Werner, P., Lopez-Martinez, D., Walte, S., Al-Hamadi, A., Gruss, S., and Picard, R.W. (2019). Automatic Recognition Methods Supporting Pain Assessment: A Survey. IEEE Trans. Affect. Comput."},{"key":"ref_6","first-page":"439","article-title":"Facial expression of pain: An evolutionary account","volume":"25","author":"Williams","year":"2002","journal-title":"Behav. Brain Sci."},{"key":"ref_7","first-page":"e59057","article-title":"Multi-Modal Signals for Analyzing Pain Responses to Thermal and Electrical Stimuli","volume":"146","author":"Gruss","year":"2019","journal-title":"J. Vis. Exp."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1109\/TAFFC.2016.2537327","article-title":"Automatic Pain Assessment with Facial Activity Descriptors","volume":"8","author":"Werner","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Baltrusaitis, T., Robinson, P., and Morency, L.-P. (2016, January 7\u201310). OpenFace: An open source facial behavior analysis toolkit. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.","DOI":"10.1109\/WACV.2016.7477553"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Othman, E., Werner, P., Saxen, F., Al-Hamadi, A., and Walter, S. (2019, January 23\u201325). Cross-Database Evaluation of Pain Recognition from Facial Video. Proceedings of the 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), Zagreb, Croatia.","DOI":"10.1109\/ISPA.2019.8868562"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Werner, P., Al-Hamadi, A., and Walter, S. (2017, January 23\u201326). Analysis of Facial Expressiveness During Experimentally Induced Heat Pain. Proceedings of the Seventh International Conference on Affective Computing and Intelligent InteractionWorkshops and Demos (ACIIW), San Antonio, TX, USA.","DOI":"10.1109\/ACIIW.2017.8272610"},{"key":"ref_12","first-page":"644","article-title":"State of the art-nonverbal communication","volume":"14","author":"Bull","year":"2001","journal-title":"Psychologist"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1037\/1528-3542.3.2.150","article-title":"What do facial expressions convey: Feeling states, behavioral intentions, or action requests?","volume":"3","author":"Horstmann","year":"2003","journal-title":"Emotion"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1037\/h0022736","article-title":"Differential communication of affect by head and body cues","volume":"2","author":"Ekman","year":"1965","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1037\/0022-3514.53.4.712","article-title":"Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion","volume":"53","author":"Ekman","year":"1987","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Smith, F.W., and Rossit, S. (2018). Identifying and detecting facial expressions of emotion in peripheral vision. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0197160"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1037\/0003-066X.48.4.384","article-title":"Facial expression and emotion","volume":"48","author":"Ekman","year":"1993","journal-title":"Am. Psychol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1016\/0304-3959(92)90213-U","article-title":"The Consistency of Facial Expressions of Pain: A Comparison Across Modalities","volume":"51","author":"Prkachin","year":"1992","journal-title":"Pain"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wells, L.J., Gillespie, S.M., and Rotshtein, P. (2016). Identification of Emotional Facial Expressions: Effects of Expression, Intensity, and Sex on Eye Gaze. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0168307"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1016\/0304-3959(87)90073-X","article-title":"Pain expression in neonates: Facial action and cry","volume":"28","author":"Grunau","year":"1987","journal-title":"Pain"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1016\/j.neuroimage.2004.11.043","article-title":"Viewing facial expressions of pain engages cortical areas involved in the direct experience of pain","volume":"25","author":"Botvinick","year":"2005","journal-title":"Neuroimage"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., and Matthews, I. (2011, January 21\u201325). Painful data: The UNBC-McMaster shoulder pain expression archived atabase. Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA.","DOI":"10.1109\/FG.2011.5771462"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/j.pain.2008.04.010","article-title":"The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain","volume":"139","author":"Prkachin","year":"2008","journal-title":"Pain"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/978-3-319-78340-6_7","article-title":"Automatic, Objective, and Efficient Measurement of Pain Using Automated Face Analysis","volume":"Volume 139","author":"Hammal","year":"2018","journal-title":"Social and Interpersonal Dynamics in Pain"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/BF02173080","article-title":"Expressing pain: The communication and interpretation of facial pain signals","volume":"19","author":"Prkachion","year":"1995","journal-title":"Nonverbal Behav."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1016\/j.jpain.2004.06.002","article-title":"On the relationship between self-report and facial expression of pain","volume":"5","author":"Kunz","year":"2004","journal-title":"Pain"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Mende-Siedlecki, P., Qu-Lee, J., Lin, J., Drain, A., and Goharzad, A. (2020). The Delaware Pain Database: A set of painful expressions and corresponding norming data. Pain, 5.","DOI":"10.1097\/PR9.0000000000000853"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Gruss, S., Treister, R., Werner, P., Traue, H.C., Crawcour, S., Andrade, A., and Walter, S. (2015). Pain Intensity Recognition Rates via Biopotential Feature Patterns with Support Vector Machines. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0140330"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Werner, P., Al-Hamadi, A., Limbrecht-Ecklundt, K., Walter, S., and Traue, H.C. (2015). Head movements and postures as pain behavior. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0192767"},{"key":"ref_30","unstructured":"Liu, M., Li, S., Shan, S., and Chen, X. (2013, January 22\u201326). AU-aware Deep Networks for facial expression recognition. Proceedings of the International Conference on Automatic Face and Gesture Recognition, Shanghai, China."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"738","DOI":"10.1016\/j.cub.2014.02.009","article-title":"Automatic decoding of facial movements reveals deceptive pain expressions","volume":"24","author":"Bartlett","year":"2014","journal-title":"Curr. Biol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1797","DOI":"10.1016\/j.imavis.2008.12.010","article-title":"Automatic coding of facial expressions displayed during posed and genuine pain","volume":"27","author":"Littlewort","year":"2009","journal-title":"Image Vis. Comput."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1542\/peds.2015-0029","article-title":"Automated Assessment of Children\u2019s Postoperative Pain Using Computer Vision","volume":"136","author":"Sikka","year":"2015","journal-title":"Pediatrics"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random Forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Werner, P., Al-Hamadi, A., Gruss, S., and Walter, S. (2019, January 3\u20136). Twofold-Multimodal Pain Recognition with the X-ITE Pain Database. Proceedings of the International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), Cambridge, UK.","DOI":"10.1109\/ACIIW.2019.8925061"},{"key":"ref_36","unstructured":"Ruder, S. (2017). An Overview of Multi-Task Learningin Deep Neural Networks. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1093\/nsr\/nwx105","article-title":"An overview of multi-task learning","volume":"5","author":"Zhang","year":"2018","journal-title":"Natl. Sci. Rev. Comput. Sci."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1155\/2009\/542964","article-title":"Assessing pain by facial expression: Facial expression as nexus","volume":"14","author":"Prkachin","year":"2009","journal-title":"Pain Res. Manag."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2898","DOI":"10.1038\/s41598-019-39685-2","article-title":"Testing a Cognitive Control Model of Human Intelligence","volume":"9","author":"Chen","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_40","unstructured":"Chen, Z., Ansari, R., and Wilkie, D.J. (2018). Automated Pain Detection from Facial Expressions using FACS: A Revie. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Saxen, F., Werner, P., and Al-Hamadi, A. (2017, January 22\u201329). Real vs. Fake Emotion Challenge: Learning to Rank Authenticity from Facial Activity Descriptors. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.","DOI":"10.1109\/ICCVW.2017.363"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Othman, E., Saxen, F., Bershadskyy, D., Werner, P., Al-Hamadi, A., and Weimann, J. (2019). Predicting the group contribution behaviour in a public goods game from Face-to-Face Communication. Sensors, 19.","DOI":"10.3390\/s19122786"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks. arXiv.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_44","unstructured":"Caruana, R. (1993, January 27\u201329). Multitask Learning: A Knowledge-Based Source of Inductive Bias. Proceedings of the ICML\u201993: Tenth International Conference on Machine Learning, San Francisco, CA, USA."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1023\/A:1007327622663","article-title":"A Bayesian\/information theoretic model of learning to learn via multiple task sampling","volume":"28","author":"Baxter","year":"1991","journal-title":"Mach. Learn."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1155\/2001\/108098","article-title":"Does Experience Influence Judgements of Pain Behaviour? Evidence from Relatives of Pain Patients and Therapists","volume":"6","author":"Prkachin","year":"2001","journal-title":"Pain Res. Manag."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1350","DOI":"10.1002\/ejp.666","article-title":"Improving recognition of pain by calling attention to its various faces","volume":"19","author":"Kunz","year":"2015","journal-title":"Eur. J. Pain"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Rash, J.A., Prkachin, K.M., Solomon, P.E., and Campbell, T. (2019). Assessing the efficacy of a manual-based intervention for improving the detection of facial pain expression. Eur. J. Pain, 23.","DOI":"10.1002\/ejp.1369"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/3273\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:58:33Z","timestamp":1760162313000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/9\/3273"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,10]]},"references-count":48,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["s21093273"],"URL":"https:\/\/doi.org\/10.3390\/s21093273","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,10]]}}}