{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T17:10:53Z","timestamp":1784740253316,"version":"3.55.0"},"reference-count":50,"publisher":"Institute of Electronics, Information and Communications Engineers (IEICE)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IEICE Trans. Inf. & Syst."],"published-print":{"date-parts":[[2021,12,1]]},"DOI":"10.1587\/transinf.2021edp7065","type":"journal-article","created":{"date-parts":[[2021,11,30]],"date-time":"2021-11-30T22:41:03Z","timestamp":1638312063000},"page":"2184-2194","source":"Crossref","is-referenced-by-count":13,"title":["Multimodal-Based Stream Integrated Neural Networks for Pain Assessment"],"prefix":"10.1587","volume":"E104.D","author":[{"given":"Ruicong","family":"ZHI","sequence":"first","affiliation":[{"name":"School of Computer and Communication Engineering, University of Science and Technology Beijing"},{"name":"Beijing Key Laboratory of Knowledge Engineering for Materials Science"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caixia","family":"ZHOU","sequence":"additional","affiliation":[{"name":"School of Computer and Communication Engineering, University of Science and Technology Beijing"},{"name":"Beijing Key Laboratory of Knowledge Engineering for Materials Science"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junwei","family":"YU","sequence":"additional","affiliation":[{"name":"School of Computer and Communication Engineering, University of Science and Technology Beijing"},{"name":"Beijing Key Laboratory of Knowledge Engineering for Materials Science"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tingting","family":"LI","sequence":"additional","affiliation":[{"name":"School of Computer and Communication Engineering, University of Science and Technology Beijing"},{"name":"Beijing Key Laboratory of Knowledge Engineering for Materials Science"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ghada","family":"ZAMZMI","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of South Florida"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"532","reference":[{"key":"1","doi-asserted-by":"crossref","unstructured":"[1] B. Gholami, W.M. Haddad, and A.R. Tannenbaum, \u201cRelevance vector machine learning for neonate pain intensity assessment using digital imaging,\u201d IEEE Trans. Biomed. Eng., vol.57, no.6,pp.1457-1466, 2010. 10.1109\/tbme.2009.2039214","DOI":"10.1109\/TBME.2009.2039214"},{"key":"2","doi-asserted-by":"crossref","unstructured":"[2] K.M. Prkachin and P.E. Solomon, \u201cThe structure, reliability and validity of pain expression: Evidence from patients with shoulder pain,\u201d Pain, vol.139, no.2, pp.267-274, 2008. 10.1016\/j.pain.2008.04.010","DOI":"10.1016\/j.pain.2008.04.010"},{"key":"3","doi-asserted-by":"crossref","unstructured":"[3] C.C. Johnston and M.E. Strada, \u201cAcute pain response in infants: a multidimensional description,\u201d Pain, vol.24, no.3, pp.373-382, 1986. 10.1016\/0304-3959(86)90123-5","DOI":"10.1016\/0304-3959(86)90123-5"},{"key":"4","unstructured":"[4] M.H. Willis, S.I. Merkel, T. Voepel-Lewis, and S. Malviya, \u201cFlacc behavioral pain assessment scale: a comparison with the child's self-report,\u201d Pediatric nursing, vol.29, no.3, p.195, 2003."},{"key":"5","doi-asserted-by":"crossref","unstructured":"[5] S. Walter, S. Gruss, H. Ehleiter, J. Tan, H.C. Traue, P. Werner, A. Al-Hamadi, S. Crawcour, A.O. Andrade, and G.M. da Silva, \u201cThe biovid heat pain database data for the advancement and systematic validation of an automated pain recognition system,\u201d 2013 IEEE international conference on cybernetics (CYBCO), pp.128-131, IEEE, 2013. 10.1109\/cybconf.2013.6617456","DOI":"10.1109\/CYBConf.2013.6617456"},{"key":"6","doi-asserted-by":"publisher","unstructured":"[6] G. Zamzmi, R. Kasturi, D. Goldgof, R. Zhi, T. Ashmeade, and Y. Sun, \u201cA review of automated pain assessment in infants: features, classification tasks, and databases,\u201d IEEE Rev. Biomed. Eng., vol.11, pp.77-96, 2017. 10.1109\/rbme.2017.2777907","DOI":"10.1109\/RBME.2017.2777907"},{"key":"7","unstructured":"[7] A.d.O. Andrade, Decomposition and analysis of electromyographic signals, Ph.D. thesis, University of Reading, 2005."},{"key":"8","doi-asserted-by":"publisher","unstructured":"[8] M. Tavakolian and A. Hadid, \u201cA spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics,\u201d International Journal of Computer Vision, vol.127, no.10, pp.1413-1425, 2019. 10.1007\/s11263-019-01191-3","DOI":"10.1007\/s11263-019-01191-3"},{"key":"9","doi-asserted-by":"publisher","unstructured":"[9] M. Tavakolian, M.B. Lopez, and L. Liu, \u201cSelf-supervised pain intensity estimation from facial videos via statistical spatiotemporal distillation,\u201d Pattern Recognition Letters, vol.140, pp.26-33, 2020. 10.1016\/j.patrec.2020.09.012","DOI":"10.1016\/j.patrec.2020.09.012"},{"key":"10","doi-asserted-by":"publisher","unstructured":"[10] Y. Huang, L. Qing, S. Xu, L. Wang, and Y. Peng, \u201cHybnet: a hybrid network structure for pain intensity estimation,\u201d The Visual Computer, pp.1-12, 2021. 10.1007\/s00371-021-02056-y","DOI":"10.1007\/s00371-021-02056-y"},{"key":"11","doi-asserted-by":"crossref","unstructured":"[11] L. Nanni, S. Brahnam, and A. Lumini, \u201cA local approach based on a local binary patterns variant texture descriptor for classifying pain states,\u201d Expert Systems with Applications, vol.37, no.12, pp.7888-7894, 2010. 10.1016\/j.eswa.2010.04.048","DOI":"10.1016\/j.eswa.2010.04.048"},{"key":"12","doi-asserted-by":"publisher","unstructured":"[12] M.S. Aung, S. Kaltwang, B. Romera-Paredes, B. Martinez, A. Singh, M. Cella, M. Valstar, H. Meng, A. Kemp, M. Shafizadeh, A.C. Elkins, N. Kanakam, A. de Rothschild, N. Tyler, P.J. Watson, A.C. de C. Williams, M. Pantic, and N. Bianchi-Berthouze, \u201cThe automatic detection of chronic pain-related expression: requirements, challenges and the multimodal emopain dataset,\u201d IEEE transactions on affective computing, vol.7, no.4, pp.435-451, 2015. 10.1109\/taffc.2015.2462830","DOI":"10.1109\/TAFFC.2015.2462830"},{"key":"13","doi-asserted-by":"crossref","unstructured":"[13] S. Agrawal and P. Khatri, \u201cFacial expression detection techniques: based on viola and jones algorithm and principal component analysis,\u201d 2015 Fifth International Conference on Advanced Computing & Communication Technologies, pp.108-112, IEEE, 2015. 10.1109\/acct.2015.32","DOI":"10.1109\/ACCT.2015.32"},{"key":"14","doi-asserted-by":"crossref","unstructured":"[14] G. Lu, X. Li, and H. Li, \u201cResearch on recognition for facial expression of pain in neonates,\u201d Acta Optica Sinica, vol.28, no.11, pp.2109-2114, 2008.","DOI":"10.3788\/AOS20082811.2109"},{"key":"15","doi-asserted-by":"crossref","unstructured":"[15] S. Kaltwang, O. Rudovic, and M. Pantic, \u201cContinuous pain intensity estimation from facial expressions,\u201d International Symposium on Visual Computing, pp.368-377, Springer, 2012. 10.1007\/978-3-642-33191-6_36","DOI":"10.1007\/978-3-642-33191-6_36"},{"key":"16","doi-asserted-by":"crossref","unstructured":"[16] A.B. Ashraf, S. Lucey, J.F. Cohn, T. Chen, Z. Ambadar, K.M. Prkachin, and P.E. Solomon, \u201cThe painful face-pain expression recognition using active appearance models,\u201d Image and vision computing, vol.27, no.12, pp.1788-1796, 2009. 10.1016\/j.imavis.2009.05.007","DOI":"10.1016\/j.imavis.2009.05.007"},{"key":"17","doi-asserted-by":"crossref","unstructured":"[17] M. Rupenga and H.B. Vadapalli, \u201cAutomatic spontaneous pain recognition using supervised classification learning algorithms,\u201d 2016 Pattern Recognition Association of South Africa and Robotics and Mechatronics International Conference (PRASA-RobMech), pp.1-6, IEEE, 2016. 10.1109\/robomech.2016.7813150","DOI":"10.1109\/RoboMech.2016.7813150"},{"key":"18","doi-asserted-by":"publisher","unstructured":"[18] P. Werner, A. Al-Hamadi, K. Limbrecht-Ecklundt, S. Walter, and H.C. Traue, \u201cHead movements and postures as pain behavior,\u201d PloS one, vol.13, no.2, p.e0192767, 2018. 10.1371\/journal.pone.0192767","DOI":"10.1371\/journal.pone.0192767"},{"key":"19","doi-asserted-by":"crossref","unstructured":"[19] A. Ruiz, O. Rudovic, X. Binefa, and M. Pantic, \u201cMulti-instance dynamic ordinal random fields for weakly-supervised pain intensity estimation,\u201d Asian Conference on Computer Vision, pp.171-186, Springer, 2016. 10.1007\/978-3-319-54184-6_11","DOI":"10.1007\/978-3-319-54184-6_11"},{"key":"20","doi-asserted-by":"crossref","unstructured":"[20] R. Yang, S. Tong, M. Bordallo, E. Boutellaa, J. Peng, X. Feng, and A. Hadid, \u201cOn pain assessment from facial videos using spatio-temporal local descriptors,\u201d 2016 Sixth International Conference on Image Processing Theory, Tools and Applications (IPTA), pp.1-6, IEEE, 2016. 10.1109\/ipta.2016.7820930","DOI":"10.1109\/IPTA.2016.7820930"},{"key":"21","doi-asserted-by":"publisher","unstructured":"[21] J. Chen, Z. Chi, and H. Fu, \u201cA new framework with multiple tasks for detecting and locating pain events in video,\u201d Computer Vision and Image Understanding, vol.155, pp.113-123, 2017. 10.1016\/j.cviu.2016.11.003","DOI":"10.1016\/j.cviu.2016.11.003"},{"key":"22","doi-asserted-by":"crossref","unstructured":"[22] P. Werner, A. Al-Hamadi, K. Limbrecht-Ecklundt, S. Walter, S. Gruss, and H.C. Traue, \u201cAutomatic pain assessment with facial activity descriptors,\u201d IEEE Transactions on Affective Computing, vol.8, no.3, pp.286-299, 2016. 10.1109\/taffc.2016.2537327","DOI":"10.1109\/TAFFC.2016.2537327"},{"key":"23","doi-asserted-by":"crossref","unstructured":"[23] D. Bourou, A. Pampouchidou, M. Tsiknakis, K. Marias, and P. Simos, \u201cVideo-based pain level assessment: Feature selection and inter-subject variability modeling,\u201d 2018 41st International Conference on Telecommunications and Signal Processing (TSP), pp.1-6, IEEE, 2018. 10.1109\/tsp.2018.8441252","DOI":"10.1109\/TSP.2018.8441252"},{"key":"24","doi-asserted-by":"crossref","unstructured":"[24] R. Kharghanian, A. Peiravi, and F. Moradi, \u201cPain detection from facial images using unsupervised feature learning approach,\u201d 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp.419-422, IEEE, 2016. 10.1109\/embc.2016.7590729","DOI":"10.1109\/EMBC.2016.7590729"},{"key":"25","doi-asserted-by":"publisher","unstructured":"[25] P. Rodriguez, G. Cucurull, J. Gonz\u00e0lez, J.M. Gonfaus, K. Nasrollahi, T.B. Moeslund, and F.X. Roca, \u201cDeep pain: Exploiting long short-term memory networks for facial expression classification,\u201d IEEE Trans. Cybern., 2017. 10.1109\/tcyb.2017.2662199","DOI":"10.1109\/TCYB.2017.2662199"},{"key":"26","doi-asserted-by":"crossref","unstructured":"[26] J. Egede, M. Valstar, and B. Martinez, \u201cFusing deep learned and hand-crafted features of appearance, shape, and dynamics for automatic pain estimation,\u201d 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017), pp.689-696, IEEE, 2017. 10.1109\/fg.2017.87","DOI":"10.1109\/FG.2017.87"},{"key":"27","doi-asserted-by":"crossref","unstructured":"[27] M. Bellantonio, M.A. Haque, P. Rodriguez, K. Nasrollahi, T. Telve, S. Escalera, J. Gonzalez, T.B. Moeslund, P. Rasti, and G.Anbarjafari, \u201cSpatio-temporal pain recognition in cnn-based super-resolved facial images,\u201d Video Analytics. Face and Facial Expression Recognition and Audience Measurement, pp.151-162, Springer, 2016. 10.1007\/978-3-319-56687-0_13","DOI":"10.1007\/978-3-319-56687-0_13"},{"key":"28","doi-asserted-by":"crossref","unstructured":"[28] J. Zhou, X. Hong, F. Su, and G. Zhao, \u201cRecurrent convolutional neural network regression for continuous pain intensity estimation in video,\u201d Proc. IEEE conference on computer vision and pattern recognition workshops, pp.84-92, 2016. 10.1109\/cvprw.2016.191","DOI":"10.1109\/CVPRW.2016.191"},{"key":"29","doi-asserted-by":"publisher","unstructured":"[29] J. Walsh, C. Eccleston, and E. Keogh, \u201cPain communication through body posture: The development and validation of a stimulus set,\u201d PAIN\u00ae, vol.155, no.11, pp.2282-2290, 2014. 10.1016\/j.pain.2014.08.019","DOI":"10.1016\/j.pain.2014.08.019"},{"key":"30","doi-asserted-by":"crossref","unstructured":"[30] T.A. Olugbade, N. Bianchi-Berthouze, N. Marquardt, and A.C. Williams, \u201cPain level recognition using kinematics and muscle activity for physical rehabilitation in chronic pain,\u201d 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), pp.243-249, IEEE, 2015. 10.1109\/acii.2015.7344578","DOI":"10.1109\/ACII.2015.7344578"},{"key":"31","unstructured":"[31] C. Wang, T.A. Olugbade, A. Mathur, A.C.D.C. Williams, N.D. Lane, and N. Bianchi-Berthouze, \u201cAutomatic detection of protective behavior in chronic pain physical rehabilitation: A recurrent neural network approach,\u201d arXiv preprint arXiv:1902.08990, 2019."},{"key":"32","doi-asserted-by":"publisher","unstructured":"[32] M. K\u00e4chele, M. Amirian, P. Thiam, P. Werner, S. Walter, G. Palm, and F. Schwenker, \u201cAdaptive confidence learning for the personalization of pain intensity estimation systems,\u201d Evolving Systems, vol.8, no.1, pp.71-83, 2017. 10.1007\/s12530-016-9158-4","DOI":"10.1007\/s12530-016-9158-4"},{"key":"33","doi-asserted-by":"crossref","unstructured":"[33] S. Walter, S. Gruss, K. Limbrecht-Ecklundt, H.C. Traue, P. Werner, A. Al-Hamadi, N. Diniz, G.M. da Silva, and A.O. Andrade, \u201cAutomatic pain quantification using autonomic parameters,\u201d Psychology & Neuroscience, vol.7, no.3, pp.363-380, 2014. 10.3922\/j.psns.2014.041","DOI":"10.3922\/j.psns.2014.041"},{"key":"34","doi-asserted-by":"crossref","unstructured":"[34] P. Werner, A. Al-Hamadi, R. Niese, S. Walter, S. Gruss, and H.C. Traue, \u201cAutomatic pain recognition from video and biomedical signals,\u201d 2014 22nd International Conference on Pattern Recognition, pp.4582-4587, IEEE, 2014. 10.1109\/icpr.2014.784","DOI":"10.1109\/ICPR.2014.784"},{"key":"35","doi-asserted-by":"crossref","unstructured":"[35] S.D. Subramaniam and B. Dass, \u201cAutomated nociceptive pain assessment using physiological signals and a hybrid deep learning network,\u201d IEEE Sensors J., 2020.","DOI":"10.1109\/JSEN.2020.3023656"},{"key":"36","doi-asserted-by":"crossref","unstructured":"[36] M. K\u00e4chele, P. Thiam, M. Amirian, P. Werner, S. Walter, F. Schwenker, and G. Palm, \u201cMultimodal data fusion for person-independent, continuous estimation of pain intensity,\u201d International Conference on Engineering Applications of Neural Networks, pp.275-285, Springer, 2015. 10.1007\/978-3-319-23983-5_26","DOI":"10.1007\/978-3-319-23983-5_26"},{"key":"37","doi-asserted-by":"crossref","unstructured":"[37] M.A. Haque, R.B. Bautista, F. Noroozi, K. Kulkarni, C.B. Laursen, R. Irani, M. Bellantonio, S. Escalera, G. Anbarjafari, K. Nasrollahi, O.K. Andersen, E.G. Spaich, and T.B. Moeslund, \u201cDeep multimodal pain recognition: a database and comparison of spatio-temporal visual modalities,\u201d 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp.250-257, IEEE, 2018. 10.1109\/fg.2018.00044","DOI":"10.1109\/FG.2018.00044"},{"key":"38","unstructured":"[38] P. Thiam, V. Kessler, M. Amirian, P. Bellmann, G. Layher, Y. Zhang, M. Velana, S. Gruss, S. Walter, H.C. Traue, et al., \u201cMulti-modal pain intensity recognition based on the senseemotion database,\u201d IEEE Transactions on Affective Computing, 2019."},{"key":"39","doi-asserted-by":"crossref","unstructured":"[39] J.C. Gower, \u201cGeneralized procrustes analysis,\u201d Psychometrika, vol.40, no.1, pp.33-51, 1975. 10.1007\/bf02291478","DOI":"10.1007\/BF02291478"},{"key":"40","doi-asserted-by":"crossref","unstructured":"[40] T. Hassner, S. Harel, E. Paz, and R. Enbar, \u201cEffective face frontalization in unconstrained images,\u201d Proc. IEEE conference on computer vision and pattern recognition, pp.4295-4304, 2015. 10.1109\/cvpr.2015.7299058","DOI":"10.1109\/CVPR.2015.7299058"},{"key":"41","unstructured":"[41] F.N. Iandola, S. Han, M.W. Moskewicz, K. Ashraf, W.J. Dally, and K. Keutzer, \u201cSqueezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5 mb model size,\u201d arXiv preprint arXiv:1602.07360, 2016."},{"key":"42","unstructured":"[42] K. Simonyan and A. Zisserman, \u201cTwo-stream convolutional networks for action recognition in videos,\u201d arXiv preprint arXiv:1406.2199, 2014."},{"key":"43","doi-asserted-by":"publisher","unstructured":"[43] D.H. Hubel and T.N. Wiesel, \u201cReceptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat,\u201d Journal of neurophysiology, vol.28, no.2, pp.229-289, 1965. 10.1152\/jn.1965.28.2.229","DOI":"10.1152\/jn.1965.28.2.229"},{"key":"44","doi-asserted-by":"crossref","unstructured":"[44] M. Livingstone and D. Hubel, \u201cSegregation of form, color, movement, and depth: anatomy, physiology, and perception,\u201d Science, vol.240, no.4853, pp.740-749, 1988. 10.1126\/science.3283936","DOI":"10.1126\/science.3283936"},{"key":"45","doi-asserted-by":"crossref","unstructured":"[45] T.-Y. Lin, P. Doll\u00e1r, R. Girshick, K. He, B. Hariharan, and S.Belongie, \u201cFeature pyramid networks for object detection,\u201d Proc. IEEE conference on computer vision and pattern recognition, pp.2117-2125, 2017. 10.1109\/cvpr.2017.106","DOI":"10.1109\/CVPR.2017.106"},{"key":"46","doi-asserted-by":"crossref","unstructured":"[46] C. Feichtenhofer, H. Fan, J. Malik, and K. He, \u201cSlowfast networks for video recognition,\u201d Proc. IEEE\/CVF International Conference on Computer Vision, pp.6202-6211, 2019. 10.1109\/iccv.2019.00630","DOI":"10.1109\/ICCV.2019.00630"},{"key":"47","doi-asserted-by":"crossref","unstructured":"[47] D. Lopez-Martinez and R. Picard, \u201cMulti-task neural networks for personalized pain recognition from physiological signals,\u201d 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp.181-184, IEEE, 2017. 10.1109\/aciiw.2017.8272611","DOI":"10.1109\/ACIIW.2017.8272611"},{"key":"48","doi-asserted-by":"crossref","unstructured":"[48] D. Lopez-Martinez and R. Picard, \u201cContinuous pain intensity estimation from autonomic signals with recurrent neural networks,\u201d 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp.5624-5627, IEEE, 2018. 10.1109\/embc.2018.8513575","DOI":"10.1109\/EMBC.2018.8513575"},{"key":"49","doi-asserted-by":"publisher","unstructured":"[49] G.M. Foody, \u201cClassification accuracy comparison: hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority,\u201d Remote Sensing of Environment, vol.113, no.8, pp.1658-1663, 2009. 10.1016\/j.rse.2009.03.014","DOI":"10.1016\/j.rse.2009.03.014"},{"key":"50","doi-asserted-by":"crossref","unstructured":"[50] P. Werner, A. Al-Hamadi, R. Niese, S. Walter, S. Gruss, and H.C. Traue, \u201cAutomatic pain recognition from video and biomedical signals,\u201d 2014 22nd International Conference on Pattern Recognition, pp.4582-4587, IEEE, 2014. 10.1109\/icpr.2014.784","DOI":"10.1109\/ICPR.2014.784"}],"container-title":["IEICE Transactions on Information and Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jstage.jst.go.jp\/article\/transinf\/E104.D\/12\/E104.D_2021EDP7065\/_pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,8]],"date-time":"2024-05-08T05:08:34Z","timestamp":1715144914000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jstage.jst.go.jp\/article\/transinf\/E104.D\/12\/E104.D_2021EDP7065\/_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,1]]},"references-count":50,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2021]]}},"URL":"https:\/\/doi.org\/10.1587\/transinf.2021edp7065","relation":{},"ISSN":["0916-8532","1745-1361"],"issn-type":[{"value":"0916-8532","type":"print"},{"value":"1745-1361","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,1]]},"article-number":"2021EDP7065"}}