{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T13:18:30Z","timestamp":1753881510160,"version":"3.41.2"},"reference-count":32,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2021,11,15]],"date-time":"2021-11-15T00:00:00Z","timestamp":1636934400000},"content-version":"vor","delay-in-days":318,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100009950","name":"Ministry of Education","doi-asserted-by":"publisher","award":["201902168015"],"award-info":[{"award-number":["201902168015"]}],"id":[{"id":"10.13039\/100009950","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Complexity"],"published-print":{"date-parts":[[2021,1]]},"abstract":"<jats:p>Microexpressions have extremely high due value in national security, public safety, medical, and other fields. However, microexpressions have characteristics that are obviously different from macroexpressions, such as short duration and weak changes, which greatly increase the difficulty of microexpression recognition work. In this paper, we propose a microexpression recognition method based on multimodal fusion through a comparative study of traditional microexpression recognition algorithms such as LBP algorithm and CNN and LSTM deep learning algorithms. The method couples the separate microexpression image information with the corresponding body temperature information to establish a multimodal fusion microexpression database. This paper firstly introduces how to build a multimodal fusion microexpression database in a laboratory environment, secondly compares the recognition accuracy of LBP, LSTM, and CNN\u2009+\u2009LSTM networks for microexpressions, and finally selects the superior CNN\u2009+\u2009LSTM network in the comparison results for model training and testing on the test set under separate microexpression database and multimodal fusion database. The experimental results show that a microexpression recognition method based on multimodal fusion designed in this paper is more accurate than unimodal recognition in multimodal recognition after feature fusion, and its recognition rate reaches 75.1%, which proves that the method is feasible and effective in improving microexpression recognition rate and has good practical value.<\/jats:p>","DOI":"10.1155\/2021\/5221950","type":"journal-article","created":{"date-parts":[[2021,11,16]],"date-time":"2021-11-16T01:05:16Z","timestamp":1637024716000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Research on a Microexpression Recognition Technology Based on Multimodal Fusion"],"prefix":"10.1155","volume":"2021","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9010-3921","authenticated-orcid":false,"given":"Jie","family":"Kang","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8725-8749","authenticated-orcid":false,"given":"Xiao Ying","family":"Chen","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9406-2653","authenticated-orcid":false,"given":"Qi Yuan","family":"Liu","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6718-9210","authenticated-orcid":false,"given":"Si Han","family":"Jin","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0997-6280","authenticated-orcid":false,"given":"Cheng Han","family":"Yang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1846-2812","authenticated-orcid":false,"given":"Cong","family":"Hu","sequence":"additional","affiliation":[]}],"member":"311","published-online":{"date-parts":[[2021,11,15]]},"reference":[{"key":"e_1_2_9_1_2","doi-asserted-by":"publisher","DOI":"10.1080\/02699939208411068"},{"key":"e_1_2_9_2_2","doi-asserted-by":"crossref","unstructured":"BaiM. Detection of micro-expression recognition based on spatio-temporal modelling and spatial attention Proceedings of the ICMI\u201920: International Conference on Multimodal Interaction October 2020 Netherlands Utrecht.","DOI":"10.1145\/3382507.3421160"},{"key":"e_1_2_9_3_2","doi-asserted-by":"crossref","unstructured":"WenJ. YangW. WangL. WeiW. TanS. andWuY. Cross-database micro expression recognition based on apex frame optical flow and multi-head self-attention Proceedings of the International Symposium on Parallel Architectures Algorithms and Programming December 2020 Springer Singapore 128\u2013139.","DOI":"10.1007\/978-981-16-0010-4_12"},{"key":"e_1_2_9_4_2","unstructured":"LiS. Research on facial micro-expression recognition method based on deep learning 2020 China University of Mining and Technology Xuzhou China Master thesis."},{"key":"e_1_2_9_5_2","doi-asserted-by":"publisher","DOI":"10.3788\/LOP57.141504"},{"key":"e_1_2_9_6_2","doi-asserted-by":"crossref","unstructured":"KhorH. Q. SeeJ. PhanR. C. W. andLinW. Enriched long-term recurrent convolutional network for facial micro-expression recognition Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) May 2018 IEEE Xian China 667\u2013674.","DOI":"10.1109\/FG.2018.00105"},{"key":"e_1_2_9_7_2","doi-asserted-by":"crossref","unstructured":"LiuY. DuH. ZhengL. andGedeonT. A neural micro-expression recognizer Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) May 2019 Lille France IEEE 1\u20134.","DOI":"10.1109\/FG.2019.8756583"},{"key":"e_1_2_9_8_2","doi-asserted-by":"publisher","DOI":"10.1037\/e574212011-002"},{"key":"e_1_2_9_9_2","doi-asserted-by":"crossref","unstructured":"ShreveM. GodavarthyS. ManoharV. GoldgofD. andSarkarS. Towards macro-and micro-expression spotting in video using strain patterns Proceedings of the 2009 Workshop on Applications of Computer Vision (WACV) December 2009 Snowbird UT USA IEEE 1\u20136.","DOI":"10.1109\/WACV.2009.5403044"},{"key":"e_1_2_9_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10919-011-0120-7"},{"key":"e_1_2_9_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/tpami.2007.1110"},{"key":"e_1_2_9_12_2","doi-asserted-by":"crossref","unstructured":"WuQ. ShenX. andFuX. The machine knows what you are hiding: an automatic micro-expression recognition system Proceedings of the International Conference on Affective Computing and Intelligent Interaction October 2011 Berlin Germany Springer 152\u2013162 https:\/\/doi.org\/10.1007\/978-3-642-24571-8_16 2-s2.0-80054843054.","DOI":"10.1007\/978-3-642-24571-8_16"},{"key":"e_1_2_9_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2012.06.008"},{"key":"e_1_2_9_14_2","first-page":"577","article-title":"Micro-expression spotting using optical flow magnitude estimation","volume":"51","author":"Jiang B.","year":"2017","journal-title":"Journal of ZheJiang University (Engineering Science)"},{"key":"e_1_2_9_15_2","first-page":"1282","article-title":"Micro-expression recognition based on local region method","volume":"39","author":"Zhang Y.","year":"2019","journal-title":"Journal of Computer Applications"},{"key":"e_1_2_9_16_2","doi-asserted-by":"crossref","unstructured":"TakalkarM. A.andXuM. Image based facial micro-expression recognition using deep learning on small datasets Proceedings of the 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA) November 2017 Sydney Australia IEEE 1\u20137.","DOI":"10.1109\/DICTA.2017.8227443"},{"key":"e_1_2_9_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/lgrs.2018.2885809"},{"key":"e_1_2_9_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/taffc.2016.2518162"},{"key":"e_1_2_9_19_2","doi-asserted-by":"crossref","unstructured":"SuW. WangY. SuF. andZhaoZ. Micro-expression recognition based on the spatio-temporal feature Proceedings of the 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) July 2018 San Diego CA USA IEEE 1\u20136.","DOI":"10.1109\/ICMEW.2018.8551494"},{"key":"e_1_2_9_20_2","doi-asserted-by":"crossref","unstructured":"LiongS. T.andWongK. Micro-expression recognition using apex frame with phase information Proceedings of the 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) December 2017 Kuala Lumpur Malaysia IEEE 534\u2013537.","DOI":"10.1109\/APSIPA.2017.8282090"},{"key":"e_1_2_9_21_2","doi-asserted-by":"publisher","DOI":"10.18280\/ts.360611"},{"key":"e_1_2_9_22_2","unstructured":"LiY. HuangX. andZhaoG. Can micro-expression be recognized based on single apex frame Proceedings of the International Conference Onlmage October 2018 Athens Greece 1EEE 3094\u20133098."},{"key":"e_1_2_9_23_2","doi-asserted-by":"crossref","unstructured":"ZhouZ. ZhaoG. andPietik\u00e4inenM. Towards a practical lipreading system Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) 2011 June 2011 Colorado Springs CO USA IEEE 137\u2013144.","DOI":"10.1109\/CVPR.2011.5995345"},{"key":"e_1_2_9_24_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1321664111"},{"key":"e_1_2_9_25_2","doi-asserted-by":"publisher","DOI":"10.1155\/2018\/5676095"},{"key":"e_1_2_9_26_2","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/2684851"},{"key":"e_1_2_9_27_2","doi-asserted-by":"publisher","DOI":"10.1155\/2013\/472971"},{"key":"e_1_2_9_28_2","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/8269683"},{"key":"e_1_2_9_29_2","doi-asserted-by":"publisher","DOI":"10.1155\/2017\/2107451"},{"key":"e_1_2_9_30_2","doi-asserted-by":"publisher","DOI":"10.1155\/2021\/6626728"},{"key":"e_1_2_9_31_2","first-page":"478","article-title":"Twitter sentiment analysis based on hopping LSTM-CNN model","author":"Li J. B.","year":"2021","journal-title":"Computer Simulation"},{"key":"e_1_2_9_32_2","first-page":"473","article-title":"LSTM can solve hard long time lag problems","author":"Hochreiter S.","year":"1997","journal-title":"Advances in Neural Information Processing Systems"}],"container-title":["Complexity"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/downloads.hindawi.com\/journals\/complexity\/2021\/5221950.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/downloads.hindawi.com\/journals\/complexity\/2021\/5221950.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1155\/2021\/5221950","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,9]],"date-time":"2024-08-09T23:21:54Z","timestamp":1723245714000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1155\/2021\/5221950"}},"subtitle":[],"editor":[{"given":"Kai","family":"Hu","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2021,1]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1]]}},"alternative-id":["10.1155\/2021\/5221950"],"URL":"https:\/\/doi.org\/10.1155\/2021\/5221950","archive":["Portico"],"relation":{},"ISSN":["1076-2787","1099-0526"],"issn-type":[{"type":"print","value":"1076-2787"},{"type":"electronic","value":"1099-0526"}],"subject":[],"published":{"date-parts":[[2021,1]]},"assertion":[{"value":"2021-07-09","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-28","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"5221950"}}