{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,19]],"date-time":"2026-10-19T10:29:12Z","timestamp":1792405752249,"version":"4.4.0"},"reference-count":25,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,3,25]],"date-time":"2023-03-25T00:00:00Z","timestamp":1679702400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004242","name":"Princess Nourah bint Abdulrahman University","doi-asserted-by":"publisher","award":["PNURSP2023R178"],"award-info":[{"award-number":["PNURSP2023R178"]}],"id":[{"id":"10.13039\/501100004242","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Brain Sciences"],"abstract":"<jats:p>Recognition of lying is a more complex cognitive process than truth-telling because of the presence of involuntary cognitive cues that are useful to lie recognition. Researchers have proposed different approaches in the literature to solve the problem of lie recognition from either handcrafted and\/or automatic lie features during court trials and police interrogations. Unfortunately, due to the cognitive complexity and the lack of involuntary cues related to lying features, the performances of these approaches suffer and their generalization ability is limited. To improve performance, this study proposed state transition patterns based on hands, body motions, and eye blinking features from real-life court trial videos. Each video frame is represented according to a computed threshold value among neighboring pixels to extract spatial\u2013temporal state transition patterns (STSTP) of the hand and face poses as involuntary cues using fully connected convolution neural network layers optimized with the weights of ResNet-152 learning. In addition, this study computed an eye aspect ratio model to obtain eye blinking features. These features were fused together as a single multi-modal STSTP feature model. The model was built using the enhanced calculated weight of bidirectional long short-term memory. The proposed approach was evaluated by comparing its performance with current state-of-the-art methods. It was found that the proposed approach improves the performance of detecting lies.<\/jats:p>","DOI":"10.3390\/brainsci13040555","type":"journal-article","created":{"date-parts":[[2023,3,27]],"date-time":"2023-03-27T03:31:48Z","timestamp":1679887908000},"page":"555","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Lie Recognition with Multi-Modal Spatial\u2013Temporal State Transition Patterns Based on Hybrid Convolutional Neural Network\u2013Bidirectional Long Short-Term Memory"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1898-7352","authenticated-orcid":false,"given":"Sunusi Bala","family":"Abdullahi","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Faculty of Engineering, King Mongkut\u2019s University of Technology Thonburi, 126 Pracha-Uthit Road, Bang Mod, Thrung Khru, Bangkok 10140, Thailand"},{"name":"Zonal Criminal Investigation Department, Nigeria Police, Louis Edet House Force Headquarters, Abuja, Nigeria, Nigeria Police, Abuja 900211, Nigeria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zakariyya Abdullahi","family":"Bature","sequence":"additional","affiliation":[{"name":"Department of Electrical and Information Engineering, Faculty of Engineering, King Mongkut\u2019s University of Technology Thonburi, 126 Pracha-Uthit Road, Bang Mod, Thrung Khru, Bangkok 10140, Thailand"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lubna A.","family":"Gabralla","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Technology, Applied College, Princess Nourah Bint Abdulrahman University, Riyadh 11671, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3446-4316","authenticated-orcid":false,"given":"Haruna","family":"Chiroma","sequence":"additional","affiliation":[{"name":"College of Computer Science and Engineering, University of Hafr Al Batin, Hafar al-Batin 31991, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1016\/j.patrec.2020.08.014","article-title":"LieToMe: Preliminary study on hand gestures for deception detection via Fisher-LSTM","volume":"138","author":"Avola","year":"2020","journal-title":"Pattern Recognit. Lett."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1016\/S0004-3702(01)00141-2","article-title":"Recognition of gestures in Arabic sign language using neuro-fuzzy systems","volume":"133","author":"Halawan","year":"2001","journal-title":"Artif. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"49167","DOI":"10.1109\/ACCESS.2022.3171850","article-title":"Biometric Information Recognition Using Artificial Intelligence Algorithms: A Performance Comparison","volume":"10","author":"Abdullahi","year":"2022","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2050068","DOI":"10.1142\/S0129065720500689","article-title":"LieToMe: An Ensemble Approach for Deception Detection from Facial Cues","volume":"31","author":"Avola","year":"2021","journal-title":"Int. J. Neural Syst."},{"key":"ref_5","first-page":"2050068","article-title":"Multimodal deception detection using real-life trial data","volume":"2022","author":"Sen","year":"2022","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Ding, M., Zhao, A., Lu, Z., Xiang, T., and Wen, J. (2019, January 15\u201320). Face-focused cross-stream network for deception detection in videos. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00799"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"15911","DOI":"10.1109\/ACCESS.2022.3148132","article-title":"American sign language words recognition using spatio-temporal prosodic and angle features: A sequential learning approach","volume":"10","author":"Abdullahi","year":"2022","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Bhaskaran, N., Nwogu, I., Frank, M.G., and Govindaraju, V. (2011, January 21\u201323). Lie to me: Deceit detection via online behavioral learning. Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA.","DOI":"10.1109\/FG.2011.5771407"},{"key":"ref_9","unstructured":"Jeffry, G., Jenkins, J.L., Burgoon, J.K., Judee, K., and Nunamaker, J.F. (2015, January 27\u201329). Deception is in the eye of the communicator: Investigating pupil diameter variations in automated deception detection interviews. Proceedings of the 2015 IEEE International Conference on Intelligence and Security Informatics (ISI), Baltimore, MD, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1186\/s41935-018-0049-7","article-title":"Validation studies on gender determination from fingerprints with special emphasis on ridge characteristics","volume":"8","author":"Thakar","year":"2022","journal-title":"Egypt. J. Forensic Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1109\/MIS.2005.85","article-title":"Deception detection through automatic, unobtrusive analysis of nonverbal behavior","volume":"20","author":"Meservy","year":"2005","journal-title":"IEEE Intell. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"P\u00e9rez-Rosas, V., Abouelenien, M., Mihalcea, R., and Burzo, M. (2015, January 9\u201313). Deception detection using real-life trial data. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, DC, USA.","DOI":"10.1145\/2818346.2820758"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1042","DOI":"10.1109\/TIFS.2016.2639344","article-title":"Detecting deceptive behavior via integration of discriminative features from multiple modalities","volume":"5","author":"Abouelenien","year":"2017","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Karimi, H., Tang, J., and Li, Y. (2018, January 10\u201313). Toward end-to-end deception detection in videos. Proceedings of the 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA.","DOI":"10.1109\/BigData.2018.8621909"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wu, Z., Singh, B., Davis, L., and Subrahmanian, V. (2018, January 2\u20137). Deception detection in videos. Proceedings of the AAAI conference on artificial intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11502"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Rill-Garc\u00eda, R., Jair, E.H., Villasenor-Pineda, L., and Reyes-Meza, V. (2019, January 16\u201317). High-level features for multimodal deception detection in videos. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00198"},{"key":"ref_17","unstructured":"Krishnamurthy, G., Majumder, N., Poria, S., and Cambria, E. (2018). A deep learning approach for multimodal deception detection. arXiv."},{"key":"ref_18","first-page":"101","article-title":"Optimizing Hammerstein-Wiener Model for Forecasting Confirmed Cases of COVID-19","volume":"52","author":"Abdullahi","year":"2022","journal-title":"IAENG Int. J. Appl. Math."},{"key":"ref_19","first-page":"7808","article-title":"Semantic parsing for automatic retail food image recognition","volume":"53","author":"Abdullahi","year":"2020","journal-title":"Int. J. Adv. Trends Comput. Sci. Eng."},{"key":"ref_20","unstructured":"Lu, S., Tsechpenakis, G., Metaxas, D.N., Jensen, M.L., and Kruse, J. (2005, January 3\u20136). Blob analysis of the head and hands: A method for deception detection. Proceedings of the 38th Annual Hawaii International Conference on System Sciences, Big Island, HI, USA."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Abdullahi, S.B., and Chamnongthai, K. (2022). American sign language words recognition of skeletal videos using processed video driven multi-stacked deep LSTM. Sensors, 22.","DOI":"10.3390\/s22041406"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1007\/s11042-020-09700-0","article-title":"Hand pose aware multimodal isolated sign language recognition","volume":"80","author":"Abdullahi","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_23","unstructured":"Reddy, B., Kim, Y., Yun, S., Seo, C., and Jang, J. (2017, January 21\u201326). Real-Time Eye Blink Detection using Facial Landmarks. Proceedings of the Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA."},{"key":"ref_24","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Avola, D., Cinque, L., Foresti, G.L., and Pannone, D. (2019, January 9\u201311). Automatic deception detection in rgb videos using facial action units. Proceedings of the 13th International Conference on Distributed Smart Cameras, Trento, Italy.","DOI":"10.1145\/3349801.3349806"}],"container-title":["Brain Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3425\/13\/4\/555\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:03:17Z","timestamp":1760122997000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3425\/13\/4\/555"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,25]]},"references-count":25,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["brainsci13040555"],"URL":"https:\/\/doi.org\/10.3390\/brainsci13040555","relation":{},"ISSN":["2076-3425"],"issn-type":[{"value":"2076-3425","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,25]]}}}