{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:42:02Z","timestamp":1760146922086,"version":"build-2065373602"},"reference-count":52,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,12,26]],"date-time":"2024-12-26T00:00:00Z","timestamp":1735171200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Detecting deceptive behaviour for surveillance and border protection is critical for a country\u2019s security. With the advancement of technology in relation to sensors and artificial intelligence, recognising deceptive behaviour could be performed automatically. Following the success of affective computing in emotion recognition from verbal and nonverbal cues, we aim to apply a similar concept for deception detection. Recognising deceptive behaviour has been attempted; however, only a few studies have analysed this behaviour from gait and body movement. This research involves a multimodal approach for deception detection from gait, where we fuse features extracted from body movement behaviours from a video signal, acoustic features from walking steps from an audio signal, and the dynamics of walking movement using an accelerometer sensor. Using the video recording of walking from the Whodunnit deception dataset, which contains 49 subjects performing scenarios that elicit deceptive behaviour, we conduct multimodal two-category (guilty\/not guilty) subject-independent classification. The classification results obtained reached an accuracy of up to 88% through feature fusion, with an average of 60% from both single and multimodal signals. Analysing body movement using single modality showed that the visual signal had the highest performance followed by the accelerometer and acoustic signals. Several fusion techniques were explored, including early, late, and hybrid fusion, where hybrid fusion not only achieved the highest classification results, but also increased the confidence of the results. Moreover, using a systematic framework for selecting the most distinguishing features of guilty gait behaviour, we were able to interpret the performance of our models. From these baseline results, we can conclude that pattern recognition techniques could help in characterising deceptive behaviour, where future work will focus on exploring the tuning and enhancement of the results and techniques.<\/jats:p>","DOI":"10.3390\/info16010006","type":"journal-article","created":{"date-parts":[[2024,12,26]],"date-time":"2024-12-26T21:13:18Z","timestamp":1735247598000},"page":"6","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["The Walk of Guilt: Multimodal Deception Detection from Nonverbal Motion Behaviour"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9391-0163","authenticated-orcid":false,"given":"Sharifa","family":"Alghowinem","sequence":"first","affiliation":[{"name":"Media Lab, Massachusetts Institute of Technology, Cambridge, MA 02139, USA"},{"name":"Research School of Computer Science, Australian National University, Canberra 2600, Australia"},{"name":"College of Computer and Information Science, Prince Sultan University, Riyadh 11586, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0605-3149","authenticated-orcid":false,"given":"Sabrina","family":"Caldwell","sequence":"additional","affiliation":[{"name":"Research School of Computer Science, Australian National University, Canberra 2600, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8170-5058","authenticated-orcid":false,"given":"Ibrahim","family":"Radwan","sequence":"additional","affiliation":[{"name":"Human-Centred Computing Laboratory (HCC Lab), University of Canberra, Canberra 2617, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Wagner","sequence":"additional","affiliation":[{"name":"Research School of Computer Science, Australian National University, Canberra 2600, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8356-4909","authenticated-orcid":false,"given":"Tom","family":"Gedeon","sequence":"additional","affiliation":[{"name":"Research School of Computer Science, Australian National University, Canberra 2600, Australia"},{"name":"Faculty of Science and Engineering, Curtin University, Perth 6102, Australia"},{"name":"John von Neumann Faculty of Informatics, Obuda University, 1034 Budapest, Hungary"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1023\/A:1015332606792","article-title":"Suspects, lies, and videotape: An analysis of authentic high-stake liars","volume":"26","author":"Mann","year":"2002","journal-title":"Law Hum. Behav."},{"key":"ref_2","unstructured":"(2019, April 09). National Association for Shoplifting Prevention (NASP). Available online: https:\/\/www.shopliftingprevention.org\/."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1109\/T-AFFC.2013.29","article-title":"Body movements for affective expression: A survey of automatic recognition and generation","volume":"4","author":"Karg","year":"2013","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1109\/TAFFC.2020.3035535","article-title":"Interpretation of Depression Detection Models via Feature Selection Methods","volume":"14","author":"Alghowinem","year":"2020","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_5","first-page":"108","article-title":"A compendium of pattern recognition techniques in face, speech and lie detection","volume":"24","author":"Cardona","year":"2015","journal-title":"Int. J. Res. Rev. Appl. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Nasri, H., Ouarda, W., and Alimi, A.M. (December, January 29). ReLiDSS: Novel lie detection system from speech signal. Proceedings of the 2016 IEEE\/ACS 13th International Conference of Computer Systems and Applications (AICCSA), Agadir, Morocco.","DOI":"10.1109\/AICCSA.2016.7945789"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Vrij, A., and Fisher, R.P. (2020). Lying and nervous behaviours Unravelling the misconception about deception and nervous behaviour. Front. Psychol., 11.","DOI":"10.3389\/fpsyg.2020.01377"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"253","DOI":"10.1007\/s10979-007-9103-y","article-title":"Increasing cognitive load to facilitate lie detection: The benefit of recalling an event in reverse order","volume":"32","author":"Vrij","year":"2008","journal-title":"Law Hum. Behav."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"P\u00e9rez-Rosas, V., Abouelenien, M., Mihalcea, R., and Burzo, M. (2015, January 9\u201313). Deception Detection Using Real-Life Trial Data. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI \u201915, Seattle, WA, USA.","DOI":"10.1145\/2818346.2820758"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Gupta, V., Agarwal, M., Arora, M., Chakraborty, T., Singh, R., and Vatsa, M. (2019, January 16\u201317). Bag-of-lies: A multimodal dataset for deception detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00016"},{"key":"ref_11","unstructured":"P\u00e9rez-Rosas, V., Mihalcea, R., Narvaez, A., and Burzo, M. (2014, January 26\u201331). A Multimodal Dataset for Deception Detection. Proceedings of the LREC, Reykjavik, Iceland."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Raiman, N., Hung, H., and Englebienne, G. (2011, January 14\u201318). Move, and i will tell you who you are: Detecting deceptive roles in low-quality data. Proceedings of the 13th International Conference on Multimodal Interfaces, Alicante, Spain.","DOI":"10.1145\/2070481.2070515"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Diana, B., Elia, M., Zurloni, V., Elia, A., Maisto, A., and Pelosi, S. (2015, January 13). Multimodal Deception Detection: A t-pattern Approach. Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection, Seattle, WA, USA.","DOI":"10.1145\/2823465.2823466"},{"key":"ref_14","unstructured":"Van Der Zee, S., Poppe, R., Taylor, P., and Anderson, R. (2015, January 5\u20138). To freeze or not to freeze: A motion-capture approach to detecting deceit. Proceedings of the Hawaii International Conference on System Sciences, Kauai, HI, USA."},{"key":"ref_15","unstructured":"Randhavane, T., Bhattacharya, U., Kapsaskis, K., Gray, K., Bera, A., and Manocha, D. (2019). The Liar\u2019s Walk: Detecting Deception with Gait and Gesture. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1109\/T-AFFC.2012.16","article-title":"Affective body expression perception and recognition: A survey","volume":"4","author":"Kleinsmith","year":"2013","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1109\/MCG.2014.106","article-title":"Automatic emotion recognition based on body movement analysis: A survey","volume":"34","author":"Zacharatos","year":"2014","journal-title":"IEEE Comput. Graph. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1007\/s12369-017-0427-6","article-title":"Automatic affect perception based on body gait and posture: A survey","volume":"9","author":"Naghdy","year":"2017","journal-title":"Int. J. Soc. Robot."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1109\/TAFFC.2018.2874986","article-title":"Survey on emotional body gesture recognition","volume":"12","author":"Noroozi","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1016\/j.cviu.2010.10.002","article-title":"A survey of vision-based methods for action representation, segmentation and recognition","volume":"115","author":"Weinland","year":"2011","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1007\/s10919-012-0130-0","article-title":"The body action and posture coding system (BAP): Development and reliability","volume":"36","author":"Dael","year":"2012","journal-title":"J. Nonverbal Behav."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Velloso, E., Bulling, A., and Gellersen, H. (2013, January 2\u20135). AutoBAP: Automatic coding of body action and posture units from wearable sensors. Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, Geneva, Switzerland.","DOI":"10.1109\/ACII.2013.29"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1111\/cgf.12598","article-title":"Emotion analysis and classification: Understanding the performers\u2019 emotions using the LMA entities","volume":"34","author":"Aristidou","year":"2015","journal-title":"Comput. Graph. Forum"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1002\/cav.1714","article-title":"Continuous body emotion recognition system during theater performances","volume":"27","author":"Senecal","year":"2016","journal-title":"Comput. Animat. Virtual Worlds"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1007\/s12193-009-0025-5","article-title":"Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis","volume":"3","author":"Kessous","year":"2010","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1027","DOI":"10.1109\/TSMCB.2010.2103557","article-title":"Automatic recognition of non-acted affective postures","volume":"41","author":"Kleinsmith","year":"2011","journal-title":"IEEE Trans. Syst. Man Cybern. Part B (Cybern.)"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1023\/A:1021435526134","article-title":"The use of body movements and gestures as cues to emotions in younger and older adults","volume":"23","author":"Montepare","year":"1999","journal-title":"J. Nonverbal Behav."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1109\/TSMCB.2010.2044040","article-title":"Recognition of affect based on gait patterns","volume":"40","author":"Karg","year":"2010","journal-title":"IEEE Trans. Syst. Man Cybern. Part B (Cybern.)"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"580","DOI":"10.1097\/PSY.0b013e3181a2515c","article-title":"Embodiment of sadness and depression\u2014Gait patterns associated with dysphoric mood","volume":"71","author":"Michalak","year":"2009","journal-title":"Psychosom. Med."},{"key":"ref_30","first-page":"89","article-title":"A survey on gait recognition","volume":"51","author":"Wan","year":"2018","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2001","DOI":"10.1109\/TBME.2015.2410142","article-title":"Acoustic gaits: Gait analysis with footstep sounds","volume":"62","author":"Altaf","year":"2015","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Huang, J., Di Troia, F., and Stamp, M. (2018, January 22\u201324). Acoustic Gait Analysis using Support Vector Machines. Proceedings of the ICISSP, Funchal, Portugal.","DOI":"10.5220\/0006730705450552"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Wang, C., Wang, X., Long, Z., Yuan, J., Qian, Y., and Li, J. (2016). Estimation of temporal gait parameters using a wearable microphone-sensor-based system. Sensors, 16.","DOI":"10.3390\/s16122167"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wang, C., Wang, X., Long, Z., Yuan, J., Qian, Y., and Li, J. (2017, January 4\u20138). Multimodal gait analysis based on wearable inertial and microphone sensors. Proceedings of the 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld\/SCALCOM\/UIC\/ATC\/CBDCom\/IOP\/SCI), San Francisco, CA, USA.","DOI":"10.1109\/UIC-ATC.2017.8397481"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/SURV.2012.110112.00192","article-title":"A survey on human activity recognition using wearable sensors","volume":"15","author":"Lara","year":"2013","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"e2258","DOI":"10.7717\/peerj.2258","article-title":"Emotion recognition based on customized smart bracelet with built-in accelerometer","volume":"4","author":"Zhang","year":"2016","journal-title":"PeerJ"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"5725","DOI":"10.3390\/s140305725","article-title":"Analyzing body movements within the laban effort framework using a single accelerometer","volume":"14","author":"Kikhia","year":"2014","journal-title":"Sensors"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Pires, I.M., Garcia, N.M., Pombo, N., Fl\u00f3rez-Revuelta, F., and Spinsante, S. (2017). Pattern recognition techniques for the identification of Activities of Daily Living using mobile device accelerometer. arXiv.","DOI":"10.7287\/peerj.preprints.27225v1"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"103619","DOI":"10.1109\/ACCESS.2019.2931050","article-title":"Hierarchical Adversarial Network for Human Pose Estimation","volume":"7","author":"Radwan","year":"2019","journal-title":"IEEE Access"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/00401706.1969.10490657","article-title":"Procedures for detecting outlying observations in samples","volume":"11","author":"Grubbs","year":"1969","journal-title":"Technometrics"},{"key":"ref_41","unstructured":"Tzanetakis, G., Essl, G., and Cook, P. (2001, January 26\u201330). Audio analysis using the discrete wavelet transform. Proceedings of the WSES International Conference Acoustics and Music: Theory and Applications (AMTA 2001), Skiathos, Greece."},{"key":"ref_42","unstructured":"Kaiser, J.F. (1990, January 3\u20136). On a simple algorithm to calculate the\u2019energy\u2019of a signal. Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Albuquerque, NM, USA."},{"key":"ref_43","unstructured":"Arai, K., Kapoor, S., and Bhatia, R. (2019, January 6\u20138). An Interacting Decision Support System to Determine a Group-Member\u2019s Role Using Automatic Behaviour Analysis. Proceedings of the Intelligent Computing, Chongqing, China."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v036.i11","article-title":"Feature selection with the boruta package","volume":"36","author":"Kursa","year":"2010","journal-title":"J. Stat. Softw."},{"key":"ref_45","unstructured":"Liu, H., and Setiono, R. (1995, January 5\u20138). Chi2: Feature selection and discretization of numeric attributes. Proceedings of the International Conference on Tools with Artificial Intelligence, Herndon, VA, USA."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","article-title":"Regularization and variable selection via the elastic net","volume":"67","author":"Zou","year":"2005","journal-title":"J. R. Stat. Soc. Ser. B (Stat. Methodol.)"},{"key":"ref_47","unstructured":"Duda, R.O., Hart, P.E., and Stork, D.G. (2000). Pattern Classification, Wiley-Interscience. [2nd ed.]."},{"key":"ref_48","first-page":"171","article-title":"Local Causal and Markov Blanket Induction for Causal Discovery and Feature Selection for Classification Part I: Algorithms and Empirical Evaluation","volume":"11","author":"Aliferis","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Zou, H., Jakovli\u0107, I., Chen, R., Zhang, D., Zhang, J., Li, W.X., and Wang, G.T. (2017). The complete mitochondrial genome of parasitic nematode Camallanus cotti: Extreme discontinuity in the rate of mitogenomic architecture evolution within the Chromadorea class. BMC Genom., 18.","DOI":"10.1186\/s12864-017-4237-x"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"478","DOI":"10.1109\/TAFFC.2016.2634527","article-title":"Multimodal Depression Detection: Fusion Analysis of Paralinguistic, Head Pose and Eye Gaze Behaviors","volume":"9","author":"Alghowinem","year":"2018","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1080\/00332747.1969.11023575","article-title":"Nonverbal leakage and clues to deception","volume":"32","author":"Ekman","year":"1969","journal-title":"Psychiatry"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1109\/MIS.2005.85","article-title":"Deception detection through automatic, unobtrusive analysis of nonverbal behavior","volume":"20","author":"Meservy","year":"2005","journal-title":"IEEE Intell. Syst."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/1\/6\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T17:00:48Z","timestamp":1760115648000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/1\/6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,26]]},"references-count":52,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1]]}},"alternative-id":["info16010006"],"URL":"https:\/\/doi.org\/10.3390\/info16010006","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2024,12,26]]}}}