{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T18:28:13Z","timestamp":1790792893050,"version":"4.1.0"},"reference-count":36,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2023,6,25]],"date-time":"2023-06-25T00:00:00Z","timestamp":1687651200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Angers Loire Metropole (ALM)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Multimodal deep learning, in the context of biometrics, encounters significant challenges due to the dependence on long speech utterances and RGB images, which are often impractical in certain situations. This paper presents a novel solution addressing these issues by leveraging ultrashort voice utterances and depth videos of the lip for person identification. The proposed method utilizes an amalgamation of residual neural networks to encode depth videos and a Time Delay Neural Network architecture to encode voice signals. In an effort to fuse information from these different modalities, we integrate self-attention and engineer a noise-resistant model that effectively manages diverse types of noise. Through rigorous testing on a benchmark dataset, our approach exhibits superior performance over existing methods, resulting in an average improvement of\u00a010%. This method is notably efficient for scenarios where extended utterances and RGB images are unfeasible or unattainable. Furthermore, its potential extends to various multimodal applications beyond just person identification.<\/jats:p>","DOI":"10.3390\/s23135890","type":"journal-article","created":{"date-parts":[[2023,6,26]],"date-time":"2023-06-26T05:28:02Z","timestamp":1687757282000},"page":"5890","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Attention-Based Fusion of Ultrashort Voice Utterances and Depth Videos for Multimodal Person Identification"],"prefix":"10.3390","volume":"23","author":[{"given":"Abderrazzaq","family":"Moufidi","sequence":"first","affiliation":[{"name":"Centre d\u2019\u00c9tudes et de Recherche pour l\u2019Aide \u00e0 la D\u00e9cision (CERADE), ESAIP, 18 Rue du 8 Mai 1945, 49124 Saint-Barth\u00e9lemy-d\u2019Anjou, France"},{"name":"Laboratoire Angevin de Recherche en Ing\u00e9nierie des Syst\u00e8mes (LARIS), UMR INRAe-IRHS, Universit\u00e9 d\u2019Angers, 62 Avenue Notre Dame du Lac, 49000 Angers, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Rousseau","sequence":"additional","affiliation":[{"name":"Laboratoire Angevin de Recherche en Ing\u00e9nierie des Syst\u00e8mes (LARIS), UMR INRAe-IRHS, Universit\u00e9 d\u2019Angers, 62 Avenue Notre Dame du Lac, 49000 Angers, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7924-0001","authenticated-orcid":false,"given":"Pejman","family":"Rasti","sequence":"additional","affiliation":[{"name":"Centre d\u2019\u00c9tudes et de Recherche pour l\u2019Aide \u00e0 la D\u00e9cision (CERADE), ESAIP, 18 Rue du 8 Mai 1945, 49124 Saint-Barth\u00e9lemy-d\u2019Anjou, France"},{"name":"Laboratoire Angevin de Recherche en Ing\u00e9nierie des Syst\u00e8mes (LARIS), UMR INRAe-IRHS, Universit\u00e9 d\u2019Angers, 62 Avenue Notre Dame du Lac, 49000 Angers, France"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,25]]},"reference":[{"key":"ref_1","first-page":"3200","article-title":"Human action recognition from various data modalities: A review","volume":"45","author":"Sun","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Li, S.Z., and Jain, A. (2009). Encyclopedia of Biometrics, Springer.","DOI":"10.1007\/978-3-642-27733-7"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Sch\u00f6nherr, L., Orth, D., Heckmann, M., and Kolossa, D. (2016, January 13\u201316). Environmentally robust audio-visual speaker identification. Proceedings of the 2016 IEEE Spoken Language Technology Workshop (SLT), San Diego, CA, USA.","DOI":"10.1109\/SLT.2016.7846282"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wei, H., Chopada, P., and Kehtarnavaz, N. (2020). C-MHAD: Continuous Multimodal Human Action Dataset of Simultaneous Video and Inertial Sensing. Sensors, 20.","DOI":"10.3390\/s20102905"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Faraj, M.I., and Bigun, J. (2006, January 20\u201324). Motion features from lip movement for person authentication. Proceedings of the 18th International Conference on Pattern Recognition (ICPR\u201906), Hong Kong, China.","DOI":"10.1109\/ICPR.2006.814"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"3831","DOI":"10.1007\/s11042-021-11613-5","article-title":"Lip as biometric and beyond: A survey","volume":"81","author":"Chowdhury","year":"2022","journal-title":"Multimed. Tools Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"104545","DOI":"10.1016\/j.imavis.2022.104545","article-title":"Bald eagle search optimization with deep transfer learning enabled age-invariant face recognition model","volume":"126","author":"Alsubai","year":"2022","journal-title":"Image Vis. Comput."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Li, D., Gao, Y., Zhu, C., Wang, Q., and Wang, R. (2023). Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy. Sensors, 23.","DOI":"10.3390\/s23042053"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Aung, H.M.L., Pluempitiwiriyawej, C., Hamamoto, K., and Wangsiripitak, S. (2022). Multimodal Biometrics Recognition Using a Deep Convolutional Neural Network with Transfer Learning in Surveillance Videos. Computation, 10.","DOI":"10.3390\/computation10070127"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Minaee, S., Abdolrashidi, A., Su, H., Bennamoun, M., and Zhang, D. (2023). Biometrics recognition using deep learning: A survey. Artif. Intell. Rev., 1\u201349.","DOI":"10.1007\/s10462-022-10237-x"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/j.neunet.2021.03.004","article-title":"Speaker recognition based on deep learning: An overview","volume":"140","author":"Bai","year":"2021","journal-title":"Neural Netw."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1016\/j.engappai.2017.06.003","article-title":"Using a probabilistic neural network for lip-based biometric verification","volume":"64","author":"Wrobel","year":"2017","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Borghi, G., Pini, S., Vezzani, R., and Cucchiara, R. (2019). Driver face verification with depth maps. Sensors, 19.","DOI":"10.3390\/s19153361"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ravanelli, M., and Bengio, Y. (2018, January 18\u201321). Speaker recognition from raw waveform with sincnet. Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.","DOI":"10.1109\/SLT.2018.8639585"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Muckenhirn, H., Magimai-Doss, M., and Marcel, S. (2018, January 2\u20136). On Learning Vocal Tract System Related Speaker Discriminative Information from Raw Signal Using CNNs. Proceedings of the Interspeech, Hyderabad, India.","DOI":"10.21437\/Interspeech.2018-1696"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Moufidi, A., Rousseau, D., and Rasti, P. (2022, January 24\u201326). Wavelet Scattering Transform Depth Benefit, An Application for Speaker Identification. Proceedings of the Artificial Neural Networks in Pattern Recognition: 10th IAPR TC3 Workshop, ANNPR 2022, Dubai, United Arab Emirates.","DOI":"10.1007\/978-3-031-20650-4_8"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Chung, J.S., Nagrani, A., and Zisserman, A. (2018). Voxceleb2: Deep speaker recognition. arXiv.","DOI":"10.21437\/Interspeech.2018-1929"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"104042","DOI":"10.1016\/j.imavis.2020.104042","article-title":"Deep multimodal fusion for semantic image segmentation: A survey","volume":"105","author":"Zhang","year":"2021","journal-title":"Image Vis. Comput."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Desplanques, B., Thienpondt, J., and Demuynck, K. (2020). Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. arXiv.","DOI":"10.21437\/Interspeech.2020-2650"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., and Khudanpur, S. (2018, January 15\u201320). X-vectors: Robust dnn embeddings for speaker recognition. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.","DOI":"10.1109\/ICASSP.2018.8461375"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ghezaiel, W., Brun, L., and L\u00e9zoray, O. (2021, January 10\u201315). Hybrid network for end-to-end text-independent speaker identification. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9413293"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Variani, E., Lei, X., McDermott, E., Moreno, I.L., and Gonzalez-Dominguez, J. (2014, January 4\u20139). Deep neural networks for small footprint text-dependent speaker verification. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.","DOI":"10.1109\/ICASSP.2014.6854363"},{"key":"ref_23","unstructured":"Zeinali, H., Wang, S., Silnova, A., Mat\u011bjka, P., and Plchot, O. (2019). But system description to voxceleb speaker recognition challenge 2019. arXiv."},{"key":"ref_24","unstructured":"Choudhury, T., Clarkson, B., Jebara, T., and Pentland, A. Multimodal person recognition using unconstrained audio and video. Proceedings of the Proceedings, International Conference on Audio-and Video-Based Person Authentication, Washington, DC, USA."},{"key":"ref_25","first-page":"965","article-title":"XM2VTSDB: The extended M2VTS database","volume":"Volume 964","author":"Messer","year":"1999","journal-title":"Proceedings of the Second International Conference on Audio and Video-Based Biometric Person Authentication"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1016\/j.media.2018.08.008","article-title":"Local spatio-temporal encoding of raw perfusion MRI for the prediction of final lesion in stroke","volume":"50","author":"Giacalone","year":"2018","journal-title":"Med. Image Anal."},{"key":"ref_27","unstructured":"Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., and Zhong, J. (2021). SpeechBrain: A general-purpose speech toolkit. arXiv."},{"key":"ref_28","unstructured":"(2023, February 15). Available online: https:\/\/huggingface.co\/speechbrain\/spkrec-xvect-voxceleb."},{"key":"ref_29","unstructured":"Garofolo, J.S. (2023, February 15). Timit Acoustic Phonetic Continuous Speech Corpus; Linguistic Data Consortium. Available online: https:\/\/ci.nii.ac.jp\/naid\/20000921365\/."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1109\/TMM.2015.2407694","article-title":"TCD-TIMIT: An audio-visual corpus of continuous speech","volume":"17","author":"Harte","year":"2015","journal-title":"IEEE Trans. Multimed."},{"key":"ref_31","first-page":"565","article-title":"Speaker identification using mel frequency cepstral coefficients","volume":"1","author":"Hasan","year":"2004","journal-title":"Variations"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1016\/j.neunet.2021.07.007","article-title":"An efficient encoder\u2013decoder model for portrait depth estimation from single images trained on pixel-accurate synthetic data","volume":"142","author":"Khan","year":"2021","journal-title":"Neural Netw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Borghi, G., Venturelli, M., Vezzani, R., and Cucchiara, R. (2017, January 21\u201326). Poseidon: Face-from-depth for driver pose estimation. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.583"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1534","DOI":"10.1109\/TSMC.2014.2331215","article-title":"KinectFaceDB: A Kinect Database for Face Recognition","volume":"44","author":"Min","year":"2014","journal-title":"Syst. Man Cybern. Syst. IEEE Trans."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"437","DOI":"10.1007\/s11263-012-0549-0","article-title":"Random Forests for Real Time 3D Face Analysis","volume":"101","author":"Fanelli","year":"2013","journal-title":"Int. J. Comput. Vis."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/13\/5890\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:00:31Z","timestamp":1760126431000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/13\/5890"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,25]]},"references-count":36,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["s23135890"],"URL":"https:\/\/doi.org\/10.3390\/s23135890","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,25]]}}}