{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T00:58:07Z","timestamp":1760057887410,"version":"build-2065373602"},"reference-count":56,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T00:00:00Z","timestamp":1741046400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Research Council of Norway","award":["#314690"],"award-info":[{"award-number":["#314690"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["BDCC"],"abstract":"<jats:p>This paper explores advancements in real-time talking-head generation, focusing on overcoming challenges in Audio Feature Extraction (AFE), which often introduces latency and limits responsiveness in real-time applications. To address these issues, we propose and implement a fully integrated system that replaces conventional AFE models with OpenAI\u2019s Whisper, leveraging its encoder to optimize processing and improve overall system efficiency. Our evaluation of two open-source real-time models across three different datasets shows that Whisper not only accelerates processing but also improves specific aspects of rendering quality, resulting in more realistic and responsive talking-head interactions. Although interviewer training systems are considered a potential application, the primary contribution of this work is the improvement of the technical foundations necessary for creating responsive AI avatars. These advancements enable more immersive interactions and expand the scope of AI-driven applications, including educational tools and simulated training environments.<\/jats:p>","DOI":"10.3390\/bdcc9030059","type":"journal-article","created":{"date-parts":[[2025,3,4]],"date-time":"2025-03-04T04:58:17Z","timestamp":1741064297000},"page":"59","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Comparative Analysis of Audio Feature Extraction for Real-Time Talking Portrait Synthesis"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9149-3829","authenticated-orcid":false,"given":"Pegah","family":"Salehi","sequence":"first","affiliation":[{"name":"SimulaMet, 0167 Oslo, Norway"},{"name":"Department of Computer Science, UiT The Arctic University of Norway, 9019 Troms\u00f8, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1805-7880","authenticated-orcid":false,"given":"Sajad Amouei","family":"Sheshkal","sequence":"additional","affiliation":[{"name":"SimulaMet, 0167 Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6026-0929","authenticated-orcid":false,"given":"Vajira","family":"Thambawita","sequence":"additional","affiliation":[{"name":"SimulaMet, 0167 Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9232-2661","authenticated-orcid":false,"given":"Sushant","family":"Gautam","sequence":"additional","affiliation":[{"name":"SimulaMet, 0167 Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5348-8546","authenticated-orcid":false,"given":"Saeed S.","family":"Sabet","sequence":"additional","affiliation":[{"name":"Forzasys AS, 0840 Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7067-6477","authenticated-orcid":false,"given":"Dag","family":"Johansen","sequence":"additional","affiliation":[{"name":"Department of Computer Science, UiT The Arctic University of Norway, 9019 Troms\u00f8, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3153-2064","authenticated-orcid":false,"given":"Michael A.","family":"Riegler","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory, 0164 Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2073-7029","authenticated-orcid":false,"given":"P\u00e5l","family":"Halvorsen","sequence":"additional","affiliation":[{"name":"SimulaMet, 0167 Oslo, Norway"},{"name":"Faculty of Technology, Art, and Design, Department of Computer Science (IT), Oslo Metropolitan University (OsloMet), 0130 Oslo, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,3,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"499","DOI":"10.3102\/0034654320933544","article-title":"Simulation-based learning in higher education: A meta-analysis","volume":"90","author":"Chernikova","year":"2020","journal-title":"Rev. Educ. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.copsyc.2020.04.011","article-title":"Psychological foundations of emerging technologies for teaching and learning in higher education","volume":"36","author":"Crompton","year":"2020","journal-title":"Curr. Opin. Psychol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"710","DOI":"10.1037\/amp0000039","article-title":"Difficulties translating research on forensic interview practices to practitioners: Finding water, leading horses, but can we get them to drink?","volume":"71","author":"Lamb","year":"2016","journal-title":"Am. Psychol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1080\/10345329.2008.12035804","article-title":"Designing effective training programs for investigative interviewers of children","volume":"20","author":"Powell","year":"2008","journal-title":"Curr. Issues Crim. Justice"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1201","DOI":"10.1016\/j.chiabu.2007.03.021","article-title":"A structured forensic interview protocol improves the quality and informativeness of investigative interviews with children: A review of research using the NICHD Investigative Interview Protocol","volume":"31","author":"Lamb","year":"2007","journal-title":"Child Abuse Negl."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1146\/annurev-lawsocsci-110413-030913","article-title":"Interviewing children","volume":"10","author":"Lyon","year":"2014","journal-title":"Annu. Rev. Law Soc. Sci."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Lamb, M.E., Brown, D.A., Hershkowitz, I., Orbach, Y., and Esplin, P.W. (2018). Tell Me What Happened: Questioning Children About Abuse, John Wiley & Sons.","DOI":"10.1002\/9781118881248"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"105685","DOI":"10.1016\/j.chiabu.2022.105685","article-title":"An overview of mock interviews as a training tool for interviewers of children","volume":"129","author":"Powell","year":"2022","journal-title":"Child Abuse Negl."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"131842","DOI":"10.1109\/ACCESS.2024.3442297","article-title":"A theoretical and empirical analysis of 2D and 3D Virtual Environments in Training for Child Interview Skills","volume":"12","author":"Salehi","year":"2024","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Salehi, P., Hassan, S.Z., Shafiee Sabet, S., Astrid Baugerud, G., Sinkerud Johnson, M., Halvorsen, P., and Riegler, M.A. (2022, January 27\u201330). Is more realistic better? A comparison of game engine and gan-based avatars for investigative interviews of children. Proceedings of the 3rd ACM Workshop on Intelligent Cross-Data Analysis and Retrieval, Newark, NJ, USA.","DOI":"10.1145\/3512731.3534209"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1145\/3503250","article-title":"Nerf: Representing scenes as neural radiance fields for view synthesis","volume":"65","author":"Mildenhall","year":"2021","journal-title":"Commun. ACM"},{"key":"ref_12","first-page":"12449","article-title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","volume":"33","author":"Baevski","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Hsu, W.N., Tsai, Y.H.H., Bolte, B., Salakhutdinov, R., and Mohamed, A. (2021, January 6\u201311). HuBERT: How much can a bad teacher benefit ASR pre-training?. Proceedings of the ICASSP 2021\u20132021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.","DOI":"10.1109\/ICASSP39728.2021.9414460"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2560","DOI":"10.1016\/j.procs.2024.02.074","article-title":"Unveiling embedded features in Wav2vec2 and HuBERT msodels for Speech Emotion Recognition","volume":"232","author":"Chakhtouna","year":"2024","journal-title":"Procedia Comput. Sci."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"123276","DOI":"10.1109\/ACCESS.2024.3426471","article-title":"Benchmarking Inference of Transformer-Based Transcription Models with Clustering on Embedded GPUs","volume":"12","author":"Schubert","year":"2024","journal-title":"IEEE Access"},{"key":"ref_16","unstructured":"Chakravarty, A. (2024). Deep Learning Models in Speech Recognition: Measuring GPU Energy Consumption, Impact of Noise and Model Quantization for Edge Deployment. arXiv."},{"key":"ref_17","unstructured":"Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023, January 23\u201329). Robust speech recognition via large-scale weak supervision. Proceedings of the International Conference on Machine Learning. PMLR, Honolulu, HI, USA."},{"key":"ref_18","unstructured":"Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., and Chen, G. (2016, January 19\u201324). Deep speech 2: End-to-end speech recognition in english and mandarin. Proceedings of the International Conference on Machine Learning. PMLR, New York, NY, USA."},{"key":"ref_19","unstructured":"Salehi, P. (2025, February 20). Whisper AFE for Talking Heads Generation, Version 1.0.0. Available online: https:\/\/github.com\/pegahs1993\/Whisper-AFE-TalkingHeadsGen."},{"key":"ref_20","unstructured":"Dalli, K.C. (2025, January 20). Technological Acceptance of an Avatar Based Interview Training Application: The Development and Technological Acceptance Study of the AvBIT Application. Available online: http:\/\/urn.kb.se\/resolve?urn=urn:nbn:se:lnu:diva-107108."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"106324","DOI":"10.1016\/j.chiabu.2023.106324","article-title":"A field assessment of child abuse investigators\u2019 engagement with a child-avatar to develop interviewing skills","volume":"143","author":"Powell","year":"2023","journal-title":"Child Abuse Negl."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1037\/law0000052","article-title":"Evaluation of a comprehensive interactive training system for investigative interviewers of children","volume":"21","author":"Benson","year":"2015","journal-title":"Psychol. Public Policy Law"},{"key":"ref_23","unstructured":"Pompedda, F. (2018). Training in Investigative Interviews of Children: Serious Gaming Paired with Feedback Improves Interview Quality. [Doctoral Dissertation, \u00c5bo Akademi University]."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"106677","DOI":"10.1016\/j.chiabu.2024.106677","article-title":"How to prepare for conversations with children about suspicions of sexual abuse? Evaluation of an interactive virtual reality training for student teachers","volume":"149","author":"Krause","year":"2024","journal-title":"Child Abuse Negl."},{"key":"ref_25","unstructured":"Meng, M., Zhao, Y., Zhang, B., Zhu, Y., Shi, W., Wen, M., and Fan, Z. (2024). A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editing. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3478513.3480484","article-title":"Live speech portraits: Real-time photorealistic talking-head animation","volume":"40","author":"Lu","year":"2021","journal-title":"ACM Trans. Graph. (ToG)"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chung, Y.A., and Glass, J. (2020, January 4\u20138). Generative pre-training for speech with autoregressive predictive coding. Proceedings of the ICASSP 2020\u20142020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9054438"},{"key":"ref_28","unstructured":"Ji, X., Lin, C., Ding, Z., Tai, Y., Yang, J., Zhu, J., Hu, X., Zhang, J., Luo, D., and Wang, C. (2024). RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3592433","article-title":"3D Gaussian Splatting for Real-Time Radiance Field Rendering","volume":"42","author":"Kerbl","year":"2023","journal-title":"ACM Trans. Graph."},{"key":"ref_30","unstructured":"Chen, B., Hu, S., Chen, Q., Du, C., Yi, R., Qian, Y., and Chen, X. (2024). GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splatting. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Cho, K., Lee, J., Yoon, H., Hong, Y., Ko, J., Ahn, S., and Kim, S. (November, January 28). GaussianTalker: Real-Time Talking Head Synthesis with 3D Gaussian Splatting. Proceedings of the ACM Multimedia 2024, Melbourne, Australia.","DOI":"10.1145\/3664647.3681627"},{"key":"ref_32","unstructured":"Tang, J., Wang, K., Zhou, H., Chen, X., He, D., Hu, T., Liu, J., Zeng, G., and Wang, J. (2022). Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Guo, Y., Chen, K., Liang, S., Liu, Y.J., Bao, H., and Zhang, J. (2021, January 11\u201317). Ad-nerf: Audio driven neural radiance fields for talking head synthesis. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00573"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Li, J., Zhang, J., Bai, X., Zhou, J., and Gu, L. (2023, January 1\u20136). Efficient region-aware neural radiance fields for high-fidelity talking portrait synthesis. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Paris, France.","DOI":"10.1109\/ICCV51070.2023.00696"},{"key":"ref_35","unstructured":"Ye, Z., He, J., Jiang, Z., Huang, R., Huang, J., Liu, J., Ren, Y., Yin, X., Ma, Z., and Zhao, Z. (2023). Geneface++: Generalized and stable real-time audio-driven 3d talking face generation. arXiv."},{"key":"ref_36","unstructured":"Ye, Z., Zhang, L., Zeng, D., Lu, Q., and Jiang, N. (2023). R2-Talker: Realistic Real-Time Talking Head Synthesis with Hash Grid Landmarks Encoding and Progressive Multilayer Conditioning. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2673","DOI":"10.1109\/78.650093","article-title":"Bidirectional recurrent neural networks","volume":"45","author":"Schuster","year":"1997","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Thies, J., Elgharib, M., Tewari, A., Theobalt, C., and Nie\u00dfner, M. (2020, January 23\u201328). Neural voice puppetry: Audio-driven facial reenactment. Proceedings of the Computer Vision\u2013ECCV 2020: 16th European Conference, Glasgow, UK.","DOI":"10.1007\/978-3-030-58517-4_42"},{"key":"ref_39","unstructured":"Hendrycks, D., and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv."},{"key":"ref_40","unstructured":"Baevski, A., Schneider, S., and Auli, M. (2019). vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1109\/TPAMI.2010.57","article-title":"Product quantization for nearest neighbor search","volume":"33","author":"Jegou","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. (2015, January 19\u201324). Librispeech: An asr corpus based on public domain audio books. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia.","DOI":"10.1109\/ICASSP.2015.7178964"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazar\u00e9, P.E., Karadayi, J., Liptchinsky, V., Collobert, R., and Fuegen, C. (2020, January 4\u20138). Libri-light: A benchmark for asr with limited or no supervision. Proceedings of the ICASSP 2020\u20142020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9052942"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1162\/tacl_a_00300","article-title":"Spanbert: Improving pre-training by representing and predicting spans","volume":"8","author":"Joshi","year":"2020","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_45","unstructured":"Ye, Z., Jiang, Z., Ren, Y., Liu, J., He, J., and Zhao, Z. (2023). Geneface: Generalized and high-fidelity audio-driven 3D talking face synthesis. arXiv."},{"key":"ref_46","unstructured":"(2024, September 28). Amazon Web Services. AWS Polly. Web Services. Amazon. Available online: https:\/\/aws.amazon.com\/polly\/."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Zhang, R., Isola, P., Efros, A.A., Shechtman, E., and Wang, O. (2018, January 18\u201322). The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00068"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Chen, L., Li, Z., Maddox, R.K., Duan, Z., and Xu, C. (2018, January 8\u201314). Lip movements generation at a glance. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_32"},{"key":"ref_49","first-page":"6626","article-title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","volume":"30","author":"Heusel","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Baltru\u0161aitis, T., Robinson, P., and Morency, L.P. (2016, January 7\u201310). Openface: An open source facial behavior analysis toolkit. Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA.","DOI":"10.1109\/WACV.2016.7477553"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Chung, J.S., and Zisserman, A. (2016, January 20\u201324). Out of time: Automated lip sync in the wild. Proceedings of the Computer Vision\u2013ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan.","DOI":"10.1007\/978-3-319-54526-4"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Attig, C., Rauh, N., Franke, T., and Krems, J.F. (2017, January 9\u201314). System latency guidelines then and now\u2013is zero latency really considered necessary?. Proceedings of the Engineering Psychology and Cognitive Ergonomics: Cognition and Design: 14th International Conference, EPCE 2017, Vancouver, BC, Canada.","DOI":"10.1007\/978-3-319-58475-1_1"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Forch, V., Franke, T., Rauh, N., and Krems, J.F. (2017, January 9\u201314). Are 100 ms fast enough? Characterizing latency perception thresholds in mouse-based interaction. Proceedings of the Engineering Psychology and Cognitive Ergonomics: Cognition and Design: 14th International Conference, EPCE 2017, Vancouver, BC, Canada.","DOI":"10.1007\/978-3-319-58475-1_4"},{"key":"ref_54","first-page":"24","article-title":"Regulating deep fakes: Legal and ethical considerations","volume":"15","author":"Meskys","year":"2020","journal-title":"J. Intellect. Prop. Law Pract."},{"key":"ref_55","unstructured":"Hu, H. (2025, February 20). Privacy Attacks and Protection in Generative Models. Available online: https:\/\/hdl.handle.net\/10993\/58928."},{"key":"ref_56","first-page":"611","article-title":"Deepfake privacy: Attitudes and regulation","volume":"116","author":"Kugler","year":"2021","journal-title":"Northwestern Univ. Law Rev."}],"container-title":["Big Data and Cognitive Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-2289\/9\/3\/59\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:46:34Z","timestamp":1760028394000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-2289\/9\/3\/59"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,4]]},"references-count":56,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["bdcc9030059"],"URL":"https:\/\/doi.org\/10.3390\/bdcc9030059","relation":{},"ISSN":["2504-2289"],"issn-type":[{"type":"electronic","value":"2504-2289"}],"subject":[],"published":{"date-parts":[[2025,3,4]]}}}