{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T23:26:45Z","timestamp":1790810805190,"version":"4.1.0"},"reference-count":115,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,2,17]],"date-time":"2021-02-17T00:00:00Z","timestamp":1613520000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Institute of Information &amp; communications Technology Planning &amp; Evaluation(IITP)","award":["P0008703"],"award-info":[{"award-number":["P0008703"]}]},{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["NRF-2017R1A2B4012736"],"award-info":[{"award-number":["NRF-2017R1A2B4012736"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Voice is one of the essential mechanisms for communicating and expressing one\u2019s intentions as a human being. There are several causes of voice inability, including disease, accident, vocal abuse, medical surgery, ageing, and environmental pollution, and the risk of voice loss continues to increase. Novel approaches should have been developed for speech recognition and production because that would seriously undermine the quality of life and sometimes leads to isolation from society. In this review, we survey mouth interface technologies which are mouth-mounted devices for speech recognition, production, and volitional control, and the corresponding research to develop artificial mouth technologies based on various sensors, including electromyography (EMG), electroencephalography (EEG), electropalatography (EPG), electromagnetic articulography (EMA), permanent magnet articulography (PMA), gyros, images and 3-axial magnetic sensors, especially with deep learning techniques. We especially research various deep learning technologies related to voice recognition, including visual speech recognition, silent speech interface, and analyze its flow, and systematize them into a taxonomy. Finally, we discuss methods to solve the communication problems of people with disabilities in speaking and future research with respect to deep learning components.<\/jats:p>","DOI":"10.3390\/s21041399","type":"journal-article","created":{"date-parts":[[2021,2,17]],"date-time":"2021-02-17T21:35:42Z","timestamp":1613597742000},"page":"1399","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":100,"title":["Biosignal Sensors and Deep Learning-Based Speech Recognition: A Review"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8586-4577","authenticated-orcid":false,"given":"Wookey","family":"Lee","sequence":"first","affiliation":[{"name":"Biomedical Science and Engineering &amp; Dept. of Industrial Security Governance &amp; IE, Inha University, 100 Inharo, Incheon 22212, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jessica Jiwon","family":"Seong","sequence":"additional","affiliation":[{"name":"Department of Industrial Security Governance, Inha University, 100 Inharo, Incheon 22212, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Busra","family":"Ozlu","sequence":"additional","affiliation":[{"name":"Biomedical Science and Engineering &amp; Department of Chemical Engineering, Inha University, 100 Inharo, Incheon 22212, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bong Sup","family":"Shim","sequence":"additional","affiliation":[{"name":"Biomedical Science and Engineering &amp; Department of Chemical Engineering, Inha University, 100 Inharo, Incheon 22212, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Azizbek","family":"Marakhimov","sequence":"additional","affiliation":[{"name":"Frontier College, Inha University, 100 Inharo, Incheon 22212, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3047-1167","authenticated-orcid":false,"given":"Suan","family":"Lee","sequence":"additional","affiliation":[{"name":"School of Computer Science, Semyung University, Jecheon 27136, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,2,17]]},"reference":[{"key":"ref_1","unstructured":"(2019, October 29). Voice Disorders: Overview. Available online: https:\/\/www.asha.org\/practice-portal\/clinical-topics\/voice-disorders\/."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Cheah, L.A., Gilbert, J.M., Gonzalez, J.A., Bai, J., Ell, S.R., Green, P.D., and Moore, R.K. (2016, January 21\u201323). Towards an Intraoral-Based Silent Speech Restoration System for Post-laryngectomy Voice Replacement. Proceedings of the International Joint Conference on Biomedical Engineering Systems and Technologies, Rome, Italy.","DOI":"10.1007\/978-3-319-54717-6_2"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Shin, Y.H., and Seo, J. (2016). Towards contactless silent speech recognition based on detection of active and visible articulators using IR-UWB radar. Sensors, 16.","DOI":"10.3390\/s16111812"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"863","DOI":"10.1007\/s11136-018-2033-y","article-title":"Communication changes with laryngectomy and impact on quality of life: A review","volume":"28","author":"Sharpe","year":"2019","journal-title":"Qual. Life Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1049\/cje.2016.01.014","article-title":"Silent speech interface design methodology and case study","volume":"25","author":"Li","year":"2016","journal-title":"Chin. J. Electron."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1016\/j.specom.2009.08.002","article-title":"Silent speech interfaces","volume":"52","author":"Denby","year":"2010","journal-title":"Speech Commun."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.specom.2018.02.002","article-title":"Updating the Silent Speech Challenge benchmark with deep learning","volume":"98","author":"Ji","year":"2018","journal-title":"Speech Commun."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2386","DOI":"10.1109\/TASLP.2017.2740000","article-title":"Silent speech recognition as an alternative communication device for persons with laryngectomy","volume":"25","author":"Meltzner","year":"2017","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2257","DOI":"10.1109\/TASLP.2017.2752365","article-title":"Biosignal-Based Spoken Communication: A Survey","volume":"25","author":"Schultz","year":"2017","journal-title":"IEEE\/ACM Trans. Audio Speech Lang. Process."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1088\/1741-2552\/aac965","article-title":"Development of sEMG sensors and algorithms for silent speech recognition","volume":"15","author":"Meltzner","year":"2018","journal-title":"J. Neural Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1016\/j.bspc.2019.02.011","article-title":"A review on EMG-based motor intention prediction of continuous human upper limb motion for human-robot collaboration","volume":"51","author":"Bi","year":"2019","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Levis, J., and Suvorov, R. (2012). Automatic speech recognition. The Encyclopedia of Applied Linguistics, Springer.","DOI":"10.1002\/9781405198431.wbeal0066"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Burileanu, D. (2008). Spoken language interfaces for embedded applications. Human Factors and Voice Interactive Systems, Springer.","DOI":"10.1007\/978-0-387-68439-0_5"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"7684","DOI":"10.1073\/pnas.1915768117","article-title":"Racial disparities in automated speech recognition","volume":"117","author":"Koenecke","year":"2020","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_15","unstructured":"Janke, M., Wand, M., and Schultz, T. (2021, February 13). A Spectral map.ping Method for EMG-Based Recognition of Silent Speech. Available online: https:\/\/www.scitepress.org\/papers\/2010\/28141\/28141.pdf."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Diener, L., and Schultz, T. (2021, February 13). Investigating Objective Intelligibility in Real-Time EMG-to-Speech Conversion. Available online: https:\/\/www.csl.uni-bremen.de\/cms\/images\/documents\/publications\/IS2018_EMG_Realtime.pdf.","DOI":"10.21437\/Interspeech.2018-2080"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41378-019-0127-5","article-title":"An epidermal sEMG tattoo-like patch as a new human\u2013machine interface for patients with loss of voice","volume":"6","author":"Liu","year":"2020","journal-title":"Microsyst. Nanoeng."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Rapin, L., Dohen, M., Polosan, M., Perrier, P., and Loevenbruck, H. (2013). An EMG study of the lip muscles during covert auditory verbal hallucinations in schizophrenia. J. Speech Lang. Hear. Res.","DOI":"10.1044\/1092-4388(2013\/12-0210)"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2375","DOI":"10.1109\/TASLP.2017.2738568","article-title":"Emg-to-speech: Direct generation of speech from facial electromyographic signals","volume":"25","author":"Janke","year":"2017","journal-title":"IEEE\/ACM Trans. Audio. Speech. Lang. Process."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1016\/j.bbe.2018.11.010","article-title":"A speech recognition system based on electromyography for the rehabilitation of dysarthric patients: A Thai syllable study","volume":"39","author":"Jong","year":"2019","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1109\/TBME.1985.325564","article-title":"A Speech Prosthesis Employing a Speech Synthesizer\u2014Vowel Discrimination from Perioral Muscle Activities and Vowel Production","volume":"BME-32","author":"Sugie","year":"1985","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1016\/j.specom.2009.12.002","article-title":"Modeling coarticulation in EMG-based continuous speech recognition","volume":"52","author":"Schultz","year":"2010","journal-title":"Speech Commun."},{"key":"ref_23","first-page":"171","article-title":"Survey on Mouth Interface for Voice Reproduction and Volitional Control","volume":"12","author":"Lee","year":"2015","journal-title":"J. Inf. Technol. Archit."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Srisuwan, N., Wand, M., Janke, M., Phukpattaranont, P., Schultz, T., and Limsakul, C. (2014, January 9\u201312). Enhancement of EMG-based Thai number words classification using frame-based time domain features with stacking filter. Proceedings of the Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific, Siem Reap, Cambodia.","DOI":"10.1109\/APSIPA.2014.7041549"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Gaddy, D., and Klein, D. (2020). Digital Voicing of Silent Speech. arXiv, Available online: https:\/\/arxiv.org\/abs\/2010.02960.","DOI":"10.18653\/v1\/2020.emnlp-main.445"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1669","DOI":"10.1002\/hed.23621","article-title":"Laryngeal replacement with an artificial larynx after total laryngectomy: The possibility of restoring larynx functionality in the future","volume":"36","author":"Debry","year":"2014","journal-title":"Head Neck"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"14717","DOI":"10.1038\/s41598-018-32614-9","article-title":"Voice-selective prediction alterations in nonclinical voice hearers","volume":"8","author":"Pinheiro","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"036020","DOI":"10.1088\/1741-2552\/aa66dd","article-title":"Single-channel in-ear-EEG detects the focus of auditory attention to concurrent tone streams and mixed speech","volume":"14","author":"Fiedler","year":"2017","journal-title":"J. Neural Eng."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"562","DOI":"10.3109\/17549507.2013.862857","article-title":"Tongue-palate contact during selected vowels in children with speech sound disorders","volume":"16","author":"Lee","year":"2014","journal-title":"Int. J. Speech. Lang. Pathol."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1080\/02699200410001663362","article-title":"Abnormal patterns of tongue-palate contact in the speech of individuals with cleft palate","volume":"18","author":"Gibbon","year":"2004","journal-title":"Clin. Linguist. Pho-netics."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1016\/j.jcomdis.2009.05.001","article-title":"Intra-speaker variability in palatometric measures of consonant articulation","volume":"42","author":"Dromey","year":"2009","journal-title":"J. Commun. Disord."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Mantie-Kozlowski, A., and Pitt, K. (2014). Treating myofunctional disorders: A multiple-baseline study of a new treatment using electropalatography. Am. J. Speech-Language Pathol.","DOI":"10.1044\/2014_AJSLP-14-0001"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"21565","DOI":"10.3390\/s141121565","article-title":"An arch-shaped intraoral tongue drive system with built-in tongue-computer interfacing SoC","volume":"14","author":"Park","year":"2014","journal-title":"Sensors"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1109\/TNSRE.2008.2003375","article-title":"A magneto-inductive sensor based wireless tongue-computer interface","volume":"16","author":"Huo","year":"2008","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Sebkhi, N., Yunusova, Y., and Ghovanloo, M. (2018, January 17\u201319). Towards Phoneme Landmarks Identification for American-English using a Multimodal Speech Capture System. Proceedings of the 2018 IEEE Biomedical Circuits and Systems Conference, BioCAS 2018\u2013Proceedings, Cleveland, OH, USA.","DOI":"10.1109\/BIOCAS.2018.8584737"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1007\/BF02345373","article-title":"Myo-electric signals to augment speech recognition","volume":"39","author":"Chan","year":"2001","journal-title":"Med. Biol. Eng. Comp."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Manabe, H., Hiraiwa, A., and Sugimura, T. (2003, January 5\u201310). Unvoice Speech Recognition Using EMG-mime Speech Recognition. Proceedings of the CHI\u201903 Extended Abstracts on Human Factors in Computing Systems, Ft. Lauderdale, FL, USA.","DOI":"10.1145\/765891.765996"},{"key":"ref_38","unstructured":"Maier-Hein, L., Metze, F., Schultz, T., and Waibel, A. (December, January 27). Session independent non-audible speech recognition using surface elec-tromyography. Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, Cancun, Mexico."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Manoni, L., Turchetti, C., Falaschetti, L., and Crippa, P. (2019). A Comparative Study of Computational Methods for Compressed Sensing Reconstruction of EMG Signal. Sensors, 19.","DOI":"10.3390\/s19163531"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1109\/86.847808","article-title":"The mental prosthesis: Assessing the speed of a P300-based brain- computer interface","volume":"8","author":"Donchin","year":"2000","journal-title":"IEEE Trans. Rehabil. Eng."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Mill\u00e1n, J.D.R., Rupp, R., M\u00fcller-Putz, G.R., Murray-Smith, R., Giugliemma, C., Tangermann, M., Vidaurre, C., Cincotti, F., K\u00fcbler, A., and Leeb, R. (2010). Combining brain-computer interfaces and assistive technologies: State-of-the-art and challenges. Front. Neurosci.","DOI":"10.3389\/fnins.2010.00161"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1080\/14639230010015843","article-title":"On the use of EEG features towards person identification via neural networks","volume":"26","author":"Poulos","year":"2001","journal-title":"Med. Inform. Internet Med."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1006\/brcg.1997.0888","article-title":"Patterns of language dominance in focal left and right hemisphere epilepsies: Relation to MRI findings, EEG, sex, and age at onset of epilepsy","volume":"33","author":"Helmstaedter","year":"1997","journal-title":"Brain Cogn."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1281","DOI":"10.1109\/SURV.2012.121912.00075","article-title":"A survey of indoor inertial positioning systems for pedestrians","volume":"15","author":"Harle","year":"2013","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1109\/MCOM.2010.5560598","article-title":"A survey of mobile phone sensing","volume":"48","author":"Lane","year":"2010","journal-title":"IEEE Commun. Mag."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1080\/14417040601123676","article-title":"Advances in EPG palate design","volume":"9","author":"Wrench","year":"2007","journal-title":"Adv. Speech. Lang. Pathol."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1016\/j.medengphy.2010.08.011","article-title":"Isolated word recognition of silent speech using magnetic implants and sensors","volume":"32","author":"Gilbert","year":"2010","journal-title":"Med. Eng. Phys."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"176","DOI":"10.3390\/s100100176","article-title":"Recent advances in sensing oropharyngeal swallowing function in Japan","volume":"10","author":"Ono","year":"2010","journal-title":"Sensors"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.specom.2012.02.001","article-title":"Small-vocabulary speech recognition using a silent speech interface based on magnetic sensing","volume":"55","author":"Hofe","year":"2013","journal-title":"Speech. Commun."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1119","DOI":"10.1016\/j.patrec.2011.02.009","article-title":"A pilot study on augmented speech communication based on Elec-tro-Magnetic Articulography","volume":"32","author":"Heracleous","year":"2011","journal-title":"Pattern. Recognit. Lett."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Van Wassenhove, V. (2013). Speech through ears and eyes: Interfacing the senses with the supramodal brain. Front. Psychol.","DOI":"10.3389\/fpsyg.2013.00388"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1186\/1743-0003-11-168","article-title":"Non-invasive control interfaces for intention detection in active movement-assistive devices","volume":"11","author":"Kooren","year":"2014","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1109\/TSMCC.2010.2089620","article-title":"On the voice-activated question answering","volume":"42","author":"Rosso","year":"2012","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1017\/S0263574714000010","article-title":"Command-based voice teleoperation of a mobile robot via a human-robot interface","volume":"33","author":"Poncela","year":"2015","journal-title":"Robotica"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Hwang, S., Jin, Y.G., and Shin, J.W. (2019). Dual Microphone Voice Activity Detection Based on Reliable Spatial Cues. Sensors, 19.","DOI":"10.3390\/s19143056"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"533","DOI":"10.1163\/156855304774195064","article-title":"Robots that can hear, understand and talk","volume":"18","author":"Prasad","year":"2004","journal-title":"Adv. Robot."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/j.csl.2016.06.007","article-title":"Building DNN acoustic models for large vocabulary speech recognition","volume":"41","author":"Maas","year":"2016","journal-title":"Comput. Speech. Lang."},{"key":"ref_58","unstructured":"Ravanelli, M., and Omologo, M. (2017). Contaminated speech training methods for robust DNN-HMM distant speech recognition. arXiv."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Zeyer, A., Irie, K., Schl\u00fcter, R., and Ney, H. (2018). Improved training of end-to-end attention models for speech recognition. arXiv.","DOI":"10.21437\/Interspeech.2018-1616"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Hori, T., Cho, J., and Watanabe, S. (2018, January 18\u201321). End-to-end speech recognition with word-based RNN language models. Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece.","DOI":"10.1109\/SLT.2018.8639693"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Sak, H., Senior, A., Rao, K., and Beaufays, F. (2015). Fast and accurate recurrent neural network acoustic models for speech recognition. arXiv.","DOI":"10.21437\/Interspeech.2015-350"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1109\/TMM.2017.2751969","article-title":"Aenet: Learning deep audio features for video analysis","volume":"20","author":"Takahashi","year":"2017","journal-title":"IEEE. Trans. Multimedia"},{"key":"ref_63","first-page":"173","article-title":"Deep speech 2: End-to-end speech recognition in english and mandarin","volume":"48","author":"Amodei","year":"2016","journal-title":"Int. Conf. Mach. Learn."},{"key":"ref_64","unstructured":"Assael, Y.M., Shillingford, B., Whiteson, S., and De Freitas, N. (2016). Lipnet: End-to-end sentence-level lipreading. arXiv."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Ephrat, A., and Peleg, S. (2017, January 5\u20139). Vid2speech: Speech reconstruction from silent video. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, LA, USA.","DOI":"10.1109\/ICASSP.2017.7953127"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Chen, Y.C., Yang, Z., Yeh., C.F., Jain., M., and Seltzer., M.L. (2020, January 4\u20138). AIPNet: Generative Adversarial Pre-training of Accent-invariant Networks for End-to-end Speech Recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, Spain.","DOI":"10.1109\/ICASSP40776.2020.9053098"},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Biadsy, F., Weiss, R.J., Moreno, P.J., Kanvesky, D., and Jia, Y. (2019). Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation. arXiv.","DOI":"10.21437\/Interspeech.2019-1789"},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Sun, C., Yang, Y., Wen, C., Xie, K., and Wen, F. (2018). Voice Quality Assessment in Communication identification for limited dataset using the deep migration hybrid model based on transfer learning. Sensors, 18.","DOI":"10.3390\/s18072399"},{"key":"ref_69","unstructured":"Wang, D., and Brown, G.J. (2006). Computational Auditory Scene Analysis: Principles, Algorithms, and Applications, Wiley-IEEE Press."},{"key":"ref_70","unstructured":"Xu, R., Ren, Z., Dai, W., Lao, D., and Kwan, C. (2004, January 20\u201322). Multimodal speech enhancement in noisy environment. Proceedings of the 2004 Int. Symp. Intell. Multimedia, Video Speech Process, Hong Kong, China."},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Kamath, S., and Loizou, P. (2002, January 13\u201317). A multi-band spectral subtraction method for enhancing speech corrupted by colored noise. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Orlando, FL, USA.","DOI":"10.1109\/ICASSP.2002.5745591"},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"1766","DOI":"10.1109\/TASL.2007.901310","article-title":"Soft mask methods for single-channel speaker separation","volume":"15","author":"Reddy","year":"2007","journal-title":"IEEE Trans. Audio. Speech. Lang. Process."},{"key":"ref_73","unstructured":"Scalart, P., and Filho, J.V. (1996, January 9). Speech enhancement based on a priori signal to noise estimation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Atlanta, GA, USA."},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"1586","DOI":"10.1109\/PROC.1979.11540","article-title":"Enhancement and Bandwidth Compression of Noisy Speech","volume":"67","author":"Lim","year":"1979","journal-title":"Proc. IEEE"},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"De Almeida, F.L., Rosa, R.L., and Rodriguez, D.Z. (2018, January 28\u201331). Voice quality assessment in communication services using deep learning. Proceedings of the 15th International Symposium on Wireless Communication Systems (ISWCS), Lisbon, Portugal.","DOI":"10.1109\/ISWCS.2018.8491055"},{"key":"ref_76","doi-asserted-by":"crossref","unstructured":"Gosztolya, G., Pint\u00e9r, \u00c1., T\u00f3th, L., Gr\u00f3sz, T., Mark\u00f3, A., and Csap\u00f3, T.G. (2019, January 14\u201319). Autoencoder-Based Articulatory-to-Acoustic Mapping for Ultrasound Silent Speech Interfaces. Proceedings of the IEEE International Joint Conference on Neural Networks, Budapest, Hungary.","DOI":"10.1109\/IJCNN.2019.8852153"},{"key":"ref_77","doi-asserted-by":"crossref","unstructured":"Cao, B., Kim, M.J., van Santen, J.P., Mau, T., and Wang, J. (2017). Integrating Articulatory Information in Deep Learning-Based Text-to-Speech Synthesis. INTERSPEECH, 254\u2013258.","DOI":"10.21437\/Interspeech.2017-1762"},{"key":"ref_78","first-page":"69","article-title":"The fisher corpus: A resource for the next generations of speech-to-text","volume":"4","author":"Cieri","year":"2004","journal-title":"LREC"},{"key":"ref_79","unstructured":"Gretter, R. (2014). Euronews: A multilingual speech corpus for ASR. LREC, 2635\u20132638. Available online: https:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.1083.2378&rep=rep1&type=pdf."},{"key":"ref_80","doi-asserted-by":"crossref","unstructured":"Angelini, B., Brugnara, F., Falavigna, D., Giuliani, D., Gretter, R., and Omologo, M. (1994, January 18\u201322). Speaker independent continuous speech recognition using an acoustic-phonetic Italian corpus. Proceedings of the Third International Conference on Spoken Language Processing, Yokohama, Japan.","DOI":"10.21437\/ICSLP.1994-362"},{"key":"ref_81","doi-asserted-by":"crossref","unstructured":"Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. (2015, January 19\u201324). Librispeech: An asr corpus based on public domain audio books. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia.","DOI":"10.1109\/ICASSP.2015.7178964"},{"key":"ref_82","unstructured":"(1994, July 02). Linguistic Data Consortium, CSR-II (wsj1) Complete. Available online: https:\/\/doi.org\/10.35111\/q7sb-vv12."},{"key":"ref_83","unstructured":"Garofalo, J., Graff, D., Paul, D., and Pallett, D. (2007, May 30). CSR-I (wsj0) Complete. Available online: https:\/\/doi.org\/10.35111\/ewkm-cg47."},{"key":"ref_84","doi-asserted-by":"crossref","unstructured":"Kingsbury, B. (2009, January 19\u201324). Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling. Proceedings of the 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan.","DOI":"10.1109\/ICASSP.2009.4960445"},{"key":"ref_85","doi-asserted-by":"crossref","unstructured":"Font, F., Roma, G., and Serra, X. (2013, January 21\u201325). Freesound technical demo. Proceedings of the 21st ACM international conference on Multimedia, Barcelona, Spain.","DOI":"10.1145\/2502081.2502245"},{"key":"ref_86","unstructured":"Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A dataset of 101 human action classes from videos in the wild. arXiv."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"2421","DOI":"10.1121\/1.2229005","article-title":"An audio-visual corpus for speech perception and automatic speech recognition","volume":"120","author":"Cooke","year":"2006","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_88","unstructured":"NIST Multimodal Information Group (2011, August 15). 2008 NIST Speaker Recognition Evaluation Training Set Part 1. Available online: https:\/\/doi.org\/10.35111\/pr4h-n676."},{"key":"ref_89","unstructured":"(2017, December 25). DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus. Available online: https:\/\/catalog.ldc.upenn.edu\/ldc93s1."},{"key":"ref_90","doi-asserted-by":"crossref","unstructured":"Lu, Y., and Li, H. (2019). Automatic Lip-Reading System Based on Deep Convolutional Neural Network and Attention-Based Long Short-Term Memory. Appl. Sci., 9.","DOI":"10.3390\/app9081599"},{"key":"ref_91","doi-asserted-by":"crossref","unstructured":"Akbari, H., Arora, H., Cao, L., and Mesgarani, N. (2018, January 15\u201320). Lip2audspec: Speech reconstruction from silent lip movements video. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.","DOI":"10.1109\/ICASSP.2018.8461856"},{"key":"ref_92","unstructured":"Li, X., and Kwan, C. (November, January 30). Geometrical feature extraction for robust speech recognition. Proceedings of the IEEE International Conference on Record of the Thirty-Ninth Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA."},{"key":"ref_93","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/j.imavis.2018.07.002","article-title":"Survey on automatic lip-reading in the era of deep learning","volume":"78","author":"Sukno","year":"2018","journal-title":"Image. Vis. Comput."},{"key":"ref_94","doi-asserted-by":"crossref","first-page":"204518","DOI":"10.1109\/ACCESS.2020.3036865","article-title":"A Survey of Research on Lipreading Technology","volume":"8","author":"Hao","year":"2020","journal-title":"IEEE Access"},{"key":"ref_95","unstructured":"Fernandez-Lopez, A., Martinez, O., and Sukno, F.M. (June, January 30). Towards estimating the upper bound of visual-speech recognition: The visual lip-reading feasibility database. Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, Washington, DC, USA."},{"key":"ref_96","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1016\/j.ins.2019.09.035","article-title":"Effective privacy preserving data publishing by vectorization","volume":"527","author":"Eom","year":"2019","journal-title":"Inform. Sci."},{"key":"ref_97","doi-asserted-by":"crossref","unstructured":"Wang, J., and Hahm, S. Speaker-independent silent speech recognition with across-speaker articulatory normalization and speaker adaptive training. Proceedings of the Annual Conference of the International Speech Communication Association\u2013Proceedings, Available online: https:\/\/www.isca-speech.org\/archive\/interspeech_2015\/i15_2415.html.","DOI":"10.21437\/Interspeech.2015-522"},{"key":"ref_98","doi-asserted-by":"crossref","first-page":"177995","DOI":"10.1109\/ACCESS.2020.3026579","article-title":"Silent Speech Interfaces for Speech Restoration: A Review","volume":"8","author":"Gomez","year":"2020","journal-title":"IEEE Access"},{"key":"ref_99","doi-asserted-by":"crossref","unstructured":"Kapur, A., Kapur, S., and Maes, P. (2018, January 7\u201311). Alterego: A personalized wearable silent speech interface. Proceedings of the 2018 International Conference Intelligent User Interfaces, Tokyo, Japan.","DOI":"10.1145\/3172944.3172977"},{"key":"ref_100","doi-asserted-by":"crossref","unstructured":"Kimura, N., Kono, M., and Rekimoto, J. (2019, January 4\u20139). SottoVoce: An ultrasound imaging-based silent speech interaction using deep neural networks. Proceedings of the CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, UK.","DOI":"10.1145\/3290605.3300376"},{"key":"ref_101","doi-asserted-by":"crossref","first-page":"9324","DOI":"10.1109\/JSEN.2019.2923585","article-title":"A deep neural network-based permanent magnet localization for tongue tracking","volume":"19","author":"Sebkhi","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_102","doi-asserted-by":"crossref","unstructured":"Kim, M., Sebkhi, N., Cao, B., Ghovanloo, M., and Wang, J. (2018, January 17\u201319). Preliminary test of a wireless magnetic tongue tracking system for silent speech interface. Proceedings of the Biomedical Circuits and Systems Conference (BioCAS), Cleveland, OH, USA.","DOI":"10.1109\/BIOCAS.2018.8584786"},{"key":"ref_103","doi-asserted-by":"crossref","unstructured":"Csap\u00f3, T.G., Al-Radhi, M.S., N\u00e9meth, G., Gosztolya, G., Gr\u00f3sz, T., T\u00f3th, L., and Mark\u00f3, A. (2019). Ultrasound-based Silent Speech Interface Built on a Continuous Vocoder. arXiv, Available online: https:\/\/arxiv.org\/abs\/1906.09885.","DOI":"10.21437\/Interspeech.2019-2046"},{"key":"ref_104","unstructured":"Cao, B., Kim, M.J., Wang, J.R., van Santen, J.P., Mau, T., and Wang, J. Articulation-to-Speech Synthesis Using Articulatory Flesh Point Sensors\u2019 Orientation Information. Proceedings of the INTERSPEECH, Available online: https:\/\/www.researchgate.net\/profile\/Jun_Wang121\/publication\/327350739_Articulation-to-Speech_Synthesis_Using_Articula-tory_Flesh_Point_Sensors\u2019_Orientation_Information\/links\/5b89a729299bf1d5a735a574\/Articulation-to-Speech-Synthesis-Using-Articulatory-Flesh-Point-Sensors-Orientation-Information.pdf."},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1080\/14640748108400802","article-title":"The role of subvocalisation in reading","volume":"33","author":"Baddeley","year":"1981","journal-title":"Q. J. Exp. Psychol."},{"key":"ref_106","unstructured":"Oord, A.V.D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A.W., and Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio. arXiv."},{"key":"ref_107","doi-asserted-by":"crossref","unstructured":"Boles., A., and Rad., P. (2017, January 18\u201321). Voice biometrics: Deep learning-based voiceprint authentication system. Proceedings of the IEEE System of Systems Engineering Conference, Waikoloa, HI, USA.","DOI":"10.1109\/SYSOSE.2017.7994971"},{"key":"ref_108","doi-asserted-by":"crossref","unstructured":"Wang, J., Samal, A., and Green, J.R. Across-speaker articulatory normalization for speaker-independent silent speech recognition. Proceedings of the Annual Conference of the International Speech Communication Association, Available online: https:\/\/www.isca-speech.org\/archive\/interspeech_2014\/i14_1179.html.","DOI":"10.21437\/Interspeech.2014-299"},{"key":"ref_109","unstructured":"Hahm, S., Wang, J., and Friedman, J. (2015). Silent speech recognition from articulatory movements using deep neural network. Int. Congr. Phon. Sci., 1\u20135. Available online: http:\/\/www.internationalphoneticassociation.org\/icphs-proceedings\/ICPhS2015\/Papers\/ICPHS0524.pdf."},{"key":"ref_110","unstructured":"Kim, M., Cao, B., Mau, T., and Wang, J. Speaker-independent silent speech recognition from flesh-point articulatory movements using an LSTM neural network. Proceedings of the IEEE\/ACM Transactions on Audio Speech and Language Processing, Available online: https:\/\/ieeexplore.ieee.org\/abstract\/document\/8114350."},{"key":"ref_111","doi-asserted-by":"crossref","unstructured":"Beigi, H. (2012). Speaker recognition: Advancements and challenges. New Trends and Developments in Biometrics, InTech.","DOI":"10.5772\/52023"},{"key":"ref_112","doi-asserted-by":"crossref","first-page":"2769","DOI":"10.21437\/Interspeech.2017-952","article-title":"Multiview Representation Learning via Deep CCA for Silent Speech Recognition","volume":"7","author":"Kim","year":"2017","journal-title":"INTERSPEECH"},{"key":"ref_113","unstructured":"Patil, P., Gujarathi, G., and Sonawane, G. (2016). Different Approaches for Artifact Removal in Electromyography based Silent Speech Interface. Int. J. Sci. Eng. Technol., 5, Available online: http:\/\/ijsetr.org\/wp-content\/uploads\/2016\/01\/IJSETR-VOL-5-ISSUE-1-282-285.pdf."},{"key":"ref_114","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0044155","article-title":"Delayed auditory feedback","volume":"60","author":"Yates","year":"1963","journal-title":"Psychol. Bull."},{"key":"ref_115","doi-asserted-by":"crossref","unstructured":"Jou, S.C., Schultz, T., Walliczek, M., Kraft, F., and Waibel, A. (2006). Towards continuous speech recognition using surface electromyography. Int. Conf. Spok. Lang. Process., 573\u2013576. Available online: https:\/\/www.isca-speech.org\/archive\/interspeech_2006\/i06_1592.html.","DOI":"10.21437\/Interspeech.2006-212"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/4\/1399\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:25:11Z","timestamp":1760160311000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/4\/1399"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,17]]},"references-count":115,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,2]]}},"alternative-id":["s21041399"],"URL":"https:\/\/doi.org\/10.3390\/s21041399","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,17]]}}}