{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T02:19:16Z","timestamp":1778725156028,"version":"3.51.4"},"reference-count":28,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:00:00Z","timestamp":1687132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Trade, Industry, and Energy (MOTIE, Korea)","award":["00144288"],"award-info":[{"award-number":["00144288"]}]},{"name":"Ministry of Trade, Industry, and Energy (MOTIE, Korea)","award":["00144290"],"award-info":[{"award-number":["00144290"]}]},{"name":"IDEC","award":["00144288"],"award-info":[{"award-number":["00144288"]}]},{"name":"IDEC","award":["00144290"],"award-info":[{"award-number":["00144290"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Keyword spotting (KWS) systems are used for human\u2013machine communications in various applications. In many cases, KWS involves a combination of wake-up-word (WUW) recognition for device activation and voice command classification tasks. These tasks present a challenge for embedded systems due to the complexity of deep learning algorithms and the need for optimized networks for each application. In this paper, we propose a depthwise separable binarized\/ternarized neural network (DS-BTNN) hardware accelerator capable of performing both WUW recognition and command classification on a single device. The design achieves significant area efficiency by redundantly utilizing bitwise operators in the computation of the binarized neural network (BNN) and ternary neural network (TNN). In a complementary metal-oxide semiconductor (CMOS) 40 nm process environment, the DS-BTNN accelerator demonstrated significant efficiency. Compared with a design approach where BNN and TNN were independently developed and subsequently integrated as two separate modules into the system, our method achieved a 49.3% area reduction while yielding an area of 0.558 mm2. The designed KWS system, which was implemented on a Xilinx UltraScale+ ZCU104 field-programmable gate array (FPGA) board, receives real-time data from the microphone, preprocesses them into a mel spectrogram, and uses this as input to the classifier. Depending on the order, the network operates as a BNN or a TNN for WUW recognition and command classification, respectively. Operating at 170 MHz, our system achieved 97.1% accuracy in BNN-based WUW recognition and 90.5% in TNN-based command classification.<\/jats:p>","DOI":"10.3390\/s23125701","type":"journal-article","created":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T02:29:19Z","timestamp":1687141759000},"page":"5701","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["FPGA Implementation of Keyword Spotting System Using Depthwise Separable Binarized and Ternarized Neural Networks"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-1982-2475","authenticated-orcid":false,"given":"Seongwoo","family":"Bae","sequence":"first","affiliation":[{"name":"School of Electronics and Information Engineering, Korea Aerospace University, Goyang-si 10540, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2693-5942","authenticated-orcid":false,"given":"Haechan","family":"Kim","sequence":"additional","affiliation":[{"name":"School of Electronics and Information Engineering, Korea Aerospace University, Goyang-si 10540, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9344-7052","authenticated-orcid":false,"given":"Seongjoo","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Semiconductor Systems Engineering, Sejong University, Seoul 05006, Republic of Korea"},{"name":"Institute of Semiconductor and System IC, Sejong University, Seoul 05006, Republic of Korea"},{"name":"Department of Convergence Engineering of Intelligent Drone, Sejong University, Seoul 05006, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2299-9911","authenticated-orcid":false,"given":"Yunho","family":"Jung","sequence":"additional","affiliation":[{"name":"School of Electronics and Information Engineering, Korea Aerospace University, Goyang-si 10540, Republic of Korea"},{"name":"Department of Smart Air Mobility, Korea Aerospace University, Goyang-si 10540, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,19]]},"reference":[{"key":"ref_1","unstructured":"Blouw, P., Malik, G., Morcos, B., Voelker, A.R., and Eliasmith, C. (2021). Hardware Aware Training for Efficient Keyword Spotting on General Purpose and Specialized Hardware. arXiv."},{"key":"ref_2","unstructured":"Zhang, Y., Suda, N., Lai, L., and Chandra, V. (2018). Hello Edge: Keyword Spotting on Microcontrollers. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1186\/s13636-020-00176-2","article-title":"A depthwise separable convolutional neural network for keyword spotting on an embedded system","volume":"2020","author":"Epp","year":"2020","journal-title":"EURASIP J. Audio Speech Music. Process."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Xiang, L., Lu, S., Wang, X., Liu, H., Pang, W., and Yu, H. (2019, January 25\u201327). Implementation of LSTM Accelerator for Speech Keywords Recognition. Proceedings of the 2019 IEEE 4th International Conference on Integrated Circuits and Microsystems (ICICM), Beijing, China.","DOI":"10.1109\/ICICM48536.2019.8977176"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Song, D., Yin, S., Ouyang, P., Liu, L., and Wei, S. (2018, January 20\u201322). Low Bits: Binary Neural Network for Vad and Wakeup. Proceedings of the 2018 5th International Conference on Information Science and Control Engineering (ICISCE), Zhengzhou, China.","DOI":"10.1109\/ICISCE.2018.00071"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1109\/JSSC.2020.3029097","article-title":"A 510-nW Wake-Up Keyword-Spotting Chip Using Serial-FFT-Based MFCC and Binarized Depthwise Separable CNN in 28-nm CMOS","volume":"56","author":"Shan","year":"2021","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zhu, L., Shan, W., Xu, J., and Lu, Y. (2021, January 13\u201322). AAD-KWS: A sub-\u03bcW keyword spotting chip with a zero-cost, acoustic activity detector from a 170 nW MFCC feature extractor in 28 nm CMOS. Proceedings of the ESSCIRC 2021\u2014IEEE 47th European Solid State Circuits Conference (ESSCIRC), Grenoble, France.","DOI":"10.1109\/ESSCIRC53450.2021.9567770"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"4733","DOI":"10.1109\/TCSI.2020.2997913","article-title":"A 22nm, 10.8 \u03bcW\/15.1 \u03bcW Dual Computing Modes High Power-Performance-Area Efficiency Domained Background Noise Aware Keyword- Spotting Processor","volume":"67","author":"Liu","year":"2020","journal-title":"IEEE Trans. Circuits Syst. Regul. Pap."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"205878","DOI":"10.1109\/ACCESS.2020.3037931","article-title":"QCNN Inspired Reconfigurable Keyword Spotting Processor With Hybrid Data-Weight Reuse Methods","volume":"8","author":"Gong","year":"2020","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Giraldo, J.S.P., Lauwereins, S., Badami, K., Van Hamme, H., and Verhelst, M. (2019, January 9\u201314). 18\u03bcW SoC for near-microphone Keyword Spotting and Speaker Verification. Proceedings of the 2019 Symposium on VLSI Circuits, Kyoto, Japan.","DOI":"10.23919\/VLSIC.2019.8777994"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"983","DOI":"10.1109\/JSSC.2017.2778702","article-title":"BRein Memory: A Single-Chip Binary\/Ternary Reconfigurable in-Memory Deep Neural Network Accelerator Achieving 1.4 TOPS at 0.6 W","volume":"53","author":"Ando","year":"2018","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1109\/JSSC.2018.2869150","article-title":"An Always-On 3.8 \u03bcJ\/86Processor with All Memory on Chip in 28-nm CMOS","volume":"54","author":"Bankman","year":"2019","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Choi, S., Lee, J., Lee, K., and Yoo, H.J. (2018, January 11\u201315). A 9.02 mW CNN-stereo-based real-time 3D hand-gesture recognition processor for smart mobile devices. Proceedings of the 2018 IEEE International Solid-State Circuits Conference\u2014(ISSCC), San Francisco, CA, USA.","DOI":"10.1109\/ISSCC.2018.8310263"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2220","DOI":"10.1109\/TVLSI.2021.3120189","article-title":"Efficient Execution of Temporal Convolutional Networks for Embedded Keyword Spotting","volume":"29","author":"Giraldo","year":"2021","journal-title":"IEEE Trans. Very Large Scale Integr. (VLSI) Syst."},{"key":"ref_15","first-page":"1662","article-title":"A 2.5 \u03bcW KWS Engine With Pruned LSTM and Embedded MFCC for IoT Applications","volume":"69","author":"Chong","year":"2022","journal-title":"IEEE Trans. Circuits Syst. II Express Briefs"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Yang, L., He, Z., and Fan, D. (2019, January 9\u201311). Binarized Depthwise Separable Neural Network for Object Tracking in FPGA. Proceedings of the GLSVLSI \u201919: Great Lakes Symposium on VLSI, Tysons Corner, VA, USA.","DOI":"10.1145\/3299874.3318034"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Gupta, H., and Gupta, D. (2016, January 14\u201315). LPC and LPCC method of feature extraction in Speech Recognition System. Proceedings of the 2016 6th International Conference\u2014Cloud System and Big Data Engineering (Confluence), Noida, India.","DOI":"10.1109\/CONFLUENCE.2016.7508171"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1738","DOI":"10.1121\/1.399423","article-title":"Perceptual linear predictive (PLP) analysis of speech","volume":"87","author":"Hermansky","year":"1990","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_19","unstructured":"Hermansky, H., Morgan, N., Bayya, A., and Kohn, P. (1991, January 4\u20136). The challenge of inverse-E: The RASTA-PLP method. Proceedings of the [1991] Conference Record of the Twenty-Fifth Asilomar Conference on Signals, Systems & Computers, Pacific Grove, CA, USA."},{"key":"ref_20","first-page":"56677","article-title":"Robust Speech Recognition System Using Conventional and Hybrid Features of MFCC, LPCC, PLP, RASTA-PLP and Hidden Markov Model Classifier in Noisy Conditions","volume":"3","author":"Elharati","year":"2015","journal-title":"J. Comput. Commun."},{"key":"ref_21","unstructured":"Ioffe, S., and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv."},{"key":"ref_22","unstructured":"Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016). Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or \u22121. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1072","DOI":"10.1016\/j.neucom.2017.09.046","article-title":"FP-BNN: Binarized neural network on FPGA","volume":"275","author":"Liang","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2679","DOI":"10.1109\/JSSC.2017.2712626","article-title":"A Neuromorphic Chip Optimized for Deep Learning and CMOS Technology With Time-Domain Analog and Digital Mixed-Signal Processing","volume":"52","author":"Miyashita","year":"2017","journal-title":"IEEE J. Solid-State Circuits"},{"key":"ref_25","unstructured":"Warden, P. (2018). Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition. arXiv."},{"key":"ref_26","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhang, Z. (2018, January 4\u20136). Improved Adam Optimizer for Deep Neural Networks. Proceedings of the 2018 IEEE\/ACM 26th International Symposium on Quality of Service (IWQoS), Banff, AB, Canada.","DOI":"10.1109\/IWQoS.2018.8624183"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"440","DOI":"10.1109\/JSAC.2003.809724","article-title":"Prototype experience for MIMO BLAST over third-generation wireless system","volume":"21","author":"Adjoudani","year":"2003","journal-title":"IEEE J. Sel. Areas Commun."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5701\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:56:08Z","timestamp":1760126168000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5701"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,19]]},"references-count":28,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["s23125701"],"URL":"https:\/\/doi.org\/10.3390\/s23125701","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,19]]}}}