{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,27]],"date-time":"2025-12-27T21:15:20Z","timestamp":1766870120785,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2022,5,23]],"date-time":"2022-05-23T00:00:00Z","timestamp":1653264000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Inha University","award":["NRF-2018R1A6A1A03025523"],"award-info":[{"award-number":["NRF-2018R1A6A1A03025523"]}]},{"name":"Basic Science Research Program of the National Research Foundation of Korea","award":["NRF-2018R1A6A1A03025523"],"award-info":[{"award-number":["NRF-2018R1A6A1A03025523"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Inertial-measurement-unit (IMU)-based human activity recognition (HAR) studies have improved their performance owing to the latest classification model. In this study, the conformer, which is a state-of-the-art (SOTA) model in the field of speech recognition, is introduced in HAR to improve the performance of the transformer-based HAR model. The transformer model has a multi-head self-attention structure that can extract temporal dependency well, similar to the recurrent neural network (RNN) series while having higher computational efficiency than the RNN series. However, recent HAR studies have shown good performance by combining an RNN-series and convolutional neural network (CNN) model. Therefore, the performance of the transformer-based HAR study can be improved by adding a CNN layer that extracts local features well. The model that improved these points is the conformer-based-model model. To evaluate the proposed model, WISDM, UCI-HAR, and PAMAP2 datasets were used. A synthetic minority oversampling technique was used for the data augmentation algorithm to improve the dataset. From the experiment, the conformer-based HAR model showed better performance than baseline models: the transformer-based-model and the 1D-CNN HAR models. Moreover, the performance of the proposed algorithm was superior to that of algorithms proposed in recent similar studies which do not use RNN-series.<\/jats:p>","DOI":"10.3390\/s22103932","type":"journal-article","created":{"date-parts":[[2022,5,24]],"date-time":"2022-05-24T03:16:55Z","timestamp":1653362215000},"page":"3932","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["Inertial-Measurement-Unit-Based Novel Human Activity Recognition Algorithm Using Conformer"],"prefix":"10.3390","volume":"22","author":[{"given":"Yeon-Wook","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Inha University, Incheon 22212, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2076-7255","authenticated-orcid":false,"given":"Woo-Hyeong","family":"Cho","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Inha University, Incheon 22212, Korea"}]},{"given":"Kyu-Sung","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Otorhinolaryngology, Inha University Hospital, Incheon 22332, Korea"}]},{"given":"Sangmin","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Inha University, Incheon 22212, Korea"},{"name":"Department of Smart Engineering Program in Biomedical Science & Engineering, Inha University, Incheon 22212, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2499621","article-title":"A tutorial on human activity recognition using body-worn inertial sensors","volume":"46","author":"Bulling","year":"2014","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"210816","DOI":"10.1109\/ACCESS.2020.3037715","article-title":"Human activity recognition using inertial, physiological and environmental sensors: A comprehensive survey","volume":"8","author":"Demrozi","year":"2020","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Sousa Lima, W., Souto, E., El-Khatib, K., Jalali, R., and Gama, J. (2019). Human activity recognition using inertial sensors in a smartphone: An overview. Sensors, 19.","DOI":"10.3390\/s19143213"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ma, R., Yan, D., Peng, H., Yang, T., Sha, X., Zhao, Y., and Liu, L. (2018, January 5\u20137). Basketball movements recognition using a wrist wearable inertial measurement unit. Proceedings of the 2018 IEEE 1st International Conference on Micro\/Nano Sensors for AI, Healthcare, and Robotics (NSENS), Shenzhen, China.","DOI":"10.1109\/NSENS.2018.8713634"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wang, Z., Shi, X., Wang, J., Gao, F., Li, J., Guo, M., and Qiu, S. (2019, January 6\u20139). Swimming motion analysis and posture recognition based on wearable inertial sensors. Proceedings of the 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).","DOI":"10.1109\/SMC.2019.8913847"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kim, Y.W., Joa, K.L., Jeong, H.Y., and Lee, S. (2021). Wearable IMU-based human activity recognition algorithm for clinical balance assessment using 1D-CNN and GRU ensemble model. Sensors, 21.","DOI":"10.3390\/s21227628"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Huang, C., Zhang, F., Xu, Z., and Wei, J. (2022). The Diverse Gait Dataset: Gait segmentation using inertial sensors for pedestrian localization with different genders, heights and walking speeds. Sensors, 22.","DOI":"10.3390\/s22041678"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/SURV.2012.110112.00192","article-title":"A survey on human activity recognition using wearable sensors","volume":"15","author":"Lara","year":"2012","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2040011","DOI":"10.1142\/S0219519420400114","article-title":"A new auto-Scoring algorithm for bance assessment with wearable IMU device based on nonlinear model","volume":"20","author":"Kim","year":"2020","journal-title":"J. Mech. Med. Biol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3070","DOI":"10.1109\/TII.2017.2712746","article-title":"Robust human activity recognition using smartphone sensors via CT-PCA and online SVM","volume":"13","author":"Chen","year":"2017","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.patrec.2018.02.010","article-title":"Deep learning for sensor-based activity recognition: A survey","volume":"119","author":"Wang","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1593","DOI":"10.1007\/s11277-021-08525-w","article-title":"A survey of deep learning-based models for human activity recognition","volume":"120","author":"Khan","year":"2021","journal-title":"Wirel. Pers. Commun."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Mekruksavanich, S., and Jitpattanakul, A. (2020, January 25\u201328). Smartwatch-based human activity recognition using hybrid lstm network. Proceedings of the 2020 IEEE Sensors, Virtual Conference, Virtual, Rotterdam, The Netherlands.","DOI":"10.1109\/SENSORS47125.2020.9278630"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"31663","DOI":"10.1007\/s11042-020-09537-7","article-title":"EnsemConvNet: A deep learning approach for human activity recognition using smartphone sensors for healthcare applications","volume":"79","author":"Mukherjee","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_17","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 30, Long Beach, CA, USA."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1748","DOI":"10.1016\/j.ijforecast.2021.03.012","article-title":"Temporal fusion transformers for interpretable multi-horizon time series forecasting","volume":"37","author":"Lim","year":"2021","journal-title":"Int. J. Forecast."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., and Xu, Y. (2022). A survey on vision transformer. IEEE Trans. Pattern Anal. Mach. Intell.","DOI":"10.1109\/TPAMI.2022.3152247"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.aiopen.2022.01.001","article-title":"Survey: Transformer based video-language pre-training","volume":"3","author":"Ruan","year":"2022","journal-title":"AI Open"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"108487","DOI":"10.1016\/j.patcog.2021.108487","article-title":"Action Transformer: A self-attention model for short-time pose-based human action recognition","volume":"124","author":"Mazzia","year":"2022","journal-title":"Pattern Recognit."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"53540","DOI":"10.1109\/ACCESS.2021.3070646","article-title":"Boosting inertial-based human activity recognition with transformers","volume":"9","author":"Shavit","year":"2021","journal-title":"IEEE Access"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Dirgov\u00e1 Lupt\u00e1kov\u00e1, I., Kubov\u010d\u00edk, M., and Posp\u00edchal, J. (2022). Wearable sensor-based human activity recognition with transformer model. Sensors, 22.","DOI":"10.20944\/preprints202202.0111.v1"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2020, January 25\u201329). Conformer: Convolution-augmented Transformer for Speech Recognition. Proceedings of the INTERSPEECH 2020, Shanghai, China.","DOI":"10.21437\/Interspeech.2020-3015"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Peng, Z., Huang, W., Gu, S., Xie, L., Wang, Y., Jiao, J., and Ye, Q. (2021, January 11\u201317). Conformer: Local features coupling global representations for visual recognition. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00042"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Chen, S., Wu, Y., Chen, Z., Wu, J., Li, J., Yoshioka, T., Wang, C., Liu, S., and Zhou, M. (2021, January 6\u201311). Continuous speech separation with conformer. Proceedings of the ICASSP 2021\u20142021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.","DOI":"10.1109\/ICASSP39728.2021.9413423"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Okai, J., Paraschiakos, S., Beekman, M., Knobbe, A., and de S\u00e1, C.R. (2019, January 23\u201327). Building robust models for human activity recognition from raw accelerometers data using gated recurrent units and long short term memory neural networks. Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).","DOI":"10.1109\/EMBC.2019.8857288"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zebin, T., Sperrin, M., Peek, N., and Casson, A.J. (2018, January 17\u201321). Human activity recognition from inertial sensor time-series using batch normalized deep LSTM recurrent networks. Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.","DOI":"10.1109\/EMBC.2018.8513115"},{"key":"ref_30","first-page":"2815","article-title":"Comparison of deep feature learning methods for human activity recognition","volume":"35","author":"Kuang","year":"2018","journal-title":"Appl. Res. Comput."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"7265","DOI":"10.1109\/JSEN.2020.2978772","article-title":"The layer-wise training convolutional neural networks using local loss for sensor-based human activity recognition","volume":"20","author":"Teng","year":"2020","journal-title":"IEEE Sens. J."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ha, S., Yun, J.M., and Choi, S. (2015, January 9\u201312). Multi-modal convolutional neural networks for activity recognition. Proceedings of the 2015 IEEE International Conference on Systems, Man, and Cybernetics, Hong Kong, China.","DOI":"10.1109\/SMC.2015.525"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ord\u00f3\u00f1ez, F.J., and Roggen, D. (2016). Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16.","DOI":"10.3390\/s16010115"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1145\/1964897.1964918","article-title":"Activity recognition using cell phone accelerometers","volume":"12","author":"Kwapisz","year":"2011","journal-title":"ACM SigKDD Explor. Newsl."},{"key":"ref_35","unstructured":"Anguita, D., Ghio, A., Oneto, L., Parra, X., and Reyes-Ortiz, J.L. (2013, January 24\u201326). A public domain dataset for human activity recognition using smartphones. Proceedings of the 21th International European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"35585","DOI":"10.1007\/s11042-020-10478-4","article-title":"Hybrid deep learning approaches for smartphone sensor-based human activity recognition","volume":"80","author":"Ghate","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"199393","DOI":"10.1109\/ACCESS.2020.3032699","article-title":"Human activity recognition based on Gramian angular field and deep convolutional neural network","volume":"8","author":"Xu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"107671","DOI":"10.1016\/j.asoc.2021.107671","article-title":"Attention induced multi-head convolutional neural network for human activity recognition","volume":"110","author":"Khan","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40537-019-0192-5","article-title":"Survey on deep learning with class imbalance","volume":"6","author":"Johnson","year":"2019","journal-title":"J. Big Data"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., and Xu, H. (2020). Time series data augmentation for deep learning: A survey. arXiv.","DOI":"10.24963\/ijcai.2021\/631"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"113696","DOI":"10.1016\/j.eswa.2020.113696","article-title":"Improving classification accuracy using data augmentation on small data sets","volume":"161","author":"Jerez","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Rok, B., and Lusa, L. (2013). SMOTE for high-dimensional class-imbalanced data. BMC Bioinform., 14.","DOI":"10.1186\/1471-2105-14-106"},{"key":"ref_43","unstructured":"Boin, J.B., Roth, N., Doshi, J., Llueca, P., and Borensztein, N. (2020). Multi-class segmentation under severe class imbalance: A case study in roof damage assessment. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Reiss, A., and Stricker, D. (2012, January 18\u201312). Introducing a new benchmarked dataset for activity monitoring. Proceedings of the 2012 16th International Symposium on Wearable Computers, Newcastle, UK.","DOI":"10.1109\/ISWC.2012.13"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"6696","DOI":"10.1007\/s11227-021-04140-5","article-title":"A lightweight neural network framework using linear grouped convolution for human activity recognition on mobile devices","volume":"78","author":"Liu","year":"2022","journal-title":"J. Supercomput."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"5889","DOI":"10.1109\/JSEN.2022.3149337","article-title":"Real-time human activity recognition using conditionally parametrized convolutions on mobile and wearable devices","volume":"22","author":"Cheng","year":"2022","journal-title":"IEEE Sens. J."},{"key":"ref_47","unstructured":"Abedin, A., Rezatofighi, H., and Ranasinghe, D.C. (2021). Guided-GAN: Adversarial Representation Learning for Activity Recognition with Wearables. arXiv."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"10463","DOI":"10.1007\/s11227-021-03688-6","article-title":"BCGAN: A CGAN-based over-sampling model using the boundary class for data balancing","volume":"77","author":"Son","year":"2021","journal-title":"J. Supercomput."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Chowdhury, S.S., Boubrahimi, S.F., and Hamdi, S.M. (2021, January 13\u201315). Time Series Data Augmentation using Time-Warped Auto-Encoders. Proceedings of the 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA.","DOI":"10.1109\/ICMLA52953.2021.00111"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/10\/3932\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:16:41Z","timestamp":1760138201000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/10\/3932"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,23]]},"references-count":49,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["s22103932"],"URL":"https:\/\/doi.org\/10.3390\/s22103932","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,5,23]]}}}