{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T03:30:41Z","timestamp":1768015841868,"version":"3.49.0"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T00:00:00Z","timestamp":1658793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001321","name":"National Research Foundation","doi-asserted-by":"publisher","award":["2018R1A5A7025409"],"award-info":[{"award-number":["2018R1A5A7025409"]}],"id":[{"id":"10.13039\/501100001321","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,8,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Although the e-scooter sharing service market is growing as a representative last-mile mobility, the accident rate is increasing proportionally as the number of users increases. This study proposes a deep learning-based personal mobility driver monitoring system that detects inattentive driving by classifying vibration data transmitted to the e-scooter when the driver fails to concentrate on driving. First, the N-back task technique is used. The driver was stimulated by external visual and auditory factors to generate a cognitive load, and vibration data were collected through a six-axis sensor. Second, the generated vibration data were pre-processed using short-time Fourier transform and wavelet transform (WT) and then converted into an image (spectrogram). Third, four multimodal convolutional neural networks such as LeNet-5, VGG16, ResNet50, and DenseNet121 were constructed and their performance was compared to find the best architecture. Experimental results show that multimodal DenseNet121 with WT can accurately classify safe, slightly anxious, and very anxious driving conditions. The proposed model can be applied to real-time monitoring and warning systems for sharing service providers and used as a basis for insurance and legal action in the case of accidents.<\/jats:p>","DOI":"10.1093\/jcde\/qwac061","type":"journal-article","created":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T16:56:24Z","timestamp":1658854584000},"page":"1397-1409","source":"Crossref","is-referenced-by-count":11,"title":["Safety monitoring system of personal mobility driving using deep learning"],"prefix":"10.1093","volume":"9","author":[{"given":"Eunji","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Mechanical Systems Engineering, Sookmyung Women\u2019s University , Seoul 04310, Republic of Korea"},{"name":"Narnia Labs , Daejeon 34141, Republic of Korea"}]},{"given":"Hanyoung","family":"Ryu","sequence":"additional","affiliation":[{"name":"Department of Mechanical Systems Engineering, Sookmyung Women\u2019s University , Seoul 04310, Republic of Korea"}]},{"given":"Hyunji","family":"Oh","sequence":"additional","affiliation":[{"name":"Department of Mechanical Systems Engineering, Sookmyung Women\u2019s University , Seoul 04310, Republic of Korea"}]},{"given":"Namwoo","family":"Kang","sequence":"additional","affiliation":[{"name":"Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology , Daejeon 34141, Republic of Korea"},{"name":"Narnia Labs , Daejeon 34141, Republic of Korea"}]}],"member":"286","published-online":{"date-parts":[[2022,7,26]]},"reference":[{"key":"2022083110582142800_bib1","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.jsv.2016.10.043","article-title":"Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks","volume":"388","author":"Abdeljaber","year":"2017","journal-title":"Journal of Sound and Vibration"},{"key":"2022083110582142800_bib3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/FarEastCon50210.2020.9271210","article-title":"Comparison of the use of wavelet transform and short-time Fourier transform for the study of geomagnetically induced current in the autotransformer neutral","volume-title":"Proceedings of the 2020 International Multi-Conference on Industrial Engineering and Modern Technologies (FarEastCon)","author":"Aksenovich","year":"2020"},{"key":"2022083110582142800_bib15","first-page":"210","article-title":"Smartroadsense: Collaborative road surface condition monitoring","volume-title":"Proceedings of the {UBICOMM} 2014: The Eighth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies","author":"Alessandroni","year":"2014"},{"key":"2022083110582142800_bib2","first-page":"1874","article-title":"Gait-based frailty assessment using image representation of IMU signals and deep CNN","volume-title":"Proceedings of the 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)","author":"Arshad","year":"2021"},{"key":"2022083110582142800_bib4","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1007\/s00034-021-01789-4","article-title":"Accuracy enhancement of epileptic seizure detection: A deep learning approach with hardware realization of STFT","volume":"41","author":"Beeraka","year":"2022","journal-title":"Circuits, Systems, and Signal Processing"},{"key":"2022083110582142800_bib5","volume-title":"The bike and scootersharing telematics market","author":"Berg Insight","year":"2020","edition":"2"},{"key":"2022083110582142800_bib47","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1007\/s42947-020-6007-4","article-title":"Sensor-based pavement diagnostic using acoustic signature for moduli estimation","volume":"13","author":"Cafiso","year":"2020","journal-title":"International Journal of Pavement Research and Technology"},{"key":"2022083110582142800_bib6","doi-asserted-by":"crossref","first-page":"e00889","DOI":"10.1016\/j.cscm.2022.e00889","article-title":"Urban road pavements monitoring and assessment using bike and e-scooter as probe vehicles","volume":"16","author":"Cafiso","year":"2022","journal-title":"Case Studies in Construction Materials"},{"key":"2022083110582142800_bib9","doi-asserted-by":"crossref","first-page":"864","DOI":"10.1109\/LCOMM.2020.3037273","article-title":"Deep STFT-CNN for spectrum sensing in cognitive radio","volume":"25","author":"Chen","year":"2020","journal-title":"IEEE Communications Letters"},{"key":"2022083110582142800_bib8","first-page":"1","article-title":"A novel pavement transverse cracks detection model using WT-CNN and STFT-CNN for smartphone data analysis","author":"Chen","year":"2021","journal-title":"International Journal of Pavement Engineering"},{"key":"2022083110582142800_bib11","first-page":"464","article-title":"Identification of voltage quality problems under different types of sag\/swell faults with fast Fourier transform analysis","volume-title":"Proceedings of the 2016 2nd International Conference on Advances in Electrical, Electronics, Information, Communication and Bio-Informatics (AEEICB)","author":"Devadasu","year":"2016"},{"key":"2022083110582142800_bib12","doi-asserted-by":"crossref","first-page":"1107","DOI":"10.1093\/jcde\/qwac048","article-title":"Wave data prediction with optimized machine learning and deep learning techniques","volume":"9","author":"Domala","year":"2022","journal-title":"Journal of Computational Design and Engineering"},{"key":"2022083110582142800_bib13","article-title":"Personal mobility devices market by type (wheelchair, scooters, handbikes, walkers, stair-lifts, power add on products and others), and end user (hospitals & clinics, ambulatory surgical centers, urgent care center, home care setting and other end users) \u2013 Global Industry Perspective Comprehensive Analysis and Forecast, 2019 \u20132026","author":"Facts & Factors","year":"2020"},{"key":"2022083110582142800_bib14","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/j.jcde.2016.12.001","article-title":"Designing a generic human-machine framework for real-time supply chain planning","volume":"4","author":"Gaudreault","year":"2017","journal-title":"Journal of Computational Design and Engineering"},{"key":"2022083110582142800_bib17","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1016\/j.aap.2006.08.013","article-title":"An on-road assessment of cognitive distraction: Impacts on drivers\u2019 visual behavior and braking performance","volume":"39","author":"Harbluk","year":"2007","journal-title":"Accident Analysis & Prevention"},{"key":"2022083110582142800_bib19","first-page":"770","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"2022083110582142800_bib18","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1109\/THMS.2019.2917194","article-title":"High cognitive load assessment in drivers through wireless electroencephalography and the validation of a modified N-back task","volume":"49","author":"He","year":"2019","journal-title":"IEEE Transactions on Human-Machine Systems"},{"key":"2022083110582142800_bib20","first-page":"4700","article-title":"Densely connected convolutional networks","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Huang","year":"2017"},{"key":"2022083110582142800_bib25","article-title":"E-scooter injuries cost taxpayers nearly $15 million in two years","author":"Hutt","year":"2021"},{"key":"2022083110582142800_bib22","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1080\/09658211003702171","article-title":"The concurrent validity of the N-back task as a working memory measure","volume":"18","author":"Jaeggi","year":"2010","journal-title":"Memory"},{"key":"2022083110582142800_bib23","doi-asserted-by":"crossref","first-page":"732","DOI":"10.3390\/diagnostics11040732","article-title":"Efficiently classifying lung sounds through depthwise separable CNN models with fused STFT and MFCC features","volume":"11","author":"Jung","year":"2021","journal-title":"Diagnostics"},{"key":"2022083110582142800_bib27","article-title":"Use the convenient electric scooter safely","author":"Korea Consumer Agency","year":"2019"},{"key":"2022083110582142800_bib28","article-title":"Micro-mobility transportation policy support project","author":"Korea Transport Institute","year":"2017"},{"key":"2022083110582142800_bib29","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"Krizhevsky","year":"2012","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2022083110582142800_bib30","article-title":"Electric scooter accidents have quadrupled in three years. Insurance in the future","author":"Kwon","year":"2020"},{"key":"2022083110582142800_bib31","article-title":"LeNet-5, convolutional neural networks","author":"LeCun","year":"2015"},{"key":"2022083110582142800_bib32","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1049\/el.2017.3538","article-title":"Arousal-valence recognition using CNN with STFT feature-combined image","volume":"54","author":"Lee","year":"2018","journal-title":"Electronics Letters"},{"key":"2022083110582142800_bib39","article-title":"E-scooters embrace AI to cut down on pedestrian collisions","author":"Lewis","year":"2021"},{"key":"2022083110582142800_bib34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/PHM.2017.8079099","article-title":"Wavelet transform based convolutional neural network for gearbox fault classification","volume-title":"Proceedings of the 2017 Prognostics and System Health Management Conference (PHM-Harbin)","author":"Liao","year":"2017"},{"key":"2022083110582142800_bib35","article-title":"Electric scooter accidents are spiking nationwide","author":"Mack","year":"2020"},{"key":"2022083110582142800_bib37","article-title":"CWT","author":"MathWorks","year":"2016"},{"key":"2022083110582142800_bib36","doi-asserted-by":"crossref","first-page":"1499","DOI":"10.1093\/jcde\/qwab054","article-title":"Recent use of deep learning techniques in clinical applications based on gait: A survey","volume":"8","author":"Matsushita","year":"2021","journal-title":"Journal of Computational Design and Engineering"},{"key":"2022083110582142800_bib38","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1093\/arclin\/acp063","article-title":"Is the N-back task a valid neuropsychological measure for assessing working memory","volume":"24","author":"Miller","year":"2009","journal-title":"Archives of Clinical Neuropsychology"},{"key":"2022083110582142800_bib10","article-title":"How to safely drive an electric scooter? Technology development companies","author":"Min-young","year":"2020"},{"key":"2022083110582142800_bib40","first-page":"689","article-title":"Multimodal deep learning","volume-title":"Proceedings of the 28th International Conference on Machine Learning (ICML 2011)","author":"Ngiam","year":"2011"},{"key":"2022083110582142800_bib41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.trf.2020.09.013","article-title":"On-to-off-path gaze shift cancellations lead to gaze concentration in cognitively loaded car drivers: A simulator study exploring gaze patterns in relation to a cognitive task and the traffic environment","volume":"75","author":"Nilsson","year":"2020","journal-title":"Transportation Research Part F: Traffic Psychology and Behaviour"},{"key":"2022083110582142800_bib42","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1207\/S15326985EP3801_8","article-title":"Cognitive load measurement as a means to advance cognitive load theory","volume":"38","author":"Paas","year":"2003","journal-title":"Educational Psychologist"},{"key":"2022083110582142800_bib43","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2009","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"2022083110582142800_bib45","doi-asserted-by":"crossref","DOI":"10.1037\/e563342012-001","article-title":"Developing a test to measure distraction potential of in-vehicle information system tasks in production vehicles","author":"Ranney","year":"2011"},{"key":"2022083110582142800_bib46","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1093\/jcde\/qwaa022","article-title":"A critical review of symbiosis approaches in the context of Industry 4.0","volume":"7","author":"Scaf\u00e0","year":"2020","journal-title":"Journal of Computational Design and Engineering"},{"key":"2022083110582142800_bib49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/AVSS.2018.8639425","article-title":"Drone detection using convolutional neural networks with acoustic stft features","volume-title":"Proceedings of the 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)","author":"Seo","year":"2018"},{"key":"2022083110582142800_bib50","article-title":"Very deep convolutional networks for large-scale image recognition","author":"Simonyan","year":"2014"},{"key":"2022083110582142800_bib51","first-page":"2141","article-title":"Improved multimodal deep learning with variation of information","volume":"27","author":"Sohn","year":"2014","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2022083110582142800_bib52","doi-asserted-by":"crossref","first-page":"1371","DOI":"10.1177\/0018720819836575","article-title":"Visual and cognitive demands of carplay, android auto, and five native infotainment systems","volume":"61","author":"Strayer","year":"2019","journal-title":"Human Factors"},{"key":"2022083110582142800_bib53","first-page":"270","article-title":"A survey on deep transfer learning","volume-title":"Proceedings of the International Conference on Artificial Neural Networks","author":"Tan","year":"2018"},{"key":"2022083110582142800_bib55","doi-asserted-by":"crossref","first-page":"167","DOI":"10.3389\/fnhum.2017.00167","article-title":"Assessing the driver\u2019s current level of working memory load with high density functional near-infrared spectroscopy: A realistic driving simulator study","volume":"11","author":"Unni","year":"2017","journal-title":"Frontiers in Human Neuroscience"},{"key":"2022083110582142800_bib56","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1016\/j.trf.2020.11.014","article-title":"A meta-analysis of the N-back task while driving and its effects on cognitive workload","volume":"76","author":"Von\u00a0Janczewski","year":"2021","journal-title":"Transportation Research Part F: Traffic Psychology and Behaviour"},{"key":"2022083110582142800_bib57","doi-asserted-by":"crossref","first-page":"1357","DOI":"10.1007\/s10033-017-0190-5","article-title":"Motor fault diagnosis based on short-time Fourier transform and convolutional neural network","volume":"30","author":"Wang","year":"2017","journal-title":"Chinese Journal of Mechanical Engineering"},{"key":"2022083110582142800_bib48","first-page":"135","article-title":"Validation of smartphone-based pavement roughness measures","volume":"1","author":"Yeganeh","year":"2017","journal-title":"AUT Journal of Civil Engineering"},{"key":"2022083110582142800_bib59","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1016\/j.jcde.2018.12.002","article-title":"Influence of a hybrid digital toolset on the creative behaviors of designers in early-stage design","volume":"6","author":"Zboinska","year":"2019","journal-title":"Journal of Computational Design and Engineering"},{"key":"2022083110582142800_bib16","doi-asserted-by":"crossref","first-page":"133","DOI":"10.3141\/2523-15","article-title":"Identifying deficient pavement sections using an improved acceleration-based metric","volume":"2523","author":"Zeng","year":"2015","journal-title":"Transportation Research Record: Journal of the Transportation Research Board"}],"container-title":["Journal of Computational Design and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jcde\/advance-article-pdf\/doi\/10.1093\/jcde\/qwac061\/45084247\/qwac061.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/9\/4\/1397\/45630640\/qwac061.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/9\/4\/1397\/45630640\/qwac061.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,31]],"date-time":"2022-08-31T10:59:08Z","timestamp":1661943548000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jcde\/article\/9\/4\/1397\/6650219"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,26]]},"references-count":51,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,8,6]]}},"URL":"https:\/\/doi.org\/10.1093\/jcde\/qwac061","relation":{},"ISSN":["2288-5048"],"issn-type":[{"value":"2288-5048","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,8]]},"published":{"date-parts":[[2022,7,26]]}}}