{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:15:27Z","timestamp":1760231727172,"version":"build-2065373602"},"reference-count":39,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2022,10,2]],"date-time":"2022-10-02T00:00:00Z","timestamp":1664668800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Research Foundation of Korea (NRF)","award":["2020R1F1A1072772"],"award-info":[{"award-number":["2020R1F1A1072772"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With many conveniences afforded by advances in smartphone technology, developing advanced data analysis methods for health-related information from smartphone users has become a fast-growing research topic in the healthcare field. Along these lines, this paper addresses smartphone sensor-based characterization of human motions with neural stochastic differential equations (NSDEs) and a Transformer model. NSDEs and modeling via Transformer networks are two of the most prominent deep learning-based modeling approaches, with significant performance yields in many applications. For the problem of modeling dynamical features, stochastic differential equations and deep neural networks are frequently used paradigms in science and engineering, respectively. Combining these two paradigms in one unified framework has drawn significant interest in the deep learning community, and NSDEs are among the leading technologies for combining these efforts. The use of attention has also become a widely adopted strategy in many deep learning applications, and a Transformer is a deep learning model that uses the mechanism of self-attention. This concept of a self-attention based Transformer was originally introduced for tasks of natural language processing (NLP), and due to its excellent performance and versatility, the scope of its applications is rapidly expanding. By utilizing the techniques of neural stochastic differential equations and a Transformer model along with data obtained from smartphone sensors, we present a deep learning method capable of efficiently characterizing human motions. For characterizing human motions, we encode the high-dimensional sequential data from smartphone sensors into latent variables in a low-dimensional latent space. The concept of the latent variable is particularly useful because it can not only carry condensed information concerning motion data, but also learn their low-dimensional representations. More precisely, we use neural stochastic differential equations for modeling transitions of human motion in a latent space, and rely on a Generative Pre-trained Transformer 2 (GPT2)-based Transformer model for approximating the intractable posterior of conditional latent variables. Our experiments show that the proposed method can yield promising results for the problem of characterizing human motion patterns and some related tasks including user identification.<\/jats:p>","DOI":"10.3390\/s22197480","type":"journal-article","created":{"date-parts":[[2022,10,10]],"date-time":"2022-10-10T03:07:28Z","timestamp":1665371248000},"page":"7480","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Smartphone Sensor-Based Human Motion Characterization with Neural Stochastic Differential Equations and Transformer Model"],"prefix":"10.3390","volume":"22","author":[{"given":"Juwon","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Control and Instrumentation Engineering, Korea University, 2511 Sejong-ro, Sejong-City 30019, Korea"}]},{"given":"Taehwan","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Control and Instrumentation Engineering, Korea University, 2511 Sejong-ro, Sejong-City 30019, Korea"}]},{"given":"Jeongho","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Control and Instrumentation Engineering, Korea University, 2511 Sejong-ro, Sejong-City 30019, Korea"}]},{"given":"Jooyoung","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Control and Instrumentation Engineering, Korea University, 2511 Sejong-ro, Sejong-City 30019, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1109\/TPAMI.2007.1167","article-title":"Gaussian process dynamical models for human motion","volume":"30","author":"Wang","year":"2007","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kim, T., Park, J., Heo, S., Sung, K., and Park, J. (2017). Characterizing dynamic walking patterns and detecting falls with wearable sensors using Gaussian process methods. Sensors, 17.","DOI":"10.3390\/s17051172"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kim, J., Lee, J., Jang, W., Lee, S., Kim, H., and Park, J. (2019). Two-stage latent dynamics modeling and filtering for characterizing individual walking and running patterns with smartphone sensors. Sensors, 19.","DOI":"10.3390\/s19122712"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"16455","DOI":"10.1038\/s41598-021-95947-y","article-title":"Human activity recognition using wearable sensors, discriminant analysis, and Long Short-Term Memory-based neural structured learning","volume":"11","author":"Uddin","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"31663","DOI":"10.1007\/s11042-020-09537-7","article-title":"EnsemConvNet: A deep learning approach for human activity recognition using smartphone sensors for healthcare applications","volume":"79","author":"Mukherjee","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1016\/j.eswa.2016.04.032","article-title":"Human activity recognition with smartphone sensors using deep learning neural networks","volume":"59","author":"Ronao","year":"2016","journal-title":"Expert Syst. Appl."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Jiang, W., and Yin, Z. (2015, January 26\u201330). Human activity recognition using wearable sensors by deep convolutional neural networks. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.","DOI":"10.1145\/2733373.2806333"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wang, N., Ambikairajah, E., Lovell, N.H., and Celler, B.G. (2007, January 23\u201326). Accelerometry based classification of walking patterns using time-frequency analysis. Proceedings of the 2007 29th Annual International Conference of the Engineering in Medicine and Biology Society (EMBS 2007), Lyon, France.","DOI":"10.1109\/IEMBS.2007.4353438"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"107671","DOI":"10.1016\/j.asoc.2021.107671","article-title":"Attention induced multi-head convolutional neural network for human activity recognition","volume":"110","author":"Khan","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Augustinov, G., Nisar, M.A., Li, F., Tabatabaei, A., Grzegorzek, M., Sohrabi, K., and Fudickar, S. (2022, July 12). Transformer-Based Recognition of Activities of Daily Living from Wearable Sensor Data. Available online: https:\/\/doi.org\/10.21203\/rs.3.rs-2015249\/v1.","DOI":"10.21203\/rs.3.rs-2015249\/v1"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Shi, S., Wang, Y., Dong, H., Gui, G., and Ohtsuki, T. (2022, January 2\u20135). Smartphone-Aided Human Activity Recognition Method Using Residual Multi-Layer Perceptron. Proceedings of the IEEE INFOCOM 2022-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), New York, NY, USA.","DOI":"10.1109\/INFOCOMWKSHPS54753.2022.9798274"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Zhang, Y., Zhang, Z., Bao, J., and Song, Y. (2018). Human activity recognition based on time series analysis using U-Net. arXiv.","DOI":"10.1109\/ACCESS.2019.2920969"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wu, J., Feng, Y., and Sun, P. (2018). Sensor fusion for recognition of activities of daily living. Sensors, 18.","DOI":"10.3390\/s18114029"},{"key":"ref_14","unstructured":"Kidger, P., Foster, J., Li, X., and Lyons, T.J. (2021, January 18\u201324). Neural sdes as infinite-dimensional gans. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_15","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_16","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"S\u00e4rkk\u00e4, S., and Solin, A. (2019). Applied Stochastic Differential Equations, Cambridge University Press.","DOI":"10.1017\/9781108186735"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Jolliffe, I. (2022, July 12). Principal Component Analysis. Encyclopedia of Statistics in Behavioral Science. Available online: https:\/\/doi.org\/10.1002\/0470013192.bsa501.","DOI":"10.1002\/0470013192.bsa501"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1007\/s11263-007-0075-7","article-title":"Incremental learning for robust visual tracking","volume":"77","author":"Ross","year":"2008","journal-title":"Int. J. Comput. Vis."},{"key":"ref_20","unstructured":"Haykin, S. (1994). Neural Networks: A Comprehensive Foundation, Prentice Hall PTR."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Krishnan, R.G., Shalit, U., and Sontag, D. (2017, January 4\u20139). Structured inference networks for nonlinear state space models. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.10779"},{"key":"ref_22","unstructured":"Zhao, T.Z., Nagabandi, A., Rakelly, K., Finn, C., and Levine, S. (2020). Meld: Meta-reinforcement learning from images via latent state models. arXiv."},{"key":"ref_23","unstructured":"Xie, Z., Liu, C., Zhang, Y., Lu, H., Wang, D., and Ding, Y. (2022, July 12). Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation. Available online: https:\/\/arxiv.org\/pdf\/2103.10693.pdf."},{"key":"ref_24","unstructured":"Oord, A.V.D., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv."},{"key":"ref_25","unstructured":"Guo, Z.D., Azar, M.G., Piot, B., Pires, B.A., and Munos, R. (2018). Neural predictive belief representations. arXiv."},{"key":"ref_26","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_27","unstructured":"Kingma, D.P., and Welling, M. (2014, May 01). Auto-Encoding Variational Bayes. The International Conference on Learning Representations (ICLR) 2014. Available online: https:\/\/arxiv.org\/pdf\/1312.6114v10.pdf."},{"key":"ref_28","unstructured":"(2022, July 12). Matlab Application. Available online: https:\/\/apps.apple.com\/us\/app\/matlab-mobile\/id370976661."},{"key":"ref_29","unstructured":"(2022, July 12). iPhone XS Max Specification. Available online: https:\/\/support.apple.com\/kb\/SP780?viewlocale=en_US&locale=ko_KR."},{"key":"ref_30","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2022, July 12). Automatic Differentiation in Pytorch., NIPS-W 2017. Available online: https:\/\/openreview.net\/forum?id=BJJsrmfCZ."},{"key":"ref_31","unstructured":"(2022, July 12). Definition of Walking and Running, Walk Jog Run Club. Available online: http:\/\/www.wjrclub.com\/terms-and-definitions.html."},{"key":"ref_32","unstructured":"(2022, July 12). Definition of Squats, Healthline Media. Available online: https:\/\/www.healthline.com\/health\/exercise-fitness\/what-muscles-do-squats-work#how-to."},{"key":"ref_33","unstructured":"(2022, July 12). Definition of Jumping Jacks, Healthline Media. Available online: https:\/\/www.healthline.com\/health\/fitness-exercise\/jumping-jacks#how-to."},{"key":"ref_34","unstructured":"Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization. arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"3783","DOI":"10.1007\/s11071-022-07365-y","article-title":"Estimating initial conditions for dynamical systems with incomplete information","volume":"108","author":"Kolic","year":"2022","journal-title":"Nonlinear Dyn."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Chen, N., Karl, M., and Van Der Smagt, P. (2016, January 15\u201317). Dynamic movement primitives in latent space of time-dependent variational autoencoders. Proceedings of the 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), Cancun, Mexico.","DOI":"10.1109\/HUMANOIDS.2016.7803340"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1080\/24709360.2017.1396742","article-title":"A tutorial on kernel density estimation and recent advances","volume":"1","author":"Chen","year":"2017","journal-title":"Biostat. Epidemiol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: Fundamental algorithms for scientific computing in Python","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nat. Methods"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1109\/MCSE.2007.55","article-title":"Matplotlib: A 2D graphics environment","volume":"9","author":"Hunter","year":"2007","journal-title":"Comput. Sci. Eng."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/19\/7480\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:45:38Z","timestamp":1760143538000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/19\/7480"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,2]]},"references-count":39,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["s22197480"],"URL":"https:\/\/doi.org\/10.3390\/s22197480","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,10,2]]}}}