{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T12:32:33Z","timestamp":1775737953710,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T00:00:00Z","timestamp":1658102400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Bayerisches Staatsministerium f\u00fcr Wirtschaft, Landesentwicklung und Energie","award":["IUK-1902-0005\/\/IUK606\/002"],"award-info":[{"award-number":["IUK-1902-0005\/\/IUK606\/002"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Efficient handwriting trajectory reconstruction (TR) requires specific writing surfaces for detecting movements of digital pens. Although several motion-based solutions have been developed to remove the necessity of writing surfaces, most of them are based on classical sensor fusion methods limited, by sensor error accumulation over time, to tracing only single strokes. In this work, we present an approach to map the movements of an IMU-enhanced digital pen to relative displacement data. Training data is collected by means of a tablet. We propose several pre-processing and data-preparation methods to synchronize data between the pen and the tablet, which are of different sampling rates, and train a convolutional neural network (CNN) to reconstruct multiple strokes without the need of writing segmentation or post-processing correction of the predicted trajectory. The proposed system learns the relative displacement of the pen tip over time from the recorded raw sensor data, achieving a normalized error rate of 0.176 relative to unit-scaled tablet ground truth (GT) trajectory. To test the effectiveness of the approach, we train a neural network for character recognition from the reconstructed trajectories, which achieved a character error rate of 19.51%. Finally, a joint model is implemented that makes use of both the IMU data and the generated trajectories, which outperforms the sensor-only-based recognition approach by 0.75%.<\/jats:p>","DOI":"10.3390\/s22145347","type":"journal-article","created":{"date-parts":[[2022,7,19]],"date-time":"2022-07-19T00:19:21Z","timestamp":1658189961000},"page":"5347","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Surface-Free Multi-Stroke Trajectory Reconstruction and Word Recognition Using an IMU-Enhanced Digital Pen"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9808-7698","authenticated-orcid":false,"given":"Mohamad","family":"Wehbi","sequence":"first","affiliation":[{"name":"Machine Learning and Data Analytics Lab., Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU), 91052 Erlangen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Luge","sequence":"additional","affiliation":[{"name":"Machine Learning and Data Analytics Lab., Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU), 91052 Erlangen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tim","family":"Hamann","sequence":"additional","affiliation":[{"name":"STABILO International GmbH, 90562 Heroldsberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3967-9578","authenticated-orcid":false,"given":"Jens","family":"Barth","sequence":"additional","affiliation":[{"name":"STABILO International GmbH, 90562 Heroldsberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter","family":"Kaempf","sequence":"additional","affiliation":[{"name":"STABILO International GmbH, 90562 Heroldsberg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5886-0597","authenticated-orcid":false,"given":"Dario","family":"Zanca","sequence":"additional","affiliation":[{"name":"Machine Learning and Data Analytics Lab., Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU), 91052 Erlangen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0417-0336","authenticated-orcid":false,"given":"Bjoern M.","family":"Eskofier","sequence":"additional","affiliation":[{"name":"Machine Learning and Data Analytics Lab., Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universit\u00e4t Erlangen-N\u00fcrnberg (FAU), 91052 Erlangen, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Singh, T.P., Gupta, S., Garg, M., Gupta, D., Alharbi, A., Alyami, H., Anand, D., Ortega-Mansilla, A., and Goyal, N. (2022). Visualization of Customized Convolutional Neural Network for Natural Language Recognition. Sensors, 22.","DOI":"10.3390\/s22082881"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Aladem, M., and Rawashdeh, S.A. (2018). Lightweight visual odometry for autonomous mobile robots. Sensors, 18.","DOI":"10.3390\/s18092837"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhang, S., Lu, S., He, R., and Bao, Z. (2021). Stereo Visual Odometry Pose Correction through Unsupervised Deep Learning. Sensors, 21.","DOI":"10.3390\/s21144735"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Klein, I. (2019). Smartphone location recognition: A deep learning-based approach. Sensors, 20.","DOI":"10.3390\/s20010214"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kim, M., Cho, J., Lee, S., and Jung, Y. (2019). IMU sensor-based hand gesture recognition for human-machine interfaces. Sensors, 19.","DOI":"10.3390\/s19183827"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1007\/s10846-017-0725-0","article-title":"Hand gesture recognition based omnidirectional wheelchair control using IMU and EMG sensors","volume":"91","author":"Kundu","year":"2018","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"107101","DOI":"10.1016\/j.asoc.2021.107101","article-title":"Att-Net: Enhanced emotion recognition system using lightweight self-attention module","volume":"102","author":"Kwon","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Luo, Y., Liu, J., and Shimamoto, S. (2021, January 9\u201312). Wearable air-writing recognition system employing dynamic time warping. Proceedings of the 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA.","DOI":"10.1109\/CCNC49032.2021.9369458"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Younas, J., Margarito, H., Bian, S., and Lukowicz, P. (2020, January 7\u20139). Finger Air Writing-Movement Reconstruction with Low-cost IMU Sensor. Proceedings of the MobiQuitous 2020-17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, Darmstadt, Germany.","DOI":"10.1145\/3448891.3448925"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Meng, X., Wang, H., and Liu, B. (2017). A robust vehicle localization approach based on gnss\/imu\/dmi\/lidar sensor fusion for autonomous vehicles. Sensors, 17.","DOI":"10.3390\/s17092140"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Gim, J., and Ahn, C. (2018). IMU-based virtual road profile sensor for vehicle localization. Sensors, 18.","DOI":"10.3390\/s18103344"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Skog, I., Nilsson, J.O., and H\u00e4ndel, P. (2014, January 6\u20137). Pedestrian tracking using an IMU array. Proceedings of the 2014 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT), Bangalore, India.","DOI":"10.1109\/CONECCT.2014.6740346"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Chen, L., and Hu, H. (2012, January 12\u201313). IMU\/GPS based pedestrian localization. Proceedings of the 2012 4th Computer Science and Electronic Engineering Conference (CEEC), Colchester, UK.","DOI":"10.1109\/CEEC.2012.6375373"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1109\/TIM.2011.2159317","article-title":"Accurate pedestrian indoor navigation by tightly coupling foot-mounted IMU and RFID measurements","volume":"61","author":"Ruiz","year":"2011","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"H\u00f6flinger, F., Zhang, R., and Reindl, L.M. (2012, January 23\u201327). Indoor-localization system using a micro-inertial measurement unit (imu). Proceedings of the 2012 European Frequency and Time Forum, Gothenburg, Sweden.","DOI":"10.1109\/EFTF.2012.6502421"},{"key":"ref_16","unstructured":"(2022, June 10). One by Wacom. Available online: https:\/\/www.wacom.com\/en-de\/products\/pen-tablets\/one-by-wacom."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1109\/THMS.2014.2365723","article-title":"Gyropen: Gyroscopes for pen-input with mobile phones","volume":"45","author":"Deselaers","year":"2014","journal-title":"IEEE Trans.-Hum.-Mach. Syst."},{"key":"ref_18","first-page":"261","article-title":"Handwriting trajectory reconstruction using low-cost imu","volume":"3","author":"Pan","year":"2018","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3494956","article-title":"Handwriting-Assistant: Reconstructing Continuous Strokes with Millimeter-level Accuracy via Attachable Inertial Sensors","volume":"5","author":"Bu","year":"2021","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Griswold-Steiner, I., Matovu, R., and Serwadda, A. (2017, January 1\u20134). Handwriting watcher: A mechanism for smartwatch-driven handwriting authentication. Proceedings of the 2017 IEEE International Joint Conference on Biometrics (IJCB), Denver, CO, USA.","DOI":"10.1109\/BTAS.2017.8272701"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1109\/TBIOM.2019.2912401","article-title":"Wearables-driven freeform handwriting authentication","volume":"1","author":"Matovu","year":"2019","journal-title":"IEEE Trans. Biom. Behav. Identity Sci."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3264929","article-title":"Handwritten signature verification using wrist-worn devices","volume":"2","author":"Levy","year":"2018","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_23","unstructured":"(2022, June 10). Samsung S-Pen. Available online: https:\/\/www.samsung.com\/de\/support\/mobile-devices\/alles-rund-um-den-s-pen\/."},{"key":"ref_24","unstructured":"(2022, June 10). Apple Pencil. Available online: https:\/\/www.apple.com\/apple-pencil\/."},{"key":"ref_25","unstructured":"(2022, June 10). Moleskine Smart Writing Set. Available online: https:\/\/www.moleskine.com\/de-de\/online-shop\/moleskine-smart\/smart-writing-system\/smart-writing-system\/smart-writing-set-schwarz-8056598851571.html."},{"key":"ref_26","unstructured":"(2022, June 10). XP-Pen. Available online: https:\/\/www.xp-pen.de\/product\/519.html."},{"key":"ref_27","unstructured":"(2022, June 10). Rocketbook. Available online: https:\/\/myrocketbook.eu\/."},{"key":"ref_28","unstructured":"Bronkhorst, M. (2021). A Pen is All You Need: Online Handwriting Recognition Using Transformers. [Bachelor\u2019s Thesis, University of Twente]."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Koellner, C., Kurz, M., and Sonnleitner, E. (2019, January 21\u201323). What did you mean? An evaluation of online character recognition approaches. Proceedings of the 2019 International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), Barcelona, Spain.","DOI":"10.1109\/WiMOB.2019.8923384"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wehbi, M., Hamann, T., Barth, J., and Eskofier, B. (2020, January 8\u201310). Digitizing handwriting with a sensor pen: A writer-independent recognizer. Proceedings of the 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), Dortmund, Germany.","DOI":"10.1109\/ICFHR2020.2020.00061"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3411842","article-title":"The onhw dataset: Online handwriting recognition from imu-enhanced ballpoint pens with machine learning","volume":"4","author":"Ott","year":"2020","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wehbi, M., Hamann, T., Barth, J., Kaempf, P., Zanca, D., and Eskofier, B. (2021, January 5\u201310). Towards an IMU-based Pen Online Handwriting Recognizer. Proceedings of the International Conference on Document Analysis and Recognition, Lausanne, Switzerland.","DOI":"10.1007\/978-3-030-86334-0_19"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Pan, T.Y., Kuo, C.H., and Hu, M.C. (2016, January 11\u201315). A noise reduction method for IMU and its application on handwriting trajectory reconstruction. Proceedings of the 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Seattle, WA, USA.","DOI":"10.1109\/ICMEW.2016.7574685"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"3508","DOI":"10.1109\/TIE.2009.2038339","article-title":"An inertial-measurement-unit-based pen with a trajectory reconstruction algorithm and its applications","volume":"57","author":"Wang","year":"2009","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_35","unstructured":"(2022, June 10). Stabilo Digipen. Available online: https:\/\/stabilodigital.com\/."},{"key":"ref_36","unstructured":"(2022, June 10). Samsun Galaxy Tab S6. Available online: https:\/\/www.samsung.com\/de\/tablets\/galaxy-tab-s\/galaxy-tab-s6-10-5-inch-gray-128gb-wi-fi-sm-t860nzaadbt\/."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Graves, A., Fern\u00e1ndez, S., Gomez, F., and Schmidhuber, J. (2006, January 25\u201329). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.","DOI":"10.1145\/1143844.1143891"},{"key":"ref_38","unstructured":"Bhunia, A.K., Bhowmick, A., Bhunia, A.K., Konwer, A., Banerjee, P., Roy, P.P., and Pal, U. (2018, January 20\u201324). Handwriting trajectory recovery using end-to-end deep encoder-decoder network. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Rabhi, B., Elbaati, A., Hamdi, Y., and Alimi, A.M. (2019, January 20\u201325). Handwriting recognition based on temporal order restored by the end-to-end system. Proceedings of the 2019 International Conference on Document Analysis and Recognition (ICDAR), Sydney, Australia.","DOI":"10.1109\/ICDAR.2019.00199"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Nguyen, H.T., Nakamura, T., Nguyen, C.T., and Nakawaga, M. (2021, January 10\u201315). Online trajectory recovery from offline handwritten Japanese kanji characters of multiple strokes. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9413294"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/14\/5347\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:52:45Z","timestamp":1760140365000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/14\/5347"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,18]]},"references-count":40,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2022,7]]}},"alternative-id":["s22145347"],"URL":"https:\/\/doi.org\/10.3390\/s22145347","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,18]]}}}