{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,20]],"date-time":"2025-10-20T10:26:26Z","timestamp":1760955986621,"version":"3.37.3"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2020,1,10]],"date-time":"2020-01-10T00:00:00Z","timestamp":1578614400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Nature Science Foundation of China","doi-asserted-by":"crossref","award":["61772338"],"award-info":[{"award-number":["61772338"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,12,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Human\u2013computer interaction through touch screens plays an increasingly important role in our daily lives. Besides smartphones and tablets, laptops are the most prevalent mobile devices for both work and leisure. To satisfy the requirements of some applications, it is desirable to re-equip a typical laptop with both handwriting and drawing capability. In this paper, we design a virtual writing tablet system, VPad, for traditional laptops without touch screens. VPad leverages two speakers and one microphone, which are available in most commodity laptops, to accurately track hand movements and recognize writing characters in the air without additional hardware. Specifically, VPad emits inaudible acoustic signals from two speakers in a laptop and then analyzes energy features and Doppler shifts of acoustic signals received by the microphone to track the trajectory of hand movements. Furthermore, we propose a state machine-based trajectory optimization method to correct the unexpected trajectory and employ a stroke direction sequence model based on probability estimation to recognize characters users write in the air. Experimental results show that VPad achieves the average error of 1.55 cm for trajectory tracking and the accuracy over 90% of character recognition merely through built-in audio devices on a laptop.<\/jats:p>","DOI":"10.1093\/comjnl\/bxz153","type":"journal-article","created":{"date-parts":[[2019,11,20]],"date-time":"2019-11-20T12:11:58Z","timestamp":1574251918000},"page":"1814-1831","source":"Crossref","is-referenced-by-count":4,"title":["Enable Traditional Laptops with Virtual Writing Capability Leveraging Acoustic Signals"],"prefix":"10.1093","volume":"64","author":[{"given":"Li","family":"Lu","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China"}]},{"given":"Jian","family":"Liu","sequence":"additional","affiliation":[{"name":"WINLAB and Department Electrical and Computer Engineering, Rutgers University, New Brunswick, NJ, USA"}]},{"given":"Jiadi","family":"Yu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China"}]},{"given":"Yingying","family":"Chen","sequence":"additional","affiliation":[{"name":"WINLAB and Department Electrical and Computer Engineering, Rutgers University, New Brunswick, NJ, USA"}]},{"given":"Yanmin","family":"Zhu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China"}]},{"given":"Linghe","family":"Kong","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China"}]},{"given":"Minglu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China"}]}],"member":"286","published-online":{"date-parts":[[2020,1,10]]},"reference":[{"year":"2018","author":"Research","key":"2021121513290244300_ref1"},{"year":"2018","author":"Drawboard","key":"2021121513290244300_ref2"},{"year":"2018","key":"2021121513290244300_ref3"},{"year":"2011","author":"Lowell","key":"2021121513290244300_ref4"},{"key":"2021121513290244300_ref5","doi-asserted-by":"crossref","DOI":"10.1145\/2659766.2661218","article-title":"Leaplook: a free-hand gestural travel technique using the leap motion finger tracker","volume-title":"Proc. of ACM SUI","author":"Codd-Downey","year":"2014"},{"key":"2021121513290244300_ref6","doi-asserted-by":"crossref","first-page":"1318","DOI":"10.1109\/TCYB.2013.2265378","article-title":"Enhanced computer vision with microsoft kinect sensor: a review","volume":"43","author":"Han","year":"2013","journal-title":"IEEE T. Cybernetics"},{"key":"2021121513290244300_ref7","doi-asserted-by":"crossref","DOI":"10.1145\/2619239.2626330","article-title":"Rf-idraw: virtual touch screen in the air using rf signals","volume-title":"Proc. of ACM SIGCOMM","author":"Wang","year":"2014"},{"key":"2021121513290244300_ref8","first-page":"77","article-title":"Widraw: enabling hands-free drawing in the air on commodity wifi devices","volume-title":"Proc. of ACM MobiCom","author":"Sun","year":"2015"},{"key":"2021121513290244300_ref9","first-page":"1911","volume-title":"Soundwave: using the doppler effect to sense gestures. Proc. of ACM CHI, Austin, Texas, May","author":"Gupta","year":"2012"},{"key":"2021121513290244300_ref10","first-page":"15","article-title":"Turning a mobile device into a mouse in the air","volume-title":"Proc. of ACM MobiSys","author":"Yun","year":"2015"},{"key":"2021121513290244300_ref11","first-page":"69","article-title":"Cat: high-precision acoustic motion tracking","volume-title":"Proc. of ACM MobiCom","author":"Mao","year":"2016"},{"key":"2021121513290244300_ref12","first-page":"82","article-title":"Device-free gesture tracking using acoustic signals","volume-title":"Proc. of ACM MobiCom","author":"Wang","year":"2016"},{"key":"2021121513290244300_ref13","first-page":"1515","article-title":"Fingerio: using active sonar for fine-grained finger tracking","volume-title":"Proc. of ACM CHI","author":"Nandakumar","year":"2016"},{"key":"2021121513290244300_ref14","first-page":"15","article-title":"Strata: fine-grained acoustic-based device-free tracking","volume-title":"Proc. of ACM MobiSys","author":"Yun","year":"2017"},{"key":"2021121513290244300_ref15","first-page":"155","article-title":"Low cost crowd counting using audio tones","volume-title":"Proc. of ACM Sensys","author":"Kannan","year":"2012"},{"key":"2021121513290244300_ref16","doi-asserted-by":"crossref","first-page":"11","DOI":"10.4304\/jcp.3.5.11-17","article-title":"Fine classification and recognition of hand written devnagari characters with regular expressions and minimum edit distance method","volume":"3","author":"Deshpande","year":"2008","journal-title":"J. Comput."},{"key":"2021121513290244300_ref17","first-page":"1","article-title":"Discrete-Time Signal Processing","volume":"23","author":"Oppenheim","year":"1999","journal-title":"Prentice Hall Signal Processing"},{"key":"2021121513290244300_ref18","first-page":"775","article-title":"Keylistener: inferring keystrokes on qwerty keyboard of touch screen through acoustic signals","volume-title":"Proc. of IEEE INFOCOM","author":"Lu","year":"2019"},{"key":"2021121513290244300_ref19","first-page":"1","volume-title":"An indirect eavesdropping attack of keystrokes on touch screen through acoustic sensing","author":"Yu","year":"2019"},{"year":"2019","key":"2021121513290244300_ref20"},{"issue":"97","key":"2021121513290244300_ref21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3351255","article-title":"I3: sensing scrolling human-computer interactions for intelligent interest inference on smartphones","volume":"3","author":"Lu","year":"2019","journal-title":"Proc. of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"2021121513290244300_ref22","first-page":"1","article-title":"L3: sensing driving conditions for vehicle lane-level localization on highways","volume-title":"Proc. of IEEE INFOCOM","author":"Wu","year":"2016"},{"key":"2021121513290244300_ref23","doi-asserted-by":"crossref","first-page":"1698","DOI":"10.1109\/TMC.2014.2352862","article-title":"Sensing human-screen interaction for energy-efficient frame rate adaptation on smartphones","volume":"14","author":"Yu","year":"2015","journal-title":"IEEE Trans. Mob. Comput."},{"year":"2019","author":"TechRadar","key":"2021121513290244300_ref24"},{"article-title":"Nintendo wii","year":"2018","author":"Nintendo","key":"2021121513290244300_ref25"},{"key":"2021121513290244300_ref26","first-page":"71","article-title":"Arraytrack: a fine-grained indoor location system","volume-title":"Proc. of USENIX NSDI","author":"Xiong","year":"2013"},{"key":"2021121513290244300_ref27","first-page":"27","article-title":"Whole-home gesture recognition using wireless signals","volume-title":"Proc. of ACM MobiCom","author":"Pu","year":"2013"},{"key":"2021121513290244300_ref28","first-page":"15","article-title":"Using mobile phones to write in air","volume-title":"Proc. of ACM MobiSys","author":"Agrawal","year":"2011"},{"key":"2021121513290244300_ref29","first-page":"85","article-title":"I am a smartwatch and i can track my user\u2019s arm","volume-title":"Proc. of ACM MobiSys","author":"Shen","year":"2016"},{"key":"2021121513290244300_ref30","first-page":"155","article-title":"Mole: motion leaks through smartwatch sensors","volume-title":"Proc. of ACM MobiCom","author":"Wang","year":"2015"},{"key":"2021121513290244300_ref31","first-page":"142","article-title":"Snooping keystrokes with mm-level audio ranging on a single phone","volume-title":"Proc. of ACM MobiCom","author":"Liu","year":"2015"},{"key":"2021121513290244300_ref32","first-page":"14","article-title":"Ubiquitous keyboard for small mobile devices: harnessing multipath fading for fine-grained keystroke localization","volume-title":"Proc. of ACM MobiSys","author":"Wang","year":"2014"},{"key":"2021121513290244300_ref33","first-page":"45","article-title":"Contactless sleep apnea detection on smartphones","volume-title":"Proc. of ACM MobiSys","author":"Nandakumar","year":"2015"},{"key":"2021121513290244300_ref34","first-page":"1466","article-title":"Lippass: lip reading-based user authentication on smartphones leveraging acoustic signals","volume-title":"Proc. of IEEE INFOCOM","author":"Lu","year":"2018"},{"key":"2021121513290244300_ref35","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1109\/TNET.2019.2891733","article-title":"Lip reading-based user authentication through acoustic sensing on smartphones","volume":"27","author":"Lu","year":"2019","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"2021121513290244300_ref36","first-page":"370","article-title":"Shake and walk: acoustic direction finding and fine-grained indoor localization using smartphones","volume-title":"Proc. of IEEE INFOCOM","author":"Huang","year":"2014"},{"key":"2021121513290244300_ref37","first-page":"565","article-title":"Airlink: sharing files between multiple devices using in-air gestures","volume-title":"Proc. of ACM UbiComp","author":"Chen","year":"2014"},{"key":"2021121513290244300_ref38","first-page":"1387","article-title":"Surfacelink: using inertial and acoustic sensing to enable multi-device interaction on a surface","volume-title":"Proc. of ACM CHI","author":"Goel","year":"2014"},{"key":"2021121513290244300_ref39","doi-asserted-by":"crossref","first-page":"1031","DOI":"10.1016\/0031-3203(90)90111-W","article-title":"On-line recognition of handprinted characters: survey and beta tests","volume":"23","author":"Nouboud","year":"1990","journal-title":"Pattern Recognit."},{"key":"2021121513290244300_ref40","first-page":"1123","article-title":"Online handwriting recognition-a survey","volume-title":"Proc. of IEEE ICPR","author":"Tappert","year":"1988"},{"key":"2021121513290244300_ref41","first-page":"49","article-title":"Online handwriting recognition with support vector machines\u2014a kernel approach","volume-title":"Proc. of IWFHR","author":"Bahlmann","year":"2002"},{"key":"2021121513290244300_ref42","first-page":"545","article-title":"Offline handwriting recognition with multidimensional recurrent neural networks","volume":"21","author":"Graves","year":"2009","journal-title":"Adv. Neural Inf. Process. Syst."}],"container-title":["The Computer Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/comjnl\/article-pdf\/64\/12\/1814\/41757997\/bxz153.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/comjnl\/article-pdf\/64\/12\/1814\/41757997\/bxz153.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,15]],"date-time":"2021-12-15T13:30:30Z","timestamp":1639575030000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/comjnl\/article\/64\/12\/1814\/5699819"}},"subtitle":[],"editor":[{"given":"Suchi","family":"Bhandarkar","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2020,1,10]]},"references-count":42,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,1,10]]},"published-print":{"date-parts":[[2021,12,13]]}},"URL":"https:\/\/doi.org\/10.1093\/comjnl\/bxz153","relation":{},"ISSN":["0010-4620","1460-2067"],"issn-type":[{"type":"print","value":"0010-4620"},{"type":"electronic","value":"1460-2067"}],"subject":[],"published-other":{"date-parts":[[2021,12]]},"published":{"date-parts":[[2020,1,10]]}}}