{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T10:53:46Z","timestamp":1790765626112,"version":"4.1.0"},"reference-count":34,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2016,8,3]],"date-time":"2016-08-03T00:00:00Z","timestamp":1470182400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation Graduate Research Fellowship","award":["DGE-1148903"],"award-info":[{"award-number":["DGE-1148903"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2016,8,3]]},"abstract":"<jats:p>Most smartphones today have a rich set of sensors that could be used to infer input (e.g., accelerometer, gyroscope, microphone); however, the primary mode of interaction is still limited to the front-facing touchscreen and several physical buttons on the case. To investigate the potential opportunities for interactions supported by built-in sensors, we present the implementation and evaluation of BeyondTouch, a family of interactions to extend and enrich the input experience of a smartphone. Using only existing sensing capabilities on a commodity smartphone, we offer the user a wide variety of additional inputs on the case and the surface adjacent to the smartphone. Although most of these interactions are implemented with machine learning methods, compact and robust rule-based detection methods can also be applied for recognizing some interactions by analyzing physical characteristics of tapping events on the phone. This article is an extended version of Zhang et al. [2015], which solely covered gestures implemented by machine learning methods. We extended our previous work by adding gestures implemented with rule-based methods, which works well with different users across devices without collecting any training data. We outline the implementation of both machine learning and rule-based methods for these interaction techniques and demonstrate empirical evidence of their effectiveness and usability. We also discuss the practicality of BeyondTouch for a variety of application scenarios and compare the two different implementation methods.<\/jats:p>","DOI":"10.1145\/2954003","type":"journal-article","created":{"date-parts":[[2016,8,4]],"date-time":"2016-08-04T13:26:34Z","timestamp":1470317194000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Beyond the Touchscreen"],"prefix":"10.1145","volume":"6","author":[{"given":"Cheng","family":"Zhang","sequence":"first","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anhong","family":"Guo","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dingtian","family":"Zhang","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Li","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caleb","family":"Southern","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rosa I.","family":"Arriaga","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregory D.","family":"Abowd","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2016,8,3]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518995"},{"key":"e_1_2_2_2_1","volume-title":"Proceedings of the 6th USENIX Conference on Hot Topics in Security (HotSec'11)","author":"Cai Liang","year":"2011","unstructured":"Liang Cai and Hao Chen . 2011 . TouchLogger: Inferring keystrokes on touch screen from smartphone motion . In Proceedings of the 6th USENIX Conference on Hot Topics in Security (HotSec'11) . 9. Liang Cai and Hao Chen. 2011. TouchLogger: Inferring keystrokes on touch screen from smartphone motion. In Proceedings of the 6th USENIX Conference on Hot Topics in Security (HotSec'11). 9."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2632048.2632090"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557120"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380184"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1449715.1449747"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047279"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2037373.2037393"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/354401.354417"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979059"},{"key":"e_1_2_2_11_1","volume-title":"Proceedings of the 9th IFIP TC13 International Conference on Human-Computer Interaction (Interact'03)","author":"Hiraoka Shigeo","year":"2003","unstructured":"Shigeo Hiraoka , Isshin Miyamoto , and Kiyoshi Tomimatsu . 2003 . Behind Touch: A text input method for mobile phones by the back and tactile sense interface . In Proceedings of the 9th IFIP TC13 International Conference on Human-Computer Interaction (Interact'03) . 131--138. Shigeo Hiraoka, Isshin Miyamoto, and Kiyoshi Tomimatsu. 2003. Behind Touch: A text input method for mobile phones by the back and tactile sense interface. In Proceedings of the 9th IFIP TC13 International Conference on Human-Computer Interaction (Interact'03). 131--138."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1709886.1709906"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357273"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647363"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2307636.2307666"},{"key":"e_1_2_2_16_1","volume-title":"The Design of Everyday Things: Revised and Expanded Edition","author":"Norman Donald A.","unstructured":"Donald A. Norman . 2013. The Design of Everyday Things: Revised and Expanded Edition . Basic Books , New York, NY . Donald A. Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books, New York, NY."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2162081.2162095"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/506443.506570"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979345"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1226969.1227023"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047235"},{"key":"e_1_2_2_22_1","volume-title":"Human-Computer Interaction\u2014INTERACT","author":"Schmieder Paul","year":"2013","unstructured":"Paul Schmieder , John Hosking , Andrew Luxton-Reilly , and Beryl Plimmer . 2013. Thumbs up: 3D gesture input on mobile phones using the front facing camera . In Human-Computer Interaction\u2014INTERACT 2013 . Lecture Notes in Computer Science, Vol. 8118 . Springer , 318--336. Paul Schmieder, John Hosking, Andrew Luxton-Reilly, and Beryl Plimmer. 2013. Thumbs up: 3D gesture input on mobile phones using the front facing camera. In Human-Computer Interaction\u2014INTERACT 2013. Lecture Notes in Computer Science, Vol. 8118. Springer, 318--336."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985726"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1851600.1851630"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1152215.1152243"},{"key":"e_1_2_2_26_1","volume-title":"Retrieved","author":"Weka","year":"2016","unstructured":"Weka. n.d. Weka 3 : Data Mining Software in Java . Retrieved June 17, 2016 , from http:\/\/www.cs.waikato.ac.nz\/ml\/weka\/. Weka. n.d. Weka 3: Data Mining Software in Java. Retrieved June 17, 2016, from http:\/\/www.cs.waikato.ac.nz\/ml\/weka\/."},{"key":"e_1_2_2_27_1","volume-title":"n.d. JTransforms. Retrieved","author":"Wendykier Piotr","year":"2016","unstructured":"Piotr Wendykier . n.d. JTransforms. Retrieved June 17, 2016 , from https:\/\/sites.google.com\/site\/piotrwendykier\/software\/jtransforms. Piotr Wendykier. n.d. JTransforms. Retrieved June 17, 2016, from https:\/\/sites.google.com\/site\/piotrwendykier\/software\/jtransforms."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2481296"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1294211.1294259"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/964696.964703"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185448.2185465"},{"key":"e_1_2_2_32_1","volume-title":"Proceedings of the Adjunct Publication of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp'13 Adjunct Publication).","author":"Zhang Cheng","year":"2013","unstructured":"Cheng Zhang . 2013 . BeyondTouch: A framework for extending input on commodity smartphones . In Proceedings of the Adjunct Publication of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp'13 Adjunct Publication). Cheng Zhang. 2013. BeyondTouch: A framework for extending input on commodity smartphones. In Proceedings of the Adjunct Publication of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp'13 Adjunct Publication)."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508468.2514735"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701374"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2954003","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2954003","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:56:22Z","timestamp":1750222582000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2954003"}},"subtitle":["An Exploration of Extending Interactions on Commodity Smartphones"],"short-title":[],"issued":{"date-parts":[[2016,8,3]]},"references-count":34,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2016,8,3]]}},"alternative-id":["10.1145\/2954003"],"URL":"https:\/\/doi.org\/10.1145\/2954003","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,8,3]]},"assertion":[{"value":"2015-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-08-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}