{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T19:39:12Z","timestamp":1785008352283,"version":"3.55.0"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,6,21]],"date-time":"2019-06-21T00:00:00Z","timestamp":1561075200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2019,6,21]]},"abstract":"<jats:p>The ubiquity of modern smartphones, because they are equipped with a wide range of sensors, poses a potential security risk---malicious actors could utilize these sensors to detect private information such as the keystrokes a user enters on a nearby keyboard. Existing studies have examined the ability of phones to predict typing on a nearby keyboard but are limited by the realism of collected typing data, the expressiveness of employed prediction models, and are typically conducted in a relatively noise-free environment. We investigate the capability of mobile phone sensor arrays (using audio and motion sensor data) for classifying keystrokes that occur on a keyboard in proximity to phones around a table, as would be common in a meeting. We develop a system of mixed convolutional and recurrent neural networks and deploy the system in a human subjects experiment with 20 users typing naturally while talking. Using leave-one-user-out cross validation, we find that mobile phone arrays have the ability to detect 41.8% of keystrokes and 27% of typed words correctly in such a noisy environment---even without user specific training. To investigate the potential threat of this attack, we further developed the machine learning models into a realtime system capable of discerning keystrokes from an array of mobile phones and evaluated the system's ability with a single user typing in varying conditions. We conclude that, in order to launch a successful attack, the attacker would need advanced knowledge of the table from which a user types, and the style of keyboard on which a user types. These constraints greatly limit the feasibility of such an attack to highly capable attackers and we therefore conclude threat level of this attack to be low, but non-zero.<\/jats:p>","DOI":"10.1145\/3328916","type":"journal-article","created":{"date-parts":[[2019,6,24]],"date-time":"2019-06-24T13:45:01Z","timestamp":1561383901000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Keyboard Snooping from Mobile Phone Arrays with Mixed Convolutional and Recurrent Neural Networks"],"prefix":"10.1145","volume":"3","author":[{"given":"Tyler","family":"Giallanza","sequence":"first","affiliation":[{"name":"Darwin Deason Institute for Cybersecurity, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Travis","family":"Siems","sequence":"additional","affiliation":[{"name":"Darwin Deason Institute for Cybersecurity, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elena","family":"Smith","sequence":"additional","affiliation":[{"name":"Darwin Deason Institute for Cybersecurity, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erik","family":"Gabrielsen","sequence":"additional","affiliation":[{"name":"Darwin Deason Institute for Cybersecurity, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ian","family":"Johnson","sequence":"additional","affiliation":[{"name":"Darwin Deason Institute for Cybersecurity, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mitchell A.","family":"Thornton","sequence":"additional","affiliation":[{"name":"Darwin Deason Institute for Cybersecurity, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric C.","family":"Larson","sequence":"additional","affiliation":[{"name":"Darwin Deason Institute for Cybersecurity, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,6,21]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Apple developer documentation: mach-absolute-time. https:\/\/developer.apple.com\/documentation\/kernel\/1462446-mach_absolute_time. Accessed: 2018-07-01.  Apple developer documentation: mach-absolute-time. https:\/\/developer.apple.com\/documentation\/kernel\/1462446-mach_absolute_time. Accessed: 2018-07-01."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2014.2339736"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2789168.2790109"},{"key":"e_1_2_1_4_1","first-page":"9","volume-title":"Proceedings of the 6th USENIX conference on Hot topics in security","author":"Cai Liang","year":"2011"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30921-2_16"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390177"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2013.6639344"},{"key":"e_1_2_1_8_1","volume-title":"Walter de Gruyter","author":"Fant Gunnar","year":"1970"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1162\/089976600300015015"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.169"},{"key":"e_1_2_1_11_1","first-page":"315","volume-title":"Proceedings of the fourteenth international conference on artificial intelligence and statistics","author":"Glorot Xavier","year":"2011"},{"key":"e_1_2_1_12_1","unstructured":"Ian Goodfellow Yoshua Bengio Aaron Courville and Yoshua Bengio. Deep learning volume 1. MIT press Cambridge 2016.  Ian Goodfellow Yoshua Bengio Aaron Courville and Yoshua Bengio. Deep learning volume 1. MIT press Cambridge 2016."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_14_1","first-page":"2042","volume-title":"Advances in neural information processing systems","author":"Hu Baotian","year":"2014"},{"key":"e_1_2_1_15_1","first-page":"765","volume-title":"IEEE International Conference on ICASSP'86","volume":"11","author":"Juang Biing-Hwang","year":"1986"},{"key":"e_1_2_1_16_1","unstructured":"Andrew Kelly. Cracking passwords using keyboard acoustics and language modeling.  Andrew Kelly. Cracking passwords using keyboard acoustics and language modeling."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2810103.2813668"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3161196"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Minh-Thang Luong Hieu Pham and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 2015.  Minh-Thang Luong Hieu Pham and Christopher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 2015.","DOI":"10.18653\/v1\/D15-1166"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2046707.2046771"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10207-017-0369-x"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2010-343"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.17487\/RFC1129"},{"key":"e_1_2_1_24_1","first-page":"797","volume-title":"2002 IEEE International Conference on","volume":"1","author":"Milner Ben"},{"key":"e_1_2_1_25_1","unstructured":"Abdel-rahman Mohamed. Deep neural network acoustic models for asr. PhD thesis 2014.  Abdel-rahman Mohamed. Deep neural network acoustic models for asr. PhD thesis 2014."},{"key":"e_1_2_1_26_1","unstructured":"Lindasalwa Muda Mumtaj Begam and Irraivan Elamvazuthi. Voice recognition algorithms using mel frequency cepstral coefficient (mfcc) and dynamic time warping (dtw) techniques. arXiv preprint arXiv:1003.4083 2010.  Lindasalwa Muda Mumtaj Begam and Irraivan Elamvazuthi. Voice recognition algorithms using mel frequency cepstral coefficient (mfcc) and dynamic time warping (dtw) techniques. arXiv preprint arXiv:1003.4083 2010."},{"key":"e_1_2_1_27_1","volume-title":"Pearson Education","author":"Oppenheim Alan V","year":"2014"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Javier Ram i rez Jos e C Segura Carmen Ben i tez Angel De La Torre and Antonio Rubio. Efficient voice activity detection algorithms using long-term speech information. speech communication 42(3-4).  Javier Ram i rez Jos e C Segura Carmen Ben i tez Angel De La Torre and Antonio Rubio. Efficient voice activity detection algorithms using long-term speech information. speech communication 42(3-4).","DOI":"10.1016\/j.specom.2003.10.002"},{"key":"e_1_2_1_29_1","unstructured":"Bharath Ramsundar Steven Kearnes Patrick Riley Dale Webster David Konerding and Vijay Pande. Massively multitask networks for drug discovery. arXiv preprint arXiv:1502.02072 2015.  Bharath Ramsundar Steven Kearnes Patrick Riley Dale Webster David Konerding and Vijay Pande. Massively multitask networks for drug discovery. arXiv preprint arXiv:1502.02072 2015."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/1886436.1886447"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/78.650093"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2516760.2516770"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1515\/popets-2016-0020"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/MNET.2004.1316761"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAU.1967.1161901"},{"key":"e_1_2_1_36_1","unstructured":"Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey etal Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 2016.  Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 2016."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185448.2185465"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2971648.2971681"},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Li Zhuang Feng Zhou and J D. Tygar. Keyboard acoustic emanations revisited. 13 01 2009.  Li Zhuang Feng Zhou and J D. Tygar. Keyboard acoustic emanations revisited. 13 01 2009.","DOI":"10.1145\/1609956.1609959"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3328916","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3328916","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:54:40Z","timestamp":1750204480000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3328916"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,21]]},"references-count":39,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,6,21]]}},"alternative-id":["10.1145\/3328916"],"URL":"https:\/\/doi.org\/10.1145\/3328916","relation":{},"ISSN":["2474-9567"],"issn-type":[{"value":"2474-9567","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,6,21]]},"assertion":[{"value":"2018-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-06-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}