{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T21:31:40Z","timestamp":1776115900259,"version":"3.50.1"},"reference-count":57,"publisher":"Association for Computing Machinery (ACM)","issue":"5","funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1750563"],"award-info":[{"award-number":["1750563"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>Interaction mining is a popular technique for capturing design and interaction data while a mobile app is being used. Over the years, researchers have leveraged interaction mining systems to build large repositories of interaction data, enabling novel, ML-based tools for UX practitioners, designers, and programmers. Existing interaction mining systems range from simple screen recorders \u2014 which are easy to use but capture sparse, unstructured data \u2014 to complex installations requiring dedicated hardware and custom OS forks \u2014 which yield rich, high-fidelity traces but are difficult to deploy outside of a lab environment. This paper presents ODIM, an on-device framework for mobile interaction mining that produces detailed trace metadata. The framework is reified in an Android implementation based on a simple APK that users can install on their personal devices. The paper outlines ODIM\u2019s design principles, describes its implementation, evaluates the system on traces collected from 100 popular apps on the Google Play Store, and discusses future avenues for scaling the utility and impact of interaction mining systems. The ODIM software, source code, and online trace repository are all freely available at interactionmining.org, for anyone to use and contribute to.<\/jats:p>","DOI":"10.1145\/3743726","type":"journal-article","created":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T14:28:48Z","timestamp":1757428128000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["On-Device Interaction Mining MHCI024"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-3260-0171","authenticated-orcid":false,"given":"Deniz","family":"Arsan","sequence":"first","affiliation":[{"name":"Siebel School of Computing and Data Science","place":["Urbana, USA"]},{"name":"University of Illinois at Urbana-Champaign","place":["Urbana, USA"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-9859-5753","authenticated-orcid":false,"given":"Carl","family":"Guo","sequence":"additional","affiliation":[{"name":"Siebel School of Computing and Data Science","place":["Champaign, USA"]},{"name":"University of Illinois at Urbana-Champaign","place":["Champaign, USA"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7749-9271","authenticated-orcid":false,"given":"Muhammad Rizky","family":"Wellyanto","sequence":"additional","affiliation":[{"name":"Siebel School of Computing and Data Science","place":["Urbana, USA"]},{"name":"University of Illinois at Urbana-Champaign","place":["Urbana, USA"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-9604-3576","authenticated-orcid":false,"given":"Erik R","family":"Ji","sequence":"additional","affiliation":[{"name":"Siebel School of Computing and Data Science","place":["Champaign, USA"]},{"name":"University of Illinois, Urbana-Champaign","place":["Champaign, USA"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0908-5388","authenticated-orcid":false,"given":"Jerry O.","family":"Talton","sequence":"additional","affiliation":[{"name":"Siebel School of Computing and Data Science","place":["Urbana, USA"]},{"name":"University of Illinois at Urbana-Champaign","place":["Urbana, USA"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2107-1647","authenticated-orcid":false,"given":"Ranjitha","family":"Kumar","sequence":"additional","affiliation":[{"name":"Siebel School of Computing and Data Science","place":["Urbana, USA"]},{"name":"University of Illinois at Urbana-Champaign","place":["Urbana, USA"]}]}],"member":"320","published-online":{"date-parts":[[2025,9,9]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"crossref","unstructured":"Saleema Amershi Maya Cakmak William\u00a0Bradley Knox and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine 35 4 (2014) 105\u2013120.","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472749.3474808"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3220134.3220135"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445762"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20074-8_18"},{"key":"e_1_3_3_7_2","doi-asserted-by":"crossref","unstructured":"Jieshan Chen Chunyang Chen Zhenchang Xing Xin Xia Liming Zhu John Grundy and Jinshui Wang. 2020. Wireframe-based UI design search through image autoencoder. ACM Transactions on Software Engineering and Methodology 29 3 (2020).","DOI":"10.1145\/3391613"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502073"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126651"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126647"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984581"},{"key":"e_1_3_3_12_2","doi-asserted-by":"crossref","unstructured":"Jing Dong Yajing Zhao and Tu Peng. 2009. A review of design pattern mining techniques. International Journal of Software Engineering and Knowledge Engineering 19 06 (2009) 823\u2013855.","DOI":"10.1142\/S021819400900443X"},{"key":"e_1_3_3_13_2","unstructured":"Bardia Doosti Tao Dong Biplab Deka and Jeffrey Nichols. 2018. A computational method for evaluating UI patterns. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1807.04191 (2018)."},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642782"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604056"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490099.3511109"},{"key":"e_1_3_3_17_2","doi-asserted-by":"crossref","unstructured":"Rebecca Fiebrink. 2017. Machine learning as meta-instrument: Human-machine partnerships shaping expressive instrumental creation. Musical Instruments in the 21st Century: Identities Configurations Practices (2017) 137\u2013151.","DOI":"10.1007\/978-981-10-2951-6_10"},{"key":"e_1_3_3_18_2","unstructured":"Google for Developers. 2025. Android Accessibility Service API Reference. https:\/\/developer.android.com\/reference\/android\/accessibilityservice\/AccessibilityService."},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(08)62386-9"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300334"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466420"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58580-8_29"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3406324.3410710"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/2628363.2628384"},{"key":"e_1_3_3_25_2","volume-title":"7th International Conference on Learning Representations","author":"Li Jianan","year":"2019","unstructured":"Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators. In 7th International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=HJxB5sRcFQ"},{"key":"e_1_3_3_26_2","first-page":"231","volume-title":"Findings of the Association for Computational Linguistics: EACL 2024","author":"Li Tao","year":"2024","unstructured":"Tao Li, Gang Li, Jingjie Zheng, Purple Wang, and Yang Li. 2024. MUG: Interactive Multimodal Grounding on User Interfaces. In Findings of the Association for Computational Linguistics: EACL 2024, Yvette Graham and Matthew Purver (Eds.). Association for Computational Linguistics, St. Julian\u2019s, Malta, 231\u2013251. https:\/\/aclanthology.org\/2024.findings-eacl.17\/"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025483"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445049"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3332165.3347899"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3210240.3210339"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.729"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.443"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00104"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242650"},{"key":"e_1_3_3_35_2","unstructured":"Yuxuan Lu Bingsheng Yao Shao Zhang Yun Wang Peng Zhang Tun Lu Toby Jia-Jun Li and Dakuo Wang. 2023. Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Tasks. arxiv:https:\/\/arXiv.org\/abs\/2311.09825\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2311.09825"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","unstructured":"Arunesh Mathur Gunes Acar Michael\u00a0J. Friedman Eli Lucherini Jonathan Mayer Marshini Chetty and Arvind Narayanan. 2019. Dark Patterns at Scale: Findings from a Crawl of 11K Shopping Websites. Proc. ACM Hum.-Comput. Interact. 3 CSCW Article MHCI024 (Nov. 2019) 32\u00a0pages. 10.1145\/3359183","DOI":"10.1145\/3359183"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3267305.3274172"},{"key":"e_1_3_3_38_2","unstructured":"Microsoft. 2024. Introducing Recall for Copilot+ PCs. https:\/\/blogs.microsoft.com\/blog\/2024\/05\/20\/introducing-recall-for-copilot-pcs\/. Accessed: 2025-06-25."},{"key":"e_1_3_3_39_2","doi-asserted-by":"crossref","unstructured":"Kevin Moran Carlos Bernal-C\u00e1rdenas Michael Curcio Richard Bonett and Denys Poshyvanyk. 2018. Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Transactions on Software Engineering 46 2 (2018) 196\u2013221.","DOI":"10.1109\/TSE.2018.2844788"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/MDM.2019.00118"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308561.3353808"},{"key":"e_1_3_3_42_2","volume-title":"Extended Abstracts of the 2010 CHI Conference on Human Factors in Computing Systems","author":"Prante Thorsten","year":"2010","unstructured":"Thorsten Prante, Jens Sauer, and Seif Lotfy. 2010. Personal Experience Trace: Orienting Oneself in One\u2019s Activities and Experiences. In Extended Abstracts of the 2010 CHI Conference on Human Factors in Computing Systems. ACM, Atlanta, GA, USA."},{"key":"e_1_3_3_43_2","unstructured":"Mishaal Rahman. 2016. \"Working As Intended\" - An Exploration into Android\u2019s Accessibility Lag. https:\/\/www.xda-developers.com\/working-as-intended-an-exploration-into-androids-accessibility-lag\/"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3234695.3236364"},{"key":"e_1_3_3_45_2","doi-asserted-by":"crossref","unstructured":"Anne\u00a0Spencer Ross Xiaoyi Zhang James Fogarty and Jacob\u00a0O Wobbrock. 2020. An epidemiology-inspired large-scale analysis of android app accessibility. ACM Transactions on Accessible Computing 13 1 (2020).","DOI":"10.1145\/3348797"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377325.3377515"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174078"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300305"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472749.3474765"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445526"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3586183.3606824"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581158"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472749.3474763"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376324"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-73039-9_14"},{"key":"e_1_3_3_56_2","doi-asserted-by":"crossref","unstructured":"Tao Zhang Ying Liu Jerry Gao Li\u00a0Peng Gao and Jing Cheng. 2020. Deep learning-based mobile application isomorphic gui identification for automated robotic testing. IEEE Software 37 4 (2020) 67\u201374.","DOI":"10.1109\/MS.2020.2987044"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445186"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242616"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3743726","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T16:08:53Z","timestamp":1757520533000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3743726"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,9]]},"references-count":57,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3743726"],"URL":"https:\/\/doi.org\/10.1145\/3743726","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,9]]},"assertion":[{"value":"2025-09-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}