{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T18:36:25Z","timestamp":1792002985675,"version":"4.4.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:00:00Z","timestamp":1750291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/publication-rights-and-licensing-policy"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"\n With the rapid development of Large Language Models (LLMs), their integration into automated mobile GUI testing has emerged as a promising research direction. However, existing LLM-based testing approaches face significant challenges, including time inefficiency and high costs due to constant LLM querying. To address these issues, this paper introduces LLMD\n roid<\/jats:sc>\n , a novel testing framework designed to enhance existing automated mobile GUI testing tools by leveraging LLMs more efficiently. The workflow of LLMD\n roid<\/jats:sc>\n comprises two main stages: Autonomous Exploration and LLM Guidance. During Autonomous Exploration, LLMD\n roid<\/jats:sc>\n utilizes existing testing tools while leveraging LLMs to summarize explored pages. When code coverage growth slows, it transitions to LLM Guidance to strategically direct testing towards unexplored functionalities. This approach minimizes LLM interactions while maximizing their impact on test coverage. We applied LLMD\n roid<\/jats:sc>\n to three popular open-source Android testing tools and evaluated it on 14 top-listed apps from Google Play. Results demonstrate an average increase of\n \n \n \n 26.16<\/mml:mn>\n %<\/mml:mtext>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n in code coverage and\n \n \n \n 29.31<\/mml:mn>\n %<\/mml:mtext>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n in activity coverage. Furthermore, our evaluation under different LLMs reveals that LLMD\n roid<\/jats:sc>\n outperform existing step-wise approaches with significant cost efficiency, achieving optimal performance at\n \n \n \n $<\/mml:mtext>\n 0.49<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n per hour using GPT-4o among tested models, with a cost-effective alternative achieving\n \n \n \n 94<\/mml:mn>\n %<\/mml:mtext>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n of this performance at just\n \n \n \n $<\/mml:mtext>\n 0.03<\/mml:mn>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n per hour. These findings highlight LLMD\n roid<\/jats:sc>\n \u2019s effectiveness in enhancing automated mobile app testing and its potential for widespread adoption.\n <\/jats:p>","DOI":"10.1145\/3715763","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T11:15:34Z","timestamp":1750331734000},"page":"1001-1022","source":"Crossref","is-referenced-by-count":29,"title":["LLMDroid: Enhancing Automated Mobile App GUI Testing Coverage with Large Language Model Guidance"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-8764-7230","authenticated-orcid":false,"given":"Chenxu","family":"Wang","sequence":"first","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5216-933X","authenticated-orcid":false,"given":"Tianming","family":"Liu","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8793-5367","authenticated-orcid":false,"given":"Yanjie","family":"Zhao","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3772-4240","authenticated-orcid":false,"given":"Minghui","family":"Yang","sequence":"additional","affiliation":[{"name":"OPPO, Dongguan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1100-8633","authenticated-orcid":false,"given":"Haoyu","family":"Wang","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"2024. AccessibilityService | Android Developers. https:\/\/developer.android.com\/reference\/android\/accessibilityservice\/AccessibilityService."},{"key":"e_1_3_1_3_2","unstructured":"2024. bytedance\/Fastbot_Android: Fastbot(2.0) is a model-based testing tool for modeling GUI transitions to discover app stability problems. https:\/\/github.com\/bytedance\/Fastbot_Android."},{"key":"e_1_3_1_4_2","unstructured":"2024. Dice-S\u00f8rensen coefficient. https:\/\/en.wikipedia.org\/wiki\/Dice-S%C3%B8rensen_coefficient."},{"key":"e_1_3_1_5_2","unstructured":"2024. Fragments | Android Developers. https:\/\/developer.android.com\/guide\/fragments."},{"key":"e_1_3_1_6_2","unstructured":"2024. GPT-3.5 | OpenAI. https:\/\/platform.openai.com\/docs\/models\/gpt-3-5-turbo."},{"key":"e_1_3_1_7_2","unstructured":"2024. GPT-4o | OpenAI. https:\/\/openai.com\/index\/hello-gpt-4o\/."},{"key":"e_1_3_1_8_2","unstructured":"2024. GPT-4o mini | OpenAI. https:\/\/openai.com\/index\/gpt-4o-mini-advancing-cost-efficient-intelligence\/."},{"key":"e_1_3_1_9_2","unstructured":"2024. Java Code Coverage Library. https:\/\/github.com\/jacoco\/jacoco."},{"key":"e_1_3_1_10_2","unstructured":"2024. UI\/Application Exerciser Monkey | Android Developers. https:\/\/developer.android.com\/studio\/test\/other-testing-tools\/monkey."},{"key":"e_1_3_1_11_2","unstructured":"2025. coinse\/droidagent: DroidAgent: Intent-Driven Mobile GUI Testing with Autonomous LLM Agents. https:\/\/github.com\/coinse\/droidagent."},{"key":"e_1_3_1_12_2","unstructured":"2025. DeepSeek. https:\/\/www.deepseek.com\/."},{"key":"e_1_3_1_13_2","unstructured":"2025. Llama. https:\/\/www.llama.com\/."},{"key":"e_1_3_1_14_2","unstructured":"2025. Pricing - OpenAI API. https:\/\/platform.openai.com\/docs\/pricing."},{"key":"e_1_3_1_15_2","unstructured":"2025. testinging6\/GPTDroid. https:\/\/github.com\/testinging6\/GPTDroid."},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/ms.2014.55"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ase.2015.89"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","unstructured":"Chenhui Cui Tao Li Junjie Wang Chunyang Chen Dave Towey and Rubing Huang. 2024. Large Language Models for Mobile GUI Text Input Generation: An Empirical Study. arXiv preprint arXiv:2404.08948 (2024). doi:10.48550\/arXiv.2404.08948","DOI":"10.48550\/arXiv.2404.08948"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","unstructured":"Zhen Dong Marcel B\u00f6hme Lucia Cojocaru and Abhik Roychoudhury. 2020. Time-travel testing of android apps. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 481\u2013492. doi:10.1145\/3377811.3380402","DOI":"10.1145\/3377811.3380402"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","unstructured":"Sidong Feng and Chunyang Chen. 2024. Prompting Is All You Need: Automated Android Bug Replay with Large Language Models. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering. 1\u201313. doi:10.1145\/3597503.3608137","DOI":"10.1145\/3597503.3608137"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/icse48619.2023.00084"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","unstructured":"Tianxiao Gu Chengnian Sun Xiaoxing Ma Chun Cao Chang Xu Yuan Yao Qirun Zhang Jian Lu and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). IEEE 269\u2013280. doi:10.1109\/icse.2019.00042","DOI":"10.1109\/icse.2019.00042"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695988"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","unstructured":"Han Hu Han Wang Ruiqi Dong Xiao Chen and Chunyang Chen. 2024. Enhancing GUI Exploration Coverage of Android Apps with Deep Link-Integrated Monkey. ACM Transactions on Software Engineering and Methodology (2024). doi:10.1145\/3664810","DOI":"10.1145\/3664810"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","unstructured":"Yongxiang Hu Xuan Wang Yingchuan Wang Yu Zhang Shiyu Guo Chaoyi Chen Xin Wang and Yangfan Zhou. 2024. AUITestAgent: Automatic Requirements Oriented GUI Function Testing. arXiv preprint arXiv:2407.09018 (2024). doi:10.48550\/arXiv.2407.09018","DOI":"10.48550\/arXiv.2407.09018"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","unstructured":"Yuchao Huang Junjie Wang Zhe Liu Yawen Wang Song Wang Chunyang Chen Yuanzhe Hu and Qing Wang. 2024. Crashtranslator: Automatically reproducing mobile application crashes directly from stack trace. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering. 1\u201313. doi:10.1145\/3597503.3623298","DOI":"10.1145\/3597503.3623298"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","unstructured":"Bangyan Ju Jin Yang Tingting Yu Tamerlan Abdullayev Yuanyuan Wu Dingbang Wang and Yu Zhao. 2024. A Study of Using Multimodal LLMs for Non-Crash Functional Bug Detection in Android Apps. arXiv preprint arXiv:2407.19053 (2024). doi:10.48550\/arXiv.2407.19053","DOI":"10.48550\/arXiv.2407.19053"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/tr.2018.2865733"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","unstructured":"Yuanhong Lan Yifei Lu Zhong Li Minxue Pan Wenhua Yang Tian Zhang and Xuandong Li. 2024. Deeply Reinforcing Android GUI Testing with Deep Reinforcement Learning. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering. 1\u201313. doi:10.1145\/3597503.3623344","DOI":"10.1145\/3597503.3623344"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","unstructured":"Yuanhong Lan Yifei Lu Minxue Pan and Xuandong Li. 2024. Navigating Mobile Testing Evaluation: A Comprehensive Statistical Analysis of Android GUI Testing Metrics. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering. 944\u2013956. doi:10.1145\/3691620.3695476","DOI":"10.1145\/3691620.3695476"},{"key":"e_1_3_1_31_2","first-page":"18893","volume-title":"International Conference on Machine Learning","author":"Lee Kenton","year":"2023","unstructured":"Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning. PMLR, 18893\u201318912."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.03003"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/icse-c.2017.8"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ase.2019.00104"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/icse48619.2023.00119"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","unstructured":"Zhe Liu Chunyang Chen Junjie Wang Mengzhuo Chen Boyu Wu Xing Che Dandan Wang and Qing Wang. 2024. Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. 1\u201313. doi:10.1145\/3597503.3639180","DOI":"10.1145\/3597503.3639180"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","unstructured":"Zhe Liu Cheng Li Chunyang Chen Junjie Wang Boyu Wu Yawen Wang Jun Hu and Qing Wang. 2024. Vision-driven Automated Mobile GUI Testing via Multimodal Large Language Model. arXiv preprint arXiv:2407.03037 (2024). doi:10.48550\/arXiv.2407.03037","DOI":"10.48550\/arXiv.2407.03037"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","unstructured":"Zhengwei Lv Chao Peng Zhao Zhang Ting Su Kai Liu and Ping Yang. 2022. Fastbot2: Reusable automated modelbased gui testing for android enhanced by reinforcement learning. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u20135. doi:10.1145\/3551349.3559505","DOI":"10.1145\/3551349.3559505"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","unstructured":"Minxue Pan An Huang Guoxin Wang Tian Zhang and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153\u2013164. doi:10.1145\/3395363.3397354","DOI":"10.1145\/3395363.3397354"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","unstructured":"Jordan Samhi and Andreas Zeller. 2024. AndroLog: Android Instrumentation and Code Coverage Analysis. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 597\u2013601. doi:10.1145\/3663529.3663806","DOI":"10.1145\/3663529.3663806"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","unstructured":"Ting Su Guozhu Meng Yuting Chen Ke Wu Weiming Yang Yao Yao Geguang Pu Yang Liu and Zhendong Su. 2017. Guided stochastic model-based GUI testing of Android apps. In Proceedings of the 2017 11th joint meeting on foundations of software engineering. 245\u2013256. doi:10.1145\/3106237.3106298","DOI":"10.1145\/3106237.3106298"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","unstructured":"Ting Su Jue Wang and Zhendong Su. 2021. Benchmarking automated GUI testing for Android against real-world bugs. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 119\u2013130. doi:10.1145\/3468264.3468620","DOI":"10.1145\/3468264.3468620"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","unstructured":"Raja Vall\u00e9e-Rai Phong Co Etienne Gagnon Laurie Hendren Patrick Lam and Vijay Sundaresan. 2010. Soot: A Java bytecode optimization framework. In CASCON First Decade High Impact Papers. 214\u2013224. doi:10.1145\/1925805.1925818","DOI":"10.1145\/1925805.1925818"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","unstructured":"Bryan Wang Gang Li and Yang Li. 2023. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1\u201317. doi:10.1145\/3544548.3580895","DOI":"10.1145\/3544548.3580895"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","unstructured":"Dingbang Wang Yu Zhao Sidong Feng Zhaoxu Zhang William GJ Halfond Chunyang Chen Xiaoxia Sun Jiangfan Shi and Tingting Yu. 2024. Feedback-Driven Automated Whole Bug Report Reproduction for Android Apps. arXiv preprint arXiv:2407.05165 (2024). doi:10.1145\/3650212.3680341","DOI":"10.1145\/3650212.3680341"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","unstructured":"Hao Wen Yuanchun Li Guohong Liu Shanhui Zhao Tao Yu Toby Jia-Jun Li Shiqi Jiang Yunhao Liu Yaqin Zhang and Yunxin Liu. 2024. AutoDroid: LLM-powered Task Automation in Android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 543\u2013557. doi:10.1145\/3636534.3649379","DOI":"10.1145\/3636534.3649379"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","unstructured":"Hao Wen Hongming Wang Jiaxuan Liu and Yuanchun Li. 2023. Droidbot-gpt: Gpt-powered ui automation for android. arXiv preprint arXiv:2304.07061 (2023). doi:10.48550\/arXiv.2304.07061","DOI":"10.48550\/arXiv.2304.07061"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","unstructured":"An Yan Zhengyuan Yang Wanrong Zhu Kevin Lin Linjie Li Jianfeng Wang Jianwei Yang Yiwu Zhong Julian McAuley Jianfeng Gao et al. 2023. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07562 (2023). doi:10.48550\/arXiv.2311.07562","DOI":"10.48550\/arXiv.2311.07562"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","unstructured":"Jiwei Yan Hao Liu Linjie Pan Jun Yan Jian Zhang and Bin Liang. 2020. Multiple-entry testing of android applications by constructing activity launching contexts. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 457\u2013468. doi:10.1145\/3377811.3380347","DOI":"10.1145\/3377811.3380347"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","unstructured":"Zhao Yang Jiaxuan Liu Yucheng Han Xin Chen Zebiao Huang Bin Fu and Gang Yu. 2023. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771 (2023). doi:10.48550\/arXiv.2312.13771","DOI":"10.48550\/arXiv.2312.13771"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ase51524.2021.9678778"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST60714.2024.00020"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715763","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T11:03:13Z","timestamp":1787482993000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715763"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":51,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3715763"],"URL":"https:\/\/doi.org\/10.1145\/3715763","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}