{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T10:04:21Z","timestamp":1784541861836,"version":"3.55.0"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2022,7,12]],"date-time":"2022-07-12T00:00:00Z","timestamp":1657584000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,10,31]]},"abstract":"<jats:p>The state space of Android apps is huge, and its thorough exploration during testing remains a significant challenge. The best exploration strategy is highly dependent on the features of the app under test. Reinforcement Learning (RL) is a machine learning technique that learns the optimal strategy to solve a task by trial and error, guided by positive or negative reward, rather than explicit supervision. Deep RL is a recent extension of RL that takes advantage of the learning capabilities of neural networks. Such capabilities make Deep RL suitable for complex exploration spaces such as one of Android apps. However, state-of-the-art, publicly available tools only support basic, Tabular RL. We have developed ARES, a Deep RL approach for black-box testing of Android apps. Experimental results show that it achieves higher coverage and fault revelation than the baselines, including state-of-the-art tools, such as TimeMachine and Q-Testing. We also investigated the reasons behind such performance qualitatively, and we have identified the key features of Android apps that make Deep RL particularly effective on them to be the presence of chained and blocking activities. Moreover, we have developed FATE to fine-tune the hyperparameters of Deep RL algorithms on simulated apps, since it is computationally expensive to carry it out on real apps.<\/jats:p>","DOI":"10.1145\/3502868","type":"journal-article","created":{"date-parts":[[2022,1,31]],"date-time":"2022-01-31T17:38:58Z","timestamp":1643650738000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":75,"title":["Deep Reinforcement Learning for Black-box Testing of Android Apps"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4651-8713","authenticated-orcid":false,"given":"Andrea","family":"Romdhana","sequence":"first","affiliation":[{"name":"DIBRIS - Universit\u00e0 degli Studi di Genova, FBK-ICT, Security &amp; Trust Unit"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessio","family":"Merlo","sequence":"additional","affiliation":[{"name":"DIBRIS - Universit\u00e0 degli Studi di Genova"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7325-0316","authenticated-orcid":false,"given":"Mariano","family":"Ceccato","sequence":"additional","affiliation":[{"name":"Universit\u00e0 di Verona"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paolo","family":"Tonella","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,7,12]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"2006. Emma.Retrieved from http:\/\/emma.sourceforge.net."},{"key":"e_1_3_2_3_2","unstructured":"2020. Appbrain. Retrieved from https:\/\/www.appbrain.com."},{"key":"e_1_3_2_4_2","unstructured":"2020. Appium. Retrieved from http:\/\/appium.io."},{"key":"e_1_3_2_5_2","unstructured":"Josh Achiam. 2018. Key Concepts in RL. Retrieved from https:\/\/spinningup.openai.com\/en\/latest\/spinningup\/rl_intro.html."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2351676.2351717"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2014.55"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2393596.2393666"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3197231.3197243"},{"key":"e_1_3_2_10_2","first-page":"369","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems","author":"Boyan Justin A.","year":"1995","unstructured":"Justin A. Boyan and Andrew W. Moore. 1995. Generalization in reinforcement learning: Safely approximating the value function. In Proceedings of the Conference on Advances in Neural Information Processing Systems. 369\u2013376."},{"key":"e_1_3_2_11_2","article-title":"Openai Gym","author":"Brockman Greg","year":"2016","unstructured":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai Gym. arXiv preprint arXiv:1606.01540 (2016).","journal-title":"arXiv preprint arXiv:1606.01540"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2015.89"},{"key":"e_1_3_2_13_2","unstructured":"Jack Clark and Dario Amodei. 2016. Faulty Reward Functions in the Wild. Retrieved from https:\/\/openai.com\/blog\/faulty-reward-functions\/."},{"key":"e_1_3_2_14_2","unstructured":"Zhen Dong Marcel Bohme Lucia Cojocaru and Abhik Roychoudhury. 2020. Github Repository: TimeMachine. Retrieved from https:\/\/github.com\/DroidTest\/TimeMachine\/blob\/master\/fuzzingandroid\/sys_event_generator\/sys_event.py."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380402"},{"key":"e_1_3_2_16_2","article-title":"Addressing function approximation error in actor-critic methods","author":"Fujimoto Scott","year":"2018","unstructured":"Scott Fujimoto, Herke van Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477 (2018).","journal-title":"arXiv preprint arXiv:1802.09477"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238225"},{"key":"e_1_3_2_18_2","unstructured":"Google. 2019. Broadcasts. Retrieved from https:\/\/developer.android.com\/guide\/components\/broadcasts."},{"key":"e_1_3_2_19_2","unstructured":"Google. 2019. Safety Net. Retrieved from https:\/\/developer.android.com\/training\/safetynet\/attestation."},{"key":"e_1_3_2_20_2","unstructured":"Google. 2020. Android Emulator. Retrieved from https:\/\/developer.android.com\/studio\/run\/emulator\/."},{"key":"e_1_3_2_21_2","unstructured":"Google. 2020. System-Level Events API 25. Retrieved from https:\/\/cs.android.com\/android\/platform\/superproject\/+\/android-7.1.2_r36:frameworks\/base\/core\/res\/AndroidManifest.xml."},{"key":"e_1_3_2_22_2","unstructured":"Google. 2020. UI\/Application Exerciser Monkey. Retrieved from https:\/\/developer.android.com\/studio\/test\/monkey."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00042"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00059"},{"key":"e_1_3_2_25_2","article-title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","author":"Haarnoja Tuomas","year":"2018","unstructured":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 (2018).","journal-title":"arXiv preprint arXiv:1801.01290"},{"key":"e_1_3_2_26_2","article-title":"Stable Baselines","author":"Hill Ashley","year":"2018","unstructured":"Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. 2018. Stable Baselines. Retrieved from https:\/\/github.com\/hill-a\/stable-baselines.","journal-title":"Retrieved from https:\/\/github.com\/hill-a\/stable-baselines"},{"issue":"2","key":"e_1_3_2_27_2","first-page":"65","article-title":"A simple sequentially rejective multiple test procedure","volume":"6","author":"Holm Sture","year":"1979","unstructured":"Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6, 2 (1979), 65\u201370.","journal-title":"Scand. J. Statist."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2018.2865733"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00020"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.21236\/ADA538760"},{"key":"e_1_3_2_31_2","article-title":"Deep reinforcement learning: An overview","author":"Li Yuxi","year":"2017","unstructured":"Yuxi Li. 2017. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274 (2017).","journal-title":"arXiv preprint arXiv:1701.07274"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00104"},{"key":"e_1_3_2_33_2","article-title":"Continuous control with deep reinforcement learning","author":"Lillicrap Timothy P.","year":"2015","unstructured":"Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).","journal-title":"arXiv preprint arXiv:1509.02971"},{"key":"e_1_3_2_34_2","volume-title":"Reinforcement Learning for Robots Using Neural Networks","author":"Lin Long-Ji","year":"1993","unstructured":"Long-Ji Lin. 1993. Reinforcement Learning for Robots Using Neural Networks. Technical Report. Carnegie-Mellon University School of Computer Science, Pittsburgh PA."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491450"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635896"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931054"},{"key":"e_1_3_2_38_2","unstructured":"Evgeny Mandrikov Marc R. Hoffmann Brock Janiczak. 2020. Jacoco Code Coverage. Retrieved from https:\/\/www.eclemma.org\/jacoco\/."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.88"},{"key":"e_1_3_2_40_2","unstructured":"Microsoft. 2020. Your Device is Rooted and You Can\u2019t Connect-Android. Retrieved from https:\/\/docs.microsoft.com\/it-it\/mem\/intune\/user-help\/your-device-is-rooted-and-you-cant-connect-android."},{"key":"e_1_3_2_41_2","article-title":"Playing Atari with deep reinforcement learning","author":"Mnih Volodymyr","year":"2013","unstructured":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).","journal-title":"arXiv preprint arXiv:1312.5602"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397354"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/11564096_32"},{"key":"e_1_3_2_44_2","first-page":"387","volume-title":"Proceedings of the 31st International Conference on Machine Learning","volume":"32","author":"Silver David","year":"2014","unstructured":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning, Vol. 32, Eric P. Xing, and Tony Jebara (Eds.). PMLR, 387\u2013395. https:\/\/proceedings.mlr.press\/v32\/silver14.html."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106298"},{"key":"e_1_3_2_46_2","unstructured":"Ting Su Guozhu Meng Yuting Chen Ke Wu Weiming Yang Yao Yao Geguang Pu Yang Liu and Zhendong Su. 2017. System-Level Events API 19. Retrieved from https:\/\/sites.google.com\/site\/stoat2017\/evaluation\/stoat-s-system-level-events."},{"key":"e_1_3_2_47_2","unstructured":"Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement learning: An introduction. MIT press."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-70972-7_16"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3502868","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3502868","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:34Z","timestamp":1750188634000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3502868"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,12]]},"references-count":48,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,10,31]]}},"alternative-id":["10.1145\/3502868"],"URL":"https:\/\/doi.org\/10.1145\/3502868","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,12]]},"assertion":[{"value":"2021-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}