{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:36:50Z","timestamp":1791459410781,"version":"4.3.4"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T00:00:00Z","timestamp":1727395200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62141215, 61932012, and 62372228"],"award-info":[{"award-number":["62141215, 61932012, and 62372228"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Technology and Innovation Commission of Shenzhen Municipality","award":["CJGJZD20200617103001003"],"award-info":[{"award-number":["CJGJZD20200617103001003"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Software applications (apps) have been playing an increasingly important role in various aspects of society. In particular, mobile apps and web apps are the most prevalent among all applications and are widely used in various industries as well as in people\u2019s daily lives. To help ensure mobile and web app quality, many approaches have been introduced to improve app GUI testing via automated exploration, including random testing, model-based testing, learning-based testing, and so on. Despite the extensive effort, existing approaches are still limited in reaching high code coverage, constructing high-quality models, and being generally applicable. Reinforcement learning-based approaches, as a group of representative and advanced approaches for automated GUI exploration testing, are faced with difficult challenges, including effective app state abstraction, reward function design, and so on. Moreover, they heavily depend on the specific execution platforms (i.e., Android or Web), thus leading to poor generalizability and being unable to adapt to different platforms.<\/jats:p>\n          <jats:p>\n            This work specifically tackles these challenges based on the high-level observation that apps from distinct platforms share commonalities in GUI design. Indeed, we propose\n            <jats:sc>PIRLTest<\/jats:sc>\n            , an effective platform-independent approach for app testing. Specifically,\n            <jats:sc>PIRLTest<\/jats:sc>\n            utilizes computer vision and reinforcement learning techniques in a novel, synergistic manner for automated testing. It extracts the GUI widgets from GUI pages and characterizes the corresponding GUI layouts, embedding the GUI pages as states. The app GUI state combines the macroscopic perspective (app GUI layout) and the microscopic perspective (app GUI widget) and attaches the critical semantic information from GUI images. This enables\n            <jats:sc>PIRLTest<\/jats:sc>\n            to be platform-independent and makes the testing approach generally applicable on different platforms.\n            <jats:sc>PIRLTest<\/jats:sc>\n            explores apps with the guidance of a curiosity-driven strategy, which uses a Q-network to estimate the values of specific state-action pairs to encourage more exploration in uncovered pages without platform dependency. The exploration will be assigned with rewards for all actions, which are designed considering both the app GUI states and the concrete widgets, to help the framework explore more uncovered pages. We conduct an empirical study on 20 mobile apps and 5 web apps, and the results show that\n            <jats:sc>PIRLTest<\/jats:sc>\n            is zero-cost when being adapted to different platforms, and can perform better than the baselines, covering 6.3\u201341.4% more code on mobile apps and 1.5\u201351.1% more code on web apps.\n            <jats:sc>PIRLTest<\/jats:sc>\n            is capable of detecting 128 unique bugs on mobile and web apps, including 100 bugs that cannot be detected by the baselines.\n          <\/jats:p>","DOI":"10.1145\/3674728","type":"journal-article","created":{"date-parts":[[2024,6,22]],"date-time":"2024-06-22T04:59:34Z","timestamp":1719032374000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcement Learning"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4640-8637","authenticated-orcid":false,"given":"Shengcheng","family":"Yu","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9930-7111","authenticated-orcid":false,"given":"Chunrong","family":"Fang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7953-3920","authenticated-orcid":false,"given":"Xin","family":"Li","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9227-3824","authenticated-orcid":false,"given":"Yuchen","family":"Ling","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9592-7022","authenticated-orcid":false,"given":"Zhenyu","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2970-1391","authenticated-orcid":false,"given":"Zhendong","family":"Su","sequence":"additional","affiliation":[{"name":"Department of Computer Science, ETH Zurich, Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,9,27]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3278186.3278187"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2351676.2351717"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2014.55"},{"key":"e_1_3_2_5_2","unstructured":"Appium. 2022. Retrieved from http:\/\/appium.io\/"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3092703.3092734"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970313"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213854"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380328"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66299-2_2"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338970"},{"key":"e_1_3_2_13_2","first-page":"1","volume-title":"Proceedings of the 7th International Conference on Learning Representations","author":"Burda Yuri","year":"2019","unstructured":"Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. 2019. Large-scale study of curiosity-driven learning. Proceedings of the 7th International Conference on Learning Representations. 1\u201315."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.1986.4767851"},{"key":"e_1_3_2_15_2","first-page":"1","article-title":"Boltzmann exploration done right","volume":"30","author":"Cesa-Bianchi Nicolo","year":"2017","unstructured":"Nicolo Cesa-Bianchi, Claudio Gentile, G\u00e1bor Lugosi, and Gergely Neu. 2017. Boltzmann exploration done right. Advances in Neural Information Processing Systems 30 (2017). 1\u201310.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180240"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3391613"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380327"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409691"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2015.89"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474624.3474634"},{"key":"e_1_3_2_22_2","first-page":"957","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering","author":"Cooper Nathan","year":"2021","unstructured":"Nathan Cooper, Carlos Bernal-C\u00e1rdenas, Oscar Chaparro, Kevin Moran, and Denys Poshyvanyk. 2021. It takes two to tango: Combining visual and textual information for detecting duplicate video-based bug reports. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering. IEEE, 957\u2013969."},{"key":"e_1_3_2_23_2","first-page":"905","volume-title":"Proceedings of the 2021 36th IEEE\/ACM International Conference on Automated Software Engineering","author":"Daragh, Faraz Yazdani Banafshe","year":"2021","unstructured":"Faraz Yazdani Banafshe Daragh, and Sam Malek. 2021. Deep GUI: Black-box GUI input generation with deep learning. In Proceedings of the 2021 36th IEEE\/ACM International Conference on Automated Software Engineering. IEEE, 905\u2013916."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380402"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510048"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786862"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970367"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.5555\/2486788.2486799"},{"key":"e_1_3_2_29_2","unstructured":"Google. 2022. Retrieved from https:\/\/developer.android.com\/studio\/test\/other-testing-tools\/monkey"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-24797-2_4"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2017.72"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00042"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330557"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISPASS.2015.7095807"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC48688.2020.00-78"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.5555\/1622737.1622748"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00020"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00021"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549083"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00104"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338930"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387503"},{"key":"e_1_3_2_43_2","unstructured":"Zhe Liu Chunyang Chen Junjie Wang Mengzhuo Chen Boyu Wu Xing Che Dandan Wang and Qing Wang. 2023. Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing. arXiv: 2305.09434. Retrieved from https:\/\/arxiv.org\/abs\/2305.09434"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416547"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3559505"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491450"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931054"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/2109205.2109208"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.28"},{"key":"e_1_3_2_50_2","unstructured":"Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin Riedmiller. 2013. Playing Atari with deep reinforcement learning. arXiv: 1312.5602. Retrieved from https:\/\/arxiv.org\/abs\/1312.5602"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2844788"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2015.32"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397354"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/2699485"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2015.08.004"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380431"},{"key":"e_1_3_2_57_2","first-page":"525","article-title":"Applying the rule of least surprise","author":"Raymond Eric Steven","year":"2003","unstructured":"Eric Steven Raymond. 2003. Applying the rule of least surprise. In The Art of Unix Programming, 525.","journal-title":"The Art of Unix Programming"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1090\/S0002-9904-1952-09620-8"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3502868"},{"key":"e_1_3_2_60_2","article-title":"The cranky user: The principle of least astonishment","author":"Seebach Peter","year":"2001","unstructured":"Peter Seebach. 2001. The cranky user: The principle of least astonishment. IBM DeveloperWorks (2001).","journal-title":"IBM DeveloperWorks"},{"key":"e_1_3_2_61_2","unstructured":"Selenium. 2022. Retrieved from https:\/\/www.selenium.dev\/"},{"key":"e_1_3_2_62_2","first-page":"1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations, Conference Track Proceedings","author":"Simonyan Karen","year":"2015","unstructured":"Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Representations, Conference Track Proceedings. 1\u201314."},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106298"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380382"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.03.003"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970312"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330551"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/0377-2217(89)90348-2"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00041"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/HotWeb.2015.14"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00161"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00090"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00139"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00074"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","unstructured":"Yu Zhao Brent Harrison and Tingting Yu. 2022. DinoDroid: Testing Android apps using deep Q-networks. arXiv: 2210.06307. Retrieved from https:\/\/doi.org\/10.1145\/3652150","DOI":"10.1145\/3652150"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00048"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00077"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3674728","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3674728","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:57:50Z","timestamp":1750294670000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3674728"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,27]]},"references-count":77,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3674728"],"URL":"https:\/\/doi.org\/10.1145\/3674728","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,27]]},"assertion":[{"value":"2023-07-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-09","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}