{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T15:02:03Z","timestamp":1781708523749,"version":"3.54.5"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"name":"National Key Research and Development Program of China","award":["2024YFF0908001"],"award-info":[{"award-number":["2024YFF0908001"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["U24A20337 and 62372228"],"award-info":[{"award-number":["U24A20337 and 62372228"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"crossref","award":["14380029"],"award-info":[{"award-number":["14380029"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Shenzhen-Hong Kong-Macau Technology Research Programme","award":["SGDX20230821091559018"],"award-info":[{"award-number":["SGDX20230821091559018"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:p>\n                    Recently, creative generative AI software has emerged as a pivotal assistant, enabling users to generate content and seek inspiration rapidly. Text-to-Image (T2I) software, one of the most widely used, synthesizes images with text input by engaging in a cross-modal process. However, despite substantial advancements in the T2I engine, T2I software still encounters errors when generating complex or non-realistic scenes, including omitting focal entities, low image realism, and mismatched text-image information. The cross-modal nature of T2I software complicates error detection for traditional testing methods, and the absence of test oracles further exacerbates the complexity of the testing process. To fill this gap, we propose\n                    <jats:italic toggle=\"yes\">ACTesting<\/jats:italic>\n                    , an\n                    <jats:italic toggle=\"yes\">A<\/jats:italic>\n                    utomated\n                    <jats:italic toggle=\"yes\">C<\/jats:italic>\n                    ross-modal\n                    <jats:italic toggle=\"yes\">Testing<\/jats:italic>\n                    Method of Text-to-Image Software, the first testing method explicitly designed for T2I software. ACTesting utilizes the metamorphic testing principle to address the oracle problem and identifies cross-modal semantic consistency as its fundamental Metamorphic Relation (MR) by employing the Entity-Relationship (ER) triples. We design three kinds of mutation operators under the guidance of MR and the adaptability density constraint to construct the new input text. After generating the images based on the text, ACTesting verifies whether MR is satisfied by detecting the ER triples across two modalities to detect the errors of T2I software. In our experiments across five popular T2I software, ACTesting effectively generates error-revealing tests, resulting in a decrease in text-image consistency by up to 20% when compared to the baseline. Additionally, an ablation study demonstrates the efficacy of the proposed mutation operators. The experimental results validate that ACTesting can reliably identify errors within T2I software.\n                  <\/jats:p>","DOI":"10.1145\/3768581","type":"journal-article","created":{"date-parts":[[2025,9,19]],"date-time":"2025-09-19T22:30:06Z","timestamp":1758321006000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["ACTesting: Automated Cross-Modal Testing Method of Text-to-Image Software"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5514-6734","authenticated-orcid":false,"given":"Siqi","family":"Gu","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9930-7111","authenticated-orcid":false,"given":"Chunrong","family":"Fang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2495-3805","authenticated-orcid":false,"given":"Quanjun","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9592-7022","authenticated-orcid":false,"given":"Zhenyu","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,17]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Alibaba. 2023. Wan Xiang. Retrieved from https:\/\/tongyi.aliyun.com\/wanxiang\/"},{"key":"e_1_3_2_3_2","unstructured":"Baidu. 2023. ERNIE-ViLG. Retrieved from https:\/\/ai.baidu.com\/tech\/creativity\/ernie_Vilg"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2018.10.009"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.18034\/ajase.v8i1.9"},{"key":"e_1_3_2_6_2","unstructured":"Bytedance. 2025. Doubao. Retrieved from https:\/\/www.doubao.com\/chat\/create-image"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00132"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3069041"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00283"},{"key":"e_1_3_2_10_2","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS)","author":"Cho Jaemin","year":"2023","unstructured":"Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023. Visual programming for step-by-step text-to-image generation and evaluation. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS)."},{"key":"e_1_3_2_11_2","unstructured":"DeepAI. 2023. DeepAI Image Generator. Retrieved from https:\/\/deepai.org\/machine-learning-model\/text2img"},{"key":"e_1_3_2_12_2","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (NAACL-HLT (1))","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (NAACL-HLT (1)). Association for Computational Linguistics, 4171\u20134186."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20059-5_34"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2020.2972974"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2021.07.019"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616278"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2270"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01043"},{"key":"e_1_3_2_19_2","first-page":"1321","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Guo Chuan","year":"2017","unstructured":"Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the International Conference on Machine Learning, 1321\u20131330."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00047"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.3021209"},{"key":"e_1_3_2_22_2","first-page":"13753","volume-title":"Proceedings of the 40th International Conference on Machine Learning (ICML \u201923)","volume":"202","author":"Huang Lianghua","year":"2023","unstructured":"Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. 2023. Composer: Creative and controllable image synthesis with composable conditions. In Proceedings of the 40th International Conference on Machine Learning (ICML \u201923). Proceedings of Machine Learning Research, Vol. 202, PMLR, 13753\u201313773."},{"key":"e_1_3_2_23_2","first-page":"284","volume-title":"Proceedings of the Asian Conference on Machine Learning","author":"Huang Wanming","year":"2019","unstructured":"Wanming Huang, Richard Yi Da Xu, and Ian Oppermann. 2019. Realistic image generation using region-phrase attention. In Proceedings of the Asian Conference on Machine Learning, 284\u2013299."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/RIVF55975.2022.10013851"},{"key":"e_1_3_2_25_2","doi-asserted-by":"crossref","unstructured":"Pin Ji Yang Feng Jia Liu Zhihong Zhao and Zhenyu Chen. 2022. ASRTest: Automated testing for deep-neural-network-driven speech recognition systems. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis 189\u2013201.","DOI":"10.1145\/3533767.3534391"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01603"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00766"},{"key":"e_1_3_2_28_2","article-title":"Holistic evaluation of text-to-image models","volume":"36","author":"Lee Tony","year":"2024","unstructured":"Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. 2024. Holistic evaluation of text-to-image models. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_29_2","unstructured":"Yingwei Li Adams Wei Yu Tianjian Meng Ben Caine Jiquan Ngiam Daiyi Peng Junyang Shen Yifeng Lu Denny Zhou Quoc V. Le et al. 2022. DeepFusion: Lidar-camera deep fusion for multi-modal 3D object detection. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition 17182\u201317191."},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","unstructured":"Zixi Liu Yang Feng Yining Yin Jingyu Sun Zhenyu Chen and Baowen Xu. 2022. QATest: A Uniform Fuzzing Framework for Question Answering Systems. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering 1\u201312.","DOI":"10.1145\/3551349.3556929"},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","unstructured":"Lei Ma Fuyuan Zhang Jiyuan Sun Minhui Xue Bo Li Felix Juefei-Xu Chao Xie Li Li Yang Liu Jianjun Zhao et al. 2018. DeepMutation: Mutation testing of deep learning systems. In Proceedings of the 2018 IEEE 29th International Symposium on Software Reliability Engineering 100\u2013111.","DOI":"10.1109\/ISSRE.2018.00021"},{"key":"e_1_3_2_32_2","unstructured":"Midjourney. 2023. Midjourney. Retrieved from https:\/\/www.midjourney.com"},{"key":"e_1_3_2_33_2","unstructured":"OpenAI. 2023. DALL \\(\\cdot\\) E 2. Retrieved from https:\/\/openai.com\/research\/dall-e"},{"key":"e_1_3_2_34_2","unstructured":"PromptHero. 2023. OpenJourney. Retrieved from http:\/\/openjourney.art\/"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-demos.14"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00160"},{"key":"e_1_3_2_37_2","first-page":"8748","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning. PMLR, 8748\u20138763."},{"key":"e_1_3_2_38_2","first-page":"8821","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Ramesh Aditya","year":"2021","unstructured":"Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. In Proceedings of the International Conference on Machine Learning. PMLR, 8821\u20138831."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01042"},{"key":"e_1_3_2_40_2","first-page":"36479","article-title":"Photorealistic text-to-image diffusion models with deep language understanding","volume":"35","author":"Saharia Chitwan","year":"2022","unstructured":"Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 36479\u201336494.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_41_2","first-page":"1","article-title":"Improved techniques for training GANs","volume":"29","author":"Salimans Tim","year":"2016","unstructured":"Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training GANs. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 29, 1\u201310.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_42_2","unstructured":"Michael Saxon Fatima Jahara Mahsa Khoshnoodi Yujie Lu Aditya Sharma and William Yang Wang. 2024. Who evaluates the evaluations? Objectively scoring text-to-image prompt coherence metrics with T2IScoreScore (TS2). arXiv:2404.04251. Retrieved from https:\/\/arxiv.org\/abs\/2404.04251"},{"key":"e_1_3_2_43_2","unstructured":"Stability.ai. 2023. Stable Diffusion. Retrieved from https:\/\/stablediffusionweb.com"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00377"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00377"},{"key":"e_1_3_2_48_2","unstructured":"Tencent. 2025. Hunyuan. Retrieved from https:\/\/hunyuan.tencent.com\/"},{"key":"e_1_3_2_49_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Theis Lucas","year":"2016","unstructured":"Lucas Theis, A\u00e4ron van den Oord, and Matthias Bethge. 2016. A note on the evaluation of generative models. In Proceedings of the International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_50_2","unstructured":"Yongqiang Tian Shiqing Ma Ming Wen Yepang Liu Shing-Chi Cheung and Xiangyu Zhang. 2019. Testing deep learning models for image analysis using object-relevant metamorphic relations. arXiv:1909.03824. Retrieved from https:\/\/arxiv.org\/abs\/1909.03824"},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Yuchi Tian Kexin Pei Suman Jana and Baishakhi Ray. 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering 303\u2013314.","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_52_2","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Trabucco Brandon","year":"2024","unstructured":"Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. 2024. Effective data augmentation with diffusion models. In Proceedings of the International Conference on Learning Representations (ICLR). OpenReview.net."},{"key":"e_1_3_2_53_2","first-page":"026638212311837","article-title":"The dangers of generative artificial intelligence","volume":"40","author":"Tredinnick Luke","year":"2023","unstructured":"Luke Tredinnick and Claire Laybats. 2023. The dangers of generative artificial intelligence. Business Information Review 40 (2023), 02663821231183756.","journal-title":"Business Information Review"},{"key":"e_1_3_2_54_2","doi-asserted-by":"crossref","unstructured":"Yixin Wan Di Wu Haoran Wang and Kai-Wei Chang. 2024. The factuality tax of diversity-intervened text-to-image generation: Benchmark and fact-augmented intervention. arXiv:2407.00377. Retrieved from https:\/\/arxiv.org\/abs\/2407.00377","DOI":"10.18653\/v1\/2024.emnlp-main.513"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00189"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01758"},{"key":"e_1_3_2_57_2","unstructured":"Xianchao Wu. 2022. Creative painting with latent diffusion models. arXiv:2209.14697. Retrieved from https:\/\/arxiv.org\/abs\/2209.14697"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-023-00380-2"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3463747"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2020.2972266"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00143"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534389"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01971"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.629"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640329"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.01526"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.244"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3768581","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T14:46:54Z","timestamp":1781707614000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3768581"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,17]]},"references-count":66,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7,31]]}},"alternative-id":["10.1145\/3768581"],"URL":"https:\/\/doi.org\/10.1145\/3768581","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,17]]},"assertion":[{"value":"2025-01-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}