{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T15:20:30Z","timestamp":1787325630391,"version":"build-2736575974"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"6","funder":[{"name":"National Key Research and Development Program of China","award":["2024YFF0908001"],"award-info":[{"award-number":["2024YFF0908001"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["U24A20337, 62372228"],"award-info":[{"award-number":["U24A20337, 62372228"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"crossref","award":["14380029"],"award-info":[{"award-number":["14380029"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Shenzhen-Hong Kong-Macau Technology Research Programme","award":["SGDX20230821091559018"],"award-info":[{"award-number":["SGDX20230821091559018"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>The openness of crowdsourced testing introduces diversity in testing results. However, it also leads to a large volume of test reports, many of which highlight the same recurring issues. While these reports provide valuable feedback, their redundancy makes it inefficient for developers to review the reports and identify bugs. Crowdsourced test report clustering has been proposed to mitigate this problem, allowing developers to focus only on the representative reports from each cluster. However, existing methods primarily rely on embedding features extracted from reports for clustering, which limits their ability to generate accurate and interpretable clusters due to a lack of deeper semantic understanding of the reports.<\/jats:p>\n                  <jats:p>\n                    To address the aforementioned challenge, we propose\n                    <jats:sc>LLMCluster<\/jats:sc>\n                    , a novel method for crowdsourced test report clustering based on Large Language Models (LLMs).\n                    <jats:sc>LLMCluster<\/jats:sc>\n                    employs an iterative clustering strategy. In each iteration,\n                    <jats:sc>LLMCluster<\/jats:sc>\n                    processes a subset of reports by instructing the LLM to disregard surface-level variations in expression, analyze the core issue in each report, and group reports addressing the same issue into new or existing clusters. After the iterative clustering process,\n                    <jats:sc>LLMCluster<\/jats:sc>\n                    applies correction algorithms to ensure the completeness and validity of the clustering result. Finally,\n                    <jats:sc>LLMCluster<\/jats:sc>\n                    utilizes the LLM to generate concise summaries for each cluster, making the results more intuitive and interpretable. Experimental results show that\n                    <jats:sc>LLMCluster<\/jats:sc>\n                    outperforms state-of-the-art methods across six commonly used clustering evaluation metrics. Additionally, the cluster summaries generated by\n                    <jats:sc>LLMCluster<\/jats:sc>\n                    semantically align well with manually written summaries.\n                  <\/jats:p>","DOI":"10.1145\/3765756","type":"journal-article","created":{"date-parts":[[2025,9,4]],"date-time":"2025-09-04T15:36:30Z","timestamp":1757000190000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["LLM-based Crowdsourced Test Report Clustering"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-9227-3824","authenticated-orcid":false,"given":"Yuchen","family":"Ling","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4640-8637","authenticated-orcid":false,"given":"Shengcheng","family":"Yu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9930-7111","authenticated-orcid":false,"given":"Chunrong","family":"Fang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-1701-7366","authenticated-orcid":false,"given":"Quan","family":"Zhou","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9592-7022","authenticated-orcid":false,"given":"Zhenyu","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,5,13]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643661.3643953"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106363"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS-C55045.2021.00018"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3641289"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS54544.2021.00098"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3663529.3663801"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786862"},{"key":"e_1_3_3_9_2","first-page":"202","volume-title":"2016 31st IEEE\/ACM International Conference on Automated Software Engineering","author":"Feng Yang","year":"2016","unstructured":"Yang Feng, James A. Jones, Zhenyu Chen, and Chunrong Fang. 2016. Multi-objective test report prioritization using image understanding. In 2016 31st IEEE\/ACM International Conference on Automated Software Engineering, 202\u2013213."},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9618-5"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00096"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS51102.2020.00062"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01908075"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106164"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00194"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3226238"},{"key":"e_1_3_3_18_2","first-page":"22199","volume-title":"Advances in Neural Information Processing Systems","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 22199\u201322213. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","unstructured":"Aobo Kong Shiwan Zhao Hao Chen Qicheng Li Yong Qin Ruiqi Sun Xin Zhou Enzhi Wang and Xiaohang Dong. 2024. Better Zero-Shot Reasoning with Role-Play Prompting. DOI: 10.48550\/arXiv.2308.07702","DOI":"10.48550\/arXiv.2308.07702"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583780.3614744"},{"key":"e_1_3_3_21_2","volume-title":"European Conference on Information Systems","author":"Leicht Niklas","year":"2016","unstructured":"Niklas Leicht, Nicolas Knop, Christoph M\u00fcller-Bloch, and Jan Marco Leimeister. 2016. When is crowdsourcing advantageous? The case of crowdsourced software testing. In European Conference on Information Systems, Research Paper 60. Retrieved from https:\/\/api.semanticscholar.org\/CorpusID:9384231"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3339000"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1142\/S021819402250053X"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660767"},{"key":"e_1_3_3_25_2","first-page":"74","article-title":"ROUGE: A package for automatic evaluation of summaries","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, 74\u201381. Retrieved from https:\/\/aclanthology.org\/W04-1013","journal-title":"Text Summarization Branches Out"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2024.107629"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3017514"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","unstructured":"Xun Liu and Zhengwei Ni. 2024. Prompt Framework for Role-playing: Generation and Evaluation. DOI: 10.48550\/arXiv.2406.00627","DOI":"10.48550\/arXiv.2406.00627"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00119"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","unstructured":"Zhe Liu Chunyang Chen Junjie Wang Mengzhuo Chen Boyu Wu Xing Che Dandan Wang and Qing Wang. 2023. Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing. DOI: 10.48550\/arXiv.2305.09434","DOI":"10.48550\/arXiv.2305.09434"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3501709.3544280"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.09.015"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1177\/0013164484441003"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3661167.3661271"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.378"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSREW60843.2023.00058"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.32"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/NLBSE59153.2023.00018"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/1806799.1806811"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2102.02503"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106668"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1953024"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2961111.2962584"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2017.8"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2019.03.003"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970300"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-16-9643-5"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","unstructured":"Jason Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph Sebastian Borgeaud Dani Yogatama Maarten Bosma Denny Zhou Donald Metzler et al. 2022. Emergent Abilities of Large Language Models. DOI: 10.48550\/arXiv.2206.07682","DOI":"10.48550\/arXiv.2206.07682"},{"key":"e_1_3_3_50_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970312"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSREW60843.2023.00040"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","unstructured":"Jules White Quchen Fu Sam Hays Michael Sandborn Carlos Olea Henry Gilbert Ashraf Elnashar Jesse Spencer-Smith and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. DOI: 10.48550\/arXiv.2302.11382","DOI":"10.48550\/arXiv.2302.11382"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/AST58925.2023.00005"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3128726"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00090"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS60937.2023.00029"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3285787"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660776"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2017.28"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","unstructured":"Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. DOI: 10.48550\/arXiv.1904.09675","DOI":"10.48550\/arXiv.1904.09675"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3765756","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T02:33:30Z","timestamp":1778726010000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3765756"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,13]]},"references-count":60,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3765756"],"URL":"https:\/\/doi.org\/10.1145\/3765756","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,13]]},"assertion":[{"value":"2025-02-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-10","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}