{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:41:45Z","timestamp":1787330505169,"version":"build-2736575974"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"9","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61972373"],"award-info":[{"award-number":["61972373"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"JST CRONOS","award":["JPMJCS24K8"],"award-info":[{"award-number":["JPMJCS24K8"]}]},{"name":"JSPS KAKENHI","award":["JP21H04877, JP23H03372, and JP24K02920"],"award-info":[{"award-number":["JP21H04877, JP23H03372, and JP24K02920"]}]},{"name":"Canada CIFAR AI Chairs Program, the Natural Sciences and Engineering Research Council of Canada, and the Autoware Foundation"},{"name":"MUST Faculty Research Grant"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>In the realm of deep learning, a variety of neuron coverage criteria for Deep Neural Networks (DNNs) have been devised to effectively assess the quality of test suites and facilitate the generation of test inputs. Recently proposed coverage criteria, incorporating representation distribution and causal relationships, have infused fresh vitality into this field. However, the focus of previous works is primarily on Convolutional Neural Networks for computer vision, leading to a research gap in exploring coverage testing for language models. Concurrently, with the rise of large language models, transformer-based language models have become increasingly dominant, and numerous ones have sprouted. Therefore, the effectiveness of coverage criteria in transformer-based language tasks, especially with the introduction of novel criteria, remains an unresolved open problem. To tackle it, this study examines these concerns by evaluating a wide range of criteria, including four well-established ones and two state-of-the-art criteria, across three types of transformer-based models: encoder-only, decoder-only, and encoder\u2013decoder models.<\/jats:p>\n                  <jats:p>Building on previous research, we conduct a comprehensive evaluation across three key areas: regarding test suite properties, (1) Error-revealing capability, i.e., sensitivity to adversarial examples; (2) Diversity, i.e., distribution diversity and sample fairness (category diversity); and regarding test suite generation, (3) Input generation guidance, i.e., the ability to guide the generation of more valuable samples. The experimental results demonstrate that the impact of coverage criteria is multifaceted. For the error-revealing capability of test suites, the additional coverage for erroneous samples over noise samples is only 0.32%. In terms of distribution diversity and sample fairness, 26 and 30 cases, respectively, out of 33 configurations are effectively evaluated. Additionally, incorporating neuron-wise coverage guidance during test suite generation slightly increases the production of adversarial samples by 4.56%.<\/jats:p>\n                  <jats:p>In conclusion, while current coverage criteria can act as an antidote for assessing simple diversity, they remain largely a placebo for the core task of revealing adversarial errors, particularly when relying on individual criterion. Consequently, their practical application requires carefully evaluating the tradeoff between computational overhead and potential benefits given the massive scale of Transformers. However, this low cost-effectiveness ultimately highlights the urgent need to explore and develop more robust and efficient criteria designed specifically for Transformer-based models.<\/jats:p>","DOI":"10.1145\/3787101","type":"journal-article","created":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T11:23:27Z","timestamp":1767612207000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Antidote or Placebo? Unraveling the Efficacy of Neuron Coverage Criteria on Testing Transformer-based Language Models"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-5063-8209","authenticated-orcid":false,"given":"Xiaoning","family":"Ren","sequence":"first","affiliation":[{"name":"School of Computer Science and Technology, University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7093-9781","authenticated-orcid":false,"given":"Jiayang","family":"Song","sequence":"additional","affiliation":[{"name":"Macau University of Science and Technology, Macau, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6031-762X","authenticated-orcid":false,"given":"Chongyang","family":"Liu","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-1665-9809","authenticated-orcid":false,"given":"Jie","family":"Li","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8621-2420","authenticated-orcid":false,"given":"Lei","family":"Ma","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Tokyo, Japan and University of Alberta, Edmonton, Alberta, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2979-7151","authenticated-orcid":false,"given":"Yinxing","family":"Xue","sequence":"additional","affiliation":[{"name":"Chinese Academy of Sciences, Institute of AI for Industries, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,21]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1017\/9781316771273"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2019.100203"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","unstructured":"Samuel R. Bowman Gabor Angeli Christopher Potts and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. arXiv:1508.05326. Retrieved from https:\/\/arxiv.org\/abs\/1508.05326","DOI":"10.18653\/v1\/D15-1075"},{"key":"e_1_3_2_5_2","unstructured":"Wieland Brendel and Matthias Bethge. 2019. Approximating cnns with bag-of-local-features models works surprisingly well on imagenet. arXiv:1904.00760. Retrieved from https:\/\/arxiv.org\/abs\/1904.00760"},{"key":"e_1_3_2_6_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_7_2","unstructured":"S\u00e9bastien Bubeck Varun Chandrasekaran Ronen Eldan Johannes Gehrke Eric Horvitz Ece Kamar Peter Lee Yin Tat Lee Yuanzhi Li Scott Lundberg et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712. Retrieved from https:\/\/arxiv.org\/abs\/2303.12712"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00059"},{"issue":"240","key":"e_1_3_2_9_2","first-page":"1","article-title":"Palm: Scaling language modeling with pathways","volume":"24","author":"Chowdhery Aakanksha","year":"2023","unstructured":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1\u2013113.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_10_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. Retrieved from https:\/\/arxiv.org\/abs\/1810.04805"},{"key":"e_1_3_2_11_2","unstructured":"Yizhen Dong Peixin Zhang Jingyi Wang Shuang Liu Jun Sun Jianye Hao Xinyu Wang Li Wang Jin Song Dong and Dai Ting. 2019. There is limited correlation between coverage and robustness for deep neural networks. arXiv:1911.05904. Retrieved from https:\/\/arxiv.org\/abs\/1911.05904"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_2_13_2","unstructured":"Xiaoning Du Xiaofei Xie Yi Li Lei Ma Jianjun Zhao and Yang Liu. 2018. Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv:1812.05339. Retrieved from https:\/\/arxiv.org\/abs\/1812.05339"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2022.106874"},{"key":"e_1_3_2_15_2","unstructured":"Robert Geirhos Patricia Rubisch Claudio Michaelis Matthias Bethge Felix A. Wichmann and Wieland Brendel. 2018. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv:1811.12231. Retrieved from https:\/\/arxiv.org\/abs\/1811.12231"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380391"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409754"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00153"},{"key":"e_1_3_2_19_2","unstructured":"Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li Fang Wang and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv:1909.10351. Retrieved from https:\/\/arxiv.org\/abs\/1909.10351"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_21_2","unstructured":"Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv:1909.11942. Retrieved from https:\/\/arxiv.org\/abs\/1909.11942"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2021.05.103"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER.2019.00031"},{"key":"e_1_3_2_24_2","unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. Retrieved from https:\/\/arxiv.org\/abs\/1907.11692"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000085"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2974682"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668044"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2019.101557"},{"key":"e_1_3_2_30_2","unstructured":"OpenAI. 2023. ChatGPT. Retrieved July 26 2023 from https:\/\/openai.com\/chatgpt"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.52202\/068431-2011"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_33_2","unstructured":"Alec Radford Karthik Narasimhan Tim Salimans and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-training. Retrieved from https:\/\/www.mikecaptain.com\/resources\/pdf\/GPT-1.pdf"},{"issue":"8","key":"e_1_3_2_34_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856"},{"key":"e_1_3_2_36_2","unstructured":"Victor Sanh Lysandre Debut Julien Chaumond and Thomas Wolf. 2019. DistilBERT a distilled version of BERT: Smaller faster cheaper and lighter. arXiv:1910.01108. Retrieved from https:\/\/arxiv.org\/abs\/1910.01108"},{"key":"e_1_3_2_37_2","unstructured":"Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv:1803.04792. Retrieved from https:\/\/arxiv.org\/abs\/1803.04792"},{"key":"e_1_3_2_38_2","unstructured":"Yuchi Tian Ziyuan Zhong Vicente Ordonez and Baishakhi Ray. 2019. Testing Deep Neural Network based Image Classifiers. arXiv:1905.07831. Retrieved from https:\/\/arxiv.org\/abs\/1905.07831"},{"key":"e_1_3_2_39_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_2_40_2","first-page":"6000","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017), 6000\u20136010.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.00013"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSSR61934.2024.00035"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490489"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2561"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409671"},{"key":"e_1_3_2_46_2","first-page":"5753","article-title":"Xlnet: Generalized autoregressive pretraining for language understanding","volume":"32","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R. Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems 32 (2019), 5753\u20135763.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00056"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00107"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00131"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3787101","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:09:34Z","timestamp":1787328574000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3787101"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,21]]},"references-count":48,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3787101"],"URL":"https:\/\/doi.org\/10.1145\/3787101","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,21]]},"assertion":[{"value":"2024-09-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-21","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}