{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T03:12:33Z","timestamp":1769742753931,"version":"3.49.0"},"reference-count":95,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,8,26]],"date-time":"2024-08-26T00:00:00Z","timestamp":1724630400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>\n            Large Language Models (LLMs) have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing accuracy robustness, the computation efficiency of LLMs, which is of paramount importance due to often vast generation demands and real-time requirements, has surprisingly received little attention. In this article, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-art LLMs. By analyzing the working mechanism and implementation of 20,543 public-accessible LLMs, we observe a fundamental property in LLMs that could be manipulated in an adversarial manner to reduce computation efficiency significantly. Our interesting observation is that the output length determines the computation efficiency of LLMs instead of the input, where the output length depends on two factors: an often sufficiently large yet pessimistic pre-configured threshold controlling the max number of iterations and a runtime-generated end of sentence (EOS) token. Our key motivation is to generate test inputs that could sufficiently delay the generation of EOS such that LLMs would have to go through enough iterations to satisfy the pre-configured threshold. We present\n            <jats:monospace>LLMEffiChecker<\/jats:monospace>\n            , which can work under both white-box setting and black-box setting. In the white-box scenario,\n            <jats:monospace>LLMEffiChecker<\/jats:monospace>\n            develops a gradient-guided technique that searches for a minimal and unnoticeable perturbation at character-level, token-level, and structure-level. In the black-box scenario,\n            <jats:monospace>LLMEffiChecker<\/jats:monospace>\n            employs a causal inference-based approach to find critical tokens and similarly applies three levels of imperceptible perturbation to them. Both the white-box and black-box settings effectively delay the appearance of EOS, compelling these inputs to reach the naturally unreachable threshold. To demonstrate the effectiveness of\n            <jats:monospace>LLMEffiChecker<\/jats:monospace>\n            , we conduct a systematic evaluation on nine publicly available LLMs: Google T5, AllenAI WMT14, Helsinki-NLP translator, Facebook FairSeq, UNICAMP-DL translator, MarianMT, Google FLAN-T5, MBZUAI LaMini-GPT, and Salesforce CodeGen. Experimental results show that\n            <jats:monospace>LLMEffiChecker<\/jats:monospace>\n            can increase on average LLMs\u2019 response latency and energy consumption by 325% to 3,244% and 344% to 3,616%, respectively, by perturbing just one character or token in the input sentence. Our case study shows that inputs generated by\n            <jats:monospace>LLMEffiChecker<\/jats:monospace>\n            significantly affect the battery power in real-world mobile devices (i.e., drain more than 30 times battery power than normal inputs).\n          <\/jats:p>","DOI":"10.1145\/3664812","type":"journal-article","created":{"date-parts":[[2024,5,13]],"date-time":"2024-05-13T11:34:28Z","timestamp":1715600068000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["<tt>LLMEffiChecker<\/tt>\n            : Understanding and Testing Efficiency Degradation of Large Language Models"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-6344-2477","authenticated-orcid":false,"given":"Xiaoning","family":"Feng","sequence":"first","affiliation":[{"name":"College of Computer Science and Technology (College of Data Science), Taiyuan University of Technology, Taiyuan, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8779-6528","authenticated-orcid":false,"given":"Xiaohong","family":"Han","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology (College of Data Science), Taiyuan University of Technology, Taiyuan, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5035-3398","authenticated-orcid":false,"given":"Simin","family":"Chen","sequence":"additional","affiliation":[{"name":"Computer Science, The University of Texas at Dallas, Richardson, United States"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5338-7347","authenticated-orcid":false,"given":"Wei","family":"Yang","sequence":"additional","affiliation":[{"name":"University of Texas at Dallas, Richardson, United States"}]}],"member":"320","published-online":{"date-parts":[[2024,8,26]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"AllenAI. 2022. Retrieved from https:\/\/huggingface.co\/allenai\/wmt16-en-de-dist-12-1"},{"key":"e_1_3_2_3_2","article-title":"A general language assistant as a laboratory for alignment","volume":"2112","author":"Askell Amanda","year":"2021","unstructured":"Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A general language assistant as a laboratory for alignment. CoRR abs\/2112.00861 (2021).","journal-title":"CoRR"},{"key":"e_1_3_2_4_2","article-title":"Program synthesis with large language models","volume":"2108","author":"Austin Jacob","year":"2021","unstructured":"Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. CoRR abs\/2108.07732 (2021).","journal-title":"CoRR"},{"key":"e_1_3_2_5_2","volume-title":"6th International Conference on Learning Representations (ICLR\u201918)","author":"Belinkov Yonatan","year":"2018","unstructured":"Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In 6th International Conference on Learning Representations (ICLR\u201918). OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=BJ8vJebC-"},{"key":"e_1_3_2_6_2","first-page":"12841","volume-title":"Annual Conference on Neural Information Processing Systems (NeurIPS\u201919)","author":"Brendel Wieland","year":"2019","unstructured":"Wieland Brendel, Jonas Rauber, Matthias K\u00fcmmerer, Ivan Ustyuzhaninov, and Matthias Bethge. 2019. Accurate, reliable and fast robustness evaluation. In Annual Conference on Neural Information Processing Systems (NeurIPS\u201919), Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d\u2019Alch\u00e9-Buc, Emily B. Fox, and Roman Garnett (Eds.). 12841\u201312851. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2019\/hash\/885fe656777008c335ac96072a45be15-Abstract.html"},{"key":"e_1_3_2_7_2","unstructured":"Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems Annual Conference on Neural Information Processing Systems (NeurIPS\u201920) Hugo Larochelle Marc\u2019Aurelio Ranzato Raia Hadsell Maria-Florina Balcan and Hsuan-Tien Lin (Eds.). December 6-12 2020 virtual 2020."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.49"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3267446"},{"key":"e_1_3_2_10_2","unstructured":"Isaac Caswell and Bowen Liang. 2020. Recent Advances in Google Translate. Retrieved from https:\/\/ai.googleblog.com\/2020\/06\/recent-advances-in-google-translate.html"},{"key":"e_1_3_2_11_2","article-title":"A survey on evaluation of large language models","volume":"2307","author":"Chang Yupeng","year":"2023","unstructured":"Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2023. A survey on evaluation of large language models. CoRR abs\/2307.03109 (2023).","journal-title":"CoRR"},{"key":"e_1_3_2_12_2","article-title":"Evaluating large language models trained on code","volume":"2107","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond\u00e9 de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. CoRR abs\/2107.03374 (2021).","journal-title":"CoRR"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409733"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.02355"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3561158"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549102"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01493"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.ACL-LONG.395"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1609\/AAAI.V34I04.5767"},{"key":"e_1_3_2_20_2","article-title":"Scaling instruction-finetuned language models","volume":"2210","author":"Chung Hyung Won","year":"2022","unstructured":"Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction-finetuned language models. CoRR abs\/2210.11416 (2022).","journal-title":"CoRR"},{"key":"e_1_3_2_21_2","first-page":"653","volume-title":"27th International Conference on Computational Linguistics (COLING\u201918)","author":"Ebrahimi Javid","year":"2018","unstructured":"Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018. On adversarial examples for character-level neural machine translation. In 27th International Conference on Computational Linguistics (COLING\u201918), Emily M. Bender, Leon Derczynski, and Pierre Isabelle (Eds.). Association for Computational Linguistics, 653\u2013663. Retrieved from https:\/\/aclanthology.org\/C18-1055\/"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P18-2006"},{"key":"e_1_3_2_23_2","volume-title":"International Conference on Language Resources and Evaluation (LREC\u201910)","author":"Eisele Andreas","year":"2010","unstructured":"Andreas Eisele and Yu Chen. 2010. MultiUN: A multilingual corpus from United Nation documents. In International Conference on Language Resources and Evaluation (LREC\u201910), Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner, and Daniel Tapias (Eds.). European Language Resources Association. Retrieved from http:\/\/www.lrec-conf.org\/proceedings\/lrec2010\/summaries\/686.html"},{"key":"e_1_3_2_24_2","article-title":"Detecting adversarial samples from artifacts","volume":"1703","author":"Feinman Reuben","year":"2017","unstructured":"Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. 2017. Detecting adversarial samples from artifacts. CoRR abs\/1703.00410 (2017).","journal-title":"CoRR"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.194"},{"key":"e_1_3_2_26_2","volume-title":"6th International Conference on Learning Representations (ICLR\u201918)","author":"Fojo Daniel","year":"2018","unstructured":"Daniel Fojo, V\u00edctor Campos, and Xavier Gir\u00f3-i-Nieto. 2018. Comparing fixed and adaptive computation time for recurrent neural networks. In 6th International Conference on Learning Representations (ICLR\u201918). OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=SkZq3vyDf"},{"key":"e_1_3_2_27_2","volume-title":"11th International Conference on Learning Representations (ICLR\u201923)","author":"Fried Daniel","year":"2023","unstructured":"Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A generative model for code infilling and synthesis. In 11th International Conference on Learning Representations (ICLR\u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=hQwb-lbM6EL"},{"key":"e_1_3_2_28_2","volume-title":"12th International Conference on Learning Representations","author":"Gao Kuofeng","year":"2024","unstructured":"Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, and Wei Liu. 2024. Inducing high energy-latency of large vision-language models with verbose images. In 12th International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=BteuUysuXX"},{"issue":"1","key":"e_1_3_2_29_2","first-page":"9","article-title":"A review of ChatGPT AI\u2019s impact on several business sectors","volume":"1","author":"George A. Shaji","year":"2023","unstructured":"A. Shaji George and A. S. Hovan George. 2023. A review of ChatGPT AI\u2019s impact on several business sectors. Partn. Univ. Int. Innov. J. 1, 1 (2023), 9\u201323.","journal-title":"Partn. Univ. Int. Innov. J."},{"key":"e_1_3_2_30_2","unstructured":"Google. 2022. Retrieved from https:\/\/huggingface.co\/t5-small"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377812.3382162"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01427"},{"key":"e_1_3_2_33_2","unstructured":"Pinjia He. 2022. RobustNLP Library. Retrieved from https:\/\/github.com\/RobustNLP\/TestTranslation"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00047"},{"key":"e_1_3_2_36_2","unstructured":"Helsinki-NLP. 2022. Retrieved from https:\/\/huggingface.co\/Helsinki-NLP\/opus-mt-en-de"},{"key":"e_1_3_2_37_2","article-title":"Training compute-optimal large language models","volume":"2203","author":"Hoffmann Jordan","year":"2022","unstructured":"Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. CoRR abs\/2203.15556 (2022).","journal-title":"CoRR"},{"key":"e_1_3_2_38_2","volume-title":"9th International Conference on Learning Representations (ICLR\u201921)","author":"Hong Sanghyun","year":"2021","unstructured":"Sanghyun Hong, Yigitcan Kaya, Ionut-Vlad Modoranu, and Tudor Dumitras. 2021. A panda? No, it\u2019s a sloth: Slowdown attacks on adaptive multi-exit neural network inference. In 9th International Conference on Learning Representations (ICLR\u201921). OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=9xC2tWEwBD"},{"key":"e_1_3_2_39_2","article-title":"MobileNets: Efficient convolutional neural networks for mobile vision applications","volume":"1704","author":"Howard Andrew G.","year":"2017","unstructured":"Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient convolutional neural networks for mobile vision applications. CoRR abs\/1704.04861 (2017).","journal-title":"CoRR"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-7138-7"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1609\/AAAI.V34I05.6311"},{"key":"e_1_3_2_42_2","unstructured":"Mintong Kang Nezihe Merve G\u00fcrel Ning Yu Dawn Song and Bo Li. 2024. C-rag: Certified generation risks for retrievalaugmented language models. arXiv preprint arXiv:2402.03181."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00194"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23138"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.EMNLP-MAIN.500"},{"key":"e_1_3_2_47_2","unstructured":"Yujia Li David H. Choi Junyoung Chung Nate Kushman Julian Schrittwieser R\u2019emi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d\u2019Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J. Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-level code generation with alphacode. CoRR abs\/2203.07814."},{"key":"e_1_3_2_48_2","unstructured":"Zeming Lin Halil Akin Roshan Rao Brian Hie Zhongkai Zhu Wenting Lu Allan dos Santos Costa Maryam Fazel-Zarandi Tom Sercu Sal Candido and Alexander Rives. 2022. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv 2022:500902 2022."},{"key":"e_1_3_2_49_2","volume-title":"Annual Conference on Neural Information Processing Systems (NeurIPS\u201923)","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In Annual Conference on Neural Information Processing Systems (NeurIPS\u201923), Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2023\/hash\/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1162\/TACL_A_00343"},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Yiheng Liu Tianle Han Siyuan Ma Jiayue Zhang Yuanyuan Yang Jiaming Tian Hao He Antong Li Mengshen He Zhengliang Liu Zihao Wu Lin Zhao Dajiang Zhu Xiang Li Ning Qiang Dinggang Shen Tianming Liu and Bao Ge. 2023. Summary of chatgpt-related research and perspective towards the future of large language models. Meta-Radiology 1 2 (2023) 100017.","DOI":"10.1016\/j.metrad.2023.100017"},{"key":"e_1_3_2_52_2","first-page":"833","volume-title":"5th Conference on Machine Translation (WMT@EMNLP\u201920)","author":"Lopes Alexandre","year":"2020","unstructured":"Alexandre Lopes, Rodrigo Frassetto Nogueira, Roberto de Alencar Lotufo, and H\u00e9lio Pedrini. 2020. Lite training strategies for Portuguese-English and English-Portuguese translation. In 5th Conference on Machine Translation (WMT@EMNLP\u201920), Lo\u00efc Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-Juss\u00e0, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, Andr\u00e9 Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri (Eds.). Association for Computational Linguistics, 833\u2013840. Retrieved from https:\/\/aclanthology.org\/2020.wmt-1.90\/"},{"key":"e_1_3_2_53_2","unstructured":"MarianMT. 2023. Marianmt: translation_en-zh. https:\/\/huggingface.co\/DDDSSS\/translation_en-zh"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1186\/S13040-023-00339-9"},{"key":"e_1_3_2_55_2","article-title":"WebGPT: Browser-assisted question-answering with human feedback","volume":"2112","author":"Nakano Reiichiro","year":"2021","unstructured":"Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. WebGPT: Browser-assisted question-answering with human feedback. CoRR abs\/2112.09332 (2021).","journal-title":"CoRR"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/W19-5333"},{"key":"e_1_3_2_57_2","volume-title":"11th International Conference on Learning Representations (ICLR\u201923)","author":"Nijkamp Erik","year":"2023","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. In 11th International Conference on Learning Representations (ICLR\u201923). OpenReview.net. Retrieved from https:\/\/openreview.net\/pdf?id=iaYcJKpY2B_"},{"key":"e_1_3_2_58_2","volume-title":"Annual Conference on Neural Information Processing Systems 2022 (NeurIPS\u201922)","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Annual Conference on Neural Information Processing Systems 2022 (NeurIPS\u201922), Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). Retrieved from http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/b1efde53be364a73914f58805a001731-Abstract-Conference.html"},{"key":"e_1_3_2_59_2","unstructured":"Papers with Code. 2024. Code generation on mbpp. https:\/\/paperswithcode.com\/sota\/code-generation-on-mbpp"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3361566"},{"key":"e_1_3_2_61_2","unstructured":"Jeff Pitman. 2021. Google Translate: One billion installs one billion stories. Retrieved from https:\/\/blog.google\/products\/translate\/one-billion-installs\/"},{"key":"e_1_3_2_62_2","unstructured":"Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1 8 (2019) 9."},{"key":"e_1_3_2_63_2","first-page":"140:1\u2013140:67","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21 (2020), 140:1\u2013140:67. Retrieved from http:\/\/jmlr.org\/papers\/v21\/20-074.html","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_64_2","doi-asserted-by":"crossref","unstructured":"Partha Pratim Ray. 2023. Chatgpt: A comprehensive review on background applications key challenges bias ethics limitations and future scope. Internet of Things and Cyber-Physical Systems 3 (2023) 121\u2013154.","DOI":"10.1016\/j.iotcps.2023.04.003"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P19-1103"},{"key":"e_1_3_2_66_2","article-title":"Code Llama: Open foundation models for code","volume":"2308","author":"Rozi\u00e8re Baptiste","year":"2023","unstructured":"Baptiste Rozi\u00e8re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J\u00e9r\u00e9my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D\u00e9fossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code Llama: Open foundation models for code. CoRR abs\/2308.12950 (2023).","journal-title":"CoRR"},{"key":"e_1_3_2_67_2","volume-title":"10th International Conference on Learning Representations (ICLR\u201922)","author":"Sanh Victor","year":"2022","unstructured":"Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault F\u00e9vry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. In 10th International Conference on Learning Representations (ICLR\u201922). OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=9Vrb9D0WI4"},{"key":"e_1_3_2_68_2","article-title":"BLOOM: A 176B-parameter open-access multilingual language model","volume":"2211","author":"Scao Teven Le","year":"2022","unstructured":"Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagn\u00e9, Alexandra Sasha Luccioni, Fran\u00e7ois Yvon, Matthias Gall\u00e9, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Beno\u00eet Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Lauren\u00e7on, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, et\u00a0al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. CoRR abs\/2211.05100 (2022).","journal-title":"CoRR"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.21437\/INTERSPEECH.2005-24"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_71_2","first-page":"3104","volume-title":"Annual Conference on Neural Information Processing Systems (NeurIPS\u201914)","author":"Sutskever Ilya","year":"2014","unstructured":"Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Annual Conference on Neural Information Processing Systems (NeurIPS\u201914), Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (Eds.). 3104\u20133112. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2014\/hash\/a14ac55a4f27472c5d894ec1c3c743d2-Abstract.html"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_73_2","article-title":"LLaMA: Open and efficient foundation language models","volume":"2302","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur\u00e9lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. CoRR abs\/2302.13971 (2023).","journal-title":"CoRR"},{"key":"e_1_3_2_74_2","article-title":"Llama 2: Open foundation and fine-tuned chat models","volume":"2307","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aur\u00e9lien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR abs\/2307.09288 (2023).","journal-title":"CoRR"},{"key":"e_1_3_2_75_2","unstructured":"Barak Turovsky. 2016. Ten years of Google Translate. Retrieved from https:\/\/www.blog.google\/products\/translate\/ten-years-of-google-translate\/"},{"key":"e_1_3_2_76_2","first-page":"5998","volume-title":"Annual Conference on Neural Information Processing Systems (NeurIPS\u201917)","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Annual Conference on Neural Information Processing Systems (NeurIPS\u201917), Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 5998\u20136008. Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468625"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380379"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01261-8_25"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.EMNLP-MAIN.685"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.888628"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_3_2_84_2","first-page":"944","volume-title":"18th Conference of the European Chapter of the Association for Computational Linguistics (EACL\u201924)","author":"Wu Minghao","year":"2024","unstructured":"Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. 2024. LaMini-LM: A diverse herd of distilled models from large-scale instructions. In 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL\u201924), Yvette Graham and Matthew Purver (Eds.). Association for Computational Linguistics, 944\u2013964. Retrieved from https:\/\/aclanthology.org\/2024.eacl-long.57"},{"key":"e_1_3_2_85_2","series-title":"Proceedings of Machine Learning Research","first-page":"38087","volume-title":"International Conference on Machine Learning (ICML\u201923)","volume":"202","author":"Xiao Guangxuan","year":"2023","unstructured":"Guangxuan Xiao, Ji Lin, Micka\u00ebl Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML\u201923)(Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 38087\u201338099. Retrieved from https:\/\/proceedings.mlr.press\/v202\/xiao23c.html"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2018.23198"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.NEUCOM.2018.09.006"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/3444944"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.ACL-MAIN.540"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P19-1472"},{"key":"e_1_3_2_91_2","article-title":"OPT: Open pre-trained transformer language models","volume":"2205","author":"Zhang Susan","year":"2022","unstructured":"Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open pre-trained transformer language models. CoRR abs\/2205.01068 (2022).","journal-title":"CoRR"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.ACL-LONG.153"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00716"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.AIOPEN.2021.12.003"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599790"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.ACL-MAIN.319"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3664812","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3664812","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:17:29Z","timestamp":1750295849000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3664812"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,26]]},"references-count":95,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3664812"],"URL":"https:\/\/doi.org\/10.1145\/3664812","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,26]]},"assertion":[{"value":"2023-12-09","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}