{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T12:39:24Z","timestamp":1784205564251,"version":"3.55.0"},"reference-count":513,"publisher":"Association for Computing Machinery (ACM)","issue":"5","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2025,10,31]]},"abstract":"<jats:p>Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multimodal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to provide not only a systematic survey but also a quick, comprehensive reference for the researchers and practitioners to draw insights from extensive, informative summaries of the existing works to advance the LLM research.<\/jats:p>","DOI":"10.1145\/3744746","type":"journal-article","created":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:20:44Z","timestamp":1750281644000},"page":"1-72","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":627,"title":["A Comprehensive Overview of Large Language Models"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4962-903X","authenticated-orcid":false,"given":"Humza","family":"Naveed","sequence":"first","affiliation":[{"name":"The University of Sydney, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5127-527X","authenticated-orcid":false,"given":"Asad Ullah","family":"Khan","sequence":"additional","affiliation":[{"name":"University of Engineering and Technology, Lahore, Pakistan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9958-180X","authenticated-orcid":false,"given":"Shi","family":"Qiu","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4374-0888","authenticated-orcid":false,"given":"Muhammad","family":"Saqib","sequence":"additional","affiliation":[{"name":"University of Technology Sydney, Sydney, Australia  and Commonwealth Scientific and Industrial Research Organisation, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0692-8411","authenticated-orcid":false,"given":"Saeed","family":"Anwar","sequence":"additional","affiliation":[{"name":"King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia and SDAIAKFUPM Joint Research Center for Artificial Intelligence, Dhahran, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2059-7206","authenticated-orcid":false,"given":"Muhammad","family":"Usman","sequence":"additional","affiliation":[{"name":"University of Ontario Institute of Technology, Oshawa, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3406-673X","authenticated-orcid":false,"given":"Naveed","family":"Akhtar","sequence":"additional","affiliation":[{"name":"University of Melbourne VCCC, Parkville, Australia and The University of Western Australia, Perth, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9343-9535","authenticated-orcid":false,"given":"Nick","family":"Barnes","sequence":"additional","affiliation":[{"name":"Australian National University, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5206-3842","authenticated-orcid":false,"given":"Ajmal","family":"Mian","sequence":"additional","affiliation":[{"name":"The University of Western Australia - Perth Campus, Perth, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,8,18]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-86523-8_41"},{"key":"e_1_3_2_3_2","article-title":"SuperGLUE: A stickier benchmark for general-purpose language understanding systems","volume":"32","author":"Wang A.","year":"2019","unstructured":"A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_4_2","unstructured":"D. Adiwardana M.-T. Luong D. R. So J. Hall N. Fiedel R. Thoppilan Z. Yang A. Kulshreshtha G. Nemade Y. Lu et al. 2020. Towards a human-like open-domain chatbot. arXiv:2001.09977. Retrieved from https:\/\/arxiv.org\/abs\/2001.09977"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1162\/daed_a_01909"},{"issue":"8","key":"e_1_3_2_6_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford A.","year":"2019","unstructured":"A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_2_7_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown T.","year":"2020","unstructured":"T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 33, 1877\u20131901.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_8_2","unstructured":"J. Devlin M.-W. Chang K. Lee and K. Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. Retrieved from https:\/\/arxiv.org\/abs\/1810.04805"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1202"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"M. Lewis Y. Liu N. Goyal M. Ghazvininejad A. Mohamed O. Levy V. Stoyanov and L. Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation translation and comprehension. arXiv:1910.13461. Retrieved from https:\/\/arxiv.org\/abs\/1910.13461","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856"},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"L. Xue N. Constant A. Roberts M. Kale R. Al-Rfou A. Siddhant A. Barua and C. Raffel. 2020. MT5: A massively multilingual pre-trained text-to-text transformer. arXiv:2010.11934. Retrieved from https:\/\/arxiv.org\/abs\/2010.11934","DOI":"10.18653\/v1\/2021.naacl-main.41"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2021.12.003"},{"key":"e_1_3_2_14_2","unstructured":"T. L. Scao A. Fan C. Akiki E. Pavlick S. Ili\u0107 D. Hesslow R. Castagn\u00e9 A. S. Luccioni F. Yvon M. Gall\u00e9 et al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100. Retrieved from https:\/\/arxiv.org\/abs\/2211.05100"},{"key":"e_1_3_2_15_2","unstructured":"S. Zhang S. Roller N. Goyal M. Artetxe M. Chen S. Chen C. Dewan M. Diab X. Li X. V. Lin et al. 2022. OPT: Open pre-trained transformer language models. arXiv:2205.01068. Retrieved from https:\/\/arxiv.org\/abs\/2205.01068"},{"key":"e_1_3_2_16_2","unstructured":"A. Chowdhery S. Narang J. Devlin M. Bosma G. Mishra A. Roberts P. Barham H. W. Chung C. Sutton S. Gehrmann et al. 2022. PaLM: Scaling language modeling with pathways. arXiv:2204.02311. Retrieved from https:\/\/arxiv.org\/abs\/2204.02311"},{"key":"e_1_3_2_17_2","unstructured":"H. W. Chung L. Hou S. Longpre B. Zoph Y. Tay W. Fedus E. Li X. Wang M. Dehghani S. Brahma et al. 2022. Scaling instruction-finetuned language models. arXiv:2210.11416. Retrieved from https:\/\/arxiv.org\/abs\/2210.11416"},{"key":"e_1_3_2_18_2","unstructured":"V. Sanh A. Webson C. Raffel S. H. Bach L. Sutawika Z. Alyafeai A. Chaffin A. Stiegler T. L. Scao A. Raja et al. 2021. Multitask prompted training enables zero-shot task generalization. arXiv:2110.08207. Retrieved from https:\/\/arxiv.org\/abs\/2110.08207"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.340"},{"key":"e_1_3_2_20_2","unstructured":"Y. Wang Y. Kordi S. Mishra A. Liu N. A. Smith D. Khashabi and H. Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv:2212.10560. Retrieved from https:\/\/arxiv.org\/abs\/2212.10560"},{"key":"e_1_3_2_21_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang L.","year":"2022","unstructured":"L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. 2022. Training language models to follow instructions with human feedback. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 27730\u201327744.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_22_2","unstructured":"H. Touvron L. Martin K. Stone P. Albert A. Almahairi Y. Babaei N. Bashlykov S. Batra P. Bhargava S. Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_2_23_2","unstructured":"J. Wei Y. Tay R. Bommasani C. Raffel B. Zoph S. Borgeaud D. Yogatama M. Bosma D. Zhou D. Metzler et al. 2022. Emergent abilities of large language models. arXiv:2206.07682. Retrieved from https:\/\/arxiv.org\/abs\/2206.07682"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-023-01659-w"},{"key":"e_1_3_2_25_2","unstructured":"D. A. Boiko R. MacKnight and G. Gomes. 2023. Emergent autonomous scientific research capabilities of large language models. arXiv:2304.05332. Retrieved from https:\/\/arxiv.org\/abs\/2304.05332"},{"key":"e_1_3_2_26_2","unstructured":"G. Izacard P. Lewis M. Lomeli L. Hosseini F. Petroni T. Schick J. Dwivedi-Yu A. Joulin S. Riedel and E. Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv:2208.03299. Retrieved from https:\/\/arxiv.org\/abs\/2208.03299"},{"key":"e_1_3_2_27_2","unstructured":"D. Driess F. Xia M. S. Sajjadi C. Lynch A. Chowdhery B. Ichter A. Wahid J. Tompson Q. Vuong T. Yu et al. 2023. PaLM-E: An embodied multimodal language model. arXiv:2303.03378. Retrieved from https:\/\/arxiv.org\/abs\/2303.03378"},{"key":"e_1_3_2_28_2","unstructured":"A. Parisi Y. Zhao and N. Fiedel. 2022. TALM: Tool augmented language models. arXiv:2205.12255. Retrieved from https:\/\/arxiv.org\/abs\/2205.12255"},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","unstructured":"B. Zhang and H. Soh. 2023. Large language models as zero-shot human models for human-robot interaction. arXiv:2303.03548. Retrieved from https:\/\/arxiv.org\/abs\/2303.03548","DOI":"10.1109\/IROS55552.2023.10341488"},{"key":"e_1_3_2_30_2","unstructured":"Q. Ye H. Xu G. Xu J. Ye M. Yan Y. Zhou J. Wang A. Hu P. Shi Y. Shi et al. 2023. mPLUG-Owl: Modularization empowers large language models with multimodality. arXiv:2304.14178. Retrieved from https:\/\/arxiv.org\/abs\/2304.14178"},{"key":"e_1_3_2_31_2","unstructured":"W. Wang Z. Chen X. Chen J. Wu X. Zhu G. Zeng P. Luo T. Lu J. Zhou Y. Qiao et al. 2023. VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks. arXiv:2305.11175. Retrieved from https:\/\/arxiv.org\/abs\/2305.11175"},{"key":"e_1_3_2_32_2","unstructured":"R. Yang L. Song Y. Li S. Zhao Y. Ge X. Li and Y. Shan. 2023. GPT4Tools: Teaching large language model to use tools via self-instruction. arXiv:2305.18752. Retrieved from https:\/\/arxiv.org\/abs\/2305.18752"},{"key":"e_1_3_2_33_2","unstructured":"E. Saravia. 2022. Prompt Engineering Guide. Retrieved December 2022 from https:\/\/github.com\/dair-ai\/Prompt-Engineering-Guide"},{"key":"e_1_3_2_34_2","unstructured":"A. Zeng X. Liu Z. Du Z. Wang H. Lai M. Ding Z. Yang Y. Xu W. Zheng X. Xia et al. 2022. GLM-130B: An open bilingual pre-trained model. arXiv:2210.02414. Retrieved from https:\/\/arxiv.org\/abs\/2210.02414"},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","unstructured":"Y. Wang H. Le A. D. Gotmare N. D. Bui J. Li and S. C. Hoi. 2023. CodeT5+: Open code large language models for code understanding and generation. arXiv:2305.07922. Retrieved from https:\/\/arxiv.org\/abs\/2305.07922","DOI":"10.18653\/v1\/2023.emnlp-main.68"},{"key":"e_1_3_2_36_2","unstructured":"S. Wang Y. Sun Y. Xiang Z. Wu S. Ding W. Gong S. Feng J. Shang Y. Zhao C. Pang et al. 2021. ERNIE 3.0 Titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation. arXiv:2112.12731. Retrieved from https:\/\/arxiv.org\/abs\/2112.12731"},{"key":"e_1_3_2_37_2","first-page":"3505","volume-title":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Rasley J.","year":"2020","unstructured":"J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. 2020. Deep speed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 3505\u20133506."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00024"},{"key":"e_1_3_2_39_2","unstructured":"J. He C. Zhou X. Ma T. Berg-Kirkpatrick and G. Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv:2110.04366. Retrieved from https:\/\/arxiv.org\/abs\/2110.04366"},{"key":"e_1_3_2_40_2","doi-asserted-by":"crossref","unstructured":"Z. Hu Y. Lan L. Wang W. Xu E.-P. Lim R. K.-W. Lee L. Bing and S. Poria. 2023. LLM-Adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv:2304.01933. Retrieved from https:\/\/arxiv.org\/abs\/2304.01933","DOI":"10.18653\/v1\/2023.emnlp-main.319"},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","unstructured":"B. Lester R. Al-Rfou and N. Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv:2104.08691. Retrieved from https:\/\/arxiv.org\/abs\/2104.08691","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"e_1_3_2_42_2","unstructured":"X. L. Li and P. Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv:2101.00190. Retrieved from https:\/\/arxiv.org\/abs\/2101.00190"},{"key":"e_1_3_2_43_2","unstructured":"X. Ma G. Fang and X. Wang. 2023. LLM-Pruner: On the structural pruning of large language models. arXiv:2305.11627. Retrieved from https:\/\/arxiv.org\/abs\/2305.11627"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i10.21408"},{"key":"e_1_3_2_45_2","first-page":"38087","volume-title":"Proceedings of the International Conference on Machine Learning","volume":"202","author":"Xiao G.","year":"2023","unstructured":"G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. 2023. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the International Conference on Machine Learning, Vol. 202, PMLR, 38087\u201338099."},{"key":"e_1_3_2_46_2","unstructured":"C. Tao L. Hou W. Zhang L. Shang X. Jiang Q. Liu P. Luo and N. Wong. 2022. Compression of generative pre-trained language models via quantization. arXiv:2203.10705. Retrieved from https:\/\/arxiv.org\/abs\/2203.10705"},{"key":"e_1_3_2_47_2","unstructured":"A. Pal D. Karkhanis M. Roberts S. Dooley A. Sundararajan and S. Naidu. 2023. Giraffe: Adventures in expanding context lengths in LLMS. arXiv:2308.10882. Retrieved from https:\/\/arxiv.org\/abs\/2308.10882"},{"key":"e_1_3_2_48_2","unstructured":"B. Peng J. Quesnelle H. Fan and E. Shippole. 2023. YaRN: Efficient context window extension of large language models. arXiv:2309.00071. Retrieved from https:\/\/arxiv.org\/abs\/2309.00071"},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","unstructured":"M. Guo J. Ainslie D. Uthus S. Ontanon J. Ni Y.-H. Sung and Y. Yang. 2021. LongT5: Efficient text-to-text transformer for long sequences. arXiv:2112.07916. Retrieved from https:\/\/arxiv.org\/abs\/2112.07916","DOI":"10.18653\/v1\/2022.findings-naacl.55"},{"key":"e_1_3_2_50_2","unstructured":"S. Chen S. Wong L. Chen and Y. Tian. 2023. Extending context window of large language models via positional interpolation. arXiv:2306.15595. Retrieved from https:\/\/arxiv.org\/abs\/2306.15595"},{"key":"e_1_3_2_51_2","unstructured":"W. X. Zhao K. Zhou J. Li T. Tang X. Wang Y. Hou Y. Min B. Zhang J. Zhang Z. Dong et al. 2023. A survey of large language models. arXiv:2303.18223. Retrieved from https:\/\/arxiv.org\/abs\/2303.18223"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3434237"},{"key":"e_1_3_2_53_2","unstructured":"B. Min H. Ross E. Sulem A. P. B. Veyseh T. H. Nguyen O. Sainz E. Agirre I. Heinz and D. Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey. arXiv:2111.01243. Retrieved from https:\/\/arxiv.org\/abs\/2111.01243"},{"key":"e_1_3_2_54_2","unstructured":"C. Zhou Q. Li C. Li J. Yu Y. Liu G. Wang K. Zhang C. Ji Q. Yan L. He et al. 2023. A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT. arXiv:2302.09419. Retrieved from https:\/\/arxiv.org\/abs\/2302.09419"},{"key":"e_1_3_2_55_2","unstructured":"Q. Dong L. Li D. Dai C. Zheng Z. Wu B. Chang X. Sun J. Xu and Z. Sui. 2022. A survey for in-context learning. arXiv:2301.00234. Retrieved from https:\/\/arxiv.org\/abs\/2301.00234"},{"key":"e_1_3_2_56_2","unstructured":"J. Huang and K. C.-C. Chang. 2022. Towards reasoning in large language models: A survey. arXiv:2212.10403. Retrieved from https:\/\/arxiv.org\/abs\/2212.10403"},{"key":"e_1_3_2_57_2","unstructured":"Y. Wang W. Zhong L. Li F. Mi X. Zeng W. Huang L. Shang X. Jiang and Q. Liu. 2023. Aligning large language models with human: A survey. arXiv:2307.12966. Retrieved from https:\/\/arxiv.org\/abs\/2307.12966"},{"key":"e_1_3_2_58_2","unstructured":"X. Zhu J. Li Y. Liu C. Ma and W. Wang. 2023. A survey on model compression for large language models. arXiv:2308.07633. Retrieved from https:\/\/arxiv.org\/abs\/2308.07633"},{"key":"e_1_3_2_59_2","unstructured":"S. Yin C. Fu S. Zhao K. Li X. Sun T. Xu and E. Chen. 2023. A survey on multimodal large language models. arXiv:2306.13549. Retrieved from https:\/\/arxiv.org\/abs\/2306.13549"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.3115\/992424.992434"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1007"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1162"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2012.6289079"},{"key":"e_1_3_2_64_2","unstructured":"S. J. Mielke Z. Alyafeai E. Salesky C. Raffel M. Dey M. Gall\u00e9 A. Raja C. Si W. Y. Lee B. Sagot et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. arXiv:2112.10508. Retrieved from https:\/\/arxiv.org\/abs\/2112.10508"},{"key":"e_1_3_2_65_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani A.","year":"2017","unstructured":"A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, \u0141. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 30.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_66_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Press O.","year":"2022","unstructured":"O. Press, N. Smith, and M. Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=R8sQPpGCv0"},{"key":"e_1_3_2_67_2","unstructured":"J. Su Y. Lu S. Pan A. Murtadha B. Wen and Y. Liu. 2021. RoFormer: Enhanced transformer with rotary position embedding. arXiv:2104.09864. Retrieved from https:\/\/arxiv.org\/abs\/2104.09864"},{"key":"e_1_3_2_68_2","unstructured":"R. Child S. Gray A. Radford and I. Sutskever. 2019. Generating long sequences with sparse transformers. arXiv:1904.10509. Retrieved from https:\/\/arxiv.org\/abs\/1904.10509"},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","first-page":"16344","DOI":"10.52202\/068431-1189","article-title":"FlashAttention: Fast and memory-efficient exact attention with IO-awareness","volume":"35","author":"Dao T.","year":"2022","unstructured":"T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R\u00e9. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 16344\u201316359.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.5555\/70405.70408"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.5555\/3104322.3104425"},{"key":"e_1_3_2_72_2","unstructured":"D. Hendrycks and K. Gimpel. 2016. Gaussian error linear units (GELUs). arXiv:1606.08415. Retrieved from https:\/\/arxiv.org\/abs\/1606.08415"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2670313"},{"key":"e_1_3_2_74_2","unstructured":"D. Krueger T. Maharaj J. Kram\u00e1r M. Pezeshki N. Ballas N. R. Ke A. Goyal Y. Bengio A. Courville and C. Pal. 2016. Zoneout: Regularizing RNNS by randomly preserving hidden activations. arXiv:1606.01305. Retrieved from https:\/\/arxiv.org\/abs\/1606.01305"},{"key":"e_1_3_2_75_2","unstructured":"N. Shazeer. 2020. GLU variants improve transformer. arXiv:2002.05202. Retrieved from https:\/\/arxiv.org\/abs\/2002.05202"},{"key":"e_1_3_2_76_2","first-page":"933","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Dauphin Y. N.","year":"2017","unstructured":"Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the International Conference on Machine Learning. PMLR, 933\u2013941."},{"key":"e_1_3_2_77_2","unstructured":"J. L. Ba J. R. Kiros and G. E. Hinton. 2016. Layer normalization. arXiv:1607.06450. Retrieved from https:\/\/arxiv.org\/abs\/1607.06450"},{"key":"e_1_3_2_78_2","article-title":"Root mean square layer normalization","volume":"32","author":"Zhang B.","year":"2019","unstructured":"B. Zhang and R. Sennrich. 2019. Root mean square layer normalization. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_79_2","unstructured":"A. Baevski and M. Auli. 2018. Adaptive input representations for neural language modeling. arXiv:1809.10853. Retrieved from https:\/\/arxiv.org\/abs\/1809.10853"},{"key":"e_1_3_2_80_2","unstructured":"H. Wang S. Ma L. Dong S. Huang D. Zhang and F. Wei. 2022. DeepNet: Scaling transformers to 1 000 layers. arXiv:2203.00555. Retrieved from https:\/\/arxiv.org\/abs\/2203.00555"},{"key":"e_1_3_2_81_2","unstructured":"M. Shoeybi M. Patwary R. Puri P. LeGresley J. Casper and B. Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053. Retrieved from https:\/\/arxiv.org\/abs\/1909.08053"},{"key":"e_1_3_2_82_2","unstructured":"BMTrain. 2025. BMTrain: Efficient Training for Big Models. Retrieved from https:\/\/github.com\/OpenBMB\/BMTrain"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_3_2_84_2","unstructured":"J. Bradbury R. Frostig P. Hawkins M. J. Johnson C. Leary D. Maclaurin G. Necula A. Paszke J. VanderPlas S. Wanderman-Milne et al. 2018. JAX: Composable transformations of Python+ Numpy programs (2018)."},{"key":"e_1_3_2_85_2","unstructured":"S. Li J. Fang Z. Bian H. Liu Y. Liu H. Huang B. Wang and Y. You. 2021. Colossal-AI: A unified deep learning system for large-scale parallel training. arXiv:2110.14883. Retrieved from https:\/\/arxiv.org\/abs\/2110.14883"},{"key":"e_1_3_2_86_2","unstructured":"J. He J. Qiu A. Zeng Z. Yang J. Zhai and J. Tang. 2021. FastMoE: A fast mixture-of-expert training system. arXiv:2103.13262. Retrieved from https:\/\/arxiv.org\/abs\/2103.13262"},{"key":"e_1_3_2_87_2","first-page":"137","volume-title":"Artificial Intelligence Technology","author":"Huawei Technologies Co., Ltd.","year":"2022","unstructured":"Huawei Technologies Co., Ltd. 2022. Huawei MindSpore AI development framework. In Artificial Intelligence Technology. Springer, 137\u2013162."},{"key":"e_1_3_2_88_2","article-title":"PyTorch: An imperative style, high-performance deep learning library","volume":"32","author":"Paszke A.","year":"2019","unstructured":"A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_89_2","first-page":"265","volume-title":"Proceedings of the Operating Systems Design and Implementation","volume":"16","author":"Abadi M.","year":"2016","unstructured":"M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the Operating Systems Design and Implementation, Vol. 16, 265\u2013283."},{"key":"e_1_3_2_90_2","unstructured":"T. Chen M. Li Y. Li M. Lin N. Wang M. Wang T. Xiao B. Xu C. Zhang and Z. Zhang. 2015. MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv:1512.01274. Retrieved from https:\/\/arxiv.org\/abs\/1512.01274"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.5555\/3586589.3586709"},{"key":"e_1_3_2_92_2","first-page":"5547","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Du N.","year":"2022","unstructured":"N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al. 2022. GLaM: Efficient scaling of language models with mixture-of-experts. In Proceedings of the International Conference on Machine Learning. PMLR, 5547\u20135569."},{"key":"e_1_3_2_93_2","unstructured":"X. Ren P. Zhou X. Meng X. Huang Y. Wang W. Wang P. Li X. Zhang A. Podolskiy G. Arshinov et al. 2023. Pangu- \\(\\sum\\) : Towards trillion parameter language model with sparse heterogeneous computing. arXiv:2303.10845. Retrieved from https:\/\/arxiv.org\/abs\/2303.10845"},{"key":"e_1_3_2_94_2","first-page":"22964","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Wang T.","year":"2022","unstructured":"T. Wang, A. Roberts, D. Hesslow, T. Le Scao, H. W. Chung, I. Beltagy, J. Launay, and C. Raffel. 2022. What language model architecture and pretraining objective works best for zero-shot generalization? In Proceedings of the International Conference on Machine Learning. PMLR, 22964\u201322984."},{"key":"e_1_3_2_95_2","article-title":"Unified language model pre-training for natural language understanding and generation","volume":"32","author":"Dong L.","year":"2019","unstructured":"L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H.-W. Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 32.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_96_2","unstructured":"J. Kaplan S. McCandlish T. Henighan T. B. Brown B. Chess R. Child S. Gray A. Radford J. Wu and D. Amodei. 2020. Scaling laws for neural language models. arXiv:2001.08361. Retrieved from https:\/\/arxiv.org\/abs\/2001.08361"},{"key":"e_1_3_2_97_2","doi-asserted-by":"crossref","unstructured":"J. Hoffmann S. Borgeaud A. Mensch E. Buchatskaya T. Cai E. Rutherford D. D. L. Casas L. A. Hendricks J. Welbl A. Clark et al. 2022. Training compute-optimal large language models. arXiv:2203.15556. Retrieved from https:\/\/arxiv.org\/abs\/2203.15556","DOI":"10.52202\/068431-2176"},{"key":"e_1_3_2_98_2","unstructured":"S. Iyer X. V. Lin R. Pasunuru T. Mihaylov D. Simig P. Yu K. Shuster T. Wang Q. Liu P. S. Koura et al. 2022. OPT-IML: Scaling language model instruction meta learning through the lens of generalization. arXiv:2212.12017. Retrieved from https:\/\/arxiv.org\/abs\/2212.12017"},{"key":"e_1_3_2_99_2","unstructured":"Z. Sun Y. Shen Q. Zhou H. Zhang Z. Chen D. Cox Y. Yang and C. Gan. 2023. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv:2305.03047. Retrieved from https:\/\/arxiv.org\/abs\/2305.03047"},{"key":"e_1_3_2_100_2","unstructured":"A. Askell Y. Bai A. Chen D. Drain D. Ganguli T. Henighan A. Jones N. Joseph B. Mann N. DasSarma et al. 2021. A general language assistant as a laboratory for alignment. arXiv:2112.00861. Retrieved from https:\/\/arxiv.org\/abs\/2112.00861"},{"key":"e_1_3_2_101_2","unstructured":"D. M. Ziegler N. Stiennon J. Wu T. B. Brown A. Radford D. Amodei P. Christiano and G. Irving. 2019. Fine-tuning language models from human preferences. arXiv:1909.08593. Retrieved from https:\/\/arxiv.org\/abs\/1909.08593"},{"key":"e_1_3_2_102_2","doi-asserted-by":"crossref","unstructured":"S. Kim S. J. Joo D. Kim J. Jang S. Ye J. Shin and M. Seo. 2023. The CoT collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. arXiv:2305.14045. Retrieved from https:\/\/arxiv.org\/abs\/2305.14045","DOI":"10.18653\/v1\/2023.emnlp-main.782"},{"key":"e_1_3_2_103_2","unstructured":"Q. Liu F. Zhou Z. Jiang L. Dou and M. Lin. 2023. From zero to hero: Examining the power of symbolic tasks in instruction tuning. arXiv:2304.07995. Retrieved from https:\/\/arxiv.org\/abs\/2304.07995"},{"key":"e_1_3_2_104_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei J.","year":"2022","unstructured":"J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, and D. Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 24824\u201324837.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_105_2","unstructured":"X. Wang J. Wei D. Schuurmans Q. Le E. Chi S. Narang A. Chowdhery and D. Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv:2203.11171. Retrieved from https:\/\/arxiv.org\/abs\/2203.11171"},{"key":"e_1_3_2_106_2","unstructured":"S. Yao D. Yu J. Zhao I. Shafran T. L. Griffiths Y. Cao and K. Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv:2305.10601. Retrieved from https:\/\/arxiv.org\/abs\/2305.10601"},{"key":"e_1_3_2_107_2","first-page":"2790","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Houlsby N.","year":"2019","unstructured":"N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning. PMLR, 2790\u20132799."},{"key":"e_1_3_2_108_2","unstructured":"S. McCandlish J. Kaplan D. Amodei and OpenAI Dota Team. 2018. An empirical model of large-batch training. arXiv:1812.06162. Retrieved from https:\/\/arxiv.org\/abs\/1812.06162"},{"key":"e_1_3_2_109_2","unstructured":"OpenAI S. Adler S. Agarwal L. Ahmad I. Akkaya F. L. Alemen D. Almeida J. Altenschmidt S. Altman S. Anadkat et al. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_110_2","unstructured":"A. Hurst A. Lerer A. P. Goucher A. Perelman A. Ramesh A. Clark A. Ostrow A. Welihinda A. Hayes A. Radford et al. 2024. GPT-4o system card. arXiv:2410.21276. Retrieved from https:\/\/arxiv.org\/abs\/2410.21276"},{"key":"e_1_3_2_111_2","unstructured":"OpenAI O3-Mini System Card. 2025. Retrieved from https:\/\/cdn.openai.com\/o3-mini-system-card-feb10.pdf"},{"key":"e_1_3_2_112_2","unstructured":"W. Zeng X. Ren T. Su H. Wang Y. Liao Z. Wang X. Jiang Z. Yang K. Wang X. Zhang et al. 2021. PanGu- \\(\\alpha\\) : Large-scale autoregressive pretrained Chinese language models with auto-parallel computation. arXiv:2104.12369. Retrieved from https:\/\/arxiv.org\/abs\/2104.12369"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2021.06.001"},{"key":"e_1_3_2_114_2","unstructured":"Y. Sun S. Wang S. Feng S. Ding C. Pang J. Shang J. Liu X. Chen Y. Zhao Y. Lu et al. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv:2107.02137. Retrieved from https:\/\/arxiv.org\/abs\/2107.02137"},{"key":"e_1_3_2_115_2","doi-asserted-by":"crossref","unstructured":"Z. Dai Z. Yang Y. Yang J. Carbonell Q. V. Le and R. Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. arXiv:1901.02860. Retrieved from https:\/\/arxiv.org\/abs\/1901.02860","DOI":"10.18653\/v1\/P19-1285"},{"key":"e_1_3_2_116_2","volume-title":"Jurassic-1: Technical Details and Evaluation","author":"Lieber O.","year":"2021","unstructured":"O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. 2021. Jurassic-1: Technical Details and Evaluation. White Paper. AI21 Labs."},{"key":"e_1_3_2_117_2","first-page":"22640","article-title":"Limits to depth efficiencies of self-attention","volume":"33","author":"Levine Y.","year":"2020","unstructured":"Y. Levine, N. Wies, O. Sharir, H. Bata, and A. Shashua. 2020. Limits to depth efficiencies of self-attention. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 33, 22640\u201322651.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_118_2","doi-asserted-by":"crossref","unstructured":"B. Kim H. Kim S.-W. Lee G. Lee D. Kwak D. H. Jeon S. Park S. Kim S. Kim D. Seo et al. 2021. What changes can large-scale language models bring? Intensive study on HyperCLOVA: Billions-scale Korean generative pretrained transformers. arXiv:2109.04650. Retrieved from https:\/\/arxiv.org\/abs\/2109.04650","DOI":"10.18653\/v1\/2021.emnlp-main.274"},{"key":"e_1_3_2_119_2","unstructured":"S. Wu X. Zhao T. Yu R. Zhang C. Shen H. Liu F. Li H. Zhu J. Luo L. Xu et al. 2021. Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning. arXiv:2110.04725. Retrieved from https:\/\/arxiv.org\/abs\/2110.04725"},{"key":"e_1_3_2_120_2","unstructured":"J. W. Rae S. Borgeaud T. Cai K. Millican J. Hoffmann F. Song J. Aslanides S. Henderson R. Ring S. Young et al. 2021. Scaling language models: Methods analysis & insights from training gopher. arXiv:2112.11446. Retrieved from https:\/\/arxiv.org\/abs\/2112.11446"},{"key":"e_1_3_2_121_2","unstructured":"S. Smith M. Patwary B. Norick P. LeGresley S. Rajbhandari J. Casper Z. Liu S. Prabhumoye G. Zerveas V. Korthikanti et al. 2022. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B a large-scale generative language model. arXiv:2201.11990. Retrieved from https:\/\/arxiv.org\/abs\/2201.11990"},{"key":"e_1_3_2_122_2","doi-asserted-by":"crossref","unstructured":"S. Black S. Biderman E. Hallahan Q. Anthony L. Gao L. Golding H. He C. Leahy K. McDonell J. Phang et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv:2204.06745. Retrieved from https:\/\/arxiv.org\/abs\/2204.06745","DOI":"10.18653\/v1\/2022.bigscience-1.9"},{"key":"e_1_3_2_123_2","unstructured":"W. Ben and K. Aran. 2021. GPT-J-6B: A 6 billion parameter autoregressive language model."},{"key":"e_1_3_2_124_2","unstructured":"P. Micikevicius S. Narang J. Alben G. Diamos E. Elsen D. Garcia B. Ginsburg M. Houston O. Kuchaiev G. Venkatesh et al. 2017. Mixed precision training. arXiv:1710.03740. Retrieved from https:\/\/arxiv.org\/abs\/1710.03740"},{"key":"e_1_3_2_125_2","unstructured":"N. Shazeer A. Mirhoseini K. Maziarz A. Davis Q. Le G. Hinton and J. Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv:1701.06538. Retrieved from https:\/\/arxiv.org\/abs\/1701.06538"},{"key":"e_1_3_2_126_2","unstructured":"S. Soltan S. Ananthakrishnan J. FitzGerald R. Gupta W. Hamza H. Khan C. Peris S. Rawls A. Rosenbaum A. Rumshisky et al. 2022. AlexaTM 20B: Few-shot learning using a large-scale multilingual Seq2Seq model. arXiv:2208.01448. Retrieved from https:\/\/arxiv.org\/abs\/2208.01448"},{"key":"e_1_3_2_127_2","unstructured":"R. Anil A. M. Dai O. Firat M. Johnson D. Lepikhin A. Passos S. Shakeri E. Taropa P. Bailey Z. Chen et al. 2023. PaLM 2 technical report. arXiv:2305.10403. Retrieved from https:\/\/arxiv.org\/abs\/2305.10403"},{"key":"e_1_3_2_128_2","doi-asserted-by":"crossref","unstructured":"Y. Tay J. Wei H. W. Chung V. Q. Tran D. R. So S. Shakeri X. Garcia H. S. Zheng J. Rao A. Chowdhery et al. 2022. Transcending scaling laws with 0.1% extra compute. arXiv:2210.11399. Retrieved from https:\/\/arxiv.org\/abs\/2210.11399","DOI":"10.18653\/v1\/2023.emnlp-main.91"},{"key":"e_1_3_2_129_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Tay Y.","year":"2022","unstructured":"Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, et al. 2022. UL2: Unifying language learning paradigms. In Proceedings of the 11th International Conference on Learning Representations."},{"key":"e_1_3_2_130_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.26"},{"key":"e_1_3_2_131_2","unstructured":"H. Touvron T. Lavril G. Izacard X. Martinet M.-A. Lachaux T. Lacroix B. Rozi\u00e8re N. Goyal E. Hambro F. Azhar et al. 2023. LLaMA: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_2_132_2","unstructured":"M. N. Rabe and C. Staats. 2021. Self-attention does not need o(n \\({}^{2}\\) ) memory. arXiv:2112.05682. Retrieved from https:\/\/arxiv.org\/abs\/2112.05682"},{"key":"e_1_3_2_133_2","first-page":"341","article-title":"Reducing activation recomputation in large transformer models","volume":"5","author":"Korthikanti V. A.","year":"2023","unstructured":"V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro. 2023. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems 5 (2023), 341\u2013353.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_2_134_2","unstructured":"A. Dubey A. Jauhri A. Pandey A. Kadian A. Al-Dahle A. Letman A. Mathur A. Schelten A. Yang A. Fan et al. 2024. The Llama 3 Herd of models. arXiv:2407.21783. Retrieved from https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"e_1_3_2_135_2","unstructured":"Mistral AI. 2024. Retrieved from https:\/\/mistral.ai\/news\/mixtral-8x22b\/"},{"key":"e_1_3_2_136_2","unstructured":"Snowflake-Labs. 2025. Retrieved from https:\/\/github.com\/Snowflake-Labs\/snowflake-arctic"},{"key":"e_1_3_2_137_2","unstructured":"Xai-org. 2025. Retrieved from https:\/\/github.com\/xai-org\/grok-1"},{"key":"e_1_3_2_138_2","unstructured":"Grok-1.5. 2024. Retrieved from https:\/\/x.ai\/blog\/grok-1.5"},{"key":"e_1_3_2_139_2","unstructured":"G. Team R. Anil S. Borgeaud Y. Wu J.-B. Alayrac J. Yu R. Soricut J. Schalkwyk A. M. Dai A. Hauth et al. 2023. Gemini: A family of highly capable multimodal models arXiv:2312.11805. Retrieved from https:\/\/arxiv.org\/abs\/2312.11805"},{"key":"e_1_3_2_140_2","unstructured":"M. Reid N. Savinov D. Teplyashin D. Lepikhin T. Lillicrap J.-B. Alayrac R. Soricut A. Lazaridou O. Firat J. Schrittwieser et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05530. Retrieved from https:\/\/arxiv.org\/abs\/2403.05530"},{"key":"e_1_3_2_141_2","unstructured":"B. Adler N. Agarwal A. Aithal D. H. Anh P. Bhattacharya A. Brundyn J. Casper B. Catanzaro S. Clay J. Cohen et al. 2024. Nemotron-4 340B technical report. arXiv:2406.11704. Retrieved from https:\/\/arxiv.org\/abs\/2406.11704"},{"key":"e_1_3_2_142_2","unstructured":"X. Bi D. Chen G. Chen S. Chen D. Dai C. Deng H. Ding K. Dong Q. Du Z. Fu et al. 2024. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv:2401.02954. Retrieved from https:\/\/arxiv.org\/abs\/2401.02954"},{"key":"e_1_3_2_143_2","unstructured":"A. DeepSeek-AI B. Liu B. Feng B. Wang B. Wang B. Liu C. Zhao C. Deng C. Ruan D. Dai et al. 2024. DeepSeek-V2: A strong economical and efficient mixture-of-experts language model. arXiv:2405.04434. Retrieved from https:\/\/arxiv.org\/abs\/2405.04434"},{"key":"e_1_3_2_144_2","unstructured":"A. Liu B. Feng B. Xue B. Wang B. Wu C. Lu C. Zhao C. Deng C. Zhang C. Ruan et al. 2024. DeepSeek-V3 technical report. arXiv:2412.19437. Retrieved from https:\/\/arxiv.org\/abs\/2412.19437"},{"key":"e_1_3_2_145_2","unstructured":"E. Nijkamp B. Pang H. Hayashi L. Tu H. Wang Y. Zhou S. Savarese C. Xiong. 2022. CodeGen: An open large language model for code with multi-turn program synthesis. arXiv:2203.13474. Retrieved from https:\/\/arxiv.org\/abs\/2203.13474"},{"key":"e_1_3_2_146_2","unstructured":"M. Chen J. Tworek H. Jun Q. Yuan H. P. O. Pinto J. Kaplan H. Edwards Y. Burda N. Joseph G. Brockman et al. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_147_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_2_148_2","unstructured":"N. Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv:1911.02150. Retrieved from https:\/\/arxiv.org\/abs\/1911.02150"},{"key":"e_1_3_2_149_2","unstructured":"R. Y. Pang and H. He. 2020. Text generation by learning from demonstrations. arXiv:2009.07839. Retrieved from https:\/\/arxiv.org\/abs\/2009.07839"},{"key":"e_1_3_2_150_2","unstructured":"R. Dabre and A. Fujita. 2020. Softmax tempering for training neural machine translation models. arXiv:2009.09372. Retrieved from https:\/\/arxiv.org\/abs\/2009.09372"},{"key":"e_1_3_2_151_2","doi-asserted-by":"crossref","unstructured":"Y. Wang W. Wang S. Joty and S. C. Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv:2109.00859. Retrieved from https:\/\/arxiv.org\/abs\/2109.00859","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_2_152_2","unstructured":"R. Li L. B. Allal Y. Zi N. Muennighoff D. Kocetkov C. Mou M. Marone C. Akiki J. Li J. Chim et al. 2023. StarCoder: May the source be with you! arXiv:2305.06161. Retrieved from https:\/\/arxiv.org\/abs\/2305.06161"},{"key":"e_1_3_2_153_2","unstructured":"R. Taylor M. Kardas G. Cucurull T. Scialom A. Hartshorn E. Saravia A. Poulton V. Kerkez and R. Stojnic. 2022. Galactica: A large language model for science. arXiv:2211.09085. Retrieved from https:\/\/arxiv.org\/abs\/2211.09085"},{"key":"e_1_3_2_154_2","unstructured":"FairScale Authors. 2021. FairScale: A General Purpose Modular PyTorch Library for High Performance and Large Scale Training. Retrieved from https:\/\/github.com\/facebookresearch\/fairscale"},{"key":"e_1_3_2_155_2","unstructured":"R. Thoppilan D. De Freitas J. Hall N. Shazeer A. Kulshreshtha H.-T. Cheng A. Jin T. Bos L. Baker Y. Du et al. 2022. LaMDA: Language models for dialog applications. arXiv:2201.08239. Retrieved from https:\/\/arxiv.org\/abs\/2201.08239"},{"key":"e_1_3_2_156_2","unstructured":"S. Wu O. Irsoy S. Lu V. Dabravolski M. Dredze S. Gehrmann P. Kambadur D. Rosenberg and G. Mann. 2023. BloombergGPT: A large language model for finance. arXiv:2303.17564. Retrieved from https:\/\/arxiv.org\/abs\/2303.17564"},{"key":"e_1_3_2_157_2","volume-title":"XuanYuan 2.0: A large Chinese financial chat model with hundreds of billions parameters","author":"Zhang X.","year":"2023","unstructured":"X. Zhang, Q. Yang, and D. Xu. 2023. XuanYuan 2.0: A large Chinese financial chat model with hundreds of billions parameters. arXiv:2305.12002. Retrieved from https:\/\/arxiv.org\/abs\/2305.12002"},{"key":"e_1_3_2_158_2","unstructured":"W. Ben. 2021. Mesh-Transformer-JAX: Model-parallel implementation of transformer language model with JAX."},{"key":"e_1_3_2_159_2","unstructured":"N. Muennighoff T. Wang L. Sutawika A. Roberts S. Biderman T. L. Scao M. S. Bari S. Shen Z.-X. Yong H. Schoelkopf et al. 2022. Crosslingual generalization through multitask finetuning. arXiv:2211.01786. Retrieved from https:\/\/arxiv.org\/abs\/2211.01786"},{"key":"e_1_3_2_160_2","doi-asserted-by":"crossref","unstructured":"D. Yin X. Liu F. Yin M. Zhong H. Bansal J. Han and K.-W. Chang. 2023. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation. arXiv:2305.14327. Retrieved from https:\/\/arxiv.org\/abs\/2305.14327","DOI":"10.18653\/v1\/2023.emnlp-main.245"},{"key":"e_1_3_2_161_2","unstructured":"P. Gao J. Han R. Zhang Z. Lin S. Geng A. Zhou W. Zhang P. Lu C. He X. Yue et al. 2023. LLaMA-Adapter V2: Parameter-efficient visual instruction model. arXiv:2304.15010. Retrieved from https:\/\/arxiv.org\/abs\/2304.15010"},{"key":"e_1_3_2_162_2","unstructured":"R. Taori I. Gulrajani T. Zhang Y. Dubois X. Li C. Guestrin P. Liang and T. B. Hashimoto. 2023. Stanford Alpaca: An instruction-Following LLaMA Model. Retrieved from https:\/\/github.com\/tatsu-lab\/stanford_alpaca"},{"key":"e_1_3_2_163_2","unstructured":"W.-L. Chiang Z. Li Z. Lin Y. Sheng Z. Wu H. Zhang L. Zheng S. Zhuang Y. Zhuang J. E. Gonzalez I. Stoica and E. P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. Retrieved March 2023 from https:\/\/lmsys.org\/blog\/2023-03-30-vicuna\/"},{"key":"e_1_3_2_164_2","unstructured":"B. Peng C. Li P. He M. Galley and J. Gao. 2023. Instruction tuning with GPT-4. arXiv:2304.03277. Retrieved from https:\/\/arxiv.org\/abs\/2304.03277"},{"key":"e_1_3_2_165_2","unstructured":"T. Liu and B. K. H. Low. 2023. Goat: Fine-tuned llama outperforms GPT-4 on arithmetic tasks. arXiv:2305.14201. Retrieved from https:\/\/arxiv.org\/abs\/2305.14201"},{"key":"e_1_3_2_166_2","unstructured":"H. Wang C. Liu N. Xi Z. Qiang S. Zhao B. Qin and T. Liu. 2023. HuaTuo: Tuning LLaMA model with Chinese medical knowledge. arXiv:2304.06975. Retrieved from https:\/\/arxiv.org\/abs\/2304.06975"},{"key":"e_1_3_2_167_2","unstructured":"C. Xu Q. Sun K. Zheng X. Geng P. Zhao J. Feng C. Tao and D. Jiang. 2023. WizardLM: Empowering large language models to follow complex instructions. arXiv:2304.12244. Retrieved from https:\/\/arxiv.org\/abs\/2304.12244"},{"key":"e_1_3_2_168_2","unstructured":"Z. Luo C. Xu P. Zhao Q. Sun X. Geng W. Hu C. Tao J. Ma Q. Lin and D. Jiang. 2023. WizardCoder: Empowering code large language models with Evol-Instruct. arXiv:2306.08568. Retrieved from https:\/\/arxiv.org\/abs\/2306.08568"},{"key":"e_1_3_2_169_2","unstructured":"J. Menick M. Trebacz V. Mikulik J. Aslanides F. Song M. Chadwick M. Glaese S. Young L. Campbell-Gillingham G. Irving et al. 2022. Teaching language models to support answers with verified quotes. arXiv:2203.11147. Retrieved from https:\/\/arxiv.org\/abs\/2203.11147"},{"key":"e_1_3_2_170_2","unstructured":"R. Nakano J. Hilton S. Balaji J. Wu L. Ouyang C. Kim C. Hesse S. Jain V. Kosaraju W. Saunders et al. 2021. WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332. Retrieved from https:\/\/arxiv.org\/abs\/2112.09332"},{"key":"e_1_3_2_171_2","unstructured":"A. Glaese N. McAleese M. Tr\u0119bacz J. Aslanides V. Firoiu T. Ewalds M. Rauh L. Weidinger M. Chadwick P. Thacker et al. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv:2209.14375. Retrieved from https:\/\/arxiv.org\/abs\/2209.14375"},{"key":"e_1_3_2_172_2","unstructured":"R. Rafailov A. Sharma E. Mitchell S. Ermon C. D. Manning and C. Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv:2305.18290. Retrieved from https:\/\/arxiv.org\/abs\/2305.18290"},{"key":"e_1_3_2_173_2","unstructured":"H. Dong W. Xiong D. Goyal R. Pan S. Diao J. Zhang K. Shum and T. Zhang. 2023. RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv:2304.06767. Retrieved from https:\/\/arxiv.org\/abs\/2304.06767"},{"key":"e_1_3_2_174_2","unstructured":"Z. Yuan H. Yuan C. Tan W. Wang S. Huang and F. Huang. 2023. RRHF: Rank responses to align language models with human feedback without tears. arXiv:2304.05302. Retrieved from https:\/\/arxiv.org\/abs\/2304.05302"},{"key":"e_1_3_2_175_2","unstructured":"F. Song B. Yu M. Li H. Yu F. Huang Y. Li and H. Wang. 2023. Preference ranking optimization for human alignment. arXiv:2306.17492. Retrieved from https:\/\/arxiv.org\/abs\/2306.17492"},{"key":"e_1_3_2_176_2","unstructured":"H. Liu C. Sferrazza and P. Abbeel. 2023. Languages are rewards: Hindsight finetuning using human feedback. arXiv:2302.02676. Retrieved from https:\/\/arxiv.org\/abs\/2302.02676"},{"key":"e_1_3_2_177_2","unstructured":"Y. Bai S. Kadavath S. Kundu A. Askell J. Kernion A. Jones A. Chen A. Goldie A. Mirhoseini C. McKinnon et al. 2022. Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073. Retrieved from https:\/\/arxiv.org\/abs\/2212.08073"},{"key":"e_1_3_2_178_2","unstructured":"Y. Dubois X. Li R. Taori T. Zhang I. Gulrajani J. Ba C. Guestrin P. Liang and T. B. Hashimoto. 2023. AlpacaFarm: A simulation framework for methods that learn from human feedback. arXiv:2305.14387. Retrieved from https:\/\/arxiv.org\/abs\/2305.14387"},{"key":"e_1_3_2_179_2","unstructured":"C. Si Z. Gan Z. Yang S. Wang J. Wang J. Boyd-Graber and L. Wang. 2022. Prompting GPT-3 to be reliable. arXiv:2210.09150. Retrieved from https:\/\/arxiv.org\/abs\/2210.09150"},{"key":"e_1_3_2_180_2","unstructured":"D. Ganguli A. Askell N. Schiefer T. Liao K. Luko\u0161i\u016bt\u0117 A. Chen A. Goldie A. Mirhoseini C. Olsson D. Hernandez et al. 2023. The capacity for moral self-correction in large language models. arXiv:2302.07459. Retrieved from https:\/\/arxiv.org\/abs\/2302.07459"},{"key":"e_1_3_2_181_2","doi-asserted-by":"crossref","unstructured":"A. Wei N. Haghtalab and J. Steinhardt. 2023. Jailbroken: How does LLM safety training fail? arXiv:2307.02483. Retrieved from https:\/\/arxiv.org\/abs\/2307.02483","DOI":"10.52202\/075280-3508"},{"key":"e_1_3_2_182_2","unstructured":"D. Ganguli L. Lovitt J. Kernion A. Askell Y. Bai S. Kadavath B. Mann E. Perez N. Schiefer K. Ndousse et al. 2022. Red teaming language models to reduce harms: Methods scaling behaviors and lessons learned. arXiv:2209.07858. Retrieved from https:\/\/arxiv.org\/abs\/2209.07858"},{"key":"e_1_3_2_183_2","unstructured":"S. Casper J. Lin J. Kwon G. Culp and D. Hadfield-Menell. 2023. Explore establish exploit: Red teaming language models from scratch. arXiv:2306.09442. Retrieved from https:\/\/arxiv.org\/abs\/2306.09442"},{"key":"e_1_3_2_184_2","doi-asserted-by":"crossref","unstructured":"E. Perez S. Huang F. Song T. Cai R. Ring J. Aslanides A. Glaese N. McAleese and G. Irving. 2022. Red teaming language models with language models. arXiv:2202.03286. Retrieved from https:\/\/arxiv.org\/abs\/2202.03286","DOI":"10.18653\/v1\/2022.emnlp-main.225"},{"key":"e_1_3_2_185_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.410"},{"key":"e_1_3_2_186_2","unstructured":"Z. Shi and A. Lipani. 2023. Don\u2019t stop pretraining? Make prompt-based fine-tuning powerful learner. arXiv:2305.01711. Retrieved from https:\/\/arxiv.org\/abs\/2305.01711"},{"key":"e_1_3_2_187_2","unstructured":"H. Gupta S. A. Sawant S. Mishra M. Nakamura A. Mitra S. Mashetty and C. Baral. 2023. Instruction tuned models are quick learners. arXiv:2306.05539. Retrieved from https:\/\/arxiv.org\/abs\/2306.05539"},{"key":"e_1_3_2_188_2","unstructured":"H. Chen Y. Zhang Q. Zhang H. Yang X. Hu X. Ma Y. Yanggong and J. Zhao. 2023. Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning. arXiv:2305.09246. Retrieved from https:\/\/arxiv.org\/abs\/2305.09246"},{"key":"e_1_3_2_189_2","unstructured":"C. Zhou P. Liu P. Xu S. Iyer J. Sun Y. Mao X. Ma A. Efrat P. Yu L. Yu et al. 2023. LIMA: Less is more for alignment. arXiv:2305.11206. Retrieved from https:\/\/arxiv.org\/abs\/2305.11206"},{"key":"e_1_3_2_190_2","unstructured":"C. Han Q. Wang W. Xiong Y. Chen H. Ji and S. Wang. 2023. LM-infinite: Simple on-the-fly length generalization for large language models. arXiv:2308.16137. Retrieved from https:\/\/arxiv.org\/abs\/2308.16137"},{"key":"e_1_3_2_191_2","doi-asserted-by":"crossref","unstructured":"J. Ainslie T. Lei M. de Jong S. Onta\u00f1\u00f3n S. Brahma Y. Zemlyanskiy D. Uthus M. Guo J. Lee-Thorp Y. Tay et al. 2023. CoLT5: Faster long-range transformers with conditional computation. arXiv:2303.09752. Retrieved from https:\/\/arxiv.org\/abs\/2303.09752","DOI":"10.18653\/v1\/2023.emnlp-main.309"},{"key":"e_1_3_2_192_2","doi-asserted-by":"crossref","unstructured":"J. Ding S. Ma L. Dong X. Zhang S. Huang W. Wang and F. Wei. 2023. LongNet: Scaling transformers to 1 000 000 000 tokens. arXiv:2307.02486. Retrieved from https:\/\/arxiv.org\/abs\/2307.02486","DOI":"10.14218\/JCTH.2022.00006S"},{"key":"e_1_3_2_193_2","unstructured":"Y. Chen S. Qian H. Tang X. Lai Z. Liu S. Han and J. Jia. 2023. LongLoRA: Efficient fine-tuning of long-context large language models. arXiv:2309.12307. Retrieved from https:\/\/arxiv.org\/abs\/2309.12307"},{"key":"e_1_3_2_194_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.352"},{"key":"e_1_3_2_195_2","unstructured":"W. Wang L. Dong H. Cheng X. Liu X. Yan J. Gao and F. Wei. 2023. Augmenting language models with long-term memory. arXiv:2306.07174. Retrieved from https:\/\/arxiv.org\/abs\/2306.07174"},{"key":"e_1_3_2_196_2","doi-asserted-by":"crossref","unstructured":"X. Xu Z. Gou W. Wu Z.-Y. Niu H. Wu H. Wang and S. Wang. 2022. Long time no see! Open-domain conversation with long-term persona memory. arXiv:2203.05797. Retrieved from https:\/\/arxiv.org\/abs\/2203.05797","DOI":"10.18653\/v1\/2022.findings-acl.207"},{"key":"e_1_3_2_197_2","first-page":"2206","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Borgeaud S.","year":"2022","unstructured":"S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of the International Conference on Machine Learning. PMLR, 2206\u20132240."},{"key":"e_1_3_2_198_2","unstructured":"W. Zhong L. Guo Q. Gao and Y. Wang. 2023. MemoryBank: Enhancing large language models with long-term memory. arXiv:2305.10250. Retrieved from https:\/\/arxiv.org\/abs\/2305.10250"},{"key":"e_1_3_2_199_2","doi-asserted-by":"crossref","unstructured":"N. Shinn F. Cassano B. Labash A. Gopinath K. Narasimhan and S. Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. arXiv:2303.11366. Retrieved from https:\/\/arxiv.org\/abs\/2303.11366","DOI":"10.52202\/075280-0377"},{"key":"e_1_3_2_200_2","unstructured":"C. Hu J. Fu C. Du S. Luo J. Zhao and H. Zhao. 2023. ChatDB: Augmenting LLMs with databases as their symbolic memory. arXiv:2306.03901. Retrieved from https:\/\/arxiv.org\/abs\/2306.03901"},{"key":"e_1_3_2_201_2","doi-asserted-by":"crossref","unstructured":"Z. Jiang F. F. Xu L. Gao Z. Sun Q. Liu J. Dwivedi-Yu Y. Yang J. Callan and G. Neubig. 2023. Active retrieval augmented generation. arXiv:2305.06983. Retrieved from https:\/\/arxiv.org\/abs\/2305.06983","DOI":"10.18653\/v1\/2023.emnlp-main.495"},{"key":"e_1_3_2_202_2","doi-asserted-by":"crossref","unstructured":"O. Ram Y. Levine I. Dalmedigos D. Muhlgay A. Shashua K. Leyton-Brown and Y. Shoham. 2023. In-context retrieval-augmented language models. arXiv:2302.00083. Retrieved from https:\/\/arxiv.org\/abs\/2302.00083","DOI":"10.1162\/tacl_a_00605"},{"key":"e_1_3_2_203_2","unstructured":"X. Li and X. Qiu. 2023. MoT: Pre-thinking and recalling enable ChatGPT to self-improve with memory-of-thoughts. arXiv:2305.05181. Retrieved from https:\/\/arxiv.org\/abs\/2305.05181"},{"key":"e_1_3_2_204_2","unstructured":"D. Schuurmans. 2023. Memory augmented large language models are computationally universal. arXiv:2301.04589. Retrieved from https:\/\/arxiv.org\/abs\/2301.04589"},{"key":"e_1_3_2_205_2","unstructured":"A. Modarressi A. Imani M. Fayyaz and H. Sch\u00fctze. 2023. RET-LLM: Towards a general read-write memory for large language models. arXiv:2305.14322. Retrieved from https:\/\/arxiv.org\/abs\/2305.14322"},{"key":"e_1_3_2_206_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000019"},{"key":"e_1_3_2_207_2","unstructured":"X. Wang J. Wei D. Schuurmans Q. Le E. Chi and D. Zhou. 2022. Rationale-augmented ensembles in language models. arXiv:2207.00747. Retrieved from https:\/\/arxiv.org\/abs\/2207.00747"},{"key":"e_1_3_2_208_2","doi-asserted-by":"crossref","unstructured":"F. Zhang B. Chen Y. Zhang J. Liu D. Zan Y. Mao J.-G. Lou and W. Chen. 2023. RepoCoder: Repository-level code completion through iterative retrieval and generation. arXiv:2303.12570. Retrieved from https:\/\/arxiv.org\/abs\/2303.12570","DOI":"10.18653\/v1\/2023.emnlp-main.151"},{"key":"e_1_3_2_209_2","doi-asserted-by":"crossref","unstructured":"B. Wang W. Ping P. Xu L. McAfee Z. Liu M. Shoeybi Y. Dong O. Kuchaiev B. Li C. Xiao et al. 2023. Shall we pretrain autoregressive language models with retrieval? A comprehensive study. arXiv:2304.06762. Retrieved from https:\/\/arxiv.org\/abs\/2304.06762","DOI":"10.18653\/v1\/2023.emnlp-main.482"},{"key":"e_1_3_2_210_2","unstructured":"L. Wang N. Yang and F. Wei. 2023. Learning to retrieve in-context examples for large language models. arXiv:2307.07164. Retrieved from https:\/\/arxiv.org\/abs\/2307.07164"},{"key":"e_1_3_2_211_2","unstructured":"J. Liu D. Shen Y. Zhang B. Dolan L. Carin and W. Chen. 2021. What makes good in-context examples for GPT-3? arXiv:2101.06804. Retrieved from https:\/\/arxiv.org\/abs\/2101.06804"},{"key":"e_1_3_2_212_2","unstructured":"O. Rubin J. Herzig and J. Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv:2112.08633. Retrieved from https:\/\/arxiv.org\/abs\/2112.08633"},{"key":"e_1_3_2_213_2","unstructured":"W. Shi S. Min M. Yasunaga M. Seo R. James M. Lewis L. Zettlemoyer and W. Yih. 2023. REPLUG: Retrieval-augmented black-box language models. arXiv:2301.12652. Retrieved from https:\/\/arxiv.org\/abs\/2301.12652"},{"key":"e_1_3_2_214_2","unstructured":"O. Rubin and J. Berant. 2023. Long-range language modeling with self-retrieval. arXiv:2306.13421. Retrieved from https:\/\/arxiv.org\/abs\/2306.13421"},{"key":"e_1_3_2_215_2","first-page":"3929","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Guu K.","year":"2020","unstructured":"K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the International Conference on Machine Learning. PMLR, 3929\u20133938."},{"key":"e_1_3_2_216_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591687"},{"key":"e_1_3_2_217_2","unstructured":"M. Komeili K. Shuster and J. Weston. 2021. Internet-augmented dialogue generation. arXiv:2107.07566. Retrieved from https:\/\/arxiv.org\/abs\/2107.07566"},{"key":"e_1_3_2_218_2","unstructured":"A. Lazaridou E. Gribovskaya W. Stokowiec and N. Grigorev. 2022. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv:2203.05115. Retrieved from https:\/\/arxiv.org\/abs\/2203.05115"},{"key":"e_1_3_2_219_2","unstructured":"D. Gao L. Ji L. Zhou K. Q. Lin J. Chen Z. Fan and M. Z. Shou. 2023. AssistGPT: A general multi-modal assistant that can plan execute inspect and learn. arXiv:2306.08640. Retrieved from https:\/\/arxiv.org\/abs\/2306.08640"},{"key":"e_1_3_2_220_2","unstructured":"P. Lu B. Peng H. Cheng M. Galley K.-W. Chang Y. N. Wu S.-C. Zhu and J. Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv:2304.09842. Retrieved from https:\/\/arxiv.org\/abs\/2304.09842"},{"key":"e_1_3_2_221_2","unstructured":"B. Paranjape S. Lundberg S. Singh H. Hajishirzi L. Zettlemoyer and M. T. Ribeiro. 2023. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv:2303.09014. Retrieved from https:\/\/arxiv.org\/abs\/2303.09014"},{"key":"e_1_3_2_222_2","unstructured":"C.-Y. Hsieh S.-A. Chen C.-L. Li Y. Fujii A. Ratner C.-Y. Lee R. Krishna and T. Pfister. 2023. Tool documentation enables zero-shot tool-usage with large language models. arXiv:2308.00675. Retrieved from https:\/\/arxiv.org\/abs\/2308.00675"},{"key":"e_1_3_2_223_2","unstructured":"Y. Song W. Xiong D. Zhu C. Li K. Wang Y. Tian and S. Li. 2023. RestGPT: Connecting large language models with real-world applications via RESTful APIs. arXiv:2306.06624. Retrieved from https:\/\/arxiv.org\/abs\/2306.06624"},{"key":"e_1_3_2_224_2","doi-asserted-by":"crossref","unstructured":"S. Hao T. Liu Z. Wang and Z. Hu. 2023. ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings. arXiv:2305.11554. Retrieved from https:\/\/arxiv.org\/abs\/2305.11554","DOI":"10.52202\/075280-1988"},{"key":"e_1_3_2_225_2","unstructured":"S. G. Patil T. Zhang X. Wang and J. E. Gonzalez. 2023. Gorilla: Large language model connected with massive APIs. arXiv:2305.15334. Retrieved from https:\/\/arxiv.org\/abs\/2305.15334"},{"key":"e_1_3_2_226_2","unstructured":"Q. Xu F. Hong B. Li C. Hu Z. Chen and J. Zhang. 2023. On the tool manipulation capability of open-source large language models. arXiv:2305.16504. Retrieved from https:\/\/arxiv.org\/abs\/2305.16504"},{"key":"e_1_3_2_227_2","unstructured":"Y. Qin S. Liang Y. Ye K. Zhu L. Yan Y. Lu Y. Lin X. Cong X. Tang B. Qian et al. 2023. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. arXiv:2307.16789. Retrieved from https:\/\/arxiv.org\/abs\/2307.16789"},{"key":"e_1_3_2_228_2","unstructured":"Y. Shen K. Song X. Tan D. Li W. Lu and Y. Zhuang. 2023. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. arXiv:2303.17580. Retrieved from https:\/\/arxiv.org\/abs\/2303.17580"},{"key":"e_1_3_2_229_2","doi-asserted-by":"crossref","unstructured":"Y. Liang C. Wu T. Song W. Wu Y. Xia Y. Liu Y. Ou S. Lu L. Ji S. Mao et al. 2023. TaskMatrix.AI: Completing tasks by connecting foundation models with millions of APIs. arXiv:2303.16434. Retrieved from https:\/\/arxiv.org\/abs\/2303.16434","DOI":"10.34133\/icomputing.0063"},{"key":"e_1_3_2_230_2","doi-asserted-by":"crossref","unstructured":"D. Sur\u00eds S. Menon and C. Vondrick. 2023. ViperGPT: Visual inference via Python execution for reasoning. arXiv:2303.08128. Retrieved from https:\/\/arxiv.org\/abs\/2303.08128","DOI":"10.1109\/ICCV51070.2023.01092"},{"key":"e_1_3_2_231_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12599-016-0444-2"},{"key":"e_1_3_2_232_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(01)00129-1"},{"key":"e_1_3_2_233_2","unstructured":"S. Hong X. Zheng J. Chen Y. Cheng J. Wang C. Zhang Z. Wang S. K. S. Yau Z. Lin L. Zhou et al. 2023. MetaGPT: Meta programming for multi-agent collaborative framework. arXiv:2308.00352. Retrieved from https:\/\/arxiv.org\/abs\/2308.00352"},{"key":"e_1_3_2_234_2","unstructured":"Z. Xi W. Chen X. Guo W. He Y. Ding B. Hong M. Zhang J. Wang S. Jin E. Zhou et al. 2023. The rise and potential of large language model based agents: A survey. arXiv:2309.07864. Retrieved from https:\/\/arxiv.org\/abs\/2309.07864"},{"key":"e_1_3_2_235_2","unstructured":"L. Wang C. Ma X. Feng Z. Zhang H. Yang J. Zhang Z. Chen J. Tang X. Chen Y. Lin et al. 2023. A survey on large language model based autonomous agents. arXiv:2308.11432. Retrieved from https:\/\/arxiv.org\/abs\/2308.11432"},{"key":"e_1_3_2_236_2","first-page":"9118","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Huang W.","year":"2022","unstructured":"W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the International Conference on Machine Learning. PMLR, 9118\u20139147."},{"key":"e_1_3_2_237_2","doi-asserted-by":"crossref","unstructured":"S. Hao Y. Gu H. Ma J. J. Hong Z. Wang D. Z. Wang and Z. Hu. 2023. Reasoning with language model is planning with world model. arXiv:2305.14992. Retrieved from https:\/\/arxiv.org\/abs\/2305.14992","DOI":"10.18653\/v1\/2023.emnlp-main.507"},{"key":"e_1_3_2_238_2","unstructured":"W. Yao S. Heinecke J. C. Niebles Z. Liu Y. Feng L. Xue R. Murthy Z. Chen J. Zhang D. Arpit et al. 2023. Retroformer: Retrospective large language agents with policy gradient optimization. arXiv:2308.02151. Retrieved from https:\/\/arxiv.org\/abs\/2308.02151"},{"key":"e_1_3_2_239_2","volume-title":"Proceedings of the 6th Annual Conference on Robot Learning","author":"Huang W.","year":"2022","unstructured":"W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, T. Jackson, N. Brown, L. Luu, S. Levine, K. Hausman, and B. Ichter. 2022. Inner monologue: Embodied reasoning through planning with language models. In Proceedings of the 6th Annual Conference on Robot Learning. Retrieved from https:\/\/openreview.net\/forum?id=3R3Pz5i0tye."},{"key":"e_1_3_2_240_2","unstructured":"C. Jin W. Tan J. Yang B. Liu R. Song L. Wang and J. Fu. 2023. AlphaBlock: Embodied finetuning for vision-language reasoning in robot manipulation. arXiv:2305.18898. Retrieved from https:\/\/arxiv.org\/abs\/2305.18898"},{"key":"e_1_3_2_241_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48891.2023.10161317"},{"key":"e_1_3_2_242_2","unstructured":"W. Yu N. Gileadi C. Fu S. Kirmani K.-H. Lee M. G. Arenas H.-T. L. Chiang T. Erez L. Hasenclever J. Humplik et al. 2023. Language to rewards for robotic skill synthesis. arXiv:2306.08647. Retrieved from https:\/\/arxiv.org\/abs\/2306.08647"},{"key":"e_1_3_2_243_2","unstructured":"X. Tang A. Zou Z. Zhang Y. Zhao X. Zhang A. Cohan and M. Gerstein. 2023. MedAgents: Large language models as collaborators for zero-shot medical reasoning. arXiv:2311.10537. Retrieved from https:\/\/arxiv.org\/abs\/2311.10537"},{"key":"e_1_3_2_244_2","first-page":"287","volume-title":"Conference on Robot Learning","author":"Brohan A.","year":"2023","unstructured":"A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. 2023. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning. PMLR, 287\u2013318."},{"key":"e_1_3_2_245_2","unstructured":"H. Ha P. Florence and S. Song. 2023. Scaling up and distilling down: Language-guided robot skill acquisition. arXiv:2307.14535. Retrieved from https:\/\/arxiv.org\/abs\/2307.14535"},{"key":"e_1_3_2_246_2","unstructured":"A. Rajvanshi K. Sikka X. Lin B. Lee H.-P. Chiu and A. Velasquez. 2023. SayNav: Grounding large language models for dynamic planning to navigation in new environments. arXiv:2309.04077. Retrieved from https:\/\/arxiv.org\/abs\/2309.04077"},{"key":"e_1_3_2_247_2","unstructured":"C. H. Song J. Wu C. Washington B. M. Sadler W.-L. Chao and Y. Su. 2022. LLM-Planner: Few-shot grounded planning for embodied agents with large language models. arXiv:2212.04088. Retrieved from https:\/\/arxiv.org\/abs\/2212.04088"},{"key":"e_1_3_2_248_2","unstructured":"V. S. Dorbala J. F. Mullen Jr and D. Manocha. 2023. Can an embodied agent find your \u201ccat-shaped mug\u201d? LLM-based zero-shot object navigation. arXiv:2303.03480. Retrieved from https:\/\/arxiv.org\/abs\/2303.03480"},{"key":"e_1_3_2_249_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48891.2023.10160969"},{"key":"e_1_3_2_250_2","doi-asserted-by":"crossref","unstructured":"Y. Ding X. Zhang C. Paxton and S. Zhang. 2023. Task and motion planning with large language models for object rearrangement. arXiv:2303.06247. Retrieved from https:\/\/arxiv.org\/abs\/2303.06247","DOI":"10.1109\/IROS55552.2023.10342169"},{"key":"e_1_3_2_251_2","unstructured":"X. Liu Y. Zheng Z. Du M. Ding Y. Qian Z. Yang and J. Tang. 2021. GPT understands too. arXiv:2103.10385. Retrieved from https:\/\/arxiv.org\/abs\/2103.10385"},{"key":"e_1_3_2_252_2","doi-asserted-by":"crossref","unstructured":"G. Chen F. Liu Z. Meng and S. Liang. 2022. Revisiting parameter-efficient tuning: Are we really there yet? arXiv:2202.07962. Retrieved from https:\/\/arxiv.org\/abs\/2202.07962","DOI":"10.18653\/v1\/2022.emnlp-main.168"},{"key":"e_1_3_2_253_2","doi-asserted-by":"crossref","unstructured":"Y. Wang S. Mukherjee X. Liu J. Gao A. H. Awadallah and J. Gao. 2022. AdaMix: Mixture-of-adapter for parameter-efficient tuning of large language models. arXiv:2205.12410. Retrieved from https:\/\/arxiv.org\/abs\/2205.12410","DOI":"10.18653\/v1\/2022.emnlp-main.388"},{"key":"e_1_3_2_254_2","unstructured":"E. J. Hu Y. Shen P. Wallis Z. Allen-Zhu Y. Li S. Wang L. Wang and W. Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv:2106.09685. Retrieved from https:\/\/arxiv.org\/abs\/2106.09685"},{"key":"e_1_3_2_255_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-short.8"},{"key":"e_1_3_2_256_2","unstructured":"A. Razdaibiedina Y. Mao R. Hou M. Khabsa M. Lewis and A. Almahairi. 2023. Progressive prompts: Continual learning for language models. arXiv:2301.12314. Retrieved from https:\/\/arxiv.org\/abs\/2301.12314"},{"key":"e_1_3_2_257_2","unstructured":"Z.-R. Zhang C. Tan H. Xu C. Wang J. Huang and S. Huang. 2023. Towards adaptive prefix tuning for parameter-efficient language model fine-tuning. arXiv:2305.15212. Retrieved from https:\/\/arxiv.org\/abs\/2305.15212"},{"key":"e_1_3_2_258_2","unstructured":"E. B. Zaken S. Ravfogel and Y. Goldberg. 2021. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv:2106.10199. Retrieved from https:\/\/arxiv.org\/abs\/2106.10199"},{"key":"e_1_3_2_259_2","unstructured":"T. Dettmers M. Lewis Y. Belkada and L. Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. arXiv:2208.07339. Retrieved from https:\/\/arxiv.org\/abs\/2208.07339"},{"key":"e_1_3_2_260_2","unstructured":"E. Frantar S. Ashkboos T. Hoefler and D. Alistarh. 2022. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv:2210.17323. Retrieved from https:\/\/arxiv.org\/abs\/2210.17323"},{"key":"e_1_3_2_261_2","doi-asserted-by":"crossref","unstructured":"X. Wei Y. Zhang Y. Li X. Zhang R. Gong J. Guo and X. Liu. 2023. Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv:2304.09145. Retrieved from https:\/\/arxiv.org\/abs\/2304.09145","DOI":"10.18653\/v1\/2023.emnlp-main.102"},{"key":"e_1_3_2_262_2","first-page":"4475","article-title":"Optimal brain compression: A framework for accurate post-training quantization and pruning","volume":"35","author":"Frantar E.","year":"2022","unstructured":"E. Frantar and D. Alistarh. 2022. Optimal brain compression: A framework for accurate post-training quantization and pruning. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 4475\u20134488.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_263_2","unstructured":"C. Lee J. Jin T. Kim H. Kim and E. Park. 2023. OWQ: Lessons learned from activation outliers for weight quantization in large language models. arXiv:2306.02272. Retrieved from https:\/\/arxiv.org\/abs\/2306.02272"},{"key":"e_1_3_2_264_2","doi-asserted-by":"crossref","unstructured":"S. J. Kwon J. Kim J. Bae K. M. Yoo J.-H. Kim B. Park B. Kim J.-W. Ha N. Sung and D. Lee. 2022. AlphaTuning: Quantization-aware parameter-efficient adaptation of large-scale pre-trained language models. arXiv:2210.03858. Retrieved from https:\/\/arxiv.org\/abs\/2210.03858","DOI":"10.18653\/v1\/2022.findings-emnlp.240"},{"key":"e_1_3_2_265_2","doi-asserted-by":"crossref","unstructured":"T. Dettmers A. Pagnoni A. Holtzman and L. Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. arXiv:2305.14314. Retrieved from https:\/\/arxiv.org\/abs\/2305.14314","DOI":"10.52202\/075280-0441"},{"key":"e_1_3_2_266_2","doi-asserted-by":"crossref","unstructured":"Z. Liu B. Oguz C. Zhao E. Chang P. Stock Y. Mehdad Y. Shi R. Krishnamoorthi and V. Chandra. 2023. LLM-QAT: Data-free quantization aware training for large language models. arXiv:2305.17888. Retrieved from https:\/\/arxiv.org\/abs\/2305.17888","DOI":"10.18653\/v1\/2024.findings-acl.26"},{"key":"e_1_3_2_267_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.430"},{"key":"e_1_3_2_268_2","doi-asserted-by":"crossref","unstructured":"J. Kim J. H. Lee S. Kim J. Park K. M. Yoo S. J. Kwon and D. Lee. 2023. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv:2305.14152. Retrieved from https:\/\/arxiv.org\/abs\/2305.14152","DOI":"10.52202\/075280-1569"},{"key":"e_1_3_2_269_2","unstructured":"M. Sun Z. Liu A. Bair and J. Z. Kolter. 2023. A simple and effective pruning approach for large language models. arXiv:2306.11695. Retrieved from https:\/\/arxiv.org\/abs\/2306.11695"},{"key":"e_1_3_2_270_2","unstructured":"Z. Wang J. Wohlwend and T. Lei. 2019. Structured pruning of large language models. arXiv:1910.04732. Retrieved from https:\/\/arxiv.org\/abs\/1910.04732"},{"key":"e_1_3_2_271_2","unstructured":"L. Yin Y. Wu Z. Zhang C.-Y. Hsieh Y. Wang Y. Jia M. Pechenizkiy Y. Liang Z. Wang and S. Liu. 2023. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. arXiv:2310.05175. Retrieved from https:\/\/arxiv.org\/abs\/2310.05175"},{"key":"e_1_3_2_272_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-acl.692"},{"key":"e_1_3_2_273_2","doi-asserted-by":"crossref","first-page":"23716","DOI":"10.52202\/068431-1723","article-title":"Flamingo: A visual language model for few-shot learning","volume":"35","author":"Alayrac J.-B.","year":"2022","unstructured":"J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. 2022. Flamingo: A visual language model for few-shot learning. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 23716\u201323736.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_274_2","unstructured":"J. Li D. Li S. Savarese and S. Hoi. 2023. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597. Retrieved from https:\/\/arxiv.org\/abs\/2301.12597"},{"key":"e_1_3_2_275_2","unstructured":"H. Liu C. Li Q. Wu and Y. J. Lee. 2023. Visual instruction tuning. arXiv:2304.08485. Retrieved from https:\/\/arxiv.org\/abs\/2304.08485"},{"key":"e_1_3_2_276_2","unstructured":"K. Li Y. He Y. Wang Y. Li W. Wang P. Luo Y. Wang L. Wang and Y. Qiao. 2023. VideoChat: Chat-centric video understanding. arXiv:2305.06355. Retrieved from https:\/\/arxiv.org\/abs\/2305.06355"},{"key":"e_1_3_2_277_2","unstructured":"M. Maaz H. Rasheed S. Khan and F. S. Khan. 2023. Video-ChatGPT: Towards detailed video understanding via large vision and language models. arXiv:2306.05424. Retrieved from https:\/\/arxiv.org\/abs\/2306.05424"},{"key":"e_1_3_2_278_2","doi-asserted-by":"crossref","unstructured":"H. Zhang X. Li and L. Bing. 2023. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. arXiv:2306.02858. Retrieved from https:\/\/arxiv.org\/abs\/2306.02858","DOI":"10.18653\/v1\/2023.emnlp-demo.49"},{"key":"e_1_3_2_279_2","doi-asserted-by":"crossref","unstructured":"X. Mei C. Meng H. Liu Q. Kong T. Ko C. Zhao M. D. Plumbley Y. Zou and W. Wang. 2023. WavCaps: A ChatGPT-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. arXiv:2303.17395. Retrieved from https:\/\/arxiv.org\/abs\/2303.17395","DOI":"10.1109\/TASLP.2024.3419446"},{"key":"e_1_3_2_280_2","unstructured":"C. Lyu M. Wu L. Wang X. Huang B. Liu Z. Du S. Shi and Z. Tu. 2023. Macaw-LLM: Multi-modal language modeling with image audio video and text integration. arXiv:2306.09093. Retrieved from https:\/\/arxiv.org\/abs\/2306.09093"},{"key":"e_1_3_2_281_2","unstructured":"D. Zhu J. Chen X. Shen X. Li and M. Elhoseiny. 2023. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592. Retrieved from https:\/\/arxiv.org\/abs\/2304.10592"},{"key":"e_1_3_2_282_2","unstructured":"A. Dosovitskiy L. Beyer A. Kolesnikov D. Weissenborn X. Zhai T. Unterthiner M. Dehghani M. Minderer G. Heigold S. Gelly et al. 2020. An image is worth 16\u2009\u00d7\u200916 words: Transformers for image recognition at scale. arXiv:2010.11929. Retrieved from https:\/\/arxiv.org\/abs\/2010.11929"},{"key":"e_1_3_2_283_2","doi-asserted-by":"crossref","unstructured":"W. Dai J. Li D. Li A. M. H. Tiong J. Zhao W. Wang B. Li P. Fung and S. Hoi. 2023. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500. Retrieved from https:\/\/arxiv.org\/abs\/2305.06500","DOI":"10.52202\/075280-2142"},{"key":"e_1_3_2_284_2","doi-asserted-by":"crossref","unstructured":"Z. Xu Y. Shen and L. Huang. 2022. MultiInstruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv:2212.10773. Retrieved from https:\/\/arxiv.org\/abs\/2212.10773","DOI":"10.18653\/v1\/2023.acl-long.641"},{"key":"e_1_3_2_285_2","unstructured":"Z. Zhao L. Guo T. Yue S. Chen S. Shao X. Zhu Z. Yuan and J. Liu. 2023. ChatBridge: Bridging modalities with large language model as a language catalyst. arXiv:2305.16103. Retrieved from https:\/\/arxiv.org\/abs\/2305.16103"},{"key":"e_1_3_2_286_2","unstructured":"L. Li Y. Yin S. Li L. Chen P. Wang S. Ren M. Li Y. Yang J. Xu X. Sun et al. 2023. M3IT: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv:2306.04387. Retrieved from https:\/\/arxiv.org\/abs\/2306.04387"},{"key":"e_1_3_2_287_2","doi-asserted-by":"crossref","unstructured":"R. Pi J. Gao S. Diao R. Pan H. Dong J. Zhang L. Yao J. Han H. Xu and L. K. T. Zhang. 2023. DetGPT: Detect what you need via reasoning. arXiv:2305.14167. Retrieved from https:\/\/arxiv.org\/abs\/2305.14167","DOI":"10.18653\/v1\/2023.emnlp-main.876"},{"key":"e_1_3_2_288_2","unstructured":"G. Luo Y. Zhou T. Ren S. Chen X. Sun and R. Ji. 2023. Cheap and quick: Efficient vision-language instruction tuning for large language models. arXiv:2305.15023. Retrieved from https:\/\/arxiv.org\/abs\/2305.15023"},{"key":"e_1_3_2_289_2","unstructured":"R. Zhang J. Han A. Zhou X. Hu S. Yan P. Lu H. Li P. Gao and Y. Qiao. 2023. LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention. arXiv:2303.16199. Retrieved from https:\/\/arxiv.org\/abs\/2303.16199"},{"key":"e_1_3_2_290_2","first-page":"28492","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Radford A.","year":"2023","unstructured":"A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the International Conference on Machine Learning. PMLR, 28492\u201328518."},{"key":"e_1_3_2_291_2","unstructured":"Z. Zhang A. Zhang M. Li H. Zhao G. Karypis and A. Smola. 2023. Multimodal chain-of-thought reasoning in language models. arXiv:2302.00923. Retrieved from https:\/\/arxiv.org\/abs\/2302.00923"},{"key":"e_1_3_2_292_2","unstructured":"J. Ge H. Luo S. Qian Y. Gan J. Fu and S. Zhan. 2023. Chain of thought prompt tuning in vision language models. arXiv:2304.07919. Retrieved from https:\/\/arxiv.org\/abs\/2304.07919"},{"key":"e_1_3_2_293_2","unstructured":"C. Wu S. Yin W. Qi X. Wang Z. Tang and N. Duan. 2023. Visual ChatGPT: Talking drawing and editing with visual foundation models. arXiv:2303.04671. Retrieved from https:\/\/arxiv.org\/abs\/2303.04671"},{"key":"e_1_3_2_294_2","unstructured":"Z. Yang L. Li J. Wang K. Lin E. Azarnasab F. Ahmed Z. Liu C. Liu M. Zeng and L. Wang. 2023. MM-REACT: Prompting ChatGPT for multimodal reasoning and action. arXiv:2303.11381. Retrieved from https:\/\/arxiv.org\/abs\/2303.11381"},{"key":"e_1_3_2_295_2","unstructured":"T. Wang J. Zhang J. Fei Y. Ge H. Zheng Y. Tang Z. Li M. Gao S. Zhao Y. Shan et al. 2023. Caption anything: Interactive image description with diverse multimodal controls. arXiv:2305.02677. Retrieved from https:\/\/arxiv.org\/abs\/2305.02677"},{"key":"e_1_3_2_296_2","doi-asserted-by":"crossref","unstructured":"X. Zhu R. Zhang B. He Z. Zeng S. Zhang and P. Gao. 2022. PointCLIP V2: Adapting CLIP for powerful 3D open-world learning. arXiv:2211.11682. Retrieved from https:\/\/arxiv.org\/abs\/2211.11682","DOI":"10.1109\/ICCV51070.2023.00249"},{"key":"e_1_3_2_297_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01436"},{"key":"e_1_3_2_298_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00680"},{"key":"e_1_3_2_299_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00644"},{"key":"e_1_3_2_300_2","doi-asserted-by":"crossref","unstructured":"H. You R. Sun Z. Wang L. Chen G. Wang H. A. Ayyubi K.-W. Chang and S.-F. Chang. 2023. IdealGPT: Iteratively decomposing vision and language reasoning via large language models. arXiv:2305.14985. Retrieved from https:\/\/arxiv.org\/abs\/2305.14985","DOI":"10.18653\/v1\/2023.findings-emnlp.755"},{"key":"e_1_3_2_301_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01460"},{"key":"e_1_3_2_302_2","unstructured":"T. Q. Nguyen and J. Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. arXiv:1910.05895. Retrieved from https:\/\/arxiv.org\/abs\/1910.05895"},{"key":"e_1_3_2_303_2","unstructured":"Y. Liu M. Ott N. Goyal J. Du M. Joshi D. Chen O. Levy M. Lewis L. Zettlemoyer and V. Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv:1907.11692. Retrieved from https:\/\/arxiv.org\/abs\/1907.11692"},{"key":"e_1_3_2_304_2","unstructured":"X. Geng A. Gudibande H. Liu E. Wallace P. Abbeel S. Levine and D. Song. 2023. Koala: A Dialogue Model for Academic Research Blog Post. Retrieved April 2023 from https:\/\/bair.berkeley.edu\/blog\/2023\/04\/03\/koala\/"},{"key":"e_1_3_2_305_2","unstructured":"L. Gao S. Biderman S. Black L. Golding T. Hoppe C. Foster J. Phang H. He A. Thite N. Nabeshima et al. 2020. The pile: An 800GB dataset of diverse text for language modeling. arXiv:2101.00027. Retrieved from https:\/\/arxiv.org\/abs\/2101.00027"},{"key":"e_1_3_2_306_2","first-page":"31809","article-title":"The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset","volume":"35","author":"Lauren\u00e7on H.","year":"2022","unstructured":"H. Lauren\u00e7on, L. Saulnier, T. Wang, C. Akiki, A. Villanova del Moral, T. Le Scao, L. Von Werra, C. Mou, E. Gonz\u00e1lez Ponferrada, H. Nguyen, et al. 2022. The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 31809\u201331826.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_307_2","unstructured":"Wikipedia. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Main_Page"},{"key":"e_1_3_2_308_2","unstructured":"Together Computer. 2023. RedPajama: An Open Source Recipe to Reproduce LLaMA Training Dataset. 2023. Retrieved April 2023 from https:\/\/github.com\/togethercomputer\/RedPajama-Data"},{"key":"e_1_3_2_309_2","unstructured":"O. Honovich T. Scialom O. Levy and T. Schick. 2022. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv:2212.09689. Retrieved from https:\/\/arxiv.org\/abs\/2212.09689"},{"key":"e_1_3_2_310_2","unstructured":"Y. Bai A. Jones K. Ndousse A. Askell A. Chen N. DasSarma D. Drain S. Fort D. Ganguli T. Henighan et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862. Retrieved from https:\/\/arxiv.org\/abs\/2204.05862"},{"key":"e_1_3_2_311_2","unstructured":"D. Hendrycks C. Burns S. Basart A. Zou M. Mazeika D. Song and J. Steinhardt. 2020. Measuring massive multitask language understanding. arXiv:2009.03300. Retrieved from https:\/\/arxiv.org\/abs\/2009.03300"},{"key":"e_1_3_2_312_2","unstructured":"A. Srivastava A. Rastogi A. Rao A. A. M. Shoeb A. Abid A. Fisch A. R. Brown A. Santoro A. Gupta A. Garriga-Alonso et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615. Retrieved from https:\/\/arxiv.org\/abs\/2206.04615"},{"key":"e_1_3_2_313_2","doi-asserted-by":"crossref","unstructured":"A. Wang A. Singh J. Michael F. Hill O. Levy and S. R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv:1804.07461. Retrieved from https:\/\/arxiv.org\/abs\/1804.07461","DOI":"10.18653\/v1\/W18-5446"},{"key":"e_1_3_2_314_2","unstructured":"Y. Yao Q. Dong J. Guan B. Cao Z. Zhang C. Xiao X. Wang F. Qi J. Bao J. Nie et al. 2021. CUGE: A Chinese language understanding and generation evaluation benchmark. arXiv:2112.13610. Retrieved from https:\/\/arxiv.org\/abs\/2112.13610"},{"key":"e_1_3_2_315_2","unstructured":"L. Xu H. Hu X. Zhang L. Li C. Cao Y. Li Y. Xu K. Sun D. Yu C. Yu et al. 2020. CLUE: A Chinese language understanding evaluation benchmark. arXiv:2004.05986. Retrieved from https:\/\/arxiv.org\/abs\/2004.05986"},{"key":"e_1_3_2_316_2","unstructured":"L. Xu X. Lu C. Yuan X. Zhang H. Xu H. Yuan G. Wei X. Pan X. Tian L. Qin et al. 2021. FewCLUE: A Chinese few-shot learning evaluation benchmark. arXiv:2107.07498. Retrieved from https:\/\/arxiv.org\/abs\/2107.07498"},{"key":"e_1_3_2_317_2","doi-asserted-by":"crossref","unstructured":"E. M. Smith M. Williamson K. Shuster J. Weston and Y.-L. Boureau. 2020. Can you put it all together: Evaluating conversational agents\u2019 ability to blend skills. arXiv:2004.08449. Retrieved from https:\/\/arxiv.org\/abs\/2004.08449","DOI":"10.18653\/v1\/2020.acl-main.183"},{"key":"e_1_3_2_318_2","unstructured":"P. Liang R. Bommasani T. Lee D. Tsipras D. Soylu M. Yasunaga Y. Zhang D. Narayanan Y. Wu A. Kumar et al. 2022. Holistic evaluation of language models. arXiv:2211.09110. Retrieved from https:\/\/arxiv.org\/abs\/2211.09110"},{"key":"e_1_3_2_319_2","unstructured":"S. Park J. Moon S. Kim W. I. Cho J. Han J. Park C. Song J. Kim Y. Song T. Oh et al. 2021. KLUE: Korean language understanding evaluation. arXiv:2105.09680. Retrieved from https:\/\/arxiv.org\/abs\/2105.09680"},{"key":"e_1_3_2_320_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00266"},{"key":"e_1_3_2_321_2","unstructured":"M. T. Pilehvar and J. Camacho-Collados. 2018. WiC: 10 000 example pairs for evaluating context-sensitive representations. arXiv:1808.09121. Retrieved from https:\/\/arxiv.org\/abs\/1808.09121"},{"key":"e_1_3_2_322_2","unstructured":"S. Merity C. Xiong J. Bradbury and R. Socher. 2016. Pointer sentinel mixture models. arXiv:1609.07843. Retrieved from https:\/\/arxiv.org\/abs\/1609.07843"},{"key":"e_1_3_2_323_2","unstructured":"J. W. Rae A. Potapenko S. M. Jayakumar and T. P. Lillicrap. 2019. Compressive transformers for long-range sequence modelling. arXiv:1911.05507. Retrieved from https:\/\/arxiv.org\/abs\/1911.05507"},{"key":"e_1_3_2_324_2","first-page":"1952","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Liu X.","year":"2018","unstructured":"X. Liu, Q. Chen, C. Deng, H. Zeng, J. Chen, D. Li, and B. Tang. 2018. LCQMC: A large-scale Chinese question matching corpus. In Proceedings of the 27th International Conference on Computational Linguistics, 1952\u20131962."},{"key":"e_1_3_2_325_2","unstructured":"S. Iyer N. Dandekar and K. Csernai. 2025. First Quora Dataset Release: Question Pairs. Retrieved from https:\/\/quoradata.quora.com\/First-Quora-Dataset-Release-Question-Pairs"},{"key":"e_1_3_2_326_2","unstructured":"R. Rudinger J. Naradowsky B. Leonard and B. Van Durme. 2018. Gender bias in coreference resolution. arXiv:1804.09301. Retrieved from https:\/\/arxiv.org\/abs\/1804.09301"},{"key":"e_1_3_2_327_2","first-page":"107","article-title":"The CommitmentBank: Investigating projection in naturally occurring discourse","volume":"23","author":"De Marneffe M.-C.","year":"2019","unstructured":"M.-C. De Marneffe, M. Simons, and J. Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung, Vol. 23, 107\u2013124.","journal-title":"Proceedings of Sinn und Bedeutung"},{"key":"e_1_3_2_328_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1430"},{"key":"e_1_3_2_329_2","unstructured":"J. Xu J. Wen X. Sun and Q. Su. 2017. A discourse-level named entity recognition and relation extraction dataset for Chinese literature text. arXiv:1711.07010. Retrieved from https:\/\/arxiv.org\/abs\/1711.07010"},{"key":"e_1_3_2_330_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1536"},{"key":"e_1_3_2_331_2","unstructured":"B. Liu D. Niu H. Wei J. Lin Y. He K. Lai and Y. Xu. 2018. Matching article pairs with graphical decomposition and convolutions. arXiv:1802.07459. Retrieved from https:\/\/arxiv.org\/abs\/1802.07459"},{"key":"e_1_3_2_332_2","unstructured":"P. Li W. Li Z. He X. Wang Y. Cao J. Zhou and W. Xu. 2016. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. arXiv:1607.06275. Retrieved from https:\/\/arxiv.org\/abs\/1607.06275"},{"key":"e_1_3_2_333_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1064"},{"key":"e_1_3_2_334_2","doi-asserted-by":"crossref","unstructured":"W. Ling D. Yogatama C. Dyer and P. Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv:1705.04146. Retrieved from https:\/\/arxiv.org\/abs\/1705.04146","DOI":"10.18653\/v1\/P17-1015"},{"key":"e_1_3_2_335_2","volume-title":". Ontonotes Release 4.0, LDC2011T03","author":"Weischedel R.","year":"2011","unstructured":"R. Weischedel, S. Pradhan, L. Ramshaw, M. Palmer, N. Xue, M. Marcus, A. Taylor, C. Greenberg, E. Hovy, R. Belvin, et al. 2011. Ontonotes Release 4.0, LDC2011T03. Linguistic Data Consortium, Philadelphia, PA."},{"key":"e_1_3_2_336_2","doi-asserted-by":"crossref","unstructured":"D. Vilares and C. G\u00f3mez-Rodr\u00edguez. 2019. HEAD-QA: A healthcare dataset for complex reasoning. arXiv:1906.04701. Retrieved from https:\/\/arxiv.org\/abs\/1906.04701","DOI":"10.18653\/v1\/P19-1092"},{"key":"e_1_3_2_337_2","doi-asserted-by":"crossref","unstructured":"S. L. Blodgett L. Green and B. O\u2019Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. arXiv:1608.08868. Retrieved from https:\/\/arxiv.org\/abs\/1608.08868","DOI":"10.18653\/v1\/D16-1120"},{"key":"e_1_3_2_338_2","doi-asserted-by":"crossref","unstructured":"N. Mostafazadeh N. Chambers X. He D. Parikh D. Batra L. Vanderwende P. Kohli and J. Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv:1604.01696. Retrieved from https:\/\/arxiv.org\/abs\/1604.01696","DOI":"10.18653\/v1\/N16-1098"},{"key":"e_1_3_2_339_2","doi-asserted-by":"crossref","unstructured":"D. Paperno G. Kruszewski A. Lazaridou Q. N. Pham R. Bernardi S. Pezzelle M. Baroni G. Boleda and R. Fern\u00e1ndez. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. arXiv:1606.06031. Retrieved from https:\/\/arxiv.org\/abs\/1606.06031","DOI":"10.18653\/v1\/P16-1144"},{"key":"e_1_3_2_340_2","doi-asserted-by":"crossref","unstructured":"B. Hu Q. Chen and F. Zhu. 2015. LCSTS: A large scale Chinese short text summarization dataset. arXiv:1506.05865. Retrieved from https:\/\/arxiv.org\/abs\/1506.05865","DOI":"10.18653\/v1\/D15-1229"},{"key":"e_1_3_2_341_2","unstructured":"Z. Shao M. Huang J. Wen W. Xu and X. Zhu. 2019. Long and diverse text generation with planning-based hierarchical variational model. arXiv:1908.06605. Retrieved from https:\/\/arxiv.org\/abs\/1908.06605"},{"key":"e_1_3_2_342_2","doi-asserted-by":"crossref","unstructured":"J. Novikova O. Du\u0161ek and V. Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. arXiv:1706.09254. Retrieved from https:\/\/arxiv.org\/abs\/1706.09254","DOI":"10.18653\/v1\/W17-5525"},{"key":"e_1_3_2_343_2","doi-asserted-by":"crossref","unstructured":"C. Zheng M. Huang and A. Sun. 2019. ChiD: A large-scale Chinese IDiom dataset for cloze test. arXiv:1906.01265. Retrieved from https:\/\/arxiv.org\/abs\/1906.01265","DOI":"10.18653\/v1\/P19-1075"},{"key":"e_1_3_2_344_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6239"},{"key":"e_1_3_2_345_2","doi-asserted-by":"crossref","unstructured":"M. Joshi E. Choi D. S. Weld and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv:1705.03551. Retrieved from https:\/\/arxiv.org\/abs\/1705.03551","DOI":"10.18653\/v1\/P17-1147"},{"key":"e_1_3_2_346_2","unstructured":"P. Clark I. Cowhey O. Etzioni T. Khot A. Sabharwal C. Schoenick and O. Tafjord. 2018. Think you have solved question answering? Try ARC the AI2 reasoning challenge. arXiv:1803.05457. Retrieved from https:\/\/arxiv.org\/abs\/1803.05457"},{"key":"e_1_3_2_347_2","doi-asserted-by":"crossref","unstructured":"S. Aroca-Ouellette C. Paik A. Roncone and K. Kann. 2021. PROST: Physical reasoning of objects through space and time. arXiv:2106.03634. Retrieved from https:\/\/arxiv.org\/abs\/2106.03634","DOI":"10.18653\/v1\/2021.findings-acl.404"},{"key":"e_1_3_2_348_2","doi-asserted-by":"crossref","unstructured":"T. Mihaylov P. Clark T. Khot and A. Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. arXiv:1809.02789. Retrieved from https:\/\/arxiv.org\/abs\/1809.02789","DOI":"10.18653\/v1\/D18-1260"},{"key":"e_1_3_2_349_2","volume-title":"Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)","author":"Ferreira T. C.","year":"2020","unstructured":"T. C. Ferreira, C. Gardent, N. Ilinykh, C. Van Der Lee, S. Mille, D. Moussallem, and A. Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task overview and evaluation results (WebNLG+). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)."},{"key":"e_1_3_2_350_2","doi-asserted-by":"crossref","unstructured":"C. Xu W. Zhou T. Ge K. Xu J. McAuley and F. Wei. 2021. Blow the dog whistle: A Chinese dataset for cant understanding with common sense and world knowledge. arXiv:2104.02704. Retrieved from https:\/\/arxiv.org\/abs\/2104.02704","DOI":"10.18653\/v1\/2021.naacl-main.172"},{"key":"e_1_3_2_351_2","doi-asserted-by":"crossref","unstructured":"G. Lai Q. Xie H. Liu Y. Yang and E. Hovy. 2017. RACE: Large-scale reading comprehension dataset from examinations. arXiv:1704.04683. Retrieved from https:\/\/arxiv.org\/abs\/1704.04683","DOI":"10.18653\/v1\/D17-1082"},{"key":"e_1_3_2_352_2","doi-asserted-by":"crossref","unstructured":"E. Choi H. He M. Iyyer M. Yatskar W-T Yih Y. Choi P. Liang and L. Zettlemoyer. 2018. QuAC: Question answering in context. arXiv:1808.07036. Retrieved from https:\/\/arxiv.org\/abs\/1808.07036","DOI":"10.18653\/v1\/D18-1241"},{"key":"e_1_3_2_353_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00370"},{"key":"e_1_3_2_354_2","first-page":"1290","volume-title":"Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning","author":"Boyd-Graber J.","year":"2012","unstructured":"J. Boyd-Graber, B. Satinoff, H. He, and H. Daum\u00e9 Iii. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, 1290\u20131301."},{"key":"e_1_3_2_355_2","doi-asserted-by":"publisher","DOI":"10.3390\/app7080767"},{"key":"e_1_3_2_356_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2883637"},{"key":"e_1_3_2_357_2","doi-asserted-by":"crossref","unstructured":"C. Xu J. Pei H. Wu Y. Liu and C. Li. 2020. MATINF: A jointly labeled large-scale dataset for classification question answering and summarization. arXiv:2004.12302. Retrieved from https:\/\/arxiv.org\/abs\/2004.12302","DOI":"10.18653\/v1\/2020.acl-main.330"},{"key":"e_1_3_2_358_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474381"},{"key":"e_1_3_2_359_2","doi-asserted-by":"crossref","unstructured":"R. Zellers A. Holtzman Y. Bisk A. Farhadi and Y. Choi. 2019. HellaSwag: Can a machine really finish your sentence? arXiv:1905.07830. Retrieved from https:\/\/arxiv.org\/abs\/1905.07830","DOI":"10.18653\/v1\/P19-1472"},{"key":"e_1_3_2_360_2","first-page":"90","volume-title":"Proceedings of the AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning","author":"Roemmele M.","year":"2011","unstructured":"M. Roemmele, C. A. Bejan, and A. S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Proceedings of the AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 90\u201395."},{"key":"e_1_3_2_361_2","volume-title":"Proceedings of the 13th International Conference on the Principles of Knowledge Representation and Reasoning","author":"Levesque H.","year":"2012","unstructured":"H. Levesque, E. Davis, and L. Morgenstern. 2012. The Winograd schema challenge. In Proceedings of the 13th International Conference on the Principles of Knowledge Representation and Reasoning."},{"key":"e_1_3_2_362_2","unstructured":"A. Talmor J. Herzig N. Lourie and J. Berant. 2018. CommonsenseQA: A question answering challenge targeting commonsense knowledge. arXiv:1811.00937. Retrieved from https:\/\/arxiv.org\/abs\/1811.00937"},{"key":"e_1_3_2_363_2","unstructured":"M. Sap H. Rashkin D. Chen R. LeBras and Y. Choi. 2019. SocialIQA: Commonsense reasoning about social interactions. arXiv:1904.09728. Retrieved from https:\/\/arxiv.org\/abs\/1904.09728"},{"key":"e_1_3_2_364_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00305"},{"key":"e_1_3_2_365_2","unstructured":"S. Zhang X. Liu J. Liu J. Gao K. Duh and B. Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv:1810.12885. Retrieved from https:\/\/arxiv.org\/abs\/1810.12885"},{"key":"e_1_3_2_366_2","doi-asserted-by":"crossref","unstructured":"P. Rajpurkar J. Zhang K. Lopyrev and P. Liang. 2016. SQuAD: 100 000+ questions for machine comprehension of text. arXiv:1606.05250. Retrieved from https:\/\/arxiv.org\/abs\/1606.05250","DOI":"10.18653\/v1\/D16-1264"},{"key":"e_1_3_2_367_2","unstructured":"C. Clark K. Lee M.-W. Chang T. Kwiatkowski M. Collins and K. Toutanova. 2019. BooLQ: Exploring the surprising difficulty of natural yes\/no questions. arXiv:1905.10044. Retrieved from https:\/\/arxiv.org\/abs\/1905.10044"},{"key":"e_1_3_2_368_2","doi-asserted-by":"crossref","unstructured":"P. Rajpurkar R. Jia and P. Liang. 2018. Know what you don\u2019t know: Unanswerable questions for SQuAD. arXiv:1806.03822. Retrieved from https:\/\/arxiv.org\/abs\/1806.03822","DOI":"10.18653\/v1\/P18-2124"},{"key":"e_1_3_2_369_2","unstructured":"D. Dua Y. Wang P. Dasigi G. Stanovsky S. Singh and M. Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv:1903.00161. Retrieved from https:\/\/arxiv.org\/abs\/1903.00161"},{"key":"e_1_3_2_370_2","first-page":"177","volume-title":"Proceedings of the Machine Learning Challenges Workshop","author":"Dagan I.","year":"2005","unstructured":"I. Dagan, O. Glickman, and B. Magnini. 2005. The PASCAL recognising textual entailment challenge. In Proceedings of the Machine Learning Challenges Workshop. Springer, 177\u2013190."},{"key":"e_1_3_2_371_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01600"},{"key":"e_1_3_2_372_2","unstructured":"Y. Cui T. Liu Z. Chen W. Ma S. Wang and G. Hu. 2017. Dataset for the first evaluation on Chinese machine reading comprehension. arXiv:1709.08299. Retrieved from https:\/\/arxiv.org\/abs\/1709.08299"},{"key":"e_1_3_2_373_2","unstructured":"Y. Cui T. Liu W. Che L. Xiao Z. Chen W. Ma S. Wang and G. Hu. 2018. A span-extraction dataset for Chinese machine reading comprehension. arXiv:1810.07366. Retrieved from https:\/\/arxiv.org\/abs\/1810.07366"},{"key":"e_1_3_2_374_2","doi-asserted-by":"crossref","unstructured":"Y. Cui T. Liu Z. Yang Z. Chen W. Ma W. Che S. Wang and G. Hu. 2020. A sentence cloze dataset for Chinese machine reading comprehension. arXiv:2004.03116. Retrieved from https:\/\/arxiv.org\/abs\/2004.03116","DOI":"10.18653\/v1\/2020.coling-main.589"},{"key":"e_1_3_2_375_2","unstructured":"Y. Li T. Liu D. Li Q. Li J. Shi and Y. Wang. 2018. Character-based BiLSTM-CRF incorporating POS and dictionaries for Chinese opinion target extraction. In Proceedings of the Asian Conference on Machine Learning. PMLR 518\u2013533."},{"key":"e_1_3_2_376_2","first-page":"252","article-title":"Looking beyond the surface: A challenge set for reading comprehension over multiple sentences","volume":"1","author":"Khashabi D.","year":"2018","unstructured":"D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Long Papers), Vol. 1, 252\u2013262.","journal-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Long Papers)"},{"key":"e_1_3_2_377_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00276"},{"key":"e_1_3_2_378_2","unstructured":"C. C. Shao T. Liu Y. Lai Y. Tseng and S. Tsai. 2018. DRCD: A Chinese machine reading comprehension dataset. arXiv:1806.00920. Retrieved from https:\/\/arxiv.org\/abs\/1806.00920"},{"key":"e_1_3_2_379_2","doi-asserted-by":"crossref","unstructured":"W. He K. Liu J. Liu Y. Lyu S. Zhao X. Xiao Y. Liu Y. Wang H. Wu Q. She et al. 2017. DuReader: A Chinese machine reading comprehension dataset from real-world applications. arXiv:1711.05073. Retrieved from https:\/\/arxiv.org\/abs\/1711.05073","DOI":"10.18653\/v1\/W18-2605"},{"key":"e_1_3_2_380_2","unstructured":"H. Tang J. Liu H. Li Y. Hong H. Wu and H. Wang. 2020. DuReaderrobust: A Chinese dataset towards evaluating the robustness of machine reading comprehension models. arXiv:2004.11142. Retrieved from https:\/\/arxiv.org\/abs\/2004.11142"},{"key":"e_1_3_2_381_2","doi-asserted-by":"crossref","unstructured":"J. Welbl N. F. Liu and M. Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv:1707.06209. Retrieved from https:\/\/arxiv.org\/abs\/1707.06209","DOI":"10.18653\/v1\/W17-4413"},{"key":"e_1_3_2_382_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080809"},{"key":"e_1_3_2_383_2","first-page":"303","volume-title":"Proceedings of the 4th International Conference of the CLEF Initiative on Information Access Evaluation. Multilinguality, Multimodality, and Visualization (CLEF \u201913)","author":"Pe\u00f1as A.","year":"2013","unstructured":"A. Pe\u00f1as, E. Hovy, P. Forner, \u00c1. Rodrigo, R. Sutcliffe, and R. Morante. 2013. QA4MRE 2011-2013: Overview of question answering for machine reading evaluation. In Proceedings of the 4th International Conference of the CLEF Initiative on Information Access Evaluation. Multilinguality, Multimodality, and Visualization (CLEF \u201913). Springer, 303\u2013320."},{"key":"e_1_3_2_384_2","unstructured":"S. Lim M. Kim and J. Lee. 2019. KorQuAD1.0: Korean QA dataset for machine reading comprehension. arXiv:1909.07005. Retrieved from https:\/\/arxiv.org\/abs\/1909.07005"},{"key":"e_1_3_2_385_2","unstructured":"C. Xiao H. Zhong Z. Guo C. Tu Z. Liu M. Sun Y. Feng X. Han Z. Hu H. Wang et al. 2018. CAIL2018: A large-scale legal dataset for judgment prediction. arXiv:1807.02478. Retrieved from https:\/\/arxiv.org\/abs\/1807.02478"},{"key":"e_1_3_2_386_2","unstructured":"D. Hendrycks S. Basart S. Kadavath M. Mazeika A. Arora E. Guo C. Burns S. Puranik H. He D. Song et al. 2021. Measuring coding challenge competence with apps. arXiv:2105.09938. Retrieved from https:\/\/arxiv.org\/abs\/2105.09938"},{"key":"e_1_3_2_387_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1088"},{"key":"e_1_3_2_388_2","unstructured":"K. Cobbe V. Kosaraju M. Bavarian M. Chen H. Jun L. Kaiser M. Plappert J. Tworek J. Hilton R. Nakano et al. 2021. Training verifiers to solve math word problems. arXiv:2110.14168. Retrieved from https:\/\/arxiv.org\/abs\/2110.14168"},{"key":"e_1_3_2_389_2","unstructured":"J. Austin A. Odena M. I. Nye M. Bosma H. Michalewski D. Dohan E. Jiang C. J. Cai M. Terry Q. V. Le and C. Sutton. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_2_390_2","unstructured":"F. Shi M. Suzgun M. Freitag X. Wang S. Srivats S. Vosoughi H. W. Chung Y. Tay S. Ruder D. Zhou et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv:2210.03057. Retrieved from https:\/\/arxiv.org\/abs\/2210.03057"},{"key":"e_1_3_2_391_2","unstructured":"S. Roy and D. Roth. 2016. Solving general arithmetic word problems. arXiv:1608.01413. Retrieved from https:\/\/arxiv.org\/abs\/1608.01413"},{"key":"e_1_3_2_392_2","unstructured":"S.-Y. Miao C.-C. Liang and K.-Y. Su. 2021. A diverse corpus for evaluating and developing English math word problem solvers. arXiv:2106.15772. Retrieved from https:\/\/arxiv.org\/abs\/2106.15772"},{"key":"e_1_3_2_393_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1136"},{"key":"e_1_3_2_394_2","doi-asserted-by":"crossref","unstructured":"A. Patel S. Bhattamishra and N. Goyal. 2021. Are NLP models really able to solve simple math word problems? arXiv:2103.07191. Retrieved from https:\/\/arxiv.org\/abs\/2103.07191","DOI":"10.18653\/v1\/2021.naacl-main.168"},{"key":"e_1_3_2_395_2","first-page":"18319","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Lai Y.","year":"2023","unstructured":"Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W-T Yih, D. Fried, S. Wang, and T. Yu. 2023. DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the International Conference on Machine Learning, PMLR, 18319\u201318345."},{"key":"e_1_3_2_396_2","unstructured":"J. Austin A. Odena M. Nye M. Bosma H. Michalewski D. Dohan E. Jiang C. Cai M. Terry Q. Le et al. 2021. Program synthesis with large language models. arXiv:2108.07732. Retrieved from https:\/\/arxiv.org\/abs\/2108.07732"},{"key":"e_1_3_2_397_2","unstructured":"Y. Nie A. Williams E. Dinan M. Bansal J. Weston and D. Kiela. 2019. Adversarial NLI: A new benchmark for natural language understanding. arXiv:1910.14599. Retrieved from https:\/\/arxiv.org\/abs\/1910.14599"},{"key":"e_1_3_2_398_2","unstructured":"A. Williams N. Nangia and S. R. Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv:1704.05426. Retrieved from https:\/\/arxiv.org\/abs\/1704.05426"},{"key":"e_1_3_2_399_2","doi-asserted-by":"crossref","unstructured":"R. T. McCoy E. Pavlick and T. Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv:1902.01007. Retrieved from https:\/\/arxiv.org\/abs\/1902.01007","DOI":"10.18653\/v1\/P19-1334"},{"key":"e_1_3_2_400_2","doi-asserted-by":"crossref","unstructured":"J. Liu L. Cui H. Liu D. Huang Y. Wang and Y. Zhang. 2020. LogiQA: A challenge dataset for machine reading comprehension with logical reasoning. arXiv:2007.08124. Retrieved from https:\/\/arxiv.org\/abs\/2007.08124","DOI":"10.24963\/ijcai.2020\/501"},{"key":"e_1_3_2_401_2","unstructured":"P. Lewis B. O\u011fuz R. Rinott S. Riedel and H. Schwenk. 2019. MLQA: Evaluating cross-lingual extractive question answering. arXiv:1910.07475. Retrieved from https:\/\/arxiv.org\/abs\/1910.07475"},{"key":"e_1_3_2_402_2","doi-asserted-by":"crossref","unstructured":"A. Conneau G. Lample R. Rinott A. Williams S. R. Bowman H. Schwenk and V. Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. arXiv:1809.05053. Retrieved from https:\/\/arxiv.org\/abs\/1809.05053","DOI":"10.18653\/v1\/D18-1269"},{"key":"e_1_3_2_403_2","unstructured":"Y. Yang Y. Zhang C. Tar and J. Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. arXiv:1908.11828. Retrieved from https:\/\/arxiv.org\/abs\/1908.11828"},{"key":"e_1_3_2_404_2","unstructured":"S. Narayan S. B. Cohen and M. Lapata. 1808. Don\u2019t give me the details just the summary! Topic-aware convolutional neural networks for extreme summarization. arXiv:1808.08745. Retrieved from https:\/\/arxiv.org\/abs\/1808.08745"},{"key":"e_1_3_2_405_2","doi-asserted-by":"crossref","unstructured":"E. M. Ponti G. Glava\u0161 O. Majewska Q. Liu I. Vuli\u0107 and A. Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. arXiv:2005.00333. Retrieved from https:\/\/arxiv.org\/abs\/2005.00333","DOI":"10.18653\/v1\/2020.emnlp-main.185"},{"key":"e_1_3_2_406_2","doi-asserted-by":"crossref","unstructured":"A. Tikhonov and M. Ryabinin. 2021. It\u2019s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. arXiv:2106.12066. Retrieved from https:\/\/arxiv.org\/abs\/2106.12066","DOI":"10.18653\/v1\/2021.findings-acl.310"},{"key":"e_1_3_2_407_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00317"},{"key":"e_1_3_2_408_2","doi-asserted-by":"crossref","unstructured":"T. Scialom P.-A. Dray S. Lamprier B. Piwowarski and J. Staiano. 2020. MLSUM: The multilingual summarization corpus. arXiv:2004.14900. Retrieved from https:\/\/arxiv.org\/abs\/2004.14900","DOI":"10.18653\/v1\/2020.emnlp-main.647"},{"key":"e_1_3_2_409_2","unstructured":"S. Lin J. Hilton and O. Evans. 2021. TruthfulQA: Measuring how models mimic human falsehoods. arXiv:2109.07958. Retrieved from https:\/\/arxiv.org\/abs\/2109.07958"},{"key":"e_1_3_2_410_2","doi-asserted-by":"crossref","unstructured":"I. Augenstein C. Lioma D. Wang L. C. Lima C. Hansen C. Hansen and J. G. Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. arXiv:1909.03242. Retrieved from https:\/\/arxiv.org\/abs\/1909.03242","DOI":"10.18653\/v1\/D19-1475"},{"key":"e_1_3_2_411_2","doi-asserted-by":"crossref","unstructured":"J. Thorne A. Vlachos C. Christodoulopoulos and A. Mittal. 2018. FEVER: A large-scale dataset for fact extraction and verification. arXiv:1803.05355. Retrieved from https:\/\/arxiv.org\/abs\/1803.05355","DOI":"10.18653\/v1\/W18-5501"},{"key":"e_1_3_2_412_2","unstructured":"I. Mollas Z. Chrysopoulou S. Karlos and G. Tsoumakas. 2020. ETHOS: An online hate speech detection dataset. arXiv:2006.08328. Retrieved from https:\/\/arxiv.org\/abs\/2006.08328"},{"key":"e_1_3_2_413_2","unstructured":"M. Nadeem A. Bethke and S. Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv:2004.09456. Retrieved from https:\/\/arxiv.org\/abs\/2004.09456"},{"key":"e_1_3_2_414_2","doi-asserted-by":"crossref","unstructured":"A. Parrish A. Chen N. Nangia V. Padmakumar J. Phang J. Thompson P. M. Htut and S. R. Bowman. 2021. BBQ: A hand-built bias benchmark for question answering. arXiv:2110.08193. Retrieved from https:\/\/arxiv.org\/abs\/2110.08193","DOI":"10.18653\/v1\/2022.findings-acl.165"},{"key":"e_1_3_2_415_2","unstructured":"J. Zhao T. Wang M. Yatskar V. Ordonez and K.-W. Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv:1804.06876. Retrieved from https:\/\/arxiv.org\/abs\/1804.06876"},{"key":"e_1_3_2_416_2","doi-asserted-by":"crossref","unstructured":"N. Nangia C. Vania R. Bhalerao and S. R. Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. arXiv:2010.00133. Retrieved from https:\/\/arxiv.org\/abs\/2010.00133","DOI":"10.18653\/v1\/2020.emnlp-main.154"},{"key":"e_1_3_2_417_2","doi-asserted-by":"crossref","unstructured":"S. Gehman S. Gururangan M. Sap Y. Choi and N. A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. arXiv:2009.11462. Retrieved from https:\/\/arxiv.org\/abs\/2009.11462","DOI":"10.18653\/v1\/2020.findings-emnlp.301"},{"key":"e_1_3_2_418_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317593"},{"key":"e_1_3_2_419_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2301"},{"key":"e_1_3_2_420_2","first-page":"1","volume-title":"Proceedings of the 5th Conference on Machine Translation","author":"Lo\u00efc B.","year":"2020","unstructured":"B. Lo\u00efc, B. Magdalena, B. Ond\u0159ej, F. Christian, G. Yvette, G. Roman, H. Barry, H. Matthias, J. Eric, K. Tom, et al. 2020. Findings of the 2020 conference on machine translation (WMT20). In Proceedings of the 5th Conference on Machine Translation. Association for Computational Linguistics, 1\u201355."},{"key":"e_1_3_2_421_2","unstructured":"W. Li F. Qi M. Sun X. Yi and J. Zhang. 2021. CCPM: A Chinese classical poetry matching dataset. arXiv:2106.01979. Retrieved from https:\/\/arxiv.org\/abs\/2106.01979"},{"key":"e_1_3_2_422_2","unstructured":"E. Dinan S. Roller K. Shuster A. Fan M. Auli and J. Weston. 2018. Wizard of Wikipedia: Knowledge-powered conversational agents. arXiv:1811.01241. Retrieved from https:\/\/arxiv.org\/abs\/1811.01241"},{"key":"e_1_3_2_423_2","unstructured":"H. Rashkin E. M. Smith M. Li and Y.-L. Boureau. 2018. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv:1811.00207. Retrieved from https:\/\/arxiv.org\/abs\/1811.00207"},{"key":"e_1_3_2_424_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-29135-8_7"},{"key":"e_1_3_2_425_2","doi-asserted-by":"crossref","unstructured":"H. Zhou C. Zheng K. Huang M. Huang and X. Zhu. 2020. KdConv: A Chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. arXiv:2004.04100. Retrieved from https:\/\/arxiv.org\/abs\/2004.04100","DOI":"10.18653\/v1\/2020.acl-main.635"},{"key":"e_1_3_2_426_2","unstructured":"L. Co. 2019. IFlytek: A multiple categories Chinese text classifier. Competition Official Website."},{"key":"e_1_3_2_427_2","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v14i1.7347"},{"key":"e_1_3_2_428_2","doi-asserted-by":"crossref","unstructured":"A. Fan Y. Jernite E. Perez D. Grangier J. Weston and M. Auli. 2019. ELI5: Long form question answering. arXiv:1907.09190. Retrieved from https:\/\/arxiv.org\/abs\/1907.09190","DOI":"10.18653\/v1\/P19-1346"},{"key":"e_1_3_2_429_2","unstructured":"Y. Wang S. Mishra P. Alipoormolabashi Y. Kordi A. Mirzaei A. Arunkumar A. Ashok A. S. Dhanasekaran A. Naik D. Stap et al. 2022. Benchmarking generalization via in-context instructions on 1 600+ language tasks. arXiv:2204.07705. Retrieved from https:\/\/arxiv.org\/abs\/2204.07705"},{"key":"e_1_3_2_430_2","doi-asserted-by":"crossref","unstructured":"T. Xie C. H. Wu P. Shi R. Zhong T. Scholak M. Yasunaga C.-S. Wu M. Zhong P. Yin S. I. Wang et al. 2022. UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv:2201.05966. Retrieved from https:\/\/arxiv.org\/abs\/2201.05966","DOI":"10.18653\/v1\/2022.emnlp-main.39"},{"key":"e_1_3_2_431_2","doi-asserted-by":"crossref","unstructured":"Q. Ye B. Y. Lin and X. Ren. 2021. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. arXiv:2104.08835. Retrieved from https:\/\/arxiv.org\/abs\/2104.08835","DOI":"10.18653\/v1\/2021.emnlp-main.572"},{"key":"e_1_3_2_432_2","unstructured":"V. Aribandi Y. Tay T. Schuster J. Rao H. S. Zheng S. V. Mehta H. Zhuang V. Q. Tran D. Bahri J. Ni et al. 2021. ExT5: Towards extreme multi-task scaling for transfer learning. arXiv:2111.10952. Retrieved from https:\/\/arxiv.org\/abs\/2111.10952"},{"key":"e_1_3_2_433_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1101"},{"key":"e_1_3_2_434_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1131"},{"key":"e_1_3_2_435_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.85"},{"key":"e_1_3_2_436_2","doi-asserted-by":"crossref","unstructured":"M. U. Hadi R. Qureshi A. Shah M. Irfan A. Zafar M. B. Shaikh N. Akhtar J. Wu S. Mirjalili et al. 2023. Large language models: A comprehensive survey of its applications challenges limitations and future prospects. Authorea Preprints\u00a01 (2023) 1\u201326.","DOI":"10.36227\/techrxiv.23589741"},{"key":"e_1_3_2_437_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599572"},{"key":"e_1_3_2_438_2","unstructured":"K. Pandya and M. Holia. 2023. Automating customer service using LangChain: Building custom open-source GPT chatbot for organizations. arXiv:2310.05421. Retrieved from https:\/\/arxiv.org\/abs\/2310.05421"},{"key":"e_1_3_2_439_2","unstructured":"J. Li B. Hui G. Qu B. Li J. Yang B. Li B. Wang B. Qin R. Cao R. Geng et al. 2023. Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs. arXiv:2305.03111. Retrieved from https:\/\/arxiv.org\/abs\/2305.03111"},{"key":"e_1_3_2_440_2","doi-asserted-by":"crossref","unstructured":"A. Rao J. Kim M. Kamineni M. Pang W. Lie and M. D. Succi. 2023. Evaluating ChatGPT as an adjunct for radiologic decision-making.\u00a0medRxiv.","DOI":"10.1101\/2023.02.02.23285399"},{"key":"e_1_3_2_441_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.43689"},{"key":"e_1_3_2_442_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00405-023-08104-8"},{"key":"e_1_3_2_443_2","doi-asserted-by":"publisher","DOI":"10.1145\/3582515.3609536"},{"key":"e_1_3_2_444_2","unstructured":"D. Bill and T. Eriksson. 2023. Fine-tuning a LLM using reinforcement learning from human feedback for a therapy chatbot application."},{"key":"e_1_3_2_445_2","unstructured":"M. Abbasian I. Azimi A. M. Rahmani and R. Jain. 2023. Conversational health agents: A personalized LLM-powered agent framework. arXiv:2310.02374. Retrieved from https:\/\/arxiv.org\/abs\/2310.02374"},{"key":"e_1_3_2_446_2","doi-asserted-by":"publisher","DOI":"10.1681\/ASN.0000000000000237"},{"key":"e_1_3_2_447_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10439-023-03306-x"},{"key":"e_1_3_2_448_2","unstructured":"Y. Du S. Zhao Y. Chen R. Bai J. Liu H. Wu H. Wang and B. Qin. 2023. The CALLA dataset: Probing LLMs\u2019 interactive knowledge acquisition from Chinese medical literature. arXiv:2309.04198. Retrieved from https:\/\/arxiv.org\/abs\/2309.04198"},{"key":"e_1_3_2_449_2","doi-asserted-by":"publisher","DOI":"10.2196\/48291"},{"key":"e_1_3_2_450_2","doi-asserted-by":"crossref","unstructured":"A. B. Mbakwe I. Lourentzou L. A. Celi O. J. Mechanic and A. Dagan. 2023. ChatGPT passing USMLE shines a spotlight on the flaws of medical education.","DOI":"10.1371\/journal.pdig.0000205"},{"key":"e_1_3_2_451_2","doi-asserted-by":"publisher","DOI":"10.3946\/kjme.2023.253"},{"key":"e_1_3_2_452_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41433-023-02759-7"},{"key":"e_1_3_2_453_2","doi-asserted-by":"publisher","DOI":"10.3390\/vaccines11071217"},{"key":"e_1_3_2_454_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2023.1166120"},{"issue":"10","key":"e_1_3_2_455_2","first-page":"875","article-title":"Contribution and performance of ChatGPT and other large language models (LLM) for scientific and research advancements: A double-edged sword","volume":"5","author":"Rane N. L.","year":"2023","unstructured":"N. L. Rane, A. Tawde, S. P. Choudhary, and J. Rane. 2023. Contribution and performance of ChatGPT and other large language models (LLM) for scientific and research advancements: A double-edged sword. International Research Journal of Modernization in Engineering Technology and Science 5, 10 (2023), 875\u2013899.","journal-title":"International Research Journal of Modernization in Engineering Technology and Science"},{"key":"e_1_3_2_456_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICALT58122.2023.00100"},{"key":"e_1_3_2_457_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.lindif.2023.102274"},{"key":"e_1_3_2_458_2","doi-asserted-by":"crossref","unstructured":"N. Rane. 2023. Enhancing the quality of teaching and learning through ChatGPT and similar large language models: Challenges future prospects and ethical considerations in education future prospects and ethical considerations in education. Future Prospects and Ethical Considerations in Education\u00a0(15 September 2023).","DOI":"10.2139\/ssrn.4599104"},{"key":"e_1_3_2_459_2","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2023.0140607"},{"key":"e_1_3_2_460_2","doi-asserted-by":"crossref","unstructured":"J. Irons C. Mason P. Cooper S. Sidra A. Reeson and C. Paris. 2023. Exploring the Impacts of ChatGPT on Future Scientific Work. SocArXiv.","DOI":"10.31235\/osf.io\/j2u9x"},{"key":"e_1_3_2_461_2","unstructured":"P. G. Schmidt and A. J. Meir. 2023. Using generative AI for literature searches and scholarly writing: Is the integrity of the scientific discourse in jeopardy? arXiv:2311.06981. Retrieved from https:\/\/arxiv.org\/abs\/2311.06981"},{"key":"e_1_3_2_462_2","unstructured":"Y. Zheng H. Y. Koh J. Ju A. T. Nguyen L. T. May G. I. Webb and S. Pan. 2023. Large language models for scientific synthesis inference and explanation. arXiv:2310.07984. Retrieved from https:\/\/arxiv.org\/abs\/2310.07984"},{"key":"e_1_3_2_463_2","doi-asserted-by":"crossref","unstructured":"B. Aczel and E.-J. Wagenmakers. 2023. Transparency guidance for ChatGPT usage in scientific writing. PsyArXiv.","DOI":"10.31234\/osf.io\/b58ex"},{"key":"e_1_3_2_464_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.rbmo.2023.04.009"},{"key":"e_1_3_2_465_2","unstructured":"S. Imani L. Du and H. Shrivastava. 2023. MathPrompter: Mathematical reasoning using large language models. arXiv:2303.05398. Retrieved from https:\/\/arxiv.org\/abs\/2303.05398"},{"key":"e_1_3_2_466_2","unstructured":"Z. Yuan H. Yuan C. Li G. Dong C. Tan and C. Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv:2308.01825. Retrieved from https:\/\/arxiv.org\/abs\/2308.01825"},{"key":"e_1_3_2_467_2","unstructured":"K. Yang A. M. Swope A. Gu R. Chalamala P. Song S. Yu S. Godil R. Prenger and A. Anandkumar. 2023. LeanDojo: Theorem proving with retrieval-augmented language models. arXiv:2306.15626. Retrieved from https:\/\/arxiv.org\/abs\/2306.15626"},{"key":"e_1_3_2_468_2","doi-asserted-by":"crossref","unstructured":"K. M. Collins A. Q. Jiang S. Frieder L. Wong M. Zilka U. Bhatt T. Lukasiewicz Y. Wu J. B. Tenenbaum W. Hart et al. 2023. Evaluating language models for mathematics through interactions. arXiv:2306.01694. Retrieved from https:\/\/arxiv.org\/abs\/2306.01694","DOI":"10.1073\/pnas.2318124121"},{"key":"e_1_3_2_469_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.metrad.2023.100017"},{"key":"e_1_3_2_470_2","doi-asserted-by":"crossref","unstructured":"J. Dr\u00e1pal H. Westermann and J. Savelka. 2023. Using large language models to support thematic analysis in empirical legal studies. arXiv:2310.18729. Retrieved from https:\/\/arxiv.org\/abs\/2310.18729","DOI":"10.2139\/ssrn.4617116"},{"key":"e_1_3_2_471_2","unstructured":"J. Savelka K. D. Ashley M. A. Gray H. Westermann and H. Xu. 2023. Explaining legal concepts with augmented large language models (GPT-4). arXiv:2306.09525. Retrieved from https:\/\/arxiv.org\/abs\/2306.09525"},{"key":"e_1_3_2_472_2","doi-asserted-by":"crossref","unstructured":"N. Guha J. Nyarko D. E. Ho C. R\u00e9 A. Chilton A. Narayana A. Chohlas-Wood A. Peters B. Waldon D. N. Rockmore et al. 2023. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models. arXiv:2308.11462. Retrieved from https:\/\/arxiv.org\/abs\/2308.11462","DOI":"10.2139\/ssrn.4583531"},{"key":"e_1_3_2_473_2","unstructured":"J. Cui Z. Li Y. Yan B. Chen and L. Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv:2306.16092. Retrieved from https:\/\/arxiv.org\/abs\/2306.16092"},{"key":"e_1_3_2_474_2","doi-asserted-by":"crossref","unstructured":"H. Yang X.-Y. Liu and C. D. Wang. 2023. FinGPT: Open-source financial large language models. arXiv:2306.06031. Retrieved from https:\/\/arxiv.org\/abs\/2306.06031","DOI":"10.2139\/ssrn.4489826"},{"key":"e_1_3_2_475_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604237.3626869"},{"key":"e_1_3_2_476_2","unstructured":"A. Lykov and D. Tsetserukou. 2023. LLM-BRAIn: AI-driven fast generation of robot behaviour tree based on large language model. arXiv:2305.19352. Retrieved from https:\/\/arxiv.org\/abs\/2305.19352"},{"key":"e_1_3_2_477_2","first-page":"905","volume-title":"Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction","author":"Billing E.","year":"2023","unstructured":"E. Billing, J. Ros\u00e9n, and M. Lamb. 2023. Language models for human-robot interaction. In Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction. ACM, 905\u2013906."},{"key":"e_1_3_2_478_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2023.3282111"},{"key":"e_1_3_2_479_2","article-title":"Leveraging commonsense knowledge from large language models for task and motion planning","author":"Ding Y.","year":"2023","unstructured":"Y. Ding, X. Zhang, C. Paxton, and S. Zhang. 2023. Leveraging commonsense knowledge from large language models for task and motion planning. In Proceedings of the RSS 2023 Workshop on Learning for Task and Motion Planning","journal-title":"Proceedings of the RSS 2023 Workshop on Learning for Task and Motion Planning"},{"key":"e_1_3_2_480_2","doi-asserted-by":"crossref","unstructured":"J. Wu R. Antonova A. Kan M. Lepert A. Zeng S. Song J. Bohg S. Rusinkiewicz and T. Funkhouser. 2023. TidyBot: Personalized robot assistance with large language models. arXiv:2305.05658. Retrieved from https:\/\/arxiv.org\/abs\/2305.05658","DOI":"10.1109\/IROS55552.2023.10341577"},{"key":"e_1_3_2_481_2","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Liu Z.","year":"2024","unstructured":"Z. Liu, C. Zhao, F. Iandola, C. Lai, Y. Tian, I. Fedorov, Y. Xiong, E. Chang, Y. Shi, R. Krishnamoorthi, et al. 2024. MobileLLM: Optimizing sub-billion parameter language models for on-device use cases. In Proceedings of the 41st International Conference on Machine Learning."},{"key":"e_1_3_2_482_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_3_2_483_2","first-page":"117","volume-title":"Proceedings of the 18th USENIX symposium on Operating Systems Design and Implementation (ODSI \u201924)","author":"Agrawal A.","year":"2024","unstructured":"A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee. 2024. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve. In Proceedings of the 18th USENIX symposium on Operating Systems Design and Implementation (ODSI \u201924), 117\u2013134."},{"key":"e_1_3_2_484_2","first-page":"521","volume-title":"Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201922)","author":"Yu G.-I.","year":"2022","unstructured":"G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun. 2022. ORCA: A distributed serving system for transformer-based generative models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201922), 521\u2013538."},{"key":"e_1_3_2_485_2","unstructured":"Y. Jiang F. Fu X. Yao G. He X. Miao A. Klimovic B. Cui B. Yuan and E. Yoneki. 2025. Demystifying cost-efficiency in LLM serving over heterogeneous GPUs. arXiv:2502.00722. Retrieved from https:\/\/arxiv.org\/abs\/2502.00722"},{"key":"e_1_3_2_486_2","unstructured":"E. Ren. 2024. Task scheduling for decentralized LLM serving in heterogeneous networks."},{"key":"e_1_3_2_487_2","unstructured":"Y. Jiang F. Fu X. Yao T. Wang B. Cui A. Klimovic and E. Yoneki. 2025. ThunderServe: High-performance and cost-efficient LLM serving in cloud environments. arXiv:2502.09334. Retrieved from https:\/\/arxiv.org\/abs\/2502.09334"},{"key":"e_1_3_2_488_2","unstructured":"R. Qin D. Liu C. Xu Z. Yan Z. Tan Z. Jia A. Nassereldine J. Li M. Jiang A. Abbasi et al. 2024. Empirical guidelines for deploying LLMs onto resource-constrained edge devices. arXiv:2406.03777. Retrieved from https:\/\/arxiv.org\/abs\/2406.03777"},{"key":"e_1_3_2_489_2","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2024.3524255"},{"key":"e_1_3_2_490_2","unstructured":"J. Haris R. Saha W. Hu and J. Cano. 2024. Designing efficient LLM accelerators for edge devices. arXiv:2408.00462. Retrieved from https:\/\/arxiv.org\/abs\/2408.00462"},{"key":"e_1_3_2_491_2","doi-asserted-by":"crossref","unstructured":"E. Strubell A. Ganesh and A. McCallum. 2019. Energy and policy considerations for deep learning in NLP. arXiv:1906.02243. Retrieved from https:\/\/arxiv.org\/abs\/1906.02243","DOI":"10.18653\/v1\/P19-1355"},{"key":"e_1_3_2_492_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"e_1_3_2_493_2","doi-asserted-by":"publisher","DOI":"10.1145\/3446776"},{"key":"e_1_3_2_494_2","doi-asserted-by":"crossref","unstructured":"M. T\u00e4nzer S. Ruder and M. Rei. 2021. Memorisation versus generalisation in pre-trained language models. arXiv:2105.00828. Retrieved from https:\/\/arxiv.org\/abs\/2105.00828","DOI":"10.18653\/v1\/2022.acl-long.521"},{"key":"e_1_3_2_495_2","unstructured":"S. M. West M. Whittaker and K. Crawford. 2019. Discriminating systems.\u00a0AI Now (2019) 1\u201333."},{"key":"e_1_3_2_496_2","unstructured":"K. Valmeekam A. Olmo S. Sreedharan and S. Kambhampati. 2022. Large language models still can\u2019t plan (a benchmark for LLMs on planning and reasoning about change). arXiv:2206.10498. Retrieved from https:\/\/arxiv.org\/abs\/2206.10498"},{"key":"e_1_3_2_497_2","unstructured":"Y. Zhang Y. Li L. Cui D. Cai L. Liu T. Fu X. Huang E. Zhao Y. Zhang Y. Chen et al. 2023. Siren\u2019s song in the AI ocean: A survey on hallucination in large language models. arXiv:2309.01219. Retrieved from https:\/\/arxiv.org\/abs\/2309.01219"},{"key":"e_1_3_2_498_2","unstructured":"A. Webson and E. Pavlick. 2021. Do prompt-based models really understand the meaning of their prompts? arXiv:2109.01247. Retrieved from https:\/\/arxiv.org\/abs\/2109.01247"},{"key":"e_1_3_2_499_2","unstructured":"S. Chen A. Zharmagambetov S. Mahloujifar K. Chaudhuri and C. Guo. 2024. Aligning LLMs to be robust against prompt injection. arXiv:2410.05451. Retrieved from https:\/\/arxiv.org\/abs\/2410.05451"},{"key":"e_1_3_2_500_2","unstructured":"X. Liu H. Cheng P. He W. Chen Y. Wang H. Poon and J. Gao. 2020. Adversarial Training for Large Neural Language Models. Retrieved April 2020 from https:\/\/www.microsoft.com\/en-us\/research\/publication\/adversarial-training-for-large-neural-language-models\/"},{"key":"e_1_3_2_501_2","unstructured":"X. Xu K. Kong N. Liu L. Cui D. Wang J. Zhang and M. Kankanhalli. 2023. An LLM can fool itself: A prompt-based adversarial attack. arXiv:2310.13345. Retrieved from https:\/\/arxiv.org\/abs\/2310.13345"},{"key":"e_1_3_2_502_2","unstructured":"O. Shaikh H. Zhang W. Held M. Bernstein and D. Yang. 2022. On second thought let\u2019s not think step by step! Bias and toxicity in zero-shot reasoning. arXiv:2212.08061. Retrieved from https:\/\/arxiv.org\/abs\/2212.08061"},{"key":"e_1_3_2_503_2","unstructured":"B. C. Das M. H. Amini and Y. Wu. 2024. Security and privacy challenges of large language models: A survey. arXiv:2402.00888. Retrieved from https:\/\/arxiv.org\/abs\/2402.00888"},{"key":"e_1_3_2_504_2","unstructured":"E. Shayegani M. A. A. Mamun Y. Fu P. Zaree Y. Dong and N. Abu-Ghazaleh. 2023. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv:2310.10844. Retrieved from https:\/\/arxiv.org\/abs\/2310.10844"},{"key":"e_1_3_2_505_2","unstructured":"H. Zhao H. Chen F. Yang N. Liu H. Deng H. Cai S. Wang D. Yin and M. Du. 2023. Explainability for large language models: A survey. arXiv:2309.01029. Retrieved from https:\/\/arxiv.org\/abs\/2309.01029"},{"key":"e_1_3_2_506_2","unstructured":"S. Huang S. Mamidanna S. Jangam Y. Zhou and L. H. Gilpin. 2023. Can large language models explain themselves? A study of LLM-generated self-explanations. arXiv:2310.11207. Retrieved from https:\/\/arxiv.org\/abs\/2310.11207"},{"key":"e_1_3_2_507_2","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3534642"},{"key":"e_1_3_2_508_2","doi-asserted-by":"crossref","unstructured":"R. Plant V. Giuffrida and D. Gkatzia. 2022. You are what you write: Preserving privacy in the era of large language models. arXiv:2204.09391. Retrieved from https:\/\/arxiv.org\/abs\/2204.09391","DOI":"10.2139\/ssrn.4417900"},{"key":"e_1_3_2_509_2","unstructured":"W. Niu Z. Kong G. Yuan W. Jiang J. Guan C. Ding P. Zhao S. Liu B. Ren and Y. Wang. 2020. Real-time execution of large-scale language models on mobile. arXiv:2009.06823. Retrieved from https:\/\/arxiv.org\/abs\/2009.06823"},{"key":"e_1_3_2_510_2","doi-asserted-by":"publisher","DOI":"10.1145\/3579371.3589038"},{"key":"e_1_3_2_511_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00873-0"},{"key":"e_1_3_2_512_2","unstructured":"J. Zhang X. Ji Z. Zhao X. Hei and K.-K. R. Choo. 2023. Ethical considerations and policy implications for large language models: Guiding responsible development and deployment. arXiv:2308.02678. Retrieved from https:\/\/arxiv.org\/abs\/2308.02678"},{"key":"e_1_3_2_513_2","doi-asserted-by":"crossref","unstructured":"J. M\u00f6kander J. Schuett H. R. Kirk and L. Floridi. 2024. Auditing large language models: A three-layered approach. AI and Ethics 4 4 (2024) 1085\u20131115.","DOI":"10.1007\/s43681-023-00289-2"},{"key":"e_1_3_2_514_2","doi-asserted-by":"crossref","unstructured":"J. FitzGerald S. Ananthakrishnan K. Arkoudas D. Bernardi A. Bhagia C. Delli Bovi J. Cao R. Chada A. Chauhan L. Chen et al. 2022. Alexa teacher model: Pretraining and distilling multi-billion-parameter encoders for natural language understanding systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining 2893\u20132902.","DOI":"10.1145\/3534678.3539173"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3744746","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T15:55:44Z","timestamp":1772466944000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3744746"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,18]]},"references-count":513,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,10,31]]}},"alternative-id":["10.1145\/3744746"],"URL":"https:\/\/doi.org\/10.1145\/3744746","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,18]]},"assertion":[{"value":"2024-10-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-26","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}