{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:34:48Z","timestamp":1791459288767,"version":"4.3.4"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T00:00:00Z","timestamp":1787270400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"Beijing Natural Science Foundation","award":["L253005"],"award-info":[{"award-number":["L253005"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62325201"],"award-info":[{"award-number":["62325201"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Key Laboratory of High Confidence Software Technologies (Peking University), the Ministry of Education, and the Center for Data Space Technology and System, Peking University"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>\n                    Large language model-specific inference engines (in short as\n                    <jats:italic toggle=\"yes\">LLM inference engines<\/jats:italic>\n                    ) have become a fundamental component of modern AI infrastructure, enabling the deployment of LLM-powered applications (LLM apps) across cloud and local devices. Despite their critical role, LLM inference engines are prone to bugs due to the immense resource demands of LLMs and the complexities of cross-platform compatibility. However, a systematic understanding of these bugs remains lacking. To bridge this gap, we present the first empirical study on bugs in LLM inference engines. We mine official repositories of five widely adopted LLM inference engines, constructing a comprehensive dataset of 929 real-world bugs. Through a rigorous open coding process, we analyze these bugs to uncover their symptoms, root causes, commonality, fix effort, fix strategies, and temporal evolution. Our findings reveal six bug symptom types and a taxonomy of 28 root causes, shedding light on the key challenges in bug detection and location within LLM inference engines. Based on these insights, we propose a series of actionable implications for researchers, inference engine vendors, and LLM app developers, along with general guidelines for developing LLM inference engines.\n                  <\/jats:p>","DOI":"10.1145\/3788873","type":"journal-article","created":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T13:16:18Z","timestamp":1769606178000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["A First Look at Bugs in LLM Inference Engines"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7625-8721","authenticated-orcid":false,"given":"Mugeng","family":"Liu","sequence":"first","affiliation":[{"name":"School of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-0134-2683","authenticated-orcid":false,"given":"Siqi","family":"Zhong","sequence":"additional","affiliation":[{"name":"Institute for Artificial Intelligence, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9465-910X","authenticated-orcid":false,"given":"Weichen","family":"Bi","sequence":"additional","affiliation":[{"name":"Institute for Artificial Intelligence, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3466-0165","authenticated-orcid":false,"given":"Yixuan","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-8607-8539","authenticated-orcid":false,"given":"Zhiyang","family":"Chen","sequence":"additional","affiliation":[{"name":"Institute for Artificial Intelligence, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4765-1893","authenticated-orcid":false,"given":"Zhenpeng","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Software, Tsinghua University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7908-8484","authenticated-orcid":false,"given":"Xuanzhe","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7866-4075","authenticated-orcid":false,"given":"Yun","family":"Ma","sequence":"additional","affiliation":[{"name":"Institute for Artificial Intelligence, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,21]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Wikipedia. 2019. Inference Engine. Retrieved October 1 2025 from https:\/\/en.wikipedia.org\/wiki\/Inference_engine"},{"key":"e_1_3_2_3_2","unstructured":"Apple. 2024. Apple Intelligence. Retrieved October 22 2024 from https:\/\/www.apple.com\/apple-intelligence\/"},{"key":"e_1_3_2_4_2","unstructured":"OpenAI. 2024. ChatGPT. Retrieved October 22 2024 from https:\/\/chat.openai.com\/"},{"key":"e_1_3_2_5_2","unstructured":"Google. 2024. Gemma. Retrieved October 22 2024 from https:\/\/ai.google.dev\/gemma\/"},{"key":"e_1_3_2_6_2","unstructured":"Meta. 2024. Llama. Retrieved October 22 2024 from https:\/\/www.llama.com\/"},{"key":"e_1_3_2_7_2","unstructured":"GGML Team. 2024. Llama.cpp. Retrieved October 22 2024 from https:\/\/github.com\/ggerganov\/llama.cpp"},{"key":"e_1_3_2_8_2","unstructured":"Ollama. 2024. Ollama. Retrieved October 22 2024 from https:\/\/github.com\/ollama\/ollama"},{"key":"e_1_3_2_9_2","unstructured":"Google Research. 2024. Unlocking 7B+ Language Models in Your Browser: A Deep Dive with Google AI Edge\u2019s MediaPipe. Retrieved October 22 2024 from https:\/\/research.google\/blog\/unlocking-7b-language-models-in-your-browser-a-deep-dive-with-google-ai-edges-mediapipe\/"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"GitHub. 2025. Awesome-LLM-Inference: LLM Train\/Inference Framework\/Design. Retrieved February 1 2025 from https:\/\/github.com\/DefTruth\/Awesome-LLM-Inference?tab=readme-ov-file#LLM-Train-Inference-Framework","DOI":"10.1109\/ICAISISAS64483.2025.11052174"},{"key":"e_1_3_2_11_2","unstructured":"GitHub. 2025. DeepSpeed: A Deep Learning Optimization Library That Makes Distributed Training and Inference Easy Efficient and Effective. Retrieved February 1 2025 from https:\/\/github.com\/deepspeedai\/DeepSpeed"},{"key":"e_1_3_2_12_2","unstructured":"GitHub. 2025. GitHub REST API. Retrieved October 1 2025 from https:\/\/docs.github.com\/en\/rest?apiVersion=2022-11-28"},{"key":"e_1_3_2_13_2","unstructured":"Scribbr. 2025. Pearson Correlation Coefficient. Retrieved February 1 2025 from https:\/\/www.scribbr.com\/statistics\/pearson-correlation-coefficient\/"},{"key":"e_1_3_2_14_2","unstructured":"GitHub. 2025. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference. Retrieved February 1 2025 from https:\/\/github.com\/NVIDIA\/TensorRT-LLM"},{"key":"e_1_3_2_15_2","unstructured":"GitHub. 2025. vLLM: A High-Throughput and Memory-Efficient Inference and Serving Engine for LLMs. Retrieved February 1 2025 from https:\/\/github.com\/vllm-project\/vllm"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587155"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409759"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00068"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598067"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623343"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510092"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00024"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00080"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_2_27_2","unstructured":"Sigma Jahan Saurabh Singh Rajput Tushar Sharma and Mohammad Masudur Rahman. 2025. Why attention fails: A taxonomy of faults in attention-based neural networks. arXiv:2508.04925. Retrieved form https:\/\/arxiv.org\/abs\/2508.04925"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583566"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00025"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3575693.3575707"},{"key":"e_1_3_2_31_2","unstructured":"Junwei Liu Kaixin Wang Yixuan Chen Xin Peng Zhenpeng Chen Lingming Zhang and Yiling Lou. 2024. Large language model-based agents for software engineering: A survey. arXiv:2409.02977. Retrieved form https:\/\/arxiv.org\/abs\/2409.02977"},{"key":"e_1_3_2_32_2","first-page":"1","article-title":"Coverage-guided tensor compiler fuzzing with joint IR-pass mutation","volume":"6","author":"Liu Jiawei","year":"2022","unstructured":"Jiawei Liu, Yuxiang Wei, Sen Yang, Yinlin Deng, and Lingming Zhang. 2022. Coverage-guided tensor compiler fuzzing with joint IR-pass mutation. Proceedings of the ACM on Programming Languages 6, OOPSLA (2022), 1\u201326.","journal-title":"Proceedings of the ACM on Programming Languages"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2024.112032"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597204"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409760"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00037"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598053"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.799955"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468591"},{"key":"e_1_3_2_41_2","unstructured":"MLC Team. 2023. MLC-LLM. Retrieved from https:\/\/github.com\/mlc-ai\/mlc-llm"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR59073.2023.00019"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409761"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510041"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468558"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00105"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380362"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2019.00020"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3624743"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3788873","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:09:51Z","timestamp":1787328591000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3788873"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,21]]},"references-count":49,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3788873"],"URL":"https:\/\/doi.org\/10.1145\/3788873","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,21]]},"assertion":[{"value":"2025-06-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-07","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}