{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T01:51:20Z","timestamp":1765504280173,"version":"3.48.0"},"publisher-location":"New York, NY, USA","reference-count":47,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,11,10]]},"DOI":"10.1145\/3746252.3761359","type":"proceedings-article","created":{"date-parts":[[2025,11,8]],"date-time":"2025-11-08T00:29:28Z","timestamp":1762561768000},"page":"1302-1312","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Towards Fully-Automated Materials Discovery via Large-Scale Synthesis Dataset and Expert-Level LLM-as-a-Judge"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-3820-7669","authenticated-orcid":false,"given":"Heegyu","family":"Kim","sequence":"first","affiliation":[{"name":"Ajou University, Suwon, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6535-0966","authenticated-orcid":false,"given":"Taeyang","family":"Jeon","sequence":"additional","affiliation":[{"name":"Ajou University, Suwon, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3570-0907","authenticated-orcid":false,"given":"Seungtaek","family":"Choi","sequence":"additional","affiliation":[{"name":"Hankuk University of Foreign Studies, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4934-523X","authenticated-orcid":false,"given":"Ji Hoon","family":"Hong","sequence":"additional","affiliation":[{"name":"Ajou University, Suwon, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5088-4133","authenticated-orcid":false,"given":"Dong Won","family":"Jeon","sequence":"additional","affiliation":[{"name":"Ajou University, Suwon, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8964-587X","authenticated-orcid":false,"given":"Ga-Yeon","family":"Baek","sequence":"additional","affiliation":[{"name":"Hanyang University, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5595-6966","authenticated-orcid":false,"given":"Gyeong-Won","family":"Kwak","sequence":"additional","affiliation":[{"name":"Hanyang University, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0141-8618","authenticated-orcid":false,"given":"Dong-Hee","family":"Lee","sequence":"additional","affiliation":[{"name":"Hanyang University, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6788-5973","authenticated-orcid":false,"given":"Jisu","family":"Bae","sequence":"additional","affiliation":[{"name":"Hanyang University, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8854-7539","authenticated-orcid":false,"given":"Chihoon","family":"Lee","sequence":"additional","affiliation":[{"name":"Hanyang University, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8446-5523","authenticated-orcid":false,"given":"Yoon-Seo","family":"Kim","sequence":"additional","affiliation":[{"name":"Hanyang University, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8567-0668","authenticated-orcid":false,"given":"Seon-Jin","family":"Choi","sequence":"additional","affiliation":[{"name":"Hanyang University, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9070-5666","authenticated-orcid":false,"given":"Jin-Seong","family":"Park","sequence":"additional","affiliation":[{"name":"Hanyang University, Seoul, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3151-0113","authenticated-orcid":false,"given":"Sung Beom","family":"Cho","sequence":"additional","affiliation":[{"name":"Ajou University, Suwon, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9134-1921","authenticated-orcid":false,"given":"Hyunsouk","family":"Cho","sequence":"additional","affiliation":[{"name":"Ajou University, Suwon, Republic of Korea"}]}],"member":"320","published-online":{"date-parts":[[2025,11,10]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Anthropic. 2025. Claude 3.7 Sonnet System Card. https:\/\/assets.anthropic.com\/m\/785e231869ea8b3b\/original\/claude-3-7-sonnet-system-card.pdf."},{"key":"e_1_3_2_1_2_1","volume-title":"Predicting the synthesizability of crystalline inorganic materials from the data of known material compositions. npj Computational Materials","author":"Antoniuk Evan R","year":"2023","unstructured":"Evan R Antoniuk, Gowoon Cheon, George Wang, Daniel Bernstein, William Cai, and Evan J Reed. 2023. Predicting the synthesizability of crystalline inorganic materials from the data of known material compositions. npj Computational Materials, Vol. 9, 1 (2023), 155."},{"key":"e_1_3_2_1_3_1","volume-title":"PyMuPDF4LLM: PDF Text Extraction Library for LLM Applications","author":"Software Artifex","year":"2024","unstructured":"Artifex Software. 2024. PyMuPDF4LLM: PDF Text Extraction Library for LLM Applications. Accessed: January 2024."},{"key":"e_1_3_2_1_4_1","unstructured":"Authors Alliance. 2024. Text and Data Mining Under U.S. Copyright Law: Landscape Flaws & Recommendations. Technical Report. https:\/\/www.authorsalliance.org\/wp-content\/uploads\/2024\/11\/Text-and-Data-Mining-Report-102024.pdf"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1021\/jacs.1c04888"},{"key":"e_1_3_2_1_6_1","volume-title":"Progress toward solid state synthesis by design. Accounts of chemical research","author":"Chamorro Juan R","year":"2018","unstructured":"Juan R Chamorro and Tyrel M McQueen. 2018. Progress toward solid state synthesis by design. Accounts of chemical research, Vol. 51, 11 (2018), 2918-2925."},{"key":"e_1_3_2_1_7_1","unstructured":"Yuan Chiang Elvis Hsieh Chia-Hong Chou and Janosh Riebesell. 2024. LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation. arXiv:2401.17244 [cs.CL] https:\/\/arxiv.org\/abs\/2401.17244"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1021\/ja00757a022"},{"key":"e_1_3_2_1_9_1","unstructured":"DeepSeek-AI Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi Xiaokang Zhang Xingkai Yu Yu Wu Z. F. Wu Zhibin Gou Zhihong Shao Zhuoshu Li Ziyi Gao Aixin Liu Bing Xue Bingxuan Wang Bochao Wu Bei Feng Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang Chong Ruan Damai Dai Deli Chen Dongjie Ji Erhang Li Fangyun Lin Fucong Dai Fuli Luo Guangbo Hao Guanting Chen Guowei Li H. Zhang Han Bao Hanwei Xu Haocheng Wang Honghui Ding Huajian Xin Huazuo Gao Hui Qu Hui Li Jianzhong Guo Jiashi Li Jiawei Wang Jingchang Chen Jingyang Yuan Junjie Qiu Junlong Li J. L. Cai Jiaqi Ni Jian Liang Jin Chen Kai Dong Kai Hu Kaige Gao Kang Guan Kexin Huang Kuai Yu Lean Wang Lecong Zhang Liang Zhao Litong Wang Liyue Zhang Lei Xu Leyi Xia Mingchuan Zhang Minghua Zhang Minghui Tang Meng Li Miaojun Wang Mingming Li Ning Tian Panpan Huang Peng Zhang Qiancheng Wang Qinyu Chen Qiushi Du Ruiqi Ge Ruisong Zhang Ruizhe Pan Runji Wang R. J. Chen R. L. Jin Ruyi Chen Shanghao Lu Shangyan Zhou Shanhuang Chen Shengfeng Ye Shiyu Wang Shuiping Yu Shunfeng Zhou Shuting Pan S. S. Li Shuang Zhou Shaoqing Wu Shengfeng Ye Tao Yun Tian Pei Tianyu Sun T. Wang Wangding Zeng Wanjia Zhao Wen Liu Wenfeng Liang Wenjun Gao Wenqin Yu Wentao Zhang W. L. Xiao Wei An Xiaodong Liu Xiaohan Wang Xiaokang Chen Xiaotao Nie Xin Cheng Xin Liu Xin Xie Xingchao Liu Xinyu Yang Xinyuan Li Xuecheng Su Xuheng Lin X. Q. Li Xiangyue Jin Xiaojin Shen Xiaosha Chen Xiaowen Sun Xiaoxiang Wang Xinnan Song Xinyi Zhou Xianzu Wang Xinxia Shan Y. K. Li Y. Q. Wang Y. X. Wei Yang Zhang Yanhong Xu Yao Li Yao Zhao Yaofeng Sun Yaohui Wang Yi Yu Yichao Zhang Yifan Shi Yiliang Xiong Ying He Yishi Piao Yisong Wang Yixuan Tan Yiyang Ma Yiyuan Liu Yongqiang Guo Yuan Ou Yuduan Wang Yue Gong Yuheng Zou Yujia He Yunfan Xiong Yuxiang Luo Yuxiang You Yuxuan Liu Yuyang Zhou Y. X. Zhu Yanhong Xu Yanping Huang Yaohui Li Yi Zheng Yuchen Zhu Yunxian Ma Ying Tang Yukun Zha Yuting Yan Z. Z. Ren Zehui Ren Zhangli Sha Zhe Fu Zhean Xu Zhenda Xie Zhengyan Zhang Zhewen Hao Zhicheng Ma Zhigang Yan Zhiyu Wu Zihui Gu Zijia Zhu Zijun Liu Zilin Li Ziwei Xie Ziyang Song Zizheng Pan Zhen Huang Zhipeng Xu Zhongyu Zhang and Zhen Zhang. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https:\/\/arxiv.org\/abs\/2501.12948"},{"key":"e_1_3_2_1_10_1","volume-title":"Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. npj Computational Materials","author":"Dunn Alexander","year":"2020","unstructured":"Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. 2020. Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. npj Computational Materials, Vol. 6, 1 (2020), 138."},{"key":"e_1_3_2_1_11_1","unstructured":"Google. 2025. Gemini 2.0 Flash \u00a0|\u00a0 Generative AI on Vertex AI \u00a0|\u00a0 Google Cloud -- cloud.google.com. https:\/\/cloud.google.com\/vertex-ai\/generative-ai\/docs\/models\/gemini\/2-0-flash."},{"key":"e_1_3_2_1_12_1","unstructured":"Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Alex Vaughan et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)."},{"key":"e_1_3_2_1_13_1","unstructured":"Jiawei Gu Xuhui Jiang Zhichao Shi Hexiang Tan Xuehao Zhai Chengjin Xu Wei Li Yinghan Shen Shengjie Ma Honghao Liu Saizhuo Wang Kun Zhang Yuanzhuo Wang Wen Gao Lionel Ni and Jian Guo. 2025. A Survey on LLM-as-a-Judge. arXiv:2411.15594 [cs.CL] https:\/\/arxiv.org\/abs\/2411.15594"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.3390\/ma16175977"},{"key":"e_1_3_2_1_15_1","unstructured":"Huggingface. [n.d.]. GitHub - huggingface\/trl: Train transformer language models with reinforcement learning. -- github.com. https:\/\/github.com\/huggingface\/trl."},{"key":"e_1_3_2_1_16_1","unstructured":"Aaron Hurst Adam Lerer Adam P Goucher Adam Perelman Aditya Ramesh Aidan Clark AJ Ostrow Akila Welihinda Alan Hayes Alec Radford et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)."},{"key":"e_1_3_2_1_17_1","volume-title":"LLMatDesign: Autonomous Materials Discovery with Large Language Models. arXiv preprint arXiv:2406.13163","author":"Jia Shuyi","year":"2024","unstructured":"Shuyi Jia, Chao Zhang, and Victor Fung. 2024. LLMatDesign: Autonomous Materials Discovery with Large Language Models. arXiv preprint arXiv:2406.13163 (2024)."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1021\/jacs.4c05840"},{"key":"e_1_3_2_1_19_1","volume-title":"Text-mined dataset of inorganic materials synthesis recipes. Scientific data","author":"Kononova Olga","year":"2019","unstructured":"Olga Kononova, Haoyan Huo, Tanjin He, Ziqin Rong, Tiago Botari, Wenhao Sun, Vahe Tshitoyan, and Gerbrand Ceder. 2019. Text-mined dataset of inorganic materials synthesis recipes. Scientific data, Vol. 6, 1 (2019), 203."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Shrinidhi Kumbhar Venkatesh Mishra Kevin Coutinho Divij Handa Ashif Iquebal and Chitta Baral. 2025. Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents. arXiv:2501.13299 [cs.CL] https:\/\/arxiv.org\/abs\/2501.13299","DOI":"10.18653\/v1\/2025.findings-naacl.420"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1039\/D4DD00074A"},{"key":"e_1_3_2_1_22_1","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et al., 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, Vol. 33 (2020), 9459-9474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_23_1","volume-title":"Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74-81.","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74-81."},{"key":"e_1_3_2_1_24_1","unstructured":"Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101 [cs.LG] https:\/\/arxiv.org\/abs\/1711.05101"},{"key":"e_1_3_2_1_25_1","first-page":"80","volume-title":"Nature","volume":"624","author":"Merchant Amil","year":"2023","unstructured":"Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. 2023. Scaling deep learning for materials discovery. Nature, Vol. 624, 7990 (2023), 80-85."},{"key":"e_1_3_2_1_26_1","unstructured":"Paulius Micikevicius Sharan Narang Jonah Alben Gregory Diamos Erich Elsen David Garcia Boris Ginsburg Michael Houston Oleksii Kuchaiev Ganesh Venkatesh et al. 2017. Mixed precision training. arXiv preprint arXiv:1710.03740 (2017)."},{"key":"e_1_3_2_1_27_1","volume-title":"Santiago Miret, NM Anoop Krishnan, et al., [n.d.]. LLaMat: Large Language Models for Materials Science. In AI for Accelerated Materials Design-Vienna","author":"Mishra Vaibhav","year":"2024","unstructured":"Vaibhav Mishra, Somaditya Singh, Mohd Zaki, Hargun Singh Grover, Santiago Miret, NM Anoop Krishnan, et al., [n.d.]. LLaMat: Large Language Models for Materials Science. In AI for Accelerated Materials Design-Vienna 2024."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1021\/acs.chemmater.3c01834"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.101"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1063\/5.0021106"},{"key":"e_1_3_2_1_31_1","unstructured":"OpenAI. 2022. Introducing text and code embeddings. https:\/\/openai.com\/index\/introducing-text-and-code-embeddings\/. [Accessed 11-02-2025]."},{"key":"e_1_3_2_1_32_1","unstructured":"OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence -- openai.com. https:\/\/openai.com\/index\/gpt-4o-mini-advancing-cost-efficient-intelligence\/."},{"key":"e_1_3_2_1_33_1","unstructured":"OpenAI. 2025. OpenAI o3-mini -- openai.com. https:\/\/openai.com\/index\/openai-o3-mini\/."},{"key":"e_1_3_2_1_34_1","volume-title":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311-318","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311-318."},{"key":"e_1_3_2_1_35_1","unstructured":"Qwen: An Yang Baosong Yang Beichen Zhang Binyuan Hui Bo Zheng Bowen Yu Chengyuan Li Dayiheng Liu Fei Huang Haoran Wei Huan Lin Jian Yang Jianhong Tu Jianwei Zhang Jianxin Yang Jiaxi Yang Jingren Zhou Junyang Lin Kai Dang Keming Lu Keqin Bao Kexin Yang Le Yu Mei Li Mingfeng Xue Pei Zhang Qin Zhu Rui Men Runji Lin Tianhao Li Tianyi Tang Tingyu Xia Xingzhang Ren Xuancheng Ren Yang Fan Yang Su Yichang Zhang Yu Wan Yuqiong Liu Zeyu Cui Zhenru Zhang and Zihan Qiu. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [cs.CL] https:\/\/arxiv.org\/abs\/2412.15115"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00024"},{"key":"e_1_3_2_1_37_1","volume-title":"Intraclass correlations: uses in assessing rater reliability. Psychological bulletin","author":"Shrout Patrick E","year":"1979","unstructured":"Patrick E Shrout and Joseph L Fleiss. 1979. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin, Vol. 86, 2 (1979), 420."},{"key":"e_1_3_2_1_38_1","volume-title":"Matsci-nlp: Evaluating scientific language models on materials science language tasks using text-to-schema modeling. arXiv preprint arXiv:2305.08264","author":"Song Yu","year":"2023","unstructured":"Yu Song, Santiago Miret, and Bang Liu. 2023. Matsci-nlp: Evaluating scientific language models on materials science language tasks using text-to-schema modeling. arXiv preprint arXiv:2305.08264 (2023)."},{"key":"e_1_3_2_1_39_1","volume-title":"A critical reflection on attempts to machine-learn materials synthesis insights from text-mined literature recipes. Faraday Discussions","author":"Sun Wenhao","year":"2025","unstructured":"Wenhao Sun and Nicholas David. 2025. A critical reflection on attempts to machine-learn materials synthesis insights from text-mined literature recipes. Faraday Discussions (2025)."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"crossref","unstructured":"Zheren Wang Olga Kononova Kevin Cruse Tanjin He Haoyan Huo Yuxing Fei Yan Zeng Yingzhi Sun Zijian Cai Wenhao Sun et al. 2022. Dataset of solution-based inorganic materials synthesis procedures extracted from the scientific literature. Scientific data Vol. 9 1 (2022) 231.","DOI":"10.1038\/s41597-022-01317-2"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"crossref","unstructured":"Thomas Wolf Lysandre Debut Victor Sanh Julien Chaumond Clement Delangue Anthony Moi Pierric Cistac Tim Rault R\u00e9mi Louf Morgan Funtowicz et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019).","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41524-023-01000-z"},{"key":"e_1_3_2_1_43_1","volume-title":"Small data machine learning in materials science. npj Computational Materials","author":"Xu Pengcheng","year":"2023","unstructured":"Pengcheng Xu, Xiaobo Ji, Minjie Li, and Wencong Lu. 2023b. Small data machine learning in materials science. npj Computational Materials, Vol. 9, 1 (2023), 42."},{"key":"e_1_3_2_1_44_1","volume-title":"Chi Thang Nguyen, and Bratin Sengupta","author":"Yanguas-Gil Angel","year":"2024","unstructured":"Angel Yanguas-Gil, Matthew T Dearing, Jeffrey W Elam, Jessica C Jones, Sungjoon Kim, Adnan Mohammad, Chi Thang Nguyen, and Bratin Sengupta. 2024. Benchmarking large language models for materials synthesis: the case of atomic layer deposition. arXiv preprint arXiv:2412.10477 (2024)."},{"key":"e_1_3_2_1_45_1","volume-title":"Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675","author":"Zhang Tianyi","year":"2019","unstructured":"Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)."},{"key":"e_1_3_2_1_46_1","volume-title":"Synthesis of a three-dimensional cross-linked Ni-V 2 O 5 nanomaterial in an ionic liquid for lithium-ion batteries. RSC advances","author":"Zhao Yu","year":"2020","unstructured":"Yu Zhao, Dongru Gao, Ruxin Guan, Hongwei Li, Ning Li, Guixian Li, and Shiyou Li. 2020. Synthesis of a three-dimensional cross-linked Ni-V 2 O 5 nanomaterial in an ionic liquid for lithium-ion batteries. RSC advances, Vol. 10, 64 (2020), 39137-39145."},{"key":"e_1_3_2_1_47_1","first-page":"46595","article-title":"Judging llm-as-a-judge with mt-bench and chatbot arena","volume":"36","author":"Zheng Lianmin","year":"2023","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al., 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, Vol. 36 (2023), 46595-46623.","journal-title":"Advances in Neural Information Processing Systems"}],"event":{"name":"CIKM '25: The 34th ACM International Conference on Information and Knowledge Management","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval","SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"],"location":"Seoul Republic of Korea","acronym":"CIKM '25"},"container-title":["Proceedings of the 34th ACM International Conference on Information and Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3746252.3761359","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T01:47:29Z","timestamp":1765504049000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3746252.3761359"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,10]]},"references-count":47,"alternative-id":["10.1145\/3746252.3761359","10.1145\/3746252"],"URL":"https:\/\/doi.org\/10.1145\/3746252.3761359","relation":{},"subject":[],"published":{"date-parts":[[2025,11,10]]},"assertion":[{"value":"2025-11-10","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}