{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T04:22:53Z","timestamp":1783052573573,"version":"3.54.6"},"reference-count":57,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,2,5]],"date-time":"2025-02-05T00:00:00Z","timestamp":1738713600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100017561","name":"Future of Life Institute","doi-asserted-by":"crossref","award":["RFP2-152"],"award-info":[{"award-number":["RFP2-152"]}],"id":[{"id":"10.13039\/100017561","id-type":"DOI","asserted-by":"crossref"}]},{"name":"EU (FEDER) and Spanish","award":["RTI2018-094403-B-C32"],"award-info":[{"award-number":["RTI2018-094403-B-C32"]}]},{"name":"ERDF A way of making Europe, as well as the European Union","award":["INNEST\/2021\/317 (Neurocal\u00e7at)"],"award-info":[{"award-number":["INNEST\/2021\/317 (Neurocal\u00e7at)"]}]},{"name":"Vic. Inv. of the Universitat Polit\u00e8cnica de Valencia","award":["DOCEMPR21, DOCEMPR22, and DOCEMPR23"],"award-info":[{"award-number":["DOCEMPR21, DOCEMPR22, and DOCEMPR23"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2025,4,30]]},"abstract":"<jats:p>\n            Can a language model predict for which questions\n            <jats:italic>another<\/jats:italic>\n            language model will answer successfully? We investigate the extent to which performance prediction is possible and dissect various factors that influence it. Our experimental setting fine-tunes DeBERTa models, which we call\n            <jats:italic>assessors<\/jats:italic>\n            , on the evaluation results of generative language models with up to 128 billion parameters, which we refer to as\n            <jats:italic>subject systems<\/jats:italic>\n            . Our analysis spans more than 100 tasks from BIG-bench. We find that the assessors can match and even exceed the subjects\u2019 confidence in both refinement and calibration, anticipating failures at near perfect levels for some tasks. We also find that for performance prediction it can be beneficial to learn from the scores on multiple tasks or to learn from the scores of multiple subjects, but both depend on the task at hand. Lastly, we find that large and small subject systems are equally predictable, showing promise for the scalability of the predictability problem.\n          <\/jats:p>","DOI":"10.1145\/3706118","type":"journal-article","created":{"date-parts":[[2024,11,28]],"date-time":"2024-11-28T15:55:49Z","timestamp":1732809349000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Analysing the Predictability of Language Model Performance"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9182-4747","authenticated-orcid":false,"given":"Wout","family":"Schellaert","sequence":"first","affiliation":[{"name":"Universitat Polit\u00e8cnica de Val\u00e8ncia, Valencia, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2902-6477","authenticated-orcid":false,"given":"Fernando","family":"Mart\u00ednez-Plumed","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Val\u00e8ncia, Valencia, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9746-7632","authenticated-orcid":false,"given":"Jos\u00e9","family":"Hern\u00e1ndez-Orallo","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Val\u00e8ncia, Valencia, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,2,5]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2021.05.008"},{"key":"e_1_3_2_3_2","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","first-page":"128","DOI":"10.4018\/978-1-60566-766-9.ch006","article-title":"Calibration of machine learning models","author":"Bella Antonio","year":"2010","unstructured":"Antonio Bella, C\u00e8sar Ferri, Jos\u00e9 Hern\u00e1ndez-Orallo, and Mar\u00eda Jos\u00e9 Ram\u00edrez-Quintana. 2010. Calibration of machine learning models. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques. IGI Global, 128\u2013146.","journal-title":"Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462571"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2"},{"key":"e_1_3_2_8_2","volume-title":"Proceedings of the 6th Annual Conference on Robot Learning","author":"Bronstein Eli","year":"2022","unstructured":"Eli Bronstein, Sirish Srinivasan, Supratik Paul, Aman Sinha, Matthew O\u2019Kelly, Payam Nikdel, and Shimon Whiteson. 2022. Embedding synthetic off-policy experience for autonomous driving via zero-shot curricula. In Proceedings of the 6th Annual Conference on Robot Learning, 188\u2013198."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.adf6369"},{"key":"e_1_3_2_10_2","unstructured":"Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay William Fedus Yunxuan Li Xuezhi Wang Mostafa Dehghani Siddhartha Brahma et\u00a0al. 2022. Scaling instruction-finetuned language models. arXiv:2210.11416. Retrieved from https:\/\/arxiv.org\/abs\/2210.11416"},{"key":"e_1_3_2_11_2","unstructured":"Kevin Clark Minh-Thang Luong Quoc V. Le and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv:2003.10555. Retrieved from https:\/\/arxiv.org\/abs\/2003.10555"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.180"},{"key":"e_1_3_2_13_2","volume-title":"Theory and Practice of Item Response Theory","author":"Ayala Rafael Jaime De","year":"2009","unstructured":"Rafael Jaime De Ayala. 2009. Theory and Practice of Item Response Theory. Guilford Publications."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.522"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.21"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2016191118"},{"key":"e_1_3_2_18_2","unstructured":"Dujian Ding Ankur Mallick Chi Wang Robert Sim Subhabrata Mukherjee Victor Ruhle Laks V. S. Lakshmanan and Ahmed Hassan Awadallah. 2024. Hybrid LLM: Cost-efficient and quality-aware query routing. arXiv:2404.14618. Retrieved from https:\/\/arxiv.org\/abs\/2404.14618"},{"key":"e_1_3_2_19_2","author":"Embretson S. E.","year":"2000","unstructured":"S. E. Embretson and S. P. Reise. 2000. Item Response Theory for Psychologists. L. Erlbaum.","journal-title":"Item Response Theory for Psychologists"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.3233\/IDA-230707"},{"key":"e_1_3_2_21_2","unstructured":"William Fedus Jeff Dean and Barret Zoph. 2022. A review of sparse expert models in deep learning. arXiv:2209.01667. Retrieved from https:\/\/arxiv.org\/abs\/2209.01667"},{"key":"e_1_3_2_22_2","unstructured":"Telmo Silva Filho Hao Song Miquel Perello-Nieto Raul Santos-Rodriguez Meelis Kull and Peter Flach. 2021. Classifier calibration: A survey. arXiv:2112.10327. Retrieved from https:\/\/arxiv.org\/abs\/2112.10327"},{"key":"e_1_3_2_23_2","unstructured":"Pengcheng He Jianfeng Gao and Weizhu Chen. 2021. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. arXiv:2111.09543. Retrieved from https:\/\/arxiv.org\/ abs\/2111.09543"},{"key":"e_1_3_2_24_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"He Pengcheng","year":"2021","unstructured":"Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with disentangled attention. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=XPZIaotutsD"},{"key":"e_1_3_2_25_2","unstructured":"Kilian Hendrickx Lorenzo Perini Dries Van der Plas Wannes Meert and Jesse Davis. 2021. Machine learning with a reject option: A survey. arXiv:2107.11277. Retrieved from https:\/\/arxiv.org\/abs\/2107.11277"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i11.21487"},{"key":"e_1_3_2_27_2","unstructured":"Qitian Jason Hu Jacob Bieker Xiuyu Li Nan Jiang Benjamin Keigwin Gaurav Ranganath Kurt Keutzer and Shriyash Kaustubh Upadhyay. 2024. ROUTERBENCH: A benchmark for multi-LLM routing system. arXiv:2403.12031. Retrieved from https:\/\/arxiv.org\/abs\/2403.12031"},{"key":"e_1_3_2_28_2","unstructured":"Andrew Ilyas Sung Min Park Logan Engstrom Guillaume Leclerc and Aleksander Madry. 2022. Datamodels: Predicting predictions from training data. arXiv:2202.00622. Retrieved from https:\/\/arxiv.org\/abs\/2202.00622"},{"key":"e_1_3_2_29_2","unstructured":"Saurav Kadavath Tom Conerly Amanda Askell Tom Henighan Dawn Drain Ethan Perez Nicholas Schiefer Zac Hatfield Dodds Nova DasSarma Eli Tran-Johnson et\u00a0al. 2022. Language models (mostly) know what they know. arXiv:2207.05221. Retrieved from https:\/\/arxiv.org\/abs\/2207.05221"},{"key":"e_1_3_2_30_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Lan Zhenzhong","year":"2020","unstructured":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_31_2","unstructured":"Chia-Hsuan Lee Hao Cheng and Mari Ostendorf. 2023. Orchestrallm: Efficient orchestration of language models for dialogue State tracking. arXiv:2311.09758. Retrieved from https:\/\/arxiv.org\/abs\/2311.09758"},{"key":"e_1_3_2_32_2","unstructured":"Percy Liang Rishi Bommasani Tony Lee Dimitris Tsipras Dilara Soylu Michihiro Yasunaga Yian Zhang Deepak Narayanan Yuhuai Wu Ananya Kumar et\u00a0al. 2022. Holistic evaluation of language models. arXiv:2211.09110. Retrieved from https:\/\/arxiv.org\/abs\/2211.09110"},{"key":"e_1_3_2_33_2","unstructured":"Stephanie Lin Jacob Hilton and Owain Evans. 2022. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research (2022). Retrieved from https:\/\/openreview.net\/forum?id=8s8K2UZGTZ"},{"key":"e_1_3_2_34_2","unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692. Retrieved from https:\/\/arxiv.org\/abs\/1907.11692"},{"key":"e_1_3_2_35_2","unstructured":"Keming Lu Hongyi Yuan Runji Lin Junyang Lin Zheng Yuan Chang Zhou and Jingren Zhou. 2023. Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv:2311.08692. Retrieved from https:\/\/arxiv.org\/abs\/2311.08692"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TG.2018.2883773"},{"key":"e_1_3_2_37_2","first-page":"1140","article-title":"Making sense of item response theory in machine learning","author":"Mart\u00ednez-Plumed Fernando","year":"2016","unstructured":"Fernando Mart\u00ednez-Plumed, Ricardo B. C. Prud\u00eancio, Adolfo Mart\u00ednez-Us\u00f3, and Jos\u00e9 Hern\u00e1ndez-Orallo. 2016. Making sense of item response theory in machine learning. In Proceedings of European Conference on Artificial Intelligence (ECAI \u201916). IOS Press, 1140\u20131148.","journal-title":"Proceedings of European Conference on Artificial Intelligence (ECAI \u201916)"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.09.004"},{"key":"e_1_3_2_39_2","unstructured":"Isaac Ong Amjad Almahairi Vincent Wu Wei-Lin Chiang Tianhao Wu Joseph E. Gonzalez M. Waleed Kadous and Ion Stoica. 2024. Routellm: Learning to route llms with preference data. arXiv:2406.18665. Retrieved from https:\/\/arxiv.org\/abs\/2406.18665"},{"key":"e_1_3_2_40_2","unstructured":"David Owen. 2024. How predictable is language model benchmark performance? arXiv:2401.04757. Retrieved from https:\/\/arxiv.org\/abs\/2401.04757"},{"key":"e_1_3_2_41_2","unstructured":"Lorenzo Pacchiardi Lucy Cheke and Jos\u00e9 Hern\u00e1ndez-Orallo. 2024. 100 instances is all you need: Predicting the success of a new LLM on unseen data by testing on a few instances. In GenAI Evaluation - KDD2024."},{"issue":"8","key":"e_1_3_2_42_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1177012413"},{"key":"e_1_3_2_44_2","first-page":"1","volume-title":"Proceedings of the 1st Edition of the Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM \u201924)","author":"Schellaert Wout","year":"2024","unstructured":"Wout Schellaert, Ronan Hamon, Fernando Mart\u00ednez-Plumed, and Jose Hernandez-Orallo. 2024. A proposal for scaling the scaling Laws. In Proceedings of the 1st Edition of the Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM \u201924).Antonio Valerio Miceli-Barone, Fazl Barez, Shay Cohen, Elena Voita, Ulrich Germann, and Michal Lukasik (Eds.), Association for Computational Linguistics, 1\u20138."},{"key":"e_1_3_2_45_2","unstructured":"Aarohi Srivastava Abhinav Rastogi Abhishek Rao Abu Awal Md Shoeb Abubakar Abid Adam Fisch Adam R. Brown Adam Santoro Aditya Gupta Adri\u00e0 Garriga-Alonso et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615. Retrieved from https:\/\/arxiv.org\/abs\/2206.04615"},{"key":"e_1_3_2_46_2","unstructured":"Romal Thoppilan Daniel De Freitas Jamie Hall Noam Shazeer Apoorv Kulshreshtha Heng-Tze Cheng Alicia Jin Taylor Bos Leslie Baker Yu Du et al. 2022. LaMDA: Language models for dialog applications. arXiv:2201.08239. Retrieved from https:\/\/arxiv.org\/abs\/2201.08239"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372834"},{"key":"e_1_3_2_48_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.92"},{"key":"e_1_3_2_50_2","article-title":"Superglue: A stickier benchmark for general-purpose language understanding systems","volume":"32","author":"Wang Alex","year":"2019","unstructured":"Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, Vol. 32. Retrieved from https:\/\/papers.nips.cc\/paper_files\/paper\/2019\/hash\/4496bf24afe7fab6f046bf4923da8de6-Abstract.html","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eng.2022.04.024"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_3_2_53_2","unstructured":"Qinyuan Ye Harvey Yiyun Fu Xiang Ren and Robin Jia. 2023. How predictable are large language model capabilities? A case study on BIG-bench. arXiv:2305.14947. Retrieved from https:\/\/arxiv.org\/abs\/2305.14947"},{"key":"e_1_3_2_54_2","unstructured":"Qiyuan Zhang Fuyuan Lyu Xue Liu and Chen Ma. 2024. Collaborative performance prediction for large language models. arXiv:2407.01300. Retrieved from https:\/\/arxiv.org\/abs\/2407.01300"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372852"},{"key":"e_1_3_2_56_2","volume-title":"Proceedings of the Workshop on AI Evaluation Beyond Metrics","volume":"3169","author":"Zhou Lexin","year":"2022","unstructured":"Lexin Zhou, Fernando Mart\u00ednez-Plumed, Jos\u00e9 Hern\u00e1ndez-Orallo, C\u00e8sar Ferri, and Wout Schellaert. 2022. Reject before you run: Small assessors anticipate big language models. In Proceedings of the Workshop on AI Evaluation Beyond Metrics. Jos\u00e9 Hern\u00e1ndez-Orallo, Lucy Cheke, Joshua Tenebaum, Tomer Ullman, Fernando Mart\u00ednez-Plumed, Danaja Rutar, John Burden, Ryan Burnell, and Wout Schellaert (Eds.), CEUR Workshop Proceedings, Vol. 3169, CEUR, Vienna, Austria."},{"key":"e_1_3_2_57_2","unstructured":"Lexin Zhou Pablo A. Moreno-Casares Fernando Mart\u00ednez-Plumed John Burden Ryan Burnell Lucy Cheke C\u00e8sar Ferri Alexandru Marcoci Behzad Mehrbakhsh Yael Moros-Daval et al. 2023. Predictable artificial intelligence. arXiv:2310.06167. Retrieved from https:\/\/arxiv.org\/abs\/2310.06167"},{"key":"e_1_3_2_58_2","unstructured":"Barret Zoph Irwan Bello Sameer Kumar Nan Du Yanping Huang Jeff Dean Noam Shazeer and William Fedus. 2022. ST-MoE: Designing stable and transferable sparse expert models. arXiv:2202.08906. Retrieved from https:\/\/arxiv.org\/abs\/2202.08906"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3706118","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3706118","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:18:03Z","timestamp":1750295883000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3706118"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,5]]},"references-count":57,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,4,30]]}},"alternative-id":["10.1145\/3706118"],"URL":"https:\/\/doi.org\/10.1145\/3706118","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,5]]},"assertion":[{"value":"2024-02-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}