{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T11:06:43Z","timestamp":1785409603006,"version":"3.56.0"},"reference-count":66,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T00:00:00Z","timestamp":1782864000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T00:00:00Z","timestamp":1782950400000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004233","name":"Universitat Polit\u00e8cnica de Val\u00e8ncia","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004233","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2026,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>An AI assessor is an external, ideally independent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can leverage information from the test results of many other AI systems and have the flexibility of being trained on any loss function or scoring rule: from squared error to toxicity metrics. Here we address the question: is it always optimal to train the assessor for the target metric? Or could it be better to train for a different metric and then map predictions back to the target metric? Using twenty regression and classification problems with tabular data, we experimentally explore this question for, respectively, regression losses and classification scores with monotonic and nonmonotonic mappings and find that, contrary to intuition, optimising for more informative metrics (i.e., yielding a better-conditioned supervision signal) is not universally preferred. Surprisingly, some monotonic transformations are promising. For example, logistic loss is useful for minimising absolute or quadratic errors in regression, and logarithmic score helps maximise quadratic or spherical scores in classification.<\/jats:p>","DOI":"10.1007\/s10994-026-07084-0","type":"journal-article","created":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T18:09:56Z","timestamp":1783015796000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["What Should an AI Assessor Optimise for?"],"prefix":"10.1007","volume":"115","author":[{"given":"Daniel","family":"Romero-Alvarado","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fernando","family":"Mart\u00ednez-Plumed","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jos\u00e9","family":"Hern\u00e1ndez-Orallo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,7,2]]},"reference":[{"key":"7084_CR1","doi-asserted-by":"publisher","unstructured":"Alpaydin, E., & Alimoglu, F. (1996). Pen-based recognition of handwritten digits. UCI machine learning repository. https:\/\/doi.org\/10.24432\/C5MG6K","DOI":"10.24432\/C5MG6K"},{"key":"7084_CR2","doi-asserted-by":"publisher","DOI":"10.24432\/C5DK5C","author":"M Bain","year":"1994","unstructured":"Bain, M., & Hoff, A. (1994). Chess (King-Rook vs. King). UCI Machine Learning Repository. https:\/\/doi.org\/10.24432\/C5DK5C","journal-title":"UCI Machine Learning Repository"},{"issue":"2","key":"7084_CR3","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1080\/00401706.1974.10489171","volume":"16","author":"AE Beaton","year":"1974","unstructured":"Beaton, A. E., & Tukey, J. W. (1974). The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data. Technometrics, 16(2), 147\u2013185. https:\/\/doi.org\/10.1080\/00401706.1974.10489171","journal-title":"Technometrics"},{"key":"7084_CR4","doi-asserted-by":"publisher","DOI":"10.24432\/C5XW20","author":"B Becker","year":"1996","unstructured":"Becker, B., & Kohavi, R. (1996). Adult. UCI Machine Learning Repository. https:\/\/doi.org\/10.24432\/C5XW20","journal-title":"UCI Machine Learning Repository"},{"key":"7084_CR5","doi-asserted-by":"publisher","unstructured":"Belagiannis, V., Rupprecht, C., Carneiro, G., & Navab, N. (2015). Robust optimization for deep regression. In Proceedings of the IEEE international conference on computer vision (ICCV), pp. 2830\u20132838. IEEE, Piscataway, NJ, USA. https:\/\/doi.org\/10.1109\/ICCV.2015.324.","DOI":"10.1109\/ICCV.2015.324"},{"key":"7084_CR6","volume-title":"Pattern recognition and machine learning","author":"CM Bishop","year":"2006","unstructured":"Bishop, C. M., & Nasrabadi, N. M. (2006). Pattern recognition and machine learning (Vol. 4). Springer."},{"key":"7084_CR7","doi-asserted-by":"publisher","unstructured":"Bock, R. (2004). MAGIC gamma telescope. UCI machine learning repository. https:\/\/doi.org\/10.24432\/C52C8B","DOI":"10.24432\/C52C8B"},{"key":"7084_CR8","doi-asserted-by":"publisher","unstructured":"Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021). On the opportunities and risks of foundation models. Preprint at arXiv:2108.07258. https:\/\/doi.org\/10.48550\/arXiv.2108.07258.","DOI":"10.48550\/arXiv.2108.07258"},{"key":"7084_CR9","doi-asserted-by":"publisher","unstructured":"Botchkarev, A. (2018). Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology. Preprint at arXiv:1809.03006. https:\/\/doi.org\/10.48550\/arXiv.1809.03006","DOI":"10.48550\/arXiv.1809.03006"},{"key":"7084_CR10","doi-asserted-by":"publisher","first-page":"045","DOI":"10.28945\/4184","volume":"14","author":"A Botchkarev","year":"2019","unstructured":"Botchkarev, A. (2019). A new typology design of performance metrics to measure errors in machine learning regression algorithms. Interdisciplinary Journal of Information, Knowledge, and Management, 14, 045\u2013076. https:\/\/doi.org\/10.28945\/4184","journal-title":"Interdisciplinary Journal of Information, Knowledge, and Management"},{"key":"7084_CR11","doi-asserted-by":"publisher","DOI":"10.1201\/9781315139470","volume-title":"Classification and regression trees","author":"L Breiman","year":"1984","unstructured":"Breiman, L., Friedman, J., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees. Taylor & Francis. https:\/\/doi.org\/10.1201\/9781315139470"},{"key":"7084_CR12","unstructured":"Bronstein, E., Srinivasan, S., Paul, S., Sinha, A., O\u2019Kelly, M., Nikdel, P., & Whiteson, S. (2022). Embedding synthetic off-policy experience for autonomous driving via zero-shot curricula. In Proceedings of the 6th conference on robot learning (CoRL). Proceedings of Machine Learning Research, vol. 205, pp. 188\u2013198. https:\/\/proceedings.mlr.press\/v205\/bronstein23a.html."},{"issue":"6641","key":"7084_CR13","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1126\/science.adf6369","volume":"380","author":"R Burnell","year":"2023","unstructured":"Burnell, R., Schellaert, W., Burden, J., Ullman, T. D., Mart\u00ednez-Plumed, F., Tenenbaum, J. B., Rutar, D., Cheke, L. G., Sohl-Dickstein, J., Mitchell, M., Kiela, D., Shanahan, M., Voorhees, E. M., Cohn, A. G., Leibo, J. Z., & Hern\u00e1ndez-Orallo, J. (2023). Rethink reporting of evaluation results in AI. Science, 380(6641), 136\u2013138. https:\/\/doi.org\/10.1126\/science.adf6369","journal-title":"Science"},{"issue":"43","key":"7084_CR14","doi-asserted-by":"publisher","first-page":"1165","DOI":"10.15585\/mmwr.mm6643a2","volume":"66","author":"NR Burrows","year":"2017","unstructured":"Burrows, N. R. (2017). Incidence of end-stage renal disease attributed to diabetes among persons with diagnosed diabetes\u2013US and Puerto Rico, 2000\u20132014. MMWR. Morbidity and Mortality Weekly Report, 66(43), 1165\u20131170. https:\/\/doi.org\/10.15585\/mmwr.mm6643a2","journal-title":"MMWR. Morbidity and Mortality Weekly Report"},{"key":"7084_CR15","doi-asserted-by":"crossref","unstructured":"Candillier, L., & Lemaire, V. (2012). Design and analysis of the Nomao challenge\u2014active learning in the real-world. In Proceedings of the ALRA workshop on active learning in real-world applications @ ECML-PKDD, Bristol, UK, pp. 1\u201315.","DOI":"10.1109\/IJCNN.2013.6706908"},{"key":"7084_CR16","doi-asserted-by":"publisher","unstructured":"Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (KDD), pp. 785\u2013794. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/2939672.2939785.","DOI":"10.1145\/2939672.2939785"},{"key":"7084_CR17","doi-asserted-by":"publisher","first-page":"623","DOI":"10.7717\/peerj-cs.623","volume":"7","author":"D Chicco","year":"2021","unstructured":"Chicco, D., Warrens, M. J., & Jurman, G. (2021). The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. PeerJ Computer Science, 7, 623. https:\/\/doi.org\/10.7717\/peerj-cs.623","journal-title":"PeerJ Computer Science"},{"issue":"3","key":"7084_CR18","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/BF00994018","volume":"20","author":"C Cortes","year":"1995","unstructured":"Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273\u2013297. https:\/\/doi.org\/10.1007\/BF00994018","journal-title":"Machine Learning"},{"issue":"10","key":"7084_CR19","doi-asserted-by":"publisher","first-page":"4915","DOI":"10.1016\/j.eswa.2014.02.026","volume":"41","author":"A Dal Pozzolo","year":"2014","unstructured":"Dal Pozzolo, A., Caelen, O., Le Borgne, Y.-A., Waterschoot, S., & Bontempi, G. (2014). Learned lessons in credit card fraud detection from a practitioner perspective. Expert Systems with Applications, 41(10), 4915\u20134928. https:\/\/doi.org\/10.1016\/j.eswa.2014.02.026","journal-title":"Expert Systems with Applications"},{"key":"7084_CR20","doi-asserted-by":"publisher","unstructured":"Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021). BOLD: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (FAccT), pp. 862\u2013872. ACM, New York, NY, USA. https:\/\/doi.org\/10.1145\/3442188.3445924.","DOI":"10.1145\/3442188.3445924"},{"key":"7084_CR21","unstructured":"Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V., Lakshmanan, L. V. S., & Awadallah, A. H. (2024). Hybrid LLM: Cost-efficient and quality-aware query routing. In Proceedings of the twelfth international conference on learning representations (ICLR), Vienna, Austria. https:\/\/openreview.net\/forum?id=02f3mUtqnM."},{"issue":"1","key":"7084_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1214\/aos\/1176344552","volume":"7","author":"B Efron","year":"1979","unstructured":"Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1\u201326. https:\/\/doi.org\/10.1214\/aos\/1176344552","journal-title":"The Annals of Statistics"},{"issue":"4","key":"7084_CR23","doi-asserted-by":"publisher","first-page":"1107","DOI":"10.1111\/j.1467-937X.2005.00363.x","volume":"72","author":"G Elliott","year":"2005","unstructured":"Elliott, G., Timmermann, A., & Komunjer, I. (2005). Estimation and testing of forecast rationality under flexible loss. The Review of Economic Studies, 72(4), 1107\u20131125. https:\/\/doi.org\/10.1111\/j.1467-937X.2005.00363.x","journal-title":"The Review of Economic Studies"},{"key":"7084_CR24","doi-asserted-by":"publisher","unstructured":"Eschmann, J. (2021). Reward function design in reinforcement learning. In Belousov, B., Abdulsamad, H., Klink, P., Parisi, S., Peters, J. (eds.) Reinforcement learning algorithms: Analysis and applications. Studies in Computational Intelligence, vol. 883, pp. 25\u201333. Springer, Cham, Switzerland. https:\/\/doi.org\/10.1007\/978-3-030-41188-6_3.","DOI":"10.1007\/978-3-030-41188-6_3"},{"key":"7084_CR25","doi-asserted-by":"publisher","unstructured":"Fabra-Boluda, R., Ferri, C., Mart\u00ednez-Plumed, F., Hern\u00e1ndez-Orallo, J., & Ram\u00edrez-Quintana, M. J. (2020). Family and prejudice: A behavioural taxonomy of ML techniques. In ECAI 2020\u201424th European conference on artificial intelligence. Frontiers in Artificial Intelligence and Applications, vol. 325, pp. 1135\u20131142. IOS Press, Amsterdam, The Netherlands. https:\/\/doi.org\/10.3233\/FAIA200211.","DOI":"10.3233\/FAIA200211"},{"issue":"3","key":"7084_CR26","doi-asserted-by":"publisher","DOI":"10.1088\/2632-2153\/ad62ab","volume":"5","author":"R Fabra-Boluda","year":"2024","unstructured":"Fabra-Boluda, R., Ferri, C., Ram\u00edrez-Quintana, M. J., & Mart\u00ednez-Plumed, F. (2024). Unveiling the robustness of machine learning families. Machine Learning: Science and Technology, 5(3), Article 035040. https:\/\/doi.org\/10.1088\/2632-2153\/ad62ab","journal-title":"Machine Learning: Science and Technology"},{"issue":"8","key":"7084_CR27","doi-asserted-by":"publisher","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","volume":"27","author":"T Fawcett","year":"2006","unstructured":"Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861\u2013874. https:\/\/doi.org\/10.1016\/j.patrec.2005.10.010","journal-title":"Pattern Recognition Letters"},{"key":"7084_CR28","doi-asserted-by":"publisher","first-page":"246","DOI":"10.2307\/2841583","volume":"15","author":"F Galton","year":"1886","unstructured":"Galton, F. (1886). Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246\u2013263. https:\/\/doi.org\/10.2307\/2841583","journal-title":"The Journal of the Anthropological Institute of Great Britain and Ireland"},{"issue":"477","key":"7084_CR29","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1198\/016214506000001437","volume":"102","author":"T Gneiting","year":"2007","unstructured":"Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359\u2013378. https:\/\/doi.org\/10.1198\/016214506000001437","journal-title":"Journal of the American Statistical Association"},{"key":"7084_CR30","doi-asserted-by":"publisher","first-page":"109","DOI":"10.22394\/1993-7601-2023-70-109-121","volume":"70","author":"S Gogolev","year":"2023","unstructured":"Gogolev, S., & Ozhegov, E. (2023). Asymmetric loss function in product-level sales forecasting: An empirical comparison. Applied Econometrics, 70, 109\u2013121. https:\/\/doi.org\/10.22394\/1993-7601-2023-70-109-121","journal-title":"Applied Econometrics"},{"issue":"12","key":"7084_CR31","doi-asserted-by":"publisher","first-page":"3395","DOI":"10.1016\/j.patcog.2013.06.014","volume":"46","author":"J Hern\u00e1ndez-Orallo","year":"2013","unstructured":"Hern\u00e1ndez-Orallo, J. (2013). ROC curves for regression. Pattern Recognition, 46(12), 3395\u20133411. https:\/\/doi.org\/10.1016\/j.patcog.2013.06.014","journal-title":"Pattern Recognition"},{"key":"7084_CR32","doi-asserted-by":"publisher","first-page":"12256","DOI":"10.1609\/aaai.v36i11.21487","volume":"36","author":"J Hern\u00e1ndez-Orallo","year":"2022","unstructured":"Hern\u00e1ndez-Orallo, J., Schellaert, W., & Mart\u00ednez-Plumed, F. (2022). Training on the test set: Mapping the system-problem space in AI. Proceedings of the AAAI Conference on Artificial Intelligence, 36, 12256\u201312261. https:\/\/doi.org\/10.1609\/aaai.v36i11.21487","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"7084_CR33","doi-asserted-by":"publisher","unstructured":"Ho, T. K. (1995). Random decision forests. In Proceedings of the 3rd international conference on document analysis and recognition (ICDAR), vol. 1, pp. 278\u2013282. IEEE, Piscataway, NJ, USA. https:\/\/doi.org\/10.1109\/ICDAR.1995.598994.","DOI":"10.1109\/ICDAR.1995.598994"},{"key":"7084_CR34","doi-asserted-by":"publisher","unstructured":"Hu, Q. J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., & Upadhyay, S. K. (2024). RouterBench: A benchmark for multi-LLM routing system. In Agentic markets workshop at ICML 2024. https:\/\/doi.org\/10.48550\/arXiv.2403.12031. Preprint at arXiv:2403.12031.","DOI":"10.48550\/arXiv.2403.12031"},{"key":"7084_CR35","doi-asserted-by":"publisher","first-page":"492","DOI":"10.1007\/978-1-4612-4380-9_35","volume-title":"Breakthroughs in statistics: Methodology and distribution","author":"PJ Huber","year":"1992","unstructured":"Huber, P. J. (1992). Robust estimation of a location parameter. In S. Kotz & N. L. Johnson (Eds.), Breakthroughs in statistics: Methodology and distribution (pp. 492\u2013518). Springer. https:\/\/doi.org\/10.1007\/978-1-4612-4380-9_35"},{"issue":"3","key":"7084_CR36","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1007\/s10994-021-05946-3","volume":"110","author":"E H\u00fcllermeier","year":"2021","unstructured":"H\u00fcllermeier, E., & Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3), 457\u2013506. https:\/\/doi.org\/10.1007\/s10994-021-05946-3","journal-title":"Machine Learning"},{"issue":"4","key":"7084_CR37","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1016\/j.ijforecast.2006.03.001","volume":"22","author":"RJ Hyndman","year":"2006","unstructured":"Hyndman, R. J., & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4), 679\u2013688. https:\/\/doi.org\/10.1016\/j.ijforecast.2006.03.001","journal-title":"International Journal of Forecasting"},{"key":"7084_CR38","doi-asserted-by":"publisher","unstructured":"Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., & Kaplan, J. (2022). Language models (mostly) know what they know. Preprint at arXiv:2207.05221. https:\/\/doi.org\/10.48550\/arXiv.2207.05221","DOI":"10.48550\/arXiv.2207.05221"},{"key":"7084_CR39","unstructured":"Kakkad, Y. (2021). Song popularity prediction dataset. GitHub. https:\/\/github.com\/yashrajkakkad\/song-popularity-prediction\/blob\/master\/Dataset\/data.csv."},{"key":"7084_CR40","unstructured":"Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. In Advances in neural information processing systems, vol. 30, pp. 3146\u20133154. Curran Associates, Inc., Red Hook, NY, USA. https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/6449f44a102fde848669bdd9eb6b76fa-Abstract.html."},{"issue":"3","key":"7084_CR41","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1016\/S0167-7152(96)00140-X","volume":"33","author":"R Kelley Pace","year":"1997","unstructured":"Kelley Pace, R., & Barry, R. (1997). Sparse spatial autoregressions. Statistics & Probability Letters, 33(3), 291\u2013297. https:\/\/doi.org\/10.1016\/S0167-7152(96)00140-X","journal-title":"Statistics & Probability Letters"},{"issue":"1","key":"7084_CR42","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1162\/evco_a_00242","volume":"27","author":"P Kerschke","year":"2019","unstructured":"Kerschke, P., Hoos, H. H., Neumann, F., & Trautmann, H. (2019). Automated algorithm selection: Survey and perspectives. Evolutionary Computation, 27(1), 3\u201345. https:\/\/doi.org\/10.1162\/evco_a_00242","journal-title":"Evolutionary Computation"},{"key":"7084_CR43","doi-asserted-by":"publisher","unstructured":"Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., & Chen, W. (2023). Making language models better reasoners with step-aware verifier. In Proceedings of the 61st annual meeting of the association for computational linguistics (Volume 1: Long Papers), pp. 5315\u20135333. Association for Computational Linguistics, Toronto, Canada. https:\/\/doi.org\/10.18653\/v1\/2023.acl-long.291.","DOI":"10.18653\/v1\/2023.acl-long.291"},{"key":"7084_CR44","doi-asserted-by":"publisher","unstructured":"Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C., & Zhou, J. (2024). Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of the 2024 conference of the North American chapter of the association for computational linguistics: human language technologies (Vol. 1, pp. 1964\u20131974). Mexico City; Mexico: Association for Computational Linguistics. https:\/\/doi.org\/10.18653\/v1\/2024.naacl-long.109","DOI":"10.18653\/v1\/2024.naacl-long.109"},{"issue":"4","key":"7084_CR45","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1007\/BF02478259","volume":"5","author":"WS McCulloch","year":"1943","unstructured":"McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5(4), 115\u2013133. https:\/\/doi.org\/10.1007\/BF02478259","journal-title":"The Bulletin of Mathematical Biophysics"},{"key":"7084_CR46","doi-asserted-by":"publisher","unstructured":"Menzies, T., & Di Stefano, J. S. (2004). How good is your blind spot sampling policy? In Proceedings of the eighth IEEE international symposium on high assurance systems engineering (HASE), pp. 129\u2013138. IEEE, Piscataway, NJ, USA. https:\/\/doi.org\/10.1109\/HASE.2004.1281739.","DOI":"10.1109\/HASE.2004.1281739"},{"key":"7084_CR47","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.dss.2014.03.001","volume":"62","author":"S Moro","year":"2014","unstructured":"Moro, S., Cortez, P., & Rita, P. (2014). A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62, 22\u201331. https:\/\/doi.org\/10.1016\/j.dss.2014.03.001","journal-title":"Decision Support Systems"},{"key":"7084_CR48","doi-asserted-by":"publisher","unstructured":"Nash, W., Sellers, T., Talbot, S., Cawthorn, A., & Ford, W. (1995). Abalone. UCI machine learning repository. https:\/\/doi.org\/10.24432\/C55C7W.","DOI":"10.24432\/C55C7W"},{"key":"7084_CR49","doi-asserted-by":"publisher","first-page":"31699","DOI":"10.1109\/ACCESS.2022.3154445","volume":"10","author":"E Ordoni","year":"2022","unstructured":"Ordoni, E., Bach, J., & Fleck, A.-K. (2022). Analyzing and predicting verification of data-aware process models\u2013A case study with spectrum auctions. IEEE Access, 10, 31699\u201331713. https:\/\/doi.org\/10.1109\/ACCESS.2022.3154445","journal-title":"IEEE Access"},{"issue":"7","key":"7084_CR50","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1090\/conm\/443\/08555","volume":"443","author":"AB Owen","year":"2007","unstructured":"Owen, A. B. (2007). A robust hybrid of lasso and ridge regression. Contemporary Mathematics, 443(7), 59\u201372. https:\/\/doi.org\/10.1090\/conm\/443\/08555","journal-title":"Contemporary Mathematics"},{"key":"7084_CR51","doi-asserted-by":"publisher","unstructured":"Pacchiardi, L., Cheke, L. G. & Hern\u00e1ndez-Orallo, J. (2024). 100 instances is all you need: Predicting the success of a new LLM on unseen data by testing on a few instances. In GenAI evaluation workshop @ KDD 2024, Barcelona, Spain. https:\/\/doi.org\/10.48550\/arXiv.2409.03563 . Preprint at arXiv:2409.03563.","DOI":"10.48550\/arXiv.2409.03563"},{"key":"7084_CR52","doi-asserted-by":"publisher","unstructured":"Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the association for computational linguistics (ACL), pp. 311\u2013318. Association for Computational Linguistics, Philadelphia, PA, USA. https:\/\/doi.org\/10.3115\/1073083.1073135.","DOI":"10.3115\/1073083.1073135"},{"key":"7084_CR53","unstructured":"Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. In Advances in neural information processing systems, vol. 31, pp. 6639\u20136649. Curran Associates, Inc., Red Hook, NY, USA. https:\/\/proceedings.neurips.cc\/paper\/2018\/hash\/14491b756b3a51daac41c24863285549-Abstract.html."},{"key":"7084_CR54","doi-asserted-by":"publisher","unstructured":"Romano, J. D., Le, T. T., La Cava, W., Gregg, J. T., Goldberg, D. J., Chakraborty, P., Ray, N. L., Himmelstein, D., Fu, W., & Moore, J. H. (2021). PMLB v1.0: an open source dataset collection for benchmarking machine learning methods. Preprint at http:\/\/arxiv.org\/abs\/2012.00058. https:\/\/doi.org\/10.48550\/arXiv.2012.00058.","DOI":"10.48550\/arXiv.2012.00058"},{"key":"7084_CR55","doi-asserted-by":"publisher","DOI":"10.1002\/0471725382","volume-title":"Robust regression and outlier detection","author":"PJ Rousseeuw","year":"2005","unstructured":"Rousseeuw, P. J., & Leroy, A. M. (2005). Robust regression and outlier detection. John Wiley & Sons. https:\/\/doi.org\/10.1002\/0471725382"},{"key":"7084_CR56","doi-asserted-by":"crossref","unstructured":"Schellaert, W., Hamon, R., Mart\u00ednez-Plumed, F., & Hern\u00e1ndez-Orallo, J. (2024). A proposal for scaling the scaling laws. In Proceedings of the first edition of the workshop on the scaling behavior of large language models (SCALE-LLM 2024), pp. 1\u20138. Association for Computational Linguistics, St. Julian\u2019s, Malta. https:\/\/aclanthology.org\/2024.scalellm-1.1\/.","DOI":"10.18653\/v1\/2024.scalellm-1.1"},{"issue":"2","key":"7084_CR57","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3706118","volume":"16","author":"W Schellaert","year":"2025","unstructured":"Schellaert, W., Mart\u00ednez-Plumed, F., & Hern\u00e1ndez-Orallo, J. (2025). Analysing the predictability of language model performance. ACM Transactions on Intelligent Systems and Technology, 16(2), 1\u201326. https:\/\/doi.org\/10.1145\/3706118","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"key":"7084_CR58","unstructured":"Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., et al. (2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research."},{"key":"7084_CR59","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1162\/15324430152748236","volume":"1","author":"ME Tipping","year":"2001","unstructured":"Tipping, M. E. (2001). Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research, 1, 211\u2013244. https:\/\/doi.org\/10.1162\/15324430152748236","journal-title":"Journal of Machine Learning Research"},{"issue":"4","key":"7084_CR60","doi-asserted-by":"publisher","first-page":"884","DOI":"10.1109\/TBME.2009.2036000","volume":"57","author":"A Tsanas","year":"2010","unstructured":"Tsanas, A., Little, M. A., McSharry, P. E., & Ramig, L. O. (2010). Accurate telemonitoring of Parkinson\u2019s disease progression by noninvasive speech tests. IEEE Transactions on Biomedical Engineering, 57(4), 884\u2013893. https:\/\/doi.org\/10.1109\/TBME.2009.2036000","journal-title":"IEEE Transactions on Biomedical Engineering"},{"key":"7084_CR61","doi-asserted-by":"publisher","DOI":"10.13026\/3bhc-9065","author":"Q Wang","year":"2023","unstructured":"Wang, Q., Zhou, Y., Ghassemi, P., Chenna, D., Chen, M., Casamento, J., Pfefer, J., & McBride, D. (2023). Facial and oral temperature data from a large set of human subject volunteers. PhysioNet. https:\/\/doi.org\/10.13026\/3bhc-9065","journal-title":"PhysioNet"},{"key":"7084_CR62","doi-asserted-by":"publisher","DOI":"10.24432\/C5V312","author":"D Whiteson","year":"2014","unstructured":"Whiteson, D. (2014). HIGGS. UCI Machine Learning Repository. https:\/\/doi.org\/10.24432\/C5V312","journal-title":"UCI Machine Learning Repository"},{"key":"7084_CR63","unstructured":"World Health Organization: Global Health Observatory data repository (2015). https:\/\/www.who.int\/data\/gho."},{"issue":"3","key":"7084_CR64","doi-asserted-by":"publisher","first-page":"3473","DOI":"10.1007\/s40747-023-01336-5","volume":"10","author":"Y Zhao","year":"2024","unstructured":"Zhao, Y., Ju, L., & Hern\u00e1ndez-Orallo, J. (2024). Team formation through an assessor: Choosing MARL agents in pursuit-evasion games. Complex & Intelligent Systems, 10(3), 3473\u20133492. https:\/\/doi.org\/10.1007\/s40747-023-01336-5","journal-title":"Complex & Intelligent Systems"},{"key":"7084_CR66","doi-asserted-by":"crossref","unstructured":"Zhou, L., Casares, P. A. M., Mart\u00ednez-Plumed, F., Burden, J., Burnell, R., Cheke, L., Ferri, C., Marcoci, A., Mehrbakhsh, B., Moros-Daval, Y., & \u00d3 h\u00c9igeartaigh, S., Rutar, D., Schellaert, W., Voudouris, K., & Hern\u00e1ndez-Orallo, J. (2026). Predictable artificial intelligence. Artificial Intelligence, 353, Article 104491.","DOI":"10.1016\/j.artint.2026.104491"},{"key":"7084_CR65","unstructured":"Zhou, L., Mart\u00ednez-Plumed, F., Hern\u00e1ndez-Orallo, J., Ferri, C., & Schellaert, W. (2022). Reject before you run: Small assessors anticipate big language models. In Proceedings of the 1st workshop on AI evaluation beyond metrics (EBeM 2022) @ IJCAI 2022. CEUR Workshop Proceedings, vol. 3169. Vienna, Austria. https:\/\/ceur-ws.org\/Vol-3169\/."}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-026-07084-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-026-07084-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-026-07084-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T10:26:04Z","timestamp":1785407164000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-026-07084-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7]]},"references-count":66,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7]]}},"alternative-id":["7084"],"URL":"https:\/\/doi.org\/10.1007\/s10994-026-07084-0","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7]]},"assertion":[{"value":"30 October 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 May 2026","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 May 2026","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 July 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no conflict of interest.","order":1,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"167"}}