{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T18:37:00Z","timestamp":1767638220161,"version":"3.48.0"},"reference-count":66,"publisher":"Maximum Academic Press","issue":"1","license":[{"start":{"date-parts":[[2010,3,1]],"date-time":"2010-03-01T00:00:00Z","timestamp":1267401600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["The Knowledge Engineering Review"],"published-print":{"date-parts":[[2010,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In machine learning and its risk-sensitive applications (e.g. medicine, engineering, business), the reliability estimates for individual predictions provide more information about the individual prediction error (the difference between the true label and regression prediction) than the average accuracy of predictive model (e.g. relative mean squared error). Furthermore, they enable the users to distinguish between more and less reliable predictions. The empirical evaluations of the existing individual reliability estimates revealed that the successful estimates\u2019 performance depends on the used regression model and on the particular problem domain. In the current paper, we focus on that problem as such and propose and empirically evaluate two approaches for automatic selection of the most appropriate estimate for a given domain and regression model: the internal cross-validation approach and the meta-learning approach. The testing results of both approaches demonstrated an advantage in the performance of dynamically chosen reliability estimates to the performance of the individual reliability estimates. The best results were achieved using the internal cross-validation procedure, where reliability estimates significantly positively correlated with the prediction error in 73% of experiments. In addition, the preliminary testing of the proposed methodology on a medical domain demonstrated the potential for its usage in practice.<\/jats:p>","DOI":"10.1017\/s0269888909990154","type":"journal-article","created":{"date-parts":[[2010,3,23]],"date-time":"2010-03-23T08:04:00Z","timestamp":1269331440000},"page":"27-47","source":"Crossref","is-referenced-by-count":10,"title":["Automatic selection of reliability estimates for individual regression predictions"],"prefix":"10.48130","volume":"25","author":[{"given":"Zoran","family":"Bosni\u0107","sequence":"first","affiliation":[],"role":[{"role":"author","vocab":"crossref"}]},{"given":"Igor","family":"Kononenko","sequence":"additional","affiliation":[],"role":[{"role":"author","vocab":"crossref"}]}],"member":"27968","published-online":{"date-parts":[[2010,3,1]]},"reference":[{"key":"S0269888909990154_ref35","doi-asserted-by":"publisher","DOI":"10.1007\/BF02294300"},{"key":"S0269888909990154_ref24","doi-asserted-by":"publisher","DOI":"10.1006\/jcss.1997.1504"},{"key":"S0269888909990154_ref37","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3860-5"},{"key":"S0269888909990154_ref30","first-page":"176","volume-title":"Advances in Neural Information Processing Systems","author":"Heskes","year":"1997"},{"key":"S0269888909990154_ref53","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-3324-9"},{"key":"S0269888909990154_ref26","unstructured":"Gammerman A. , Vovk V. , Vapnik V. 1998. Learning by transduction. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, Madison, Wisconsin, 148\u2013155."},{"key":"S0269888909990154_ref62","unstructured":"Whitehead S. D. 1991. A complexity analysis of cooperative mechanisms in reinforcement learning. In AAAI, 607\u2013613."},{"key":"S0269888909990154_ref33","unstructured":"Kleijnen J. 2001. Experimental designs for sensitivity analysis of simulation models. Tutorial at the Eurosim 2001 Conference."},{"key":"S0269888909990154_ref52","unstructured":"Seeger M. 2000. Learning with Labeled and Unlabeled Data. Technical report. http:\/\/www.dai.ed.ac.uk\/seeger\/papers.html."},{"key":"S0269888909990154_ref32","first-page":"152","volume-title":"Computational Learning Theory","author":"Kearns","year":"1997"},{"key":"S0269888909990154_ref54","unstructured":"Smola A. J. , Sch\u00f6lkopf B. 1998. A Tutorial on Support Vector Regression. NeuroCOLT2 Technical Report NC2-TR-1998-030."},{"key":"S0269888909990154_ref50","doi-asserted-by":"crossref","unstructured":"Schaffer C. 1993. Selecting a classification method by cross-validation. In Fourth International Workshop on Artificial Intelligence & Statistics, 15\u201325.","DOI":"10.1007\/BF00993106"},{"key":"S0269888909990154_ref27","doi-asserted-by":"publisher","DOI":"10.1016\/S0031-3203(00)00150-3"},{"key":"S0269888909990154_ref51","unstructured":"Schmidhuber J. , Storck J. 1993. Reinforcement Driven Information Acquisition in Nondeterministic Environments. Technical Report. Fakultat fur Informatik, Technische Universit at Munchen."},{"key":"S0269888909990154_ref40","first-page":"176","volume-title":"Machine Learning, Neural and Statistical Classification","author":"Michie","year":"1994"},{"key":"S0269888909990154_ref14","doi-asserted-by":"crossref","unstructured":"Carney J. , Cunningham P. 1999. Confidence and prediction intervals for neural network ensembles. In Proceedings of IJCNN\u201999, The International Joint Conference on Neural Networks, Washington, USA, 1215\u20131218.","DOI":"10.1109\/IJCNN.1999.831133"},{"key":"S0269888909990154_ref64","doi-asserted-by":"crossref","DOI":"10.1201\/9781420010404","volume-title":"Generalized Additive Models: An Introduction with R","author":"Wood","year":"2006"},{"key":"S0269888909990154_ref48","first-page":"160","volume-title":"Advances in Neural Information Processing Systems","author":"Schaal","year":"1994"},{"key":"S0269888909990154_ref47","unstructured":"Saunders C. , Gammerman A. , Vovk V. 1999. Transduction with confidence and credibility. In Proceedings of IJCAI\u201999, 2, 722\u2013726."},{"key":"S0269888909990154_ref49","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300016963"},{"key":"S0269888909990154_ref17","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511801389"},{"key":"S0269888909990154_ref58","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-2440-0"},{"key":"S0269888909990154_ref67","unstructured":"Schmidhuber J , Zhao J. , Wiering M. 1996. Simple principles of metalearning, Technical Report IDSIA-69-96, Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale, 1\u201323."},{"key":"S0269888909990154_ref4","doi-asserted-by":"crossref","unstructured":"Blum A. , Mitchell T. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory, Madison, Wisconsin, 92\u2013100.","DOI":"10.1145\/279943.279962"},{"volume-title":"A Language and Environment for Statistical Computing","year":"2006","key":"S0269888909990154_ref44"},{"key":"S0269888909990154_ref2","unstructured":"Asuncion A. , Newman D. J. 2007. UCI machine learning repository, http:\/\/www.ics.uci.edu\/~mlearn\/MLRepository.html, Irvine, CA: University of California, School of Information and Computer Science."},{"key":"S0269888909990154_ref55","first-page":"671","article-title":"Model search and inference by bootstrap bumping","volume":"8","author":"Tibshirani","year":"1999","journal-title":"Journal of Computational and Graphical Statistics"},{"key":"S0269888909990154_ref12","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"S0269888909990154_ref3","unstructured":"Birattari M. , Bontempi H. , Bersini H. 1998. Local learning for data analysis. In Proceedings of the 8th Belgian-Dutch Conference on Machine Learning, Wageningen, The Netherlands, 55\u201361."},{"key":"S0269888909990154_ref28","unstructured":"Goldman S. , Zhou Y. 2000. Enhancing supervised learning with unlabeled data. In Proc. 17th International Conf. on Machine Learning, Morgan Kaufmann, San Francisco, CA, 327\u2013334."},{"key":"S0269888909990154_ref41","unstructured":"Mitchell T. 1999. The role of unlabelled data in supervised learning. In Proceedings of the 6th International Colloquium of Cognitive Science, San Sebastian, Spain."},{"key":"S0269888909990154_ref57","first-page":"227","article-title":"Learning to predict the leave-one-out error of kernel based classifiers","author":"Tsuda","year":"2001","journal-title":"Lecture Notes in Computer Science"},{"key":"S0269888909990154_ref8","first-page":"99","volume-title":"Proceedings of Eurocon 2003","author":"Bosni\u0107","year":"2003"},{"key":"S0269888909990154_ref20","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-2953-2"},{"key":"S0269888909990154_ref36","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-36755-1_19"},{"key":"S0269888909990154_ref60","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-4493-1"},{"key":"S0269888909990154_ref15","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007379606734"},{"key":"S0269888909990154_ref7","unstructured":"Bosni\u0107 Z. , Kononenko I. 2008b. Comparison of approaches for estimating reliability of individual regression predictions. Data & Knowledge Engineering 67(3), 504\u2013516."},{"key":"S0269888909990154_ref1","doi-asserted-by":"crossref","unstructured":"Aha D. W. 1992. Generalizing from case studies: A case study. In Proceedings of the Ninth International Workshop on Machine Learning (ML 1992), Aberdeen, Scotland, UK, 1\u201310.","DOI":"10.1016\/B978-1-55860-247-2.50006-1"},{"key":"S0269888909990154_ref21","first-page":"112","volume-title":"Proc. NIPS\u201993, Neural Information Processing Systems","author":"de Sa","year":"1993"},{"key":"S0269888909990154_ref9","first-page":"499","article-title":"Stability and generalization","volume":"2","author":"Bousquet","year":"2002","journal-title":"Journal of Machine Learning Research"},{"key":"S0269888909990154_ref25","doi-asserted-by":"crossref","unstructured":"Gama J. , Brazdil P. 1995. Characterization of classification algorithms. In Progress in Artificial Intelligence, 7th Portuguese Conference on Artificial Intelligence, EPIA-95, Pinto-Ferreira, C. & Mamede, N. (eds), 189\u2013200. Springer-Verlag.","DOI":"10.1007\/3-540-60428-6_16"},{"key":"S0269888909990154_ref10","unstructured":"Breierova L. , Choudhari M. 1996. An introduction to sensitivity analysis. MIT System Dynamics in Education Project."},{"key":"S0269888909990154_ref43","first-page":"19","article-title":"A survey of connectionist network reuse through transfer","author":"Pratt","year":"1998","journal-title":"Learning to Learn"},{"key":"S0269888909990154_ref16","unstructured":"Chang C. , Lin C. 2001. LIBSVM: A Library for Support Vector Machines. Software available at http:\/\/www.csie.ntu.edu.tw\/cjlin\/libsvm."},{"key":"S0269888909990154_ref6","doi-asserted-by":"publisher","DOI":"10.1515\/JISYS.2008.17.1-3.297"},{"key":"S0269888909990154_ref39","first-page":"1","volume-title":"Learning from Data: Artificial Intelligence and Statistics","author":"Merz","year":"1996"},{"key":"S0269888909990154_ref11","doi-asserted-by":"publisher","DOI":"10.1007\/BF00058655"},{"key":"S0269888909990154_ref5","unstructured":"Bosni\u0107 Z. , Kononenko I. 2007. Estimation of individual prediction reliability using the local sensitivity analysis. Applied Intelligence 29(3), 187\u2013203."},{"key":"S0269888909990154_ref59","doi-asserted-by":"publisher","DOI":"10.1023\/A:1019956318069"},{"volume-title":"Classification and Regression Trees","year":"1984","author":"Breiman","key":"S0269888909990154_ref13"},{"volume-title":"Generalized Additive Models","year":"1990","author":"Hastie","key":"S0269888909990154_ref29"},{"key":"S0269888909990154_ref34","doi-asserted-by":"publisher","DOI":"10.1533\/9780857099440"},{"key":"S0269888909990154_ref61","unstructured":"Weigend A. , Nix D. 1994. Predictions with confidence intervals (local error bars). In Proceedings of the International Conference on Neural Information Processing (ICONIP\u201994), Seoul, Korea, 847\u2013852."},{"key":"S0269888909990154_ref19","first-page":"705","volume-title":"Advances in Neural Information Processing Systems","author":"Cohn","year":"1995"},{"key":"S0269888909990154_ref42","unstructured":"Nouretdinov I. , Melluish T. , Vovk V. 2001. Ridge regression confidence machine. In Proc. 18th International Conf. on Machine Learning, Morgan Kaufmann, San Francisco, CA, 385\u2013392."},{"key":"S0269888909990154_ref63","first-page":"241","volume-title":"Neural Networks","author":"Wolpert","year":"1992"},{"key":"S0269888909990154_ref18","first-page":"566","volume-title":"Advances in Neural Information Processing Systems","author":"Cohn","year":"1990"},{"key":"S0269888909990154_ref23","unstructured":"Elidan G. , Ninio M. , Friedman N. , Schuurmans D. 2002. Data perturbation for escaping local maxima in learning. In Proceedings of the Eighteenth National Conference on Artificial Intelligence and Fourteenth Conference on Innovative Applications of Artificial Intelligence, July 28 - August 1, 2002, Edmonton, Alberta, Canada, 132\u2013139. AAAI Press."},{"key":"S0269888909990154_ref31","doi-asserted-by":"publisher","DOI":"10.1109\/34.310693"},{"key":"S0269888909990154_ref38","unstructured":"Linden A. , Weber F. 1992. Implementing inner drive by competence reflection. In Proceedings of the 2nd International Conference on Simulation of Adaptive Behavior, Hawaii, 321\u2013326."},{"volume-title":"Learning Internal Representations by Error Propagation","year":"1986","author":"Rumelhart","key":"S0269888909990154_ref45"},{"key":"S0269888909990154_ref22","unstructured":"Department of Statistics at Carnegie Mellon University 2005. Statlib \u2013 Data, Software and News from the Statistics Community. http:\/\/lib.stat.cmu.edu\/."},{"volume-title":"Data Mining with R: Learning by Case Studies","year":"2003","author":"Torgo","key":"S0269888909990154_ref56"},{"key":"S0269888909990154_ref65","doi-asserted-by":"publisher","DOI":"10.1109\/34.588027"},{"key":"S0269888909990154_ref66","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/BF00993472","article-title":"Evaluation and Selection of Biases in Machine Learning","volume":"20","author":"DesJardins","year":"1995","journal-title":"Machine Learning"}],"container-title":["The Knowledge Engineering Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0269888909990154","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T14:43:57Z","timestamp":1767624237000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0269888909990154\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,3]]},"references-count":66,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2010,3]]}},"alternative-id":["S0269888909990154"],"URL":"https:\/\/doi.org\/10.1017\/s0269888909990154","relation":{},"ISSN":["0269-8889","1469-8005"],"issn-type":[{"type":"print","value":"0269-8889"},{"type":"electronic","value":"1469-8005"}],"subject":[],"published":{"date-parts":[[2010,3]]}}}