{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:10:53Z","timestamp":1760238653986,"version":"build-2065373602"},"reference-count":49,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2020,9,1]],"date-time":"2020-09-01T00:00:00Z","timestamp":1598918400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>For incremental machine-learning applications it is often important to robustly estimate the system accuracy during training, especially if humans perform the supervised teaching. Cross-validation and interleaved test\/train error are here the standard supervised approaches. We propose a novel semi-supervised accuracy estimation approach that clearly outperforms these two methods. We introduce the Configram Estimation (CGEM) approach to predict the accuracy of any classifier that delivers confidences. By calculating classification confidences for unseen samples, it is possible to train an offline regression model, capable of predicting the classifier\u2019s accuracy on novel data in a semi-supervised fashion. We evaluate our method with several diverse classifiers and on analytical and real-world benchmark data sets for both incremental and active learning. The results show that our novel method improves accuracy estimation over standard methods and requires less supervised training data after deployment of the model. We demonstrate the application of our approach to a challenging robot object recognition task, where the human teacher can use our method to judge sufficient training.<\/jats:p>","DOI":"10.3390\/make2030018","type":"journal-article","created":{"date-parts":[[2020,9,1]],"date-time":"2020-09-01T08:53:43Z","timestamp":1598950423000},"page":"327-346","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Beyond Cross-Validation\u2014Accuracy Estimation for Incremental and Active Learning Models"],"prefix":"10.3390","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4903-3933","authenticated-orcid":false,"given":"Christian","family":"Limberg","sequence":"first","affiliation":[{"name":"Research Institute for Cognition and Robotics, Bielefeld University, 33615 Bielefeld, Germany"},{"name":"HONDA Research Institute Europe GmbH, 63073 Offenbach, Germany"}]},{"given":"Heiko","family":"Wersing","sequence":"additional","affiliation":[{"name":"HONDA Research Institute Europe GmbH, 63073 Offenbach, Germany"}]},{"given":"Helge","family":"Ritter","sequence":"additional","affiliation":[{"name":"Research Institute for Cognition and Robotics, Bielefeld University, 33615 Bielefeld, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2020,9,1]]},"reference":[{"key":"ref_1","unstructured":"Wu, B., Hu, B., and Lin, H. (2017). A Learning Based Optimal Human Robot Collaboration with Linear Temporal Logic Constraints. arXiv."},{"key":"ref_2","unstructured":"Settles, B. (2010). Active Learning Literature Survey, Technical Report for University of Wisconsin."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","article-title":"Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI","volume":"58","author":"Arrieta","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"42200","DOI":"10.1109\/ACCESS.2020.2976199","article-title":"Explainable machine learning for scientific insights and discoveries","volume":"8","author":"Roscher","year":"2020","journal-title":"IEEE Access"},{"key":"ref_6","unstructured":"Gunning, D. (2020, August 28). Explainable Artificial Intelligence (xai). Available online: https:\/\/www.esd.whs.mil\/Portals\/54\/Documents\/FOID\/Reading%20Room\/DARPA\/15-F-0059_CLIQR_QUEST_FISCAL_YEAR_2012_RPT.pdf."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2522","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"Lundberg","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy should I trust you?\u201d Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1016\/S1071-5819(03)00038-7","article-title":"The role of trust in automation reliance","volume":"58","author":"Dzindolet","year":"2003","journal-title":"Int. J.-Hum.-Comput. Stud."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Yin, M., Wortman Vaughan, J., and Wallach, H. (2019, January 4\u20139). Understanding the effect of accuracy on trust in machine learning models. Proceedings of the 2019 Chi Conference on Human Factors in Computing Systems, Glasgow, UK.","DOI":"10.1145\/3290605.3300509"},{"key":"ref_11","unstructured":"Schmidt, P., and Biessmann, F. (2019). Quantifying interpretability and trust in machine learning systems. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Welinder, P., Welling, M., and Perona, P. (2013, January 23\u201328). A Lazy Man\u2019s Approach to Benchmarking: Semisupervised Classifier Evaluation and Recalibration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, OR, USA.","DOI":"10.1109\/CVPR.2013.419"},{"key":"ref_13","first-page":"371","article-title":"A Tutorial on Conformal Prediction","volume":"9","author":"Shafer","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_14","unstructured":"Jiang, H., Kim, B., Guan, M., and Gupta, M. (2018). To trust or not to trust a classifier. Advances in Neural Information Processing Systems, Neural Information Processing Systems Foundation, Inc. ( NIPS )."},{"key":"ref_15","unstructured":"Platanios, E., Blum, A., and Mitchell, T. (2014, January 3\u20136). Estimating Accuracy from Unlabeled Data. Proceedings of the Association for Uncertainty in Artificial Intelligence, UAI, Toronto, ON, Canada."},{"key":"ref_16","first-page":"1323","article-title":"newblock Unsupervised Supervised Learning I: Estimating Classification and Regression Errors without Labels","volume":"11","author":"Donmez","year":"2010","journal-title":"JMLR"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Aghazadeh, O., and Carlsson, S. (2013, January 9\u201313). Properties of Datasets Predict the Performance of Classifiers. Proceedings of the British Machine Vision Conference, BMVC 2013, Bristol, UK.","DOI":"10.5244\/C.27.44"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.neunet.2019.01.012","article-title":"Continual lifelong learning with neural networks: A review","volume":"113","author":"Parisi","year":"2019","journal-title":"Neural Netw."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1016\/j.neucom.2017.06.084","article-title":"Incremental on-line learning: A review and comparison of state of the art algorithms","volume":"275","author":"Losing","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1901","DOI":"10.1109\/TNN.2011.2171713","article-title":"Incremental Learning From Stream Data","volume":"22","author":"He","year":"2011","journal-title":"IEEE Trans. Neural Networks"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1109\/TNNLS.2012.2236570","article-title":"Active Learning With Drifting Streaming Data","volume":"25","author":"Zliobaite","year":"2014","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_22","unstructured":"Lomonaco, V., and Maltoni, D. (2017). Core50: A new dataset and benchmark for continuous object recognition. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3784","DOI":"10.1109\/TNNLS.2017.2736643","article-title":"Credit Card Fraud Detection: A Realistic Modeling and a Novel Learning Strategy","volume":"29","author":"Pozzolo","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1517","DOI":"10.1109\/TNN.2011.2160459","article-title":"Incremental Learning of Concept Drift in Nonstationary Environments","volume":"22","author":"Elwell","year":"2011","journal-title":"IEEE Trans. Neural Netw."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2283","DOI":"10.1109\/TKDE.2012.136","article-title":"Incremental Learning of Concept Drift from Streaming Imbalanced Data","volume":"25","author":"Ditzler","year":"2013","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Pesaranghader, A., Viktor, H.L., and Paquet, E. (2018, January 8\u201313). McDiarmid drift detection methods for evolving data streams. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.","DOI":"10.1109\/IJCNN.2018.8489260"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1145\/3373464.3373470","article-title":"Machine learning for streaming data: State of the art, challenges, and opportunities","volume":"21","author":"Gomes","year":"2019","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.neucom.2017.04.070","article-title":"Unsupervised real-time anomaly detection for streaming data","volume":"262","author":"Ahmad","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Constantinopoulos, C., and Likas, A. (2006, January 10\u201314). Active Learning with the Probabilistic RBF Classifier. Proceedings of the International Conference on Artificial Neural Networks (ICANN), Athens, Greece.","DOI":"10.1007\/11840817_38"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"K\u00e4ding, C., Freytag, A., Rodner, E., Bodesheim, P., and Denzler, J. (2015, January 7\u201312). Active learning and discovery of object categories in the presence of unnameable instances. Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299063"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Seung, S., Opper, M., and Sompolinsky, H. (1992, January 27\u201329). Query by Committee. Proceedings of the Fifth Annual Workshop on Computational Learning Theory, Pennsylvania, PA, USA.","DOI":"10.1145\/130385.130417"},{"key":"ref_32","unstructured":"Konyushkova, K., Sznitman, R., and Fua, P. (2017, January 4\u20139). Learning active learning from data. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_33","unstructured":"Bachman, P., Sordoni, A., and Trischler, A. (2017). Learning algorithms for active learning. arXiv."},{"key":"ref_34","unstructured":"Purushotham, S., and Tripathy, B. (2011, January 9\u201311). Evaluation of classifier models using stratified tenfold cross validation techniques. Proceedings of the International Conference on Computing and Communication Systems, Vellore, India."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Losing, V., Hammer, B., and Wersing, H. (2015, January 12\u201316). Interactive online learning for obstacle classification on a mobile robot. Proceedings of the International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland.","DOI":"10.1109\/IJCNN.2015.7280610"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Draper, N.R., and Smith, H. (1998). Applied Regression Analysis, John Wiley & Sons.","DOI":"10.1002\/9781118625590"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1007\/BF00994018","article-title":"Support-Vector Networks","volume":"20","author":"Cortes","year":"1995","journal-title":"Mach. Learn."},{"key":"ref_38","first-page":"61","article-title":"Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods","volume":"10","author":"Platt","year":"1999","journal-title":"Adv. Large Margin Classif."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_40","unstructured":"Sato, A., and Yamada, K. (1995, January 27\u201330). Generalized Learning Vector Quantization. Proceedings of the 8th International Conference on Neural Information Processing Systems (NIPS), Denver, CO, USA."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1215","DOI":"10.1016\/j.neucom.2006.10.149","article-title":"Margin-based active learning for LVQ networks","volume":"70","author":"Schleif","year":"2007","journal-title":"Neurocomputing"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Kohonen, T. (1990, January 17\u201321). Improved versions of learning vector quantization. Proceedings of the IJCNN International Joint Conference on Neural Networks, San Diego, CA, USA.","DOI":"10.1109\/IJCNN.1990.137622"},{"key":"ref_43","unstructured":"Street, W.N., Wolberg, W.H., and Mangasarian, O.L. (1993, January 1\u20134). Nuclear feature extraction for breast tumor diagnosis. Biomedical image processing and biomedical visualization. Proceedings of the International Society for Optics and Photonics, San Jose, CA, USA."},{"key":"ref_44","unstructured":"Fei-Fei, L., Fergus, R., and Perona, P. (July, January 27). Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC, USA."},{"key":"ref_45","unstructured":"Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., and Zisserman, A. (2020, August 28). The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. Available online: http:\/\/host.robots.ox.ac.uk\/pascal\/VOC\/voc2012\/."},{"key":"ref_46","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Liu, Z., Luo, P., Wang, X., and Tang, X. (2015, January 7\u201313). Deep Learning Face Attributes in the Wild. Proceedings of the International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.425"},{"key":"ref_48","unstructured":"Limberg, C., Wersing, H., and Ritter, H.J. (2018, January 25\u201327). Efficient accuracy estimation for instance-based incremental active learning. Proceedings of the European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Limberg, C., Krieger, K., Wersing, H., and Ritter, H.J. (2019, January 17\u201319). Active Learning for Image Recognition Using a Visualization-Based User Interface. Proceedings of the International Conference on Artificial Neural Networks (ICANN), Munich, Germany.","DOI":"10.1007\/978-3-030-30484-3_40"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/2\/3\/18\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:05:34Z","timestamp":1760177134000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/2\/3\/18"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,1]]},"references-count":49,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["make2030018"],"URL":"https:\/\/doi.org\/10.3390\/make2030018","relation":{},"ISSN":["2504-4990"],"issn-type":[{"type":"electronic","value":"2504-4990"}],"subject":[],"published":{"date-parts":[[2020,9,1]]}}}