{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T06:27:29Z","timestamp":1751956049679,"version":"3.37.3"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"9","license":[{"start":{"date-parts":[[2021,6,10]],"date-time":"2021-06-10T00:00:00Z","timestamp":1623283200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,6,10]],"date-time":"2021-06-10T00:00:00Z","timestamp":1623283200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Sign Process Syst"],"published-print":{"date-parts":[[2021,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper proposes a new boundary uncertainty-based estimation method that has significantly higher accuracy, scalability, and applicability than our previously proposed boundary uncertainty estimation method. In our previous work, we introduced a new classifier evaluation metric that we termed \u201cboundary uncertainty.\u201d The name \u201cboundary uncertainty\u201d comes from evaluating the classifier based solely on measuring the equality between class posterior probabilities along the classifier boundary; satisfaction of such equality can be described as \u201cuncertainty\u201d along the classifier boundary. We also introduced a method to estimate this new evaluation metric. By focusing solely on the classifier boundary to evaluate its uncertainty, boundary uncertainty defines an easier estimation target that can be accurately estimated based directly on a finite training set without using a validation set. Regardless of the dataset, boundary uncertainty is defined between 0 and 1, where 1 indicates whether probability estimation for the Bayes error is achieved. We call our previous boundary uncertainty estimation method \u201cProposal 1\u201d in order to contrast it with the new method introduced in this paper, which we call \u201cProposal 2.\u201d Using Proposal 1, we performed successful classifier evaluation on real-world data and supported it with theoretical analysis. However, Proposal 1 suffered from accuracy, scalability, and applicability limitations owing to the difficulty of finding the location of a classifier boundary in a multidimensional sample space. The novelty of Proposal 2 is that it locally reformalizes boundary uncertainty in a single dimension that focuses on the classifier boundary. This convenient reduction with a focus toward the classifier boundary provides the new method\u2019s significant improvements. In classifier evaluation experiments on Support Vector Machines (SVM) and MultiLayer Perceptron (MLP), we demonstrate that Proposal 2 offers a competitive classifier evaluation accuracy compared to a benchmark Cross Validation (CV) method as well as much higher scalability than both CV and Proposal 1.<\/jats:p>","DOI":"10.1007\/s11265-021-01671-1","type":"journal-article","created":{"date-parts":[[2021,6,10]],"date-time":"2021-06-10T08:08:48Z","timestamp":1623312528000},"page":"1057-1084","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["An Improved Boundary Uncertainty-Based Estimation for Classifier Evaluation"],"prefix":"10.1007","volume":"93","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3598-5036","authenticated-orcid":false,"given":"David","family":"Ha","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shigeru","family":"Katagiri","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hideyuki","family":"Watanabe","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miho","family":"Ohsaki","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,6,10]]},"reference":[{"key":"1671_CR1","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1007\/978-0-387-84858-7","volume-title":"Optimism of the error rate (in the elements of statistical learning","author":"T Hastie","year":"2009","unstructured":"Hastie, T., Tibshirani, R., & Friedman, J. (2009). Optimism of the error rate (in the elements of statistical learning, 2nd edn., (pp. 228\u2013230). New York: Springer Science+Business Media.","edition":"2nd edn."},{"key":"1671_CR2","unstructured":"Vapnik, V. (1998). The structural risk minimization principle. In Vapnik, E. (Ed.) Statistical learning theory. 1st edn. (pp. 219\u2013268). New York: Wiley."},{"issue":"NIPS 1991","key":"1671_CR3","first-page":"471","volume":"4","author":"I Guyon","year":"1991","unstructured":"Guyon, I., Vapnik, V., Boser, B., Bottou, L., & Solla, S. A. (1991). Structural risk minimization for character minimization. Proceedings of Advances in Neural Information Processing Systems, 4(NIPS 1991), 471\u2013479.","journal-title":"Proceedings of Advances in Neural Information Processing Systems"},{"issue":"2","key":"1671_CR4","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","volume":"36","author":"M Stone","year":"1974","unstructured":"Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society, 36(2), 111\u2013147.","journal-title":"Journal of the Royal Statistical Society"},{"key":"1671_CR5","doi-asserted-by":"crossref","unstructured":"Chapelle, O., & Vapnik, V. (2000). Model selection for support vector machines. Advances in Neural Information Processing Systems, 12.","DOI":"10.1162\/089976600300015042"},{"key":"1671_CR6","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-4541-9","volume-title":"An introduction to the bootstrap","author":"B Efron","year":"1993","unstructured":"Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York: Springer Science+Business Media."},{"key":"1671_CR7","first-page":"97","volume":"25","author":"S Demyanov","year":"2012","unstructured":"Demyanov, S., Bailey, J., Ramamohanarao, K., & Leckie, C. (2012). AIC And BIC based approaches for SVM parameter value estimation with RBF kernels. Journal of Machine Learning Research, 25, 97\u2013112.","journal-title":"Journal of Machine Learning Research"},{"key":"1671_CR8","doi-asserted-by":"crossref","unstructured":"Devroye, L., Gyorfi, L., & Lugosi, G. (1997). 24 deleted estimates of the error probability. In A probabilistic theory of pattern recognition (pp. 407\u2013419): Springer.","DOI":"10.1007\/978-1-4612-0711-5_24"},{"issue":"12","key":"1671_CR9","doi-asserted-by":"publisher","first-page":"3043","DOI":"10.1109\/78.175747","volume":"30","author":"B-H Juang","year":"1992","unstructured":"Juang, B.-H., & Katagiri, S. (1992). Discriminative learning for minimum error classification. IEEE Transactions on Signal Processing, 30(12), 3043\u20133054.","journal-title":"IEEE Transactions on Signal Processing"},{"key":"1671_CR10","doi-asserted-by":"crossref","unstructured":"Ohashi, T., Watanabe, H., Tokuno, J., Katagiri, S., Ohsaki, M., Matsuda, S., & Kashioka, H. (2012). Increasing virtual samples through loss smoothness determination in large geometric margin minimum classification error training. In Proceedings of 2012 international conference on acoustics, speech and signal processing (ICASSP 2012) (pp. 2081\u20132084).","DOI":"10.1109\/ICASSP.2012.6288320"},{"issue":"6","key":"1671_CR11","doi-asserted-by":"publisher","first-page":"716","DOI":"10.1109\/TAC.1974.1100705","volume":"19","author":"H Akaike","year":"1974","unstructured":"Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716\u2013723.","journal-title":"IEEE Transactions on Automatic Control"},{"issue":"2","key":"1671_CR12","first-page":"461","volume":"6","author":"G Shwarz","year":"1978","unstructured":"Shwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461\u2013464.","journal-title":"The Annals of Statistics"},{"key":"1671_CR13","unstructured":"Bishop, C. (2006). 3.4 Bayesian model selection. In Pattern recognition and machine learning (pp. 161\u2013165): Bishop."},{"issue":"4","key":"1671_CR14","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1109\/TPAMI.2006.82","volume":"28","author":"G Bouchard","year":"2006","unstructured":"Bouchard, G., & Celleux, G. (2006). Selection of generative models in classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 544\u2013554.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"1671_CR15","unstructured":"Guo, Y., & Greiner, R. (2005). Discriminative model selection for belief net structures. In Proceedings of AAAI twentieth national conference on artificial intelligence (AAAI-05) (pp. 770\u2013776)."},{"key":"1671_CR16","doi-asserted-by":"crossref","unstructured":"Ha, D., Watanabe, H., Tomotoshi, Y., Delattre, E., Katagiri, S., & Ohsaki, M. (2018). Optimality analysis of boundary-uncertainty-based classifier model parameter status selection method. In Proceedings of 2018 ACM international conference on signal processing and machine learning.","DOI":"10.1145\/3297067.3297076"},{"key":"1671_CR17","unstructured":"Tomotoshi, Y., Ha, D., Delattre, E., Watanabe, H., Katagiri, S., & Ohsaki, M. (2019). Optimal Classifier status selection using class boundary uncertainty measure for prototype-based and neural network classifier. In International symposium on integrated uncertainty in knowledge modeling and decision making."},{"key":"1671_CR18","doi-asserted-by":"crossref","unstructured":"Ha, D., Tomotoshi, Y., Senda, M., Watanabe, H., Katagiri, S., & Ohsaki, M. (2019). A practical method based on bayes boundary-ness for optimal classifier parameter status selection. Journal of Signal Processing Systems.","DOI":"10.1007\/s11265-019-01451-y"},{"key":"1671_CR19","unstructured":"Bishop, C. (2006). 1.5.1. minimizing the misclassification rate. In Pattern recognition and machine learning (pp. 39\u201340): Bishop."},{"key":"1671_CR20","unstructured":"Bishop, C. (2006). 1.4.4. inference and decision. In Pattern recognition and machine learning (pp. 42\u201344): Bishop."},{"key":"1671_CR21","first-page":"985","volume":"8","author":"M Sugiyama","year":"2007","unstructured":"Sugiyama, M., Kraudelat, M., & Muller, K.R. (2007). Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8, 985\u20131005.","journal-title":"Journal of Machine Learning Research"},{"key":"1671_CR22","doi-asserted-by":"crossref","unstructured":"Variani, E., Bagby, T., McDermott, E., & Bacchiani, M. (2017). End-to-end training of acoustic models for large vocabulary continuous speech recognition with tensorflow. Interspeech 2017.","DOI":"10.21437\/Interspeech.2017-1284"},{"key":"1671_CR23","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1016\/S0885-2308(03)00037-8","volume":"18","author":"E McDermott","year":"2004","unstructured":"McDermott, E., & Katagiri, S. (2004). A derivation of minimum classification error from the theoretical classification risk using parzen estimation. Elsevier Computer Speech and Language, 18, 107\u2013122.","journal-title":"Elsevier Computer Speech and Language"},{"key":"1671_CR24","unstructured":"Bishop, C. (2006). 2.5.1. Kernel density estimator. In Pattern recognition and machine learning (pp. 122\u2013124): Bishop."},{"key":"1671_CR25","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-3324-9","volume-title":"Density estimation for statistics and data analysis","author":"B Silverman","year":"1986","unstructured":"Silverman, B. (1986). Density estimation for statistics and data analysis. London: Chapman and Hall."},{"key":"1671_CR26","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1002\/hbm.1031","volume":"13","author":"C Goutte","year":"2001","unstructured":"Goutte, C., Hansen, L.K., Liptrot, M.G., & Rostrup, E. (2001). Feature-space clustering for fMRI meta-analysis. Human Brain Mapping, 13, 165\u2013183.","journal-title":"Human Brain Mapping"},{"key":"1671_CR27","first-page":"1485","volume":"2","author":"C Burges","year":"2009","unstructured":"Burges, C. (2009). A tutorial for support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2, 1485\u20131510.","journal-title":"Data Mining and Knowledge Discovery"},{"key":"1671_CR28","first-page":"10","volume":"1703","author":"M Sugiyama","year":"2010","unstructured":"Sugiyama, M., Suzuki, T., & Kanamori, T. (2010). Density ratio estimation: a comprehensive review. RIMS Kokyuroku, 1703, 10\u201331.","journal-title":"RIMS Kokyuroku"},{"key":"1671_CR29","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1613\/jair.953","volume":"16","author":"NV Chawla","year":"2002","unstructured":"Chawla, N.V., Bowyer, K.W., Hall, L.O., & Kegelmeyer, W.P. (2002). SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321\u2013357.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"1671_CR30","volume-title":"Introduction to algorithms","author":"T Cormen","year":"1990","unstructured":"Cormen, T., Leiserson, C., & Rivest, R. (1990). Introduction to algorithms. Cambridge: MIT Press and McGraw-Hill."},{"key":"1671_CR31","unstructured":"Bottou, L., & Lin, C.J. (2007). Support vector machine solvers. In Large-scale Kernel machines (pp. 301\u2013320): MIT Press and McGraw-Hill."}],"container-title":["Journal of Signal Processing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11265-021-01671-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11265-021-01671-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11265-021-01671-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,1]],"date-time":"2024-09-01T14:46:23Z","timestamp":1725201983000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11265-021-01671-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,10]]},"references-count":31,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2021,9]]}},"alternative-id":["1671"],"URL":"https:\/\/doi.org\/10.1007\/s11265-021-01671-1","relation":{},"ISSN":["1939-8018","1939-8115"],"issn-type":[{"type":"print","value":"1939-8018"},{"type":"electronic","value":"1939-8115"}],"subject":[],"published":{"date-parts":[[2021,6,10]]},"assertion":[{"value":"27 October 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 May 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 May 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 June 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}