{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T07:22:56Z","timestamp":1774941776838,"version":"3.50.1"},"reference-count":49,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2011,7,8]],"date-time":"2011-07-08T00:00:00Z","timestamp":1310083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Generalisation error estimation is an important issue in machine learning. Cross-validation traditionally used for this purpose requires building multiple models and repeating the whole procedure many times in order to produce reliable error estimates. It is however possible to accurately estimate the error using only a single model, if the training and test data are chosen appropriately. This paper investigates the possibility of using various probability density function divergence measures for the purpose of representative data sampling. As it turned out, the first difficulty one needs to deal with is estimation of the divergence itself. In contrast to other publications on this subject, the experimental results provided in this study show that in many cases it is not possible unless samples consisting of thousands of instances are used. Exhaustive experiments on the divergence guided representative data sampling have been performed using 26 publicly available benchmark datasets and 70 PDF divergence estimators, and their results have been analysed and discussed.<\/jats:p>","DOI":"10.3390\/e13071229","type":"journal-article","created":{"date-parts":[[2011,7,8]],"date-time":"2011-07-08T16:49:06Z","timestamp":1310143746000},"page":"1229-1266","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":25,"title":["On Accuracy of PDF Divergence Estimators and Their Applicability to Representative Data Sampling"],"prefix":"10.3390","volume":"13","author":[{"given":"Marcin","family":"Budka","sequence":"first","affiliation":[{"name":"Smart Technology Research Group, Bournemouth University, School of Design, Engineering and Computing, Poole House, Talbot Campus, Fern Barrow, Poole BH12 5BB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bogdan","family":"Gabrys","sequence":"additional","affiliation":[{"name":"Smart Technology Research Group, Bournemouth University, School of Design, Engineering and Computing, Poole House, Talbot Campus, Fern Barrow, Poole BH12 5BB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katarzyna","family":"Musial","sequence":"additional","affiliation":[{"name":"Smart Technology Research Group, Bournemouth University, School of Design, Engineering and Computing, Poole House, Talbot Campus, Fern Barrow, Poole BH12 5BB, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2011,7,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Budka, M., and Gabrys, B. (2010, January 18\u201323). Correntropy-based density-preserving data sampling as an alternative to standard cross-validation. Proceedings of the International Joint Conference on Neural Networks, IJCNN 2010, part of the IEEE World Congress on Computational Intelligence, WCCI 2010, Barcelona, Spain.","DOI":"10.1109\/IJCNN.2010.5596717"},{"key":"ref_2","unstructured":"Budka, M., and Gabrys, B. (2011). Density Preserving Sampling (DPS) for error estimation and model selection. IEEE Trans. Pattern Anal. Mach. Intell., submitted for publication."},{"key":"ref_3","first-page":"1137","article-title":"A study of cross-validation and bootstrap for accuracy estimation and model selection","volume":"Volume 2","author":"Kohavi","year":"1995","journal-title":"Proceedings of the 14th International Joint Conference on Artificial Intelligence"},{"key":"ref_4","unstructured":"Liu, W., Pokharel, P., and Principe, J. (2006, January 16\u201321). Correntropy: A Localized Similarity Measure. Proceedings of the International Joint Conference on Neural Networks, Vancouver, Canada."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1214\/aoms\/1177704472","article-title":"On estimation of a probability density function and mode","volume":"33","author":"Parzen","year":"1962","journal-title":"Ann. Math. Stat."},{"key":"ref_6","unstructured":"Duda, R., Hart, P., and Stork, D. (2001). Pattern Classification, John Wiley & Sons. [2nd ed.]."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"716","DOI":"10.1109\/TAC.1974.1100705","article-title":"A new look at the statistical model identification","volume":"19","author":"Akaike","year":"1974","journal-title":"IEEE Trans. Automat. Contr."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3314","DOI":"10.1109\/TSP.2004.837416","article-title":"A small sample model selection criterion based on Kullback\u2019s symmetric divergence","volume":"52","author":"Seghouane","year":"2004","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1111\/j.2517-6161.1977.tb01603.x","article-title":"An asymptotic equivalence of choice of model by cross-validation and Akaike\u2019s criterion","volume":"39","author":"Stone","year":"1977","journal-title":"J. Roy. Stat. Soc. B"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"5847","DOI":"10.1109\/TIT.2010.2068870","article-title":"Estimating divergence functionals and the likelihood ratio by convex risk minimization","volume":"56","author":"Nguyen","year":"2010","journal-title":"IEEE Trans. Inform. Theor."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1016\/j.jfranklin.2006.03.018","article-title":"The Cauchy-Schwarz divergence and Parzen windowing: Connections to graph theory and Mercer kernels","volume":"343","author":"Jenssen","year":"2006","journal-title":"J. Franklin Inst."},{"key":"ref_12","unstructured":"Turlach, B. (1993). Bandwidth selection in kernel density estimation: A review. CORE and Institut de Statistique, 23\u2013493."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1109\/TC.1976.1674577","article-title":"On the choice of smoothing parameters for Parzen estimators of probability density functions","volume":"100","author":"Duin","year":"1976","journal-title":"IEEE Trans. Comput."},{"key":"ref_14","unstructured":"Silverman, B. (1998). Density Estimation for Statistics and Data Analysis, Chapman & Hall\/CRC Press."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1111\/j.2517-6161.1991.tb01857.x","article-title":"A Reliable Data-Based Bandwidth Selection Method for Kernel Density Estimation","volume":"53","author":"Sheather","year":"1991","journal-title":"J. Roy. Stat. Soc. B"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1080\/01621459.1996.10476701","article-title":"A Brief Survey of Bandwidth Selection for Density Estimation","volume":"91","author":"Jones","year":"1996","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_17","unstructured":"Ghosh, J., Lambert, D., Skillicorn, D., and Srivastava, J. (2006, January 20\u201322). Fast optimal bandwidth selection for kernel density estimation. Proceedings of the 6th SIAM International Conference on Data Mining, Bethesda, Maryland, USA."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Perez\u2013Cruz, F. (2008, January 6\u201311). Kullback-Leibler divergence estimation of continuous distributions. Proceedings of the IEEE International Symposium on Information Theory, Toronto, Canada.","DOI":"10.1109\/ISIT.2008.4595271"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3390\/e12061532","article-title":"Families of Alpha-Beta-and Gamma-Divergences: Flexible and Robust Measures of Similarities","volume":"12","author":"Cichocki","year":"2010","journal-title":"Entropy"},{"key":"ref_20","unstructured":"Kullback, S. (1997). Information Theory and Statistics, Dover Publications Inc."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On information and sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Le Cam, L., and Yang, G. (2000). Asymptotics in Statistics: Some Basic Concepts, Springer Verlag.","DOI":"10.1007\/978-1-4612-1166-2"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/34.19040","article-title":"The reduced Parzen classifier","volume":"11","author":"Fukunaga","year":"1989","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1109\/97.566704","article-title":"Infomax and maximum likelihood for blind source separation","volume":"4","author":"Cardoso","year":"1997","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2009","DOI":"10.1109\/5.720250","article-title":"Blind signal separation: statistical principles","volume":"86","author":"Cardoso","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/0031-3203(95)00067-4","article-title":"A comparative study of texture measures with classification based on featured distributions","volume":"29","author":"Ojala","year":"1996","journal-title":"Pattern Recogn."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1214\/aos\/1028144844","article-title":"Classification by pairwise coupling","volume":"26","author":"Hastie","year":"1998","journal-title":"Ann. Stat."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1109\/83.806616","article-title":"Image compression via joint statistical characterization in the wavelet domain","volume":"8","author":"Buccigrossi","year":"1999","journal-title":"IEEE Trans. Image Process."},{"key":"ref_29","first-page":"1385","article-title":"A Kullback-Leibler divergence based kernel for SVM classification in multimedia applications","volume":"16","author":"Moreno","year":"2004","journal-title":"Adv. Neural Inform. Process. Syst."},{"key":"ref_30","unstructured":"MacKay, D. (2003). Information Theory, Inference, and Learning Algorithms, Cambridge University Press."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Wang, Q., Kulkarni, S., and Verdu, S. (2006, January 9\u201314). A nearest-neighbor approach to estimating divergence between continuous random vectors. Proceedings of the IEEE International Symposium on Information Theory, Seattle, WA, USA.","DOI":"10.1109\/ISIT.2006.261842"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hershey, J., and Olsen, P. (2007, January 15\u201320). Approximating the Kullback-Leibler divergence between Gaussian mixture models. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Honolulu, Hawaii.","DOI":"10.1109\/ICASSP.2007.366913"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1109\/TNN.2006.882813","article-title":"The AIC criterion and symmetrizing the Kullback-Leibler divergence","volume":"18","author":"Seghouane","year":"2007","journal-title":"IEEE Trans. Neural Network"},{"key":"ref_34","first-page":"453","article-title":"An invariant form for the prior probability in estimation problems","volume":"186","author":"Jeffreys","year":"1946","journal-title":"Proc. Roy. Soc. Lond. Math. Phys. Sci. A"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1109\/18.61115","article-title":"Divergence measures based on the Shannon entropy","volume":"37","author":"Lin","year":"1991","journal-title":"IEEE Trans. Inform. Theor."},{"key":"ref_36","first-page":"1265","article-title":"A divisive information theoretic feature clustering algorithm for text classification","volume":"3","author":"Dhillon","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_37","unstructured":"Subramaniam, S., Palpanas, T., Papadopoulos, D., Kalogeraki, V., and Gunopulos, D. (2006, January 12\u201315). Online outlier detection in sensor data using non-parametric models. Proceedings of the 32nd international conference on Very large data bases, Seoul, Korea."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Rao, S., Liu, W., Principe, J., and de Medeiros Martins, A. (2006, January 6\u20138). Information theoretic mean shift algorithm. Proceedings of the 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing, Arlington, VA, USA.","DOI":"10.1109\/MLSP.2006.275540"},{"key":"ref_39","unstructured":"Haykin, S. (2000). Unsupervised Adaptive Filtering, John Wiley & Sons."},{"key":"ref_40","unstructured":"Jenssen, R., Erdogmus, D., Principe, J., and Eltoft, T. (2005, January 18\u201323). The Laplacian spectral classifier. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Philadelphia, PA, USA."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Rangarajan, A., Vemurl, B., and Yuille, A. Energy Minimization Methods in Computer Vision and Pattern Recognition, Springer. Lect. Notes Comput. Sci., 2005, 3257, 34\u201345.","DOI":"10.1007\/11585978"},{"key":"ref_42","unstructured":"Kapur, J. (1994). Measures of Information and Their Applications, John Wiley & Sons."},{"key":"ref_43","unstructured":"Zhou, S., and Chellappa, R. (July, January 27). Kullback-Leibler distance between two Gaussian densities in reproducing kernel Hilbert space. Proceedings of the IEEE International Symposium on Information Theory, Chicago, IL, USA."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Kuncheva, L. (2000). Fuzzy Classifier Design, Physica Verlag.","DOI":"10.1007\/978-3-7908-1850-5"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Ripley, B. (1996). Pattern Recognition and Neural Networks, Cambridge University Press.","DOI":"10.1017\/CBO9780511812651"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1007\/s11047-007-9064-6","article-title":"A framework for machine learning based on dynamic physical fields","volume":"8","author":"Ruta","year":"2009","journal-title":"Nat. Comput."},{"key":"ref_47","unstructured":"Minka, T. (2001). A family of algorithms for approximate Bayesian inference. [PhD thesis, MIT, Cambridge]."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Goldberger, J., Gordon, S., and Greenspan, H. (2003, January 13\u201316). An efficient image similarity measure based on approximations of KL-divergence between two Gaussian mixtures. Proceedings of the 9th IEEE International Conference on Computer Vision, Nice, France.","DOI":"10.1109\/ICCV.2003.1238387"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/13\/7\/1229\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:56:42Z","timestamp":1760219802000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/13\/7\/1229"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,7,8]]},"references-count":49,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2011,7]]}},"alternative-id":["e13071229"],"URL":"https:\/\/doi.org\/10.3390\/e13071229","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2011,7,8]]}}}