{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T04:58:30Z","timestamp":1770958710352,"version":"3.50.1"},"reference-count":30,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2014,2,17]],"date-time":"2014-02-17T00:00:00Z","timestamp":1392595200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Estimating a discrepancy between two probability distributions from samples is an important task in statistics and machine learning. There are mainly two classes of discrepancy measures: distance measures based on the density difference, such as the Lp-distances, and divergence measures based on the density ratio, such as the \u03a6-divergences. The intersection of these two classes is the L1-distance measure, and thus, it can be estimated either based on the density difference or the density ratio. In this paper, we first show that the Bregman scores, which are widely employed for the estimation of probability densities in statistical data analysis, allows us to estimate the density difference and the density ratio directly without separately estimating each probability distribution. We then theoretically elucidate the robustness of these estimators and present numerical experiments.<\/jats:p>","DOI":"10.3390\/e16020921","type":"journal-article","created":{"date-parts":[[2014,2,17]],"date-time":"2014-02-17T13:44:35Z","timestamp":1392644675000},"page":"921-942","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Statistical Analysis of Distance Estimators with Density Differences and Density Ratios"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6878-5850","authenticated-orcid":false,"given":"Takafumi","family":"Kanamori","sequence":"first","affiliation":[{"name":"Nagoya University, Furocho, Chikusaku, Nagoya 464-8603, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Masashi","family":"Sugiyama","sequence":"additional","affiliation":[{"name":"Tokyo Institute of Technology, 2-12-1 O-okayama, Meguro-ku, Tokyo 152-8552, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2014,2,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"99","DOI":"10.5626\/JCSE.2013.7.2.99","article-title":"Direct divergence approximation between probability distributions and its applications in machine learning","volume":"7","author":"Sugiyama","year":"2013","journal-title":"J. Comput. Sci. Eng"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1016\/S0378-3758(00)00115-4","article-title":"Improving predictive inference under covariate shift by weighting the log-likelihood function","volume":"90","author":"Shimodaira","year":"2000","journal-title":"J. Stat. Plan. Infer"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Sugiyama, M., and Kawanabe, M (2012). Machine Learning in Non-Stationary Environments : Introduction to Covariate Shift Adaptation (Adaptive Computation and Machine Learning), MIT Press.","DOI":"10.7551\/mitpress\/9780262017091.001.0001"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Sugiyama, M., Suzuki, T., and Kanamori, T (2012). Density Ratio Estimation in Machine Learning, Cambridge University Press.","DOI":"10.1017\/CBO9781139035613"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Hido, S., Tsuboi, Y., Kashima, H., Sugiyama, M., and Kanamori, T (2008, January 15\u201319). Inlier-based Outlier Detection via Direct Density Ratio Estimation. Pisa, Italy.","DOI":"10.1109\/ICDM.2008.49"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"708","DOI":"10.1109\/TIT.2011.2163380","article-title":"f-Divergence estimation and two-sample homogeneity test under semiparametric density-ratio models","volume":"58","author":"Kanamori","year":"2012","journal-title":"IEEE Trans. Inform. Theor"},{"key":"ref_7","unstructured":"Kanamori, T., Hido, S., and Sugiyama, M (2009). Advances in Neural Information Processing Systems 21, MIT Press."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5847","DOI":"10.1109\/TIT.2010.2068870","article-title":"Estimating divergence functionals and the likelihood ratio by convex risk minimization","volume":"56","author":"Nguyen","year":"2010","journal-title":"IEEE Trans. Inform. Theor"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"619","DOI":"10.1093\/biomet\/85.3.619","article-title":"Inferences for case-control and semiparametric two-sample density ratio models","volume":"85","author":"Qin","year":"1998","journal-title":"Biometrika"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2734","DOI":"10.1162\/NECO_a_00492","article-title":"Density-difference estimation","volume":"25","author":"Sugiyama","year":"2013","journal-title":"Neural. Comput"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"699","DOI":"10.1007\/s10463-008-0197-x","article-title":"Direct importance estimation for covariate shift adaptation","volume":"60","author":"Sugiyama","year":"2008","journal-title":"Ann. Inst. Stat. Math"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly proper scoring rules, prediction, and estimation","volume":"102","author":"Gneiting","year":"2007","journal-title":"J. Am. Stat. Assoc"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1111\/j.2517-6161.1966.tb00626.x","article-title":"A general class of coefficients of divergence of one distribution from Another","volume":"28","author":"Ali","year":"1966","journal-title":"J. Roy. Stat. Soc. Series B"},{"key":"ref_14","first-page":"229","article-title":"Information-type measures of difference of probability distributions and indirect observation","volume":"2","year":"1967","journal-title":"Stud. Sci. Math. Hung"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1175\/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2","article-title":"Verification of forecasts expressed in terms of probability","volume":"78","author":"Brier","year":"1950","journal-title":"Mon. Weather Rev"},{"key":"ref_16","unstructured":"Godambe, V.P., and Sprott, D.A. (1971). Foundations of Statistical Inference, Dove."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1437","DOI":"10.1162\/089976604323057452","article-title":"Information geometry of U-Boost and Bregman divergence","volume":"16","author":"Murata","year":"2004","journal-title":"Neural Comput"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1214\/12-AOS971","article-title":"Proper local scoring rules","volume":"40","author":"Parry","year":"2012","journal-title":"Ann. Stat"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Hendrickson, A.D., and Buehler, R.J. (1921). Proper scores for probability forecasters. Ann. Mathe. Stat, 42.","DOI":"10.1214\/aoms\/1177693057"},{"key":"ref_20","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, Wiley-Interscience."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1093\/biomet\/85.3.549","article-title":"Robust and efficient estimation by minimising a density power divergence","volume":"85","author":"Basu","year":"1998","journal-title":"Biometrika"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Basu, A., Shioya, H., and Park, C (2010). Statistical Inference: The Minimum Distance Approach, Taylor & Francis.","DOI":"10.1201\/b10956"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2053","DOI":"10.1016\/j.jmva.2008.02.004","article-title":"Robust parameter estimation with a small bias against heavy contamination","volume":"99","author":"Fujisawa","year":"2008","journal-title":"J. Multivar. Anal"},{"key":"ref_24","unstructured":"Hampel, F.R., Rousseeuw, P.J., Ronchetti, E.M., and Stahel, W.A. (1986). Robust Statistics. The Approach Based on Influence Functions, John Wiley and Sons, Inc."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"262","DOI":"10.3390\/e12020262","article-title":"Entropy and divergence associated with power function and the statistical application","volume":"12","author":"Eguchi","year":"2010","journal-title":"Entropy"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Maronna, R., Martin, R., and Yohai, V (2006). Robust Statistics: Theory and Methods, Wiley.","DOI":"10.1002\/0470010940"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"974","DOI":"10.1198\/016214507000000617","article-title":"Robust truncated hinge loss support vector machines","volume":"102","author":"Wu","year":"2007","journal-title":"J. Am. Stat. Assoc"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Xu, H., Caramanis, C., Mannor, S., and Yun, S (2009, January 15\u201318). Risk Sensitive Robust Support Vector Machines. Shanghai, China.","DOI":"10.1109\/CDC.2009.5400598"},{"key":"ref_29","unstructured":"Xu, L., Crammer, K., and Schuurmans, D Robust Support Vector Machine Training via Convex Outlier Ablation."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kanamori, T., and Fujisawa, H (Bernoulli, 2014). Affine invariant divergences associated with composite scores and its applications, Bernoulli, submitted.","DOI":"10.3150\/13-BEJ557"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/16\/2\/921\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:08:12Z","timestamp":1760216892000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/16\/2\/921"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,2,17]]},"references-count":30,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2014,2]]}},"alternative-id":["e16020921"],"URL":"https:\/\/doi.org\/10.3390\/e16020921","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,2,17]]}}}