{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T15:34:49Z","timestamp":1780673689713,"version":"3.54.1"},"reference-count":16,"publisher":"MIT Press - Journals","issue":"11","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2014,11]]},"abstract":"<jats:p> In this letter, we introduce an estimator of K\u00fcllback-Leibler divergence based on two independent samples. We show that on any finite alphabet, this estimator has an exponentially decaying bias and that it is consistent and asymptotically normal. To explain the importance of this estimator, we provide a thorough analysis of the more standard plug-in estimator. We show that it is consistent and asymptotically normal, but with an infinite bias. Moreover, if we modify the plug-in estimator to remove the rare events that cause the bias to become infinite, the bias still decays at a rate no faster than [Formula: see text]. Further, we extend our results to estimating the symmetrized K\u00fcllback-Leibler divergence. We conclude by providing simulation results, which show that the asymptotic properties of these estimators hold even for relatively small sample sizes. <\/jats:p>","DOI":"10.1162\/neco_a_00646","type":"journal-article","created":{"date-parts":[[2014,7,24]],"date-time":"2014-07-24T15:06:10Z","timestamp":1406214370000},"page":"2570-2593","source":"Crossref","is-referenced-by-count":21,"title":["Nonparametric Estimation of K\u00fcllback-Leibler Divergence"],"prefix":"10.1162","volume":"26","author":[{"given":"Zhiyi","family":"Zhang","sequence":"first","affiliation":[{"name":"Department of Mathematics and Statistics, University of North Carolina at Charlotte, Charlotte, NC 28223, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Grabchak","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Statistics, University of North Carolina at Charlotte, Charlotte, NC 28223, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"B1","volume-title":"Elements of information theory","author":"Cover T. M.","year":"2006","edition":"2"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729694"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1111\/j.1600-0706.2011.19267.x"},{"key":"B4","first-page":"95","volume-title":"Information theory in psychology: Problems and methods","author":"Miller G.","year":"1955"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1162\/003465305775098170"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2010.2068870"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.1162\/089976603321780272"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb00917.x"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1992.tb01875.x"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1109\/97.995827"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2009.2016060"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00266"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2012.2217393"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.3390\/e15061999"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2011.2179702"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1016\/j.jspi.2009.12.023"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/NECO_a_00646","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:40:30Z","timestamp":1615585230000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/26\/11\/2570-2593\/8012"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,11]]},"references-count":16,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2014,11]]}},"alternative-id":["10.1162\/NECO_a_00646"],"URL":"https:\/\/doi.org\/10.1162\/neco_a_00646","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,11]]}}}