{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,20]],"date-time":"2026-06-20T00:40:38Z","timestamp":1781916038955,"version":"3.54.5"},"reference-count":51,"publisher":"Emerald","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2004,11,15]]},"abstract":"<jats:p>This tutorial is concerned with applications of information theory concepts in statistics, in the finite alphabet setting. The information measure known as information divergence or Kullback-Leibler distance or relative entropy plays a key role, often with a geometric flavor as an analogue of squared Euclidean distance, as in the concepts of I-projection, I-radius and I-centroid. The topics covered include large deviations, hypothesis testing, maximum likelihood estimation in exponential families, analysis of contingency tables, and iterative algorithms with an \u201cinformation geometry\u201d background. Also, an introduction is provided to the theory of universal coding, and to statistical inference via the minimum description length principle motivated by that theory.<\/jats:p>","DOI":"10.1561\/0100000004","type":"journal-article","created":{"date-parts":[[2005,2,3]],"date-time":"2005-02-03T05:30:16Z","timestamp":1107408616000},"page":"417-528","source":"Crossref","is-referenced-by-count":359,"title":["Information Theory and Statistics: A Tutorial"],"prefix":"10.1108","volume":"1","author":[{"given":"I.","family":"Csisz\u00e1r","sequence":"first","affiliation":[{"name":"R\u00e9enyi Institute of Mathematics, Hungarian Academy of Sciences POB 127, H-1364 Budapest,","place":["Hungary"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"P.\tC.","family":"Shields","sequence":"additional","affiliation":[{"name":"University of Toledo , Ohio,","place":["USA"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"140","published-online":{"date-parts":[[2004,11,15]]},"reference":[{"key":"2026042013140921700_ref001","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1111\/j.2517-6161.1966.tb00626.x","article-title":"A general class of coefficients of divergence of one distribution from another","volume":"28","author":"Ali","year":"1966","journal-title":"J. Royal Stat. Soc., Ser.B"},{"key":"2026042013140921700_ref002","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4612-5056-2","volume-title":"Differential-Geometrical Methods in Statistics","author":"Amari","year":"1985"},{"key":"2026042013140921700_ref003","volume-title":"Logically smooth density estimation","author":"Barron","year":"1985"},{"key":"2026042013140921700_ref004","first-page":"373","article-title":"Beziehung zwischen dem zweiten Hauptsatze der mechanischen W\u00e4rmetheorie und der Wahrscheinlichkeitsrechnung respektive den S\u00e4tzen \u00fcber das W\u00e4rmegleichgewicht","volume":"76","author":"Boltzmann","year":"1877","journal-title":"Wien. Ber"},{"key":"2026042013140921700_ref005","author":"Cencov","journal-title":"Statistical Decision Rules and Optimal Inference"},{"key":"2026042013140921700_ref006","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1214\/aoms\/1177729330","article-title":"A measure of asymptotic efficiency for tests of a hypothesis based on a sum of observations","volume":"23","author":"Chernoff","year":"1952","journal-title":"Annals Math. Statist"},{"key":"2026042013140921700_ref007","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1109\/TIT.1984.1056869","article-title":"An algorithm for maximizing expected log investment return","volume":"30","author":"Cover","year":"1984","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref008","volume-title":"Elements of Information Theory","author":"Cover","year":"1991"},{"key":"2026042013140921700_ref009","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4899-2887-0","volume-title":"Theoretical Statistics","author":"Cox","year":"1974"},{"key":"2026042013140921700_ref010","first-page":"85","article-title":"Eine informationtheoretische Ungleichung und ihre Anwendung auf den Beweis der Ergodizit\u00e4t von Markoffschen Ketten","volume":"8","author":"Csisz\u00e1r","year":"1963","journal-title":"Publ. Math. Inst. Hungar. Acad. Sci"},{"key":"2026042013140921700_ref011","first-page":"299","article-title":"Information-type measures of difference of probability distributions and indirect observations","volume":"2","author":"Csisz\u00e1r","year":"1967","journal-title":"Studia Sci. Math. Hungar"},{"key":"2026042013140921700_ref012","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1214\/aop\/1176996454","article-title":"I-divergence geometry of probability distributions and minimization problems","volume":"3","author":"Csisz\u00e1r","year":"1975","journal-title":"Annals Probab"},{"key":"2026042013140921700_ref013","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1214\/aos\/1176347279","article-title":"A geometric interpretation of Darroch and Ratcliff\u2019s generalized iterative scaling","volume":"17","author":"Csisz\u00e1r","year":"1989","journal-title":"Annals Statist"},{"key":"2026042013140921700_ref014","doi-asserted-by":"crossref","first-page":"2031","DOI":"10.1214\/aos\/1176348385","article-title":"Why least squares and maximum entropy? An axiomatic approach to linear inverse problems","volume":"19","author":"Csisz\u00e1r","year":"1991","journal-title":"Annals Statist"},{"key":"2026042013140921700_ref015","volume-title":"Information Theory: Coding Theorems for Discrete Memoryless Systems","author":"Csisz\u00e1r","year":"1981"},{"key":"2026042013140921700_ref016","doi-asserted-by":"crossref","first-page":"1474","DOI":"10.1109\/TIT.2003.810633","article-title":"Information projections revisited","volume":"49","author":"Csisz\u00e1r","year":"2003","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref017","doi-asserted-by":"crossref","first-page":"1601","DOI":"10.1214\/aos\/1015957472","article-title":"The consistency of the BIC Markov order estimator","volume":"28","author":"Csisz\u00e1r","year":"2000","journal-title":"Annals Statist"},{"issue":"1","key":"2026042013140921700_ref018","first-page":"205","article-title":"Information geometry and alternating minimization procedures","volume":"Suppl.","author":"Csisz\u00e1r","year":"1984","journal-title":"Statistics and Decisions,"},{"key":"2026042013140921700_ref019","doi-asserted-by":"crossref","first-page":"1470","DOI":"10.1214\/aoms\/1177692379","article-title":"Generalized iterative scaling for log-linear models","volume":"43","author":"Darroch","year":"1972","journal-title":"Annals Math. Statist"},{"key":"2026042013140921700_ref020","doi-asserted-by":"crossref","first-page":"783","DOI":"10.1109\/TIT.1973.1055092","article-title":"Universal noiseless coding","volume":"19","author":"Davisson","year":"1973","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref021","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1109\/TIT.1980.1056167","article-title":"A source matching approach to finding minimax codes","volume":"26","author":"Davisson","year":"1980","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref022","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1109\/TIT.1981.1056355","article-title":"Efficient universal noiseless source codes","volume":"27","author":"Davisson","year":"1981","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref023","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum likelihood from incomplete data via the EM algorithm","volume":"39","author":"Dempster","year":"1977","journal-title":"J. Royal Stat. Soc., Ser.B"},{"key":"2026042013140921700_ref024","volume-title":"Order estimation for functions of Markov chains","author":"Finesso","year":"1990"},{"issue":"2","key":"2026042013140921700_ref025","first-page":"3","article-title":"Coding in the case of unknown and changing message statistics","volume":"2","author":"Fitingof","year":"1966","journal-title":"Probl. Inform. Transmission"},{"key":"2026042013140921700_ref026","volume-title":"The Information in Contingency Tables","author":"Gokhale","year":"1978"},{"key":"2026042013140921700_ref027","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1214\/aoms\/1177700150","article-title":"Asymptotically optimal tests for multinomial distributions","volume":"36","author":"Hoeffding","year":"1965","journal-title":"Annals Math. Statist"},{"key":"2026042013140921700_ref028","first-page":"179","article-title":"Contingency tables with given marginals","volume":"55","author":"Ireland","year":"1968","journal-title":"Bio-metrica"},{"key":"2026042013140921700_ref029","volume-title":"Lectures in Information Theory","author":"Krichevsky","year":"1970"},{"key":"2026042013140921700_ref030","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1109\/TIT.1981.1056331","article-title":"The performance of universal coding","volume":"27","author":"Krichevsky","year":"1981","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref031","first-page":"E15","article-title":"Telefoonverkeersrekening","volume":"52","author":"Kruithof","year":"1937","journal-title":"De Ingenieur"},{"key":"2026042013140921700_ref032","volume-title":"Information Theory and Statistics","author":"Kullback","year":"1959"},{"key":"2026042013140921700_ref033","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On information and sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Annals Math. Statist"},{"key":"2026042013140921700_ref034","volume-title":"Convex Statistical Distances","author":"Liese","year":"1987"},{"key":"2026042013140921700_ref035","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1147\/rd.203.0198","article-title":"Generalized Kraft inequality and arithmetic coding","volume":"20","author":"Rissanen","year":"1976","journal-title":"IBM J. Res. Devel"},{"key":"2026042013140921700_ref036","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1016\/0005-1098(78)90005-5","article-title":"Modeling by shortest data description","volume":"14","author":"Rissanen","year":"1978","journal-title":"Automatica"},{"key":"2026042013140921700_ref037","doi-asserted-by":"crossref","first-page":"348","DOI":"10.1109\/TIT.1982.1056467","article-title":"Tight lower bounds for optimum code length","volume":"28","author":"Rissanen","year":"1982","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref038","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1109\/TIT.1984.1056936","article-title":"Universal coding, information, prediction and estimation","volume":"30","author":"Rissanen","year":"1984","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref039","volume-title":"Stochastic Complexity in Statistical Inquiry","author":"Rissanen","year":"1989"},{"key":"2026042013140921700_ref040","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1147\/rd.232.0149","article-title":"Arithmetic coding","volume":"23","author":"Rissanen","year":"1979","journal-title":"IBM J. Res. Devel"},{"issue":"3","key":"2026042013140921700_ref041","first-page":"24","article-title":"Twice-universal coding","volume":"20","author":"Ryabko","year":"1984","journal-title":"Probl. Inform. Transmission"},{"key":"2026042013140921700_ref042","first-page":"11","article-title":"On the probability of large deviations of random variables","volume":"42","author":"Sanov","year":"1957","journal-title":"Mat. Sbornik"},{"key":"2026042013140921700_ref043","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1214\/aos\/1176344136","article-title":"Estimating the dimension of a model","volume":"6","author":"Schwarz","year":"1978","journal-title":"Annals Statist"},{"key":"2026042013140921700_ref044","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Techn. J"},{"key":"2026042013140921700_ref045","doi-asserted-by":"crossref","DOI":"10.1090\/gsm\/013","volume-title":"The ergodic theory of discrete sample paths","author":"Shields","year":"1996"},{"key":"2026042013140921700_ref046","first-page":"175","article-title":"Coding of discrete sources with unknown statistics","volume":"23","author":"Shtarkov","year":"1977","journal-title":"Topics in Information Theory. Colloquia Math. Soc. J. Bolyai"},{"key":"2026042013140921700_ref047","first-page":"291","article-title":"An information theoretic identity and a problem involving capacity","volume":"2","author":"Tops\u0153","year":"1967","journal-title":"Studia Sci. Math. Hungar"},{"key":"2026042013140921700_ref048","first-page":"16","article-title":"Redundancy of universal coding of arbitrary Markov sources","volume":"10","author":"Trofimov","year":"1974","journal-title":"Probl. Inform. Transmission"},{"key":"2026042013140921700_ref049","volume-title":"Sequential Analysis","author":"Wald","year":"1947"},{"key":"2026042013140921700_ref050","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1109\/18.382012","article-title":"The context weighting method: basic properties","volume":"41","author":"Willems","year":"1995","journal-title":"IEEE Trans. Inform. Theory"},{"key":"2026042013140921700_ref051","article-title":"A universal algorithm for sequential data compression","volume":"24","author":"Ziv","year":"1977","journal-title":"IEEE Trans. Inform. Theory"}],"container-title":["Foundations and Trends\u00ae in Communications and Information Theory"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/ftcit\/article-pdf\/1\/4\/417\/11531740\/0100000004en.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/www.emerald.com\/ftcit\/article-pdf\/1\/4\/417\/11531740\/0100000004en.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T14:10:36Z","timestamp":1777471836000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.emerald.com\/ftcit\/article\/1\/4\/417\/1360943\/Information-Theory-and-Statistics-A-Tutorial"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2004,11,15]]},"references-count":51,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2004,11,15]]}},"URL":"https:\/\/doi.org\/10.1561\/0100000004","relation":{},"ISSN":["1567-2190","1567-2328"],"issn-type":[{"value":"1567-2190","type":"print"},{"value":"1567-2328","type":"electronic"}],"subject":[],"published":{"date-parts":[[2004,11,15]]}}}