{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:49:27Z","timestamp":1760240967017,"version":"build-2065373602"},"reference-count":72,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2019,10,21]],"date-time":"2019-10-21T00:00:00Z","timestamp":1571616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>This paper is focused on the derivation of data-processing and majorization inequalities for f-divergences, and their applications in information theory and statistics. For the accessibility of the material, the main results are first introduced without proofs, followed by exemplifications of the theorems with further related analytical results, interpretations, and information-theoretic applications. One application refers to the performance analysis of list decoding with either fixed or variable list sizes; some earlier bounds on the list decoding error probability are reproduced in a unified way, and new bounds are obtained and exemplified numerically. Another application is related to a study of the quality of approximating a probability mass function, induced by the leaves of a Tunstall tree, by an equiprobable distribution. The compression rates of finite-length Tunstall codes are further analyzed for asserting their closeness to the Shannon entropy of a memoryless and stationary discrete source. Almost all the analysis is relegated to the appendices, which form the major part of this manuscript.<\/jats:p>","DOI":"10.3390\/e21101022","type":"journal-article","created":{"date-parts":[[2019,10,21]],"date-time":"2019-10-21T11:37:55Z","timestamp":1571657875000},"page":"1022","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["On Data-Processing and Majorization Inequalities for f-Divergences with Applications"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5681-1273","authenticated-orcid":false,"given":"Igal","family":"Sason","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering, Technion\u2014Israel Institute of Technology, Haifa 3200003, Israel"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1111\/j.2517-6161.1966.tb00626.x","article-title":"A general class of coefficients of divergence of one distribution from another","volume":"28","author":"Ali","year":"1966","journal-title":"J. R. Stat. Soc."},{"key":"ref_2","first-page":"85","article-title":"Eine Informationstheoretische Ungleichung und ihre Anwendung auf den Bewis der Ergodizit\u00e4t von Markhoffschen Ketten","volume":"8","year":"1963","journal-title":"Publ. Math. Inst. Hungar. Acad. Sci."},{"key":"ref_3","first-page":"185","article-title":"A note on Jensen\u2019s inequality","volume":"1","year":"1966","journal-title":"Studia Scientiarum Mathematicarum Hungarica"},{"key":"ref_4","first-page":"299","article-title":"Information-type measures of difference of probability distributions and indirect observations","volume":"2","year":"1967","journal-title":"Studia Scientiarum Mathematicarum Hungarica"},{"key":"ref_5","first-page":"329","article-title":"On topological properties of f-divergences","volume":"2","year":"1967","journal-title":"Studia Scientiarum Mathematicarum Hungarica"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/BF02018661","article-title":"A class of measures of informativity of observation channels","volume":"2","year":"1972","journal-title":"Periodica Mathematicarum Hungarica"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1143\/JPSJ.18.328","article-title":"Markov processes and the H-theorem","volume":"18","author":"Morimoto","year":"1963","journal-title":"J. Phys. Soc. Jpn."},{"unstructured":"Liese, F., and Vajda, I. (1987). Convex Statistical Distances. Teubner-Texte Zur Mathematik, Springer.","key":"ref_8"},{"unstructured":"Chapman and Hall\/CRC, Taylor &amp (2006). Statistical Inference Based on Divergence Measures, Francis Group.","key":"ref_9"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1288","DOI":"10.1109\/18.605597","article-title":"About distances of discrete distributions satisfying the data processing theorem of information theory","volume":"43","author":"Pardo","year":"1997","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1080\/02331880902986919","article-title":"On divergences of finite measures and their applicability in statistics and information theory","volume":"44","author":"Stummer","year":"2010","journal-title":"Statistics"},{"unstructured":"Vajda, I. (1989). Theory of Statistical Inference and Information, Kluwer Academic Publishers.","key":"ref_12"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1109\/TIT.1973.1055015","article-title":"On functionals satisfying a data-processing theorem","volume":"19","author":"Ziv","year":"1973","journal-title":"IEEE Trans. Inf. Theory"},{"doi-asserted-by":"crossref","unstructured":"Longo, G. (1975). A generalization of the rate-distortion theory and applications. Information Theory\u2014New Trends and Open Problems, Springer.","key":"ref_14","DOI":"10.1007\/978-3-7091-2730-8"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"4926","DOI":"10.1109\/TIT.2011.2159052","article-title":"Data processing theorems and the second law of thermodynamics","volume":"57","author":"Merhav","year":"2011","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"4394","DOI":"10.1109\/TIT.2006.881731","article-title":"On divergences and informations in statistics and information theory","volume":"52","author":"Liese","year":"2006","journal-title":"IEEE Trans. Inf. Theory"},{"doi-asserted-by":"crossref","unstructured":"Csisz\u00e1r, I., and K\u00f6rner, J. (2011). Information Theory: Coding Theorems for Discrete Memoryless Systems, Cambridge University Press. [2nd ed.].","key":"ref_17","DOI":"10.1017\/CBO9780511921889"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"925","DOI":"10.1214\/aop\/1176995937","article-title":"Spreading of sets in product spaces and hypercontraction of the Markov operator","volume":"4","author":"Ahlswede","year":"1976","journal-title":"Ann. Probab."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1879","DOI":"10.1109\/TIT.2017.2782359","article-title":"Strong data processing inequalities for input constrained additive noise channels","volume":"64","author":"Calmon","year":"2018","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1016\/0024-3795(93)90331-H","article-title":"Relative entropy under mappings by stochastic matrices","volume":"179","author":"Cohen","year":"1993","journal-title":"Linear Algebra Appl."},{"unstructured":"Cohen, J.E., Kemperman, J.H.B., and Zb\u0103ganu, Gh. (1998). Comparison of Stochastic Matrices with Applications in Information Theory, Statistics, Economics and Population Sciences, Birkh\u00e4user.","key":"ref_21"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"5704","DOI":"10.1109\/TIT.2018.2839743","article-title":"Comparison of channels: Criteria for domination by a symmetric channel","volume":"64","author":"Makur","year":"2018","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1109\/TIT.2015.2482978","article-title":"Dissipation of information in channels with input constraints","volume":"62","author":"Polyanskiy","year":"2016","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3355","DOI":"10.1109\/TIT.2016.2549542","article-title":"Strong data processing inequalities and \u03a6-Sobolev inequalities for discrete channels","volume":"62","author":"Raginsky","year":"2016","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/978-1-4939-7005-6_7","article-title":"Strong data processing inequalities for channels and Bayesian networks","volume":"Volume 161","author":"Carlen","year":"2017","journal-title":"Convexity and Concentration"},{"unstructured":"Makur, A., and Zheng, L. (2018). Linear bounds between contraction coefficients for f-divergences. arXiv.","key":"ref_26"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1080\/14786440009463897","article-title":"On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling","volume":"50","author":"Pearson","year":"1900","journal-title":"Lond. Edinb. Dublin Philos. Mag. J. Sci."},{"unstructured":"Neyman, J. (1945, January 13\u201318). Contribution to the theory of the \u03c72 test. Proceedings of the First Berkeley Symposium on Mathematical Statistics and Probability, Berkeley, CA, USA.","key":"ref_28"},{"unstructured":"Sarmanov, O.V. (1962). Maximum correlation coefficient (non-symmetric case). Selected Translations in Mathematical Statistics and Probability, American Mathematical Society.","key":"ref_29"},{"doi-asserted-by":"crossref","unstructured":"Marshall, A.W., Olkin, I., and Arnold, B.C. (2011). Inequalities: Theory of Majorization and Its Applications, Springer. [2nd ed.].","key":"ref_30","DOI":"10.1007\/978-0-387-68276-1"},{"doi-asserted-by":"crossref","unstructured":"Steele, J.M. (2004). The Cauchy-Schwarz Master Class, Cambridge University Press.","key":"ref_31","DOI":"10.1017\/CBO9780511817106"},{"doi-asserted-by":"crossref","unstructured":"Bhatia, R. (1997). Matrix Analysis, Springer.","key":"ref_32","DOI":"10.1007\/978-1-4612-0653-8"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2220","DOI":"10.1109\/TIT.2017.2787181","article-title":"Bounds on the entropy of a function of a random variable and their applications","volume":"64","author":"Cicalese","year":"2018","journal-title":"IEEE Trans. Inf. Theory"},{"doi-asserted-by":"crossref","unstructured":"Sason, I. (2018). Tight bounds on the R\u00e9nyi entropy via majorization with applications to guessing and compression. Entropy, 20.","key":"ref_34","DOI":"10.3390\/e20120896"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"5930","DOI":"10.1109\/TIT.2010.2080891","article-title":"On the interplay between conditional entropy and error probability","volume":"56","author":"Ho","year":"2010","journal-title":"IEEE Trans. Inf. Theory"},{"doi-asserted-by":"crossref","unstructured":"Ho, S.W., and Verd\u00fa, S. (2015, January 14\u201319). Convexity\/concavity of the R\u00e9nyi entropy and \u03b1-mutual information. Proceedings of the 2015 IEEE International Symposium on Information Theory, Hong Kong, China.","key":"ref_36","DOI":"10.1109\/ISIT.2015.7282554"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1007\/BF02124750","article-title":"On the Lambert W function","volume":"5","author":"Corless","year":"1996","journal-title":"Adv. Comput. Math."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"3772","DOI":"10.1109\/TIT.2006.878151","article-title":"A note on approximation of uniform distributions from variable-to-fixed length codes","volume":"52","author":"Cicalese","year":"2006","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1007\/BF01016429","article-title":"Possible generalization of the Boltzmann-Gibbs statistics","volume":"52","author":"Tsallis","year":"1988","journal-title":"J. Stat. Phys."},{"key":"ref_40","first-page":"547","article-title":"On measures of entropy and information","volume":"Volume 1","year":"1961","journal-title":"Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"3436","DOI":"10.1109\/TIT.2019.2894519","article-title":"Minimum-entropy couplings and their applications","volume":"65","author":"Cicalese","year":"2019","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/TIT.2017.2757496","article-title":"Arimoto-R\u00e9nyi conditional entropy and Bayesian M-ary hypothesis testing","volume":"64","author":"Sason","year":"2018","journal-title":"IEEE Trans. Inf. Theory"},{"unstructured":"Amari, S., and Nagaoka, H. (2000). Methods of Information Geometry, Oxford University Press.","key":"ref_43"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3390\/e12061532","article-title":"Families of Alpha- Beta- and Gamma- divergences: Flexible and robust measures of similarities","volume":"12","author":"Cichocki","year":"2010","journal-title":"Entropy"},{"doi-asserted-by":"crossref","unstructured":"Sason, I. (2018). On f-divergences: Integral representations, local behavior, and inequalities. Entropy, 20.","key":"ref_45","DOI":"10.3390\/e20050383"},{"unstructured":"Fano, R.M. (1952). Class Notes for Course 6.574: Transmission of Information, MIT.","key":"ref_46"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1007\/BF00535682","article-title":"Bounds on conditional probabilities with applications in multi-user communication","volume":"34","author":"Ahlswede","year":"1977","journal-title":"Z. Wahrscheinlichkeitstheorie verw. Gebiete"},{"doi-asserted-by":"crossref","unstructured":"Raginsky, M., and Sason, I. (2019). Concentration of measure inequalities in information theory, communications and coding: Third edition. Foundations and Trends (FnT) in Communications and Information Theory, NOW Publishers.","key":"ref_48","DOI":"10.1561\/9781680835359"},{"key":"ref_49","first-page":"7687","article-title":"On Bayes risk lower bounds","volume":"17","author":"Chen","year":"2016","journal-title":"J. Mach. Learn. Res."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"2386","DOI":"10.1109\/TIT.2011.2110791","article-title":"Lower bounds for the minimax risk using f-divergences, and applications","volume":"57","author":"Guntuboyina","year":"2011","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1850","DOI":"10.1109\/TIT.2008.920242","article-title":"State amplification","volume":"54","author":"Kim","year":"2008","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_52","first-page":"41","article-title":"Information measures and capacity of order \u03b1 for discrete memoryless channels","volume":"Volume 16","author":"Elias","year":"1977","journal-title":"Topics in Information Theory\u20142nd Colloquium"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1109\/TIT.1975.1055469","article-title":"Source coding with side information and a converse for degraded broadcast channels","volume":"21","author":"Ahlswede","year":"1975","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_54","first-page":"2629","article-title":"E\u03b3 resolvability","volume":"63","author":"Liu","year":"2017","journal-title":"IEEE Trans. Inf. Theory"},{"doi-asserted-by":"crossref","unstructured":"Br\u00e9maud, P. (2017). Discrete Probability Models and Methods: Probability on Graphs and Trees, Markov Chains and Random Fields, Entropy and Coding, Springer.","key":"ref_55","DOI":"10.1007\/978-3-319-43476-6"},{"unstructured":"Tunstall, B.K. (1967). Synthesis of Noiseless Compression Codes. [Ph.D. Thesis, Georgia Institute of Technology].","key":"ref_56"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1214\/aoms\/1177704567","article-title":"Uncertainty, information and sequential experiments","volume":"33","author":"DeGroot","year":"1962","journal-title":"Ann. Math. Stat."},{"unstructured":"Roberts, A.W., and Varberg, D.E. (1973). Convex Functions, Academic Press.","key":"ref_58"},{"unstructured":"Rockafellar, R.T. (1996). Convex Analysis, Princeton University Press.","key":"ref_59"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"4387","DOI":"10.1109\/TIT.2019.2904508","article-title":"An exact expression for the gap in the data processing inequality for f-divergences","volume":"65","author":"Collet","year":"2019","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/0041-5553(67)90040-7","article-title":"The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming","volume":"7","author":"Bregman","year":"1967","journal-title":"USSR Comput. Math. Math. Phys."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"5973","DOI":"10.1109\/TIT.2016.2603151","article-title":"f-divergence inequalities","volume":"62","author":"Sason","year":"2016","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"5377","DOI":"10.1109\/TIT.2010.2068710","article-title":"On Pinsker\u2019s and Vajda\u2019s type inequalities for Csisz\u00e1r\u2019s f-divergences","volume":"56","author":"Gilardoni","year":"2010","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1111\/j.1751-5823.2002.tb00178.x","article-title":"On choosing and bounding probability metrics","volume":"70","author":"Gibbs","year":"2002","journal-title":"Int. Stat. Rev."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1007\/s10474-018-0848-1","article-title":"Second and third order moment inequalities for probability distributions","volume":"155","author":"Simic","year":"2018","journal-title":"Acta Math. Hung."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"3797","DOI":"10.1109\/TIT.2014.2320500","article-title":"R\u00e9nyi divergence and Kullback\u2013Leibler divergence","volume":"60","year":"2014","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"1860","DOI":"10.1109\/TIT.2003.813509","article-title":"On asymptotic properties of information-theoretic divergences","volume":"49","author":"Pardo","year":"2003","journal-title":"IEEE Trans. Inf. Theory"},{"doi-asserted-by":"crossref","unstructured":"Beck, A. (2014). Introduction to Nonlinear Optimization: Theory, Algorithms and Applications with Matlab, SIAM-Society for Industrial and Applied Mathematics.","key":"ref_68","DOI":"10.1137\/1.9781611973655"},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"037359","DOI":"10.1155\/2007\/37359","article-title":"On logarithmic convexity for differences of power means","volume":"2007","author":"Simic","year":"2008","journal-title":"J. Inequalities Appl."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"857","DOI":"10.1016\/S1631-073X(03)00215-2","article-title":"Dual representation of \u03c6-divergences and applications","volume":"336","author":"Keziou","year":"2003","journal-title":"C. R. Math."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"5847","DOI":"10.1109\/TIT.2010.2068870","article-title":"Estimating divergence functionals and the likelihood ratio by convex risk minimization","volume":"56","author":"Nguyen","year":"2010","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1109\/TIT.1972.1054899","article-title":"On variable-length-to-block coding","volume":"18","author":"Jelineck","year":"1972","journal-title":"IEEE Trans. Inf. Theory"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/10\/1022\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:28:12Z","timestamp":1760189292000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/21\/10\/1022"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,21]]},"references-count":72,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2019,10]]}},"alternative-id":["e21101022"],"URL":"https:\/\/doi.org\/10.3390\/e21101022","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2019,10,21]]}}}