{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:12:33Z","timestamp":1760242353843,"version":"build-2065373602"},"reference-count":62,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2017,5,3]],"date-time":"2017-05-03T00:00:00Z","timestamp":1493769600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Logarithmic score and information divergence appear in information theory, statistics, statistical mechanics, and portfolio theory. We demonstrate that all these topics involve some kind of optimization that leads directly to regret functions and such regret functions are often given by Bregman divergences. If a regret function also fulfills a sufficiency condition it must be proportional to information divergence. We will demonstrate that sufficiency is equivalent to the apparently weaker notion of locality and it is also equivalent to the apparently stronger notion of monotonicity. These sufficiency conditions have quite different relevance in the different areas of application, and often they are not fulfilled. Therefore sufficiency conditions can be used to explain when results from one area can be transferred directly to another and when one will experience differences.<\/jats:p>","DOI":"10.3390\/e19050206","type":"journal-article","created":{"date-parts":[[2017,5,3]],"date-time":"2017-05-03T12:24:47Z","timestamp":1493814287000},"page":"206","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Divergence and Sufficiency for Convex Optimization"],"prefix":"10.3390","volume":"19","author":[{"given":"Peter","family":"Harremo\u00ebs","sequence":"first","affiliation":[{"name":"GSK Department, Copenhagen Business College, N\u00f8rre Voldgade 34, 1358 Copenhagen K, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2017,5,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On Information and Sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Ann. Math. Stat."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"620","DOI":"10.1103\/PhysRev.106.620","article-title":"Information Theory and Statistical Mechanics, I","volume":"106","author":"Jaynes","year":"1957","journal-title":"Phys. Rev."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1103\/PhysRev.108.171","article-title":"Information Theory and Statistical Mechanics, II","volume":"108","author":"Jaynes","year":"1957","journal-title":"Phys. Rev."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Skilling, J. (1989). Clearing up mysteries\u2014The original goal. Maximum Entropy and Bayesian Methods, Kluwer.","DOI":"10.1007\/978-94-015-7860-8"},{"key":"ref_5","unstructured":"Liese, F., and Vajda, I. (1987). Convex Statistical Distances, Teubner."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2743","DOI":"10.1109\/18.720554","article-title":"The Minimum Description Length Principle in Coding and Modeling","volume":"44","author":"Barron","year":"1998","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Csisz\u00e1r, I., and Shields, P. (2004). Information Theory and Statistics: A Tutorial, Now Publishers Inc.. Foundations and Trends in Communications and Information Theory.","DOI":"10.1561\/9781933019543"},{"key":"ref_8","first-page":"1367","article-title":"Game Theory, Maximum Entropy, Minimum Discrepancy, and Robust Bayesian Decision Theory","volume":"32","author":"Dawid","year":"2004","journal-title":"Ann. Math. Stat."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Gr\u00fcnwald, P. (2007). The Minimum Description Length Principle, MIT Press.","DOI":"10.7551\/mitpress\/4643.001.0001"},{"key":"ref_10","unstructured":"Holevo, A.S. (1982). Probabilistic and Statistical Aspects of Quantum Theory, North-Holland."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Krumm, M., Barnum, H., Barrett, J., and M\u00fcller, M. (arXiv, 2016). Thermodynamics and the structure of quantum theory, arXiv.","DOI":"10.1088\/1367-2630\/aa68ef"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"123029","DOI":"10.1088\/1367-2630\/16\/12\/123029","article-title":"Higher-order interference and single-system postulates characterizing quantum theory","volume":"16","author":"Barnum","year":"2014","journal-title":"New J. Phys."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Harremo\u00ebs, P. (arXiv, 2016). Maximum Entropy and Sufficiency, arXiv.","DOI":"10.1063\/1.4985352"},{"key":"ref_14","unstructured":"Harremo\u00ebs, P. (arXiv, 2017). Quantum information on Spectral Sets, arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Barnum, H., Lee, C.M., Scandolo, C.M., and Selby, J.H. (arXiv, 2017). Ruling out higher-order interference from purity principles, arXiv.","DOI":"10.3390\/e19060253"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1080\/01621459.1951.10500768","article-title":"The Theory of Statistical Decision","volume":"46","author":"Servage","year":"1951","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"961","DOI":"10.1287\/opre.30.5.961","article-title":"Regret in decision making under uncertainty","volume":"30","author":"Bell","year":"1982","journal-title":"Oper. Res."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Fishburn, P.C. (1982). The Foundations of Expected Utility, Springer.","DOI":"10.1007\/978-94-017-3329-8"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"805","DOI":"10.2307\/2232669","article-title":"Regret theory: An alternative theory of rational choice under uncertainty","volume":"92","author":"Loomes","year":"1982","journal-title":"Econ. J."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"95","DOI":"10.3982\/TE738","article-title":"Transitive regret","volume":"6","author":"Bikhchandani","year":"2011","journal-title":"Theor. Econ."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1142","DOI":"10.1137\/S0363012995281742","article-title":"Proximal Minimization Methods with Generalized Bregman Functions","volume":"35","author":"Kiwiel","year":"1997","journal-title":"SIAM J. Control Optim."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1287\/moor.22.2.326","article-title":"Free-steering Relaxation Methods for Problems with Strictly Convex Costs and Linear Constraints","volume":"22","author":"Kiwiel","year":"1997","journal-title":"Math. Oper. Res."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Rockafellar, R.T. (1970). Convex Analysis, Princeton University Press.","DOI":"10.1515\/9781400873173"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1916","DOI":"10.1214\/aoms\/1177693057","article-title":"Proper scores for probability forecasters","volume":"42","author":"Hendrickson","year":"1971","journal-title":"Ann. Math. Stat."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1109\/TIT.1985.1057082","article-title":"Cross Entropy, Dissimilarity Measures, and Characterizations of Quadratic Entropy","volume":"31","author":"Rao","year":"1985","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_26","first-page":"1705","article-title":"Clustering with Bregman Divergences","volume":"6","author":"Banerjee","year":"2005","journal-title":"J. Mach. Learn. Res."},{"key":"ref_27","unstructured":"Kraft, L.G. (1949). A Device for Quanitizing, Grouping and Coding Amplitude Modulated Pulses. [Master\u2019s Thesis, Department of Electrical Engineering, MIT University]."},{"key":"ref_28","unstructured":"Han, T.S., and Kobayashi, K. (2002). Mathematics of Information and Coding, American Mathematical Society. Translations of Mathematical Monographs."},{"key":"ref_29","unstructured":"De Finetti, B. (1974). Theory of Probability, Wiley."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"654","DOI":"10.1073\/pnas.42.9.654","article-title":"Measures of the value of information","volume":"42","author":"McCarthy","year":"1956","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1198\/016214506000001437","article-title":"Strictly Proper Scoring Rules, Prediction, and Estimation","volume":"102","author":"Gneiting","year":"2007","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_32","unstructured":"Ovcharov, E.Y. (arXiv, 2015). Proper Scoring Rules and Bregman Divergences, arXiv."},{"key":"ref_33","unstructured":"Gundersen, T. (2011). An Introduction to the Concept of Exergy and Energy Quality, Norwegian University of Science and Technology. Lecture notes."},{"key":"ref_34","unstructured":"Harremo\u00ebs, P. (1993). Time and Conditional Independence, IMFUFA Roskilde University. IMFUFA-Tekst."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1002\/j.1538-7305.1956.tb03809.x","article-title":"A New Interpretation of Information Rate","volume":"35","author":"Kelly","year":"1956","journal-title":"Bell Syst. Tech. J."},{"key":"ref_36","unstructured":"Cover, T.M., and Thomas, J.A. (1991). Elements of Information Theory, Wiley."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.1467-9965.1991.tb00002.x","article-title":"Universal portfolios","volume":"1","author":"Cover","year":"1991","journal-title":"Math. Finance"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/0034-4877(70)90009-1","article-title":"On the Shannon Entropy and Related Functionals on Convex Sets","volume":"1","author":"Uhlmann","year":"1970","journal-title":"Rep. Math. Phys."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1777","DOI":"10.1007\/s00023-017-0550-9","article-title":"Monotonicity of the Quantum Relative Entropy under Positive Maps","volume":"18","author":"Reeb","year":"2017","journal-title":"Annales Henri Poincar\u00e9"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Christandl, M., and M\u00fcller-Hermes, A. (arXiv, 2016). Relative Entropy Bounds on Quantum, Private and Repeater Capacities, arXiv.","DOI":"10.1007\/s00220-017-2885-y"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1142\/S0129055X03001576","article-title":"Monotonicity of Quantum Relative Entropy Revisited","volume":"15","author":"Petz","year":"2003","journal-title":"Rev. Math. Phys."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1093\/qmath\/39.1.97","article-title":"Sufficiency of Channels over von Neumann algebras","volume":"39","author":"Petz","year":"1988","journal-title":"Q. J. Math. Oxf."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1007\/s00220-005-1510-7","article-title":"Sufficiency in quantum statistical inference","volume":"263","author":"Petz","year":"2006","journal-title":"Commun. Math. Phys."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Harremo\u00ebs, P., and Tishby, N. (2007, January 24\u201329). The Information Bottleneck Revisited or How to Choose a Good Distortion Measure. Proceedings of the IEEE International Symposium on Information Theory, Nice, France.","DOI":"10.1109\/ISIT.2007.4557285"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"7616","DOI":"10.1109\/TIT.2014.2360184","article-title":"Information Measures: The Curious Case of the Binary Alphabet","volume":"60","author":"Jiao","year":"2014","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"085303","DOI":"10.1088\/1751-8121\/aa5661","article-title":"Preservation of a quantum R\u00e9nyi relative entropy implies existence of a recovery map","volume":"50","year":"2017","journal-title":"J. Phys. A Math. Theor."},{"key":"ref_47","unstructured":"Tishby, N., Pereira, F., and Bialek, W. (1999, January 22\u201324). The information bottleneck method. Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, Urbana, Illinois, USA."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"No, A., and Weissman, T. (2015, January 14\u201319). Universality of logarithmic loss in lossy compression. Proceedings of the 2015 IEEE International Symposium on Information Theory (ISIT), Hongkong, China.","DOI":"10.1109\/ISIT.2015.7282839"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1214\/12-AOS972","article-title":"Proper local scoring rules on discrete sample spaces","volume":"40","author":"Dawid","year":"2012","journal-title":"Ann. Stat."},{"key":"ref_50","first-page":"686","article-title":"Expected Information as Expected Utility","volume":"7","author":"Bernardo","year":"1978","journal-title":"Ann. Stat."},{"key":"ref_51","first-page":"2032","article-title":"Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems","volume":"19","year":"1991","journal-title":"Ann. Stat."},{"key":"ref_52","unstructured":"Bartlett, P., Gr\u00fcnwald, P., Harremo\u00ebs, P., Hedayati, F., and Kotlowski, W. (2013, January 12\u201314). Horizon-Independent Optimal Prediction with Log-Loss in Exponential Families. Proceedings of the Conference on Learning Theory (COLT 2013), Princeton, NJ, USA."},{"key":"ref_53","first-page":"571","article-title":"A Guide to Entropy and the Second Law of Thermodynamics","volume":"45","author":"Lieb","year":"1998","journal-title":"Not. AMS"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Alon, N., Bourgain, J., Connes, A., Gromov, M., and Milman, V. (2010). The Mathematics of the Second Law of Thermodynamics. Visions in Mathematics, Birkh\u00e4user.","DOI":"10.1007\/978-3-0346-0422-2"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Marletto, C. (arXiv, 2016). Constructor Theory of Thermodynamics, arXiv.","DOI":"10.1007\/978-3-319-43760-6_6"},{"key":"ref_56","first-page":"23","article-title":"Joint and Separate Convexity of the Bregman Distance","volume":"Volume 8","author":"Reich","year":"2001","journal-title":"Inherently Parallel Algorithms in Feasibility and Optimization and Their Applications"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1399","DOI":"10.1007\/s11005-015-0784-8","article-title":"Characterisation of Matrix Entropies","volume":"105","author":"Hansen","year":"2015","journal-title":"Lett. Math. Phys."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1007\/s11005-015-0757-y","article-title":"On the Joint Convexity of the Bregman Divergence of Matrices","volume":"105","author":"Pitrik","year":"2015","journal-title":"Lett. Math. Phys."},{"key":"ref_59","first-page":"553","article-title":"Game theoretical optimization inspired by information theory","volume":"43","year":"2008","journal-title":"J. Glob. Optim."},{"key":"ref_60","unstructured":"Tops\u00f8e, F. (2011, January 7\u201310). Cognition and Inference in an Abstract Setting. Proceedings of the Fourth Workshop on Information Theoretic Methods in Science and Engineering (WITMSE 2011), Helsinki, Finland."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"20140540","DOI":"10.1098\/rspa.2014.0540","article-title":"Constructor theory of information","volume":"471","author":"Deutch","year":"2014","journal-title":"Proc. R. Soc. A"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"4925","DOI":"10.1109\/TIT.2009.2030485","article-title":"\u03b1-Divergence Is Unique, Belonging to Both f-Divergence and Bregman Divergence Classes","volume":"55","author":"Amari","year":"2009","journal-title":"IEEE Trans. Inf. Theory"}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/19\/5\/206\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:34:31Z","timestamp":1760207671000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/19\/5\/206"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,5,3]]},"references-count":62,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2017,5]]}},"alternative-id":["e19050206"],"URL":"https:\/\/doi.org\/10.3390\/e19050206","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2017,5,3]]}}}