{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,22]],"date-time":"2025-11-22T11:23:39Z","timestamp":1763810619357},"reference-count":61,"publisher":"MIT Press - Journals","issue":"9","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We investigate the use of parameterized families of information-theoretic measures to generalize the loss functions of generative adversarial networks (GANs) with the objective of improving performance. A new generator loss function, least kth-order GAN (LkGAN), is introduced, generalizing the least squares GANs (LSGANs) by using a kth-order absolute error distortion measure with k\u22651 (which recovers the LSGAN loss function when k=2). It is shown that minimizing this generalized loss function under an (unconstrained) optimal discriminator is equivalent to minimizing the kth-order Pearson-Vajda divergence. Another novel GAN generator loss function is next proposed in terms of R\u00e9nyi cross-entropy functionals with order \u03b1&amp;gt;0, \u03b1\u22601. It is demonstrated that this R\u00e9nyi-centric generalized loss function, which provably reduces to the original GAN loss function as \u03b1\u21921, preserves the equilibrium point satisfied by the original GAN based on the Jensen-R\u00e9nyi divergence, a natural extension of the Jensen-Shannon divergence.<\/jats:p>\n               <jats:p>Experimental results indicate that the proposed loss functions, applied to the MNIST and CelebA data sets, under both DCGAN and StyleGAN architectures, confer performance benefits by virtue of the extra degrees of freedom provided by the parameters k and \u03b1, respectively. More specifically, experiments show improvements with regard to the quality of the generated images as measured by the Fr\u00e9chet inception distance score and training stability. While it was applied to GANs in this study, the proposed approach is generic and can be used in other applications of information theory to deep learning, for example, the issues of fairness or privacy in artificial intelligence.<\/jats:p>","DOI":"10.1162\/neco_a_01416","type":"journal-article","created":{"date-parts":[[2021,6,30]],"date-time":"2021-06-30T21:34:33Z","timestamp":1625088873000},"page":"2473-2510","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":15,"title":["Least kth-Order and R\u00e9nyi Generative Adversarial Networks"],"prefix":"10.1162","volume":"33","author":[{"given":"Himesh","family":"Bhatia","sequence":"first","affiliation":[{"name":"Department of Mathematics and Statistics, Queens University, ON K7L 3N6, Canada himesh.bhatia@queensu.ca"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William","family":"Paul","sequence":"additional","affiliation":[{"name":"Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723, U.S.A. william.paul@jhuapl.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fady","family":"Alajaji","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Statistics, Queens University, ON K7L 3N6, Canada fa@queensu.ca"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bahman","family":"Gharesifard","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Statistics, Queens University, ON K7L 3N6, Canada bahman.gharesifard@queensu.ca"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Philippe","family":"Burlina","sequence":"additional","affiliation":[{"name":"Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723, and Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, U.S.A. philippe.burlina@jhuapl.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2021,8,19]]},"reference":[{"key":"2021121723253932600_B1","author":"Abadi","year":"2015","journal-title":"TensorFlow: Large-scale machine learning on heterogeneous systems."},{"key":"2021121723253932600_B2","author":"Achille","year":"2019","journal-title":"Where is the information in a deep neural network?"},{"issue":"4","key":"2021121723253932600_B3","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1109\/TIT.2004.825040","article-title":"Csisz\u00e1r's cutoff rates for the general hypothesis testing problem","volume":"50","author":"Alajaji","year":"2004","journal-title":"IEEE Transactions on Information Theory"},{"key":"2021121723253932600_B4","first-page":"1","article-title":"Deep variational information bottleneck","author":"Alemi","year":"2017","journal-title":"Proceedings of the 5th International Conference on Learning Representations"},{"issue":"1","key":"2021121723253932600_B5","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1109\/18.481781","article-title":"An inequality on guessing and its applications to sequential decoding","volume":"42","author":"Arikan","year":"1996","journal-title":"IEEE Transactions on Information Theory"},{"key":"2021121723253932600_B6","first-page":"214","article-title":"Wasserstein generative adversarial networks","author":"Arjovsky","year":"2017","journal-title":"Proceedings of the 34th International Conference on Machine Learning"},{"issue":"3","key":"2021121723253932600_B7","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1109\/TIT.1978.1055890","article-title":"R\u00e9nyi's entropy and the probability of error","volume":"24","author":"Ben-Bassat","year":"2006","journal-title":"IEEE Transactions on Information Theory"},{"key":"2021121723253932600_B8","author":"Bhatia","year":"2020","journal-title":"R\u00e9nyi generative adversarial networks"},{"key":"2021121723253932600_B9","author":"Burlina","year":"2020","journal-title":"Addressing artificial intelligence bias in retinal disease diagnostics."},{"key":"2021121723253932600_B10","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1016\/S0019-9958(65)90332-3","article-title":"A coding theorem and R\u00e9nyi's entropy","volume":"9","author":"Campbell","year":"1965","journal-title":"Information and Control"},{"key":"2021121723253932600_B11","first-page":"661","article-title":"Symmetric variational autoencoder and connections to adversarial learning","author":"Chen","year":"2018","journal-title":"Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics"},{"key":"2021121723253932600_B12","first-page":"2172","volume-title":"Advances in neural information processing systems","author":"Chen","year":"2016"},{"key":"2021121723253932600_B13","first-page":"2494","article-title":"Cumulant generating function of codeword lengths in optimal lossless compression.","author":"Courtade","year":"2014","journal-title":"Proceedings of the IEEE International Symposium on Information Theory"},{"issue":"1","key":"2021121723253932600_B14","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/MSP.2017.2765202","article-title":"Generative adversarial networks: An overview","volume":"35","author":"Creswell","year":"2018","journal-title":"IEEE Signal Processing Magazine"},{"key":"2021121723253932600_B15","first-page":"299","article-title":"Information-type measures of difference of probability distributions and indirect observations","volume":"2","author":"Csisz\u00e1r","year":"1967","journal-title":"Studia Sci. Math. Hungarica"},{"issue":"1","key":"2021121723253932600_B16","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/18.370121","article-title":"Generalized cutoff rates and R\u00e9nyi's information measures","volume":"41","author":"Csisz\u00e1r","year":"1995","journal-title":"IEEE Transactions on Information Theory"},{"key":"2021121723253932600_B17","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-14090-7","author":"Engel","year":"2011","journal-title":"Density functional theory"},{"key":"2021121723253932600_B18","first-page":"96","article-title":"Robust generalization via \u03b1-mutual information.","author":"Esposito","year":"2020","journal-title":"Proceedings of the International Zurich Seminar on Information and Communication"},{"key":"2021121723253932600_B19","first-page":"5248","volume-title":"Advances in neural information processing systems","author":"Farnia","year":"2018"},{"key":"2021121723253932600_B20","author":"Goodfellow","year":"2016","journal-title":"NIPS 2016 tutorial: Generative adversarial networks."},{"key":"2021121723253932600_B21","first-page":"2672","volume-title":"Advances in neural information processing systems","author":"Goodfellow","year":"2014"},{"key":"2021121723253932600_B22","first-page":"3069","article-title":"Flow-GAN: Combining maximum likelihood and adversarial learning in generative models","author":"Grover","year":"2018","journal-title":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence"},{"key":"2021121723253932600_B23","article-title":"Jensen-R\u00e9nyi divergence measure: Theoretical and computational perspectives","author":"Hamza","year":"2003","journal-title":"Proceedings of the IEEE International Symposium on Information Theory"},{"key":"2021121723253932600_B24","doi-asserted-by":"crossref","first-page":"1211","DOI":"10.1109\/TSP.2003.810305","article-title":"A generalized divergence measure for robust image registration","volume":"51","author":"He","year":"2003","journal-title":"IEEE Transactions on Signal Processing"},{"key":"2021121723253932600_B25","first-page":"6626","volume-title":"Advances in neural information processing systems","author":"Heusel","year":"2017"},{"key":"2021121723253932600_B26","author":"Huang","year":"2018","journal-title":"Generative adversarial privacy"},{"key":"2021121723253932600_B27","first-page":"4401","article-title":"A style-based generator architecture for generative adversarial networks","author":"Karras","year":"2019","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"2021121723253932600_B28","first-page":"10215","volume-title":"Advances in neural information processing systems","author":"Kingma","year":"2018"},{"key":"2021121723253932600_B29","first-page":"1","article-title":"Auto-encoding variational Bayes","author":"Kingma","year":"2014","journal-title":"Proceedings of the 2nd International Conference on Learning Representations"},{"key":"2021121723253932600_B30","first-page":"1","article-title":"On Jensen-R\u00e9nyi and Jeffreys-R\u00e9nyi type f-divergences induced by convex functions.","volume":"548","author":"Kluza","year":"2019","journal-title":"Physica A: Statistical Mechanics and Its Applications"},{"key":"2021121723253932600_B31","author":"LeCun","year":"1998","journal-title":"MNIST handwritten digit database."},{"key":"2021121723253932600_B32","first-page":"1","author":"Lee","year":"2021","journal-title":"Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision"},{"key":"2021121723253932600_B33","first-page":"3302","article-title":"Adversarial learning of a sampler based on an unnormalized distribution","author":"Li","year":"2019","journal-title":"Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics 2019"},{"key":"2021121723253932600_B34","first-page":"2052","article-title":"Dropout inference in Bayesian neural networks with alpha-divergences","author":"Li","year":"2017","journal-title":"Proceedings of the 34th International Conference on Machine Learning"},{"key":"2021121723253932600_B35","first-page":"1073","volume-title":"Advances in neural information processing systems","author":"Li","year":"2016"},{"key":"2021121723253932600_B36","first-page":"1","article-title":"Deep learning face attributes in the wild","author":"Liu","year":"2015","journal-title":"Proceedings of International Conference on Computer Vision"},{"key":"2021121723253932600_B37","first-page":"1","article-title":"Least squares generative adversarial networks.","author":"Mao","year":"2017","journal-title":"Proceedings of the IEEE International Conference on Computer Vision"},{"key":"2021121723253932600_B38","author":"Mao","year":"2017","journal-title":"On the effectiveness of least squares generative adversarial networks."},{"key":"2021121723253932600_B39","first-page":"3481","author":"Mescheder","year":"2018","journal-title":"Proceedings of the 35th International Conference on Machine Learning"},{"key":"2021121723253932600_B40","first-page":"247","article-title":"Divergence based learning vector quantization","author":"Mwebaze","year":"2010","journal-title":"Proceedings of the 18th European Symposium on Artificial Neural Networks"},{"key":"2021121723253932600_B41","author":"Nielsen","year":"2019","journal-title":"On a generalization of the Jensen-Shannon divergence."},{"key":"2021121723253932600_B42","first-page":"10","article-title":"On the chi square and higher-order chi distances for approximating f-divergences.","author":"Nielsen","year":"2013","journal-title":"IEEE Signal Processing Letters"},{"key":"2021121723253932600_B43","first-page":"271","volume-title":"Advances in neural information processing systems","author":"Nowozin","year":"2016"},{"key":"2021121723253932600_B44","author":"Oord","year":"2016","journal-title":"Wavenet: A generative model for raw audio."},{"key":"2021121723253932600_B45","author":"Pantazis","year":"2020","journal-title":"Cumulant GAN"},{"issue":"3","key":"2021121723253932600_B46","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1162\/neco_a_01359","article-title":"Unsupervised discovery, control, and disentanglement of semantic attributes with applications to anomaly detection","volume":"33","author":"Paul","year":"2021","journal-title":"Neural Computation"},{"key":"2021121723253932600_B47","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4419-1570-2","author":"Principe","year":"2010","journal-title":"Information theoretic learning: R\u00e9nyi's entropy and kernel perspectives"},{"key":"2021121723253932600_B48","first-page":"613","article-title":"R\u00e9nyi entropy rate for discrete Markov sources.","author":"Rached","year":"1999","journal-title":"Proceedings of the 33rd Conference on Information Sciences and Systems"},{"key":"2021121723253932600_B49","first-page":"97","article-title":"Unsupervised representation learning with deep convolutional generative adversarial networks","author":"Radford","year":"2017","journal-title":"Proceedings of the 9th International Conference on Image and Graphics"},{"key":"2021121723253932600_B50","first-page":"547","article-title":"On measures of entropy and information.","author":"R\u00e9nyi","year":"1961","journal-title":"Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability"},{"issue":"1","key":"2021121723253932600_B51","doi-asserted-by":"crossref","DOI":"10.1007\/s42979-020-00403-9","article-title":"RGAN: R\u00e9nyi generative adversarial network","volume":"2","author":"Sarraf","year":"2021","journal-title":"SN Computer Science"},{"key":"2021121723253932600_B52","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3390\/e20050383","article-title":"On f-divergences: Integral representations, local behavior, and inequalities","volume":"20","author":"Sason","year":"2018","journal-title":"Entropy"},{"key":"2021121723253932600_B53","first-page":"1","article-title":"Deep learning and the information bottleneck principle","author":"Tishby","year":"2015","journal-title":"Proceedings of the 2015 IEEE Information Theory Workshop"},{"key":"2021121723253932600_B54","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3390\/e21010046","article-title":"The case for shifting the R\u00e9nyi entropy.","volume":"21","author":"Valverde-Albacete","year":"2019","journal-title":"Entropy"},{"issue":"7","key":"2021121723253932600_B55","doi-asserted-by":"crossref","first-page":"3797","DOI":"10.1109\/TIT.2014.2320500","article-title":"R\u00e9nyi divergence and Kullback-Leibler divergence","volume":"60","author":"van Erwen","year":"2014","journal-title":"IEEE Transactions on Information Theory"},{"key":"2021121723253932600_B56","first-page":"1","author":"Verd\u00fa","year":"2015","journal-title":"Proceedings of the 2015 IEEE Information Theory and Applications Workshop"},{"key":"2021121723253932600_B57","author":"Wang","year":"2020","journal-title":"Generative adversarial networks in computer vision: A survey and taxonomy."},{"key":"2021121723253932600_B58","author":"Wiatrak","year":"2020","journal-title":"Stabilizing generative adversarial network training: A survey."},{"key":"2021121723253932600_B59","author":"Wickstrom","year":"2019","journal-title":"Information plane analysis of deep neural networks via matrix-based R\u00e9nyi's entropy and tensor kernels."},{"key":"2021121723253932600_B60","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3390\/e22020151","article-title":"On the information bottleneck problems: Models, connections, applications and information theoretic views","volume":"22","author":"Zaidi","year":"2020","journal-title":"Entropy"},{"key":"2021121723253932600_B61","doi-asserted-by":"crossref","first-page":"6901","DOI":"10.1609\/aaai.v34i04.6172","article-title":"Bridging maximum likelihood and adversarial learning via alpha-divergence","author":"Zhao","year":"2020","journal-title":"Proceedings of the 34th AAAI Conference on Artificial Intelligence"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/33\/9\/2473\/1978131\/neco_a_01416.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/33\/9\/2473\/1978131\/neco_a_01416.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,17]],"date-time":"2021-12-17T23:26:54Z","timestamp":1639783614000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/33\/9\/2473\/102618\/Least-kth-Order-and-Renyi-Generative-Adversarial"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,19]]},"references-count":61,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,8,19]]},"published-print":{"date-parts":[[2021,8,19]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01416","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,9]]},"published":{"date-parts":[[2021,8,19]]}}}