{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T22:23:34Z","timestamp":1767651814465,"version":"3.41.0"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2012,7,1]],"date-time":"2012-07-01T00:00:00Z","timestamp":1341100800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2012,7]]},"abstract":"<jats:p>We present an extension to Jaynes\u2019 maximum entropy principle that incorporates latent variables. The principle of<jats:italic>latent maximum entropy<\/jats:italic>we propose is different from both Jaynes\u2019 maximum entropy principle and maximum likelihood estimation, but can yield better estimates in the presence of hidden variables and limited training data. We first show that solving for a latent maximum entropy model poses a hard nonlinear constrained optimization problem in general. However, we then show that feasible solutions to this problem can be obtained efficiently for the special case of log-linear models---which forms the basis for an efficient approximation to the latent maximum entropy principle. We derive an algorithm that combines expectation-maximization with iterative scaling to produce feasible log-linear solutions. This algorithm can be interpreted as an alternating minimization algorithm in the information divergence, and reveals an intimate connection between the latent maximum entropy and maximum likelihood principles. To select a final model, we generate a series of feasible candidates, calculate the entropy of each, and choose the model that attains the highest entropy. Our experimental results show that estimation based on the latent maximum entropy principle generally gives better results than maximum likelihood when estimating latent variable models on small observed data samples.<\/jats:p>","DOI":"10.1145\/2297456.2297460","type":"journal-article","created":{"date-parts":[[2012,8,14]],"date-time":"2012-08-14T12:18:55Z","timestamp":1344946735000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["The Latent Maximum Entropy Principle"],"prefix":"10.1145","volume":"6","author":[{"given":"Shaojun","family":"Wang","sequence":"first","affiliation":[{"name":"Wright State University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dale","family":"Schuurmans","sequence":"additional","affiliation":[{"name":"University of Alberta"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yunxin","family":"Zhao","sequence":"additional","affiliation":[{"name":"University of Missouri at Columbia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2012,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog0901_7"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(95)00003-8"},{"key":"e_1_2_1_3_1","unstructured":"Amari S. and Nagaoka H. 2000. Methods of Information Geometry. American Mathematical Society. Amari S. and Nagaoka H. 2000. Methods of Information Geometry . American Mathematical Society."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/234285.234289"},{"key":"e_1_2_1_5_1","volume-title":"Nonlinear Programming","author":"Bertsekas D.","unstructured":"Bertsekas , D. 1999. Nonlinear Programming 2 nd Ed. Athena Scientific . Bertsekas, D. 1999. Nonlinear Programming 2nd Ed. Athena Scientific.","edition":"2"},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","first-page":"601","DOI":"10.7551\/mitpress\/1120.003.0082","article-title":"Latent Dirichlet allocation","volume":"14","author":"Blei D.","year":"2002","unstructured":"Blei , D. , Ng , A. , and Jordan , M. 2002 . Latent Dirichlet allocation . In Advances Neural Information Processing Systems 14 , 601 -- 608 . Blei, D., Ng, A., and Jordan, M. 2002. Latent Dirichlet allocation. In Advances Neural Information Processing Systems 14, 601--608.","journal-title":"Advances Neural Information Processing Systems"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/72.143375"},{"key":"e_1_2_1_8_1","volume-title":"Parallel Optimization: Theory, Algorithms, and Applications","author":"Censor Y.","year":"1997","unstructured":"Censor , Y. and Zenios , S . 1997 . Parallel Optimization: Theory, Algorithms, and Applications . Oxford University Press . Censor, Y. and Zenios, S. 1997. Parallel Optimization: Theory, Algorithms, and Applications. Oxford University Press."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1013912006537"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Cover T. and Thomas J. 1991. Elements of Information Theory. Wiley. Cover T. and Thomas J. 1991. Elements of Information Theory . Wiley.","DOI":"10.1002\/0471200611"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1214\/aop\/1176996454"},{"key":"e_1_2_1_12_1","first-page":"205","article-title":"Information geometry and alternating minimization procedures","volume":"1","author":"Csiszar I.","year":"1984","unstructured":"Csiszar , I. and Tusnady , G. 1984 . Information geometry and alternating minimization procedures . Statistics and Decisions. Supplement Issue 1 , 205 -- 237 . Csiszar, I. and Tusnady, G. 1984. Information geometry and alternating minimization procedures. Statistics and Decisions. Supplement Issue 1, 205--237.","journal-title":"Statistics and Decisions. Supplement Issue"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177692379"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.588021"},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum likelihood estimation from incomplete data via the EM algorithm","volume":"39","author":"Dempster A.","year":"1977","unstructured":"Dempster , A. , Laird , N. , and Rubin , D. 1977 . Maximum likelihood estimation from incomplete data via the EM algorithm . J. Royal Stat. Soc. Series B , 39 , 1 -- 38 . Dempster, A., Laird, N., and Rubin, D. 1977. Maximum likelihood estimation from incomplete data via the EM algorithm. J. Royal Stat. Soc. Series B, 39, 1--38.","journal-title":"J. Royal Stat. Soc. Series B"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073085"},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"Fang S. Rajasekera J. and Tsao H. 1997. Entropy Optimization and Mathematical Programming Kluwer. Fang S. Rajasekera J. and Tsao H. 1997. Entropy Optimization and Mathematical Programming Kluwer.","DOI":"10.1007\/978-1-4615-6131-6"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-1809.1936.tb02137.x"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859918"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/89.279278"},{"key":"e_1_2_1_21_1","unstructured":"Golan A. Miller D. and Judge G. 1996. Maximum Entropy Econometrics: Robust Estimation with Limited Data. Wiley. Golan A. Miller D. and Judge G. 1996. Maximum Entropy Econometrics: Robust Estimation with Limited Data. Wiley."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073086"},{"key":"e_1_2_1_23_1","unstructured":"Graca J. Ganchev K. and Taskar B. 2007. Expectation maximization and posterior constraints. In Advances in Neural Information Processing Systems (NIPS). Graca J. Ganchev K. and Taskar B. 2007. Expectation maximization and posterior constraints. In Advances in Neural Information Processing Systems (NIPS) ."},{"volume-title":"Loglinear Models with Latent Variables","author":"Hagenaars J.","key":"e_1_2_1_24_1","unstructured":"Hagenaars , J. 1993. Loglinear Models with Latent Variables . Sage Publications . Hagenaars, J. 1993. Loglinear Models with Latent Variables. Sage Publications."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007617005950"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1756033"},{"key":"e_1_2_1_27_1","unstructured":"Jaakkola T. Meila M. and Jebara T. 1999. Maximum entropy discrimination. In Advances in Neural Information Processing Systems. 470--476. Jaakkola T. Meila M. and Jebara T. 1999. Maximum entropy discrimination. In Advances in Neural Information Processing Systems . 470--476."},{"volume-title":"Papers on Probability, Statistics, and Statistical Physics","author":"Jaynes E.","key":"e_1_2_1_28_1","unstructured":"Jaynes , E. 1983. Papers on Probability, Statistics, and Statistical Physics . R. Rosenkrantz Ed., D. Reidel Publishing . Jaynes, E. 1983. Papers on Probability, Statistics, and Statistical Physics. R. Rosenkrantz Ed., D. Reidel Publishing."},{"volume-title":"Statistical Methods for Speech Recognition","author":"Jelinek F.","key":"e_1_2_1_30_1","unstructured":"Jelinek , F. 1998. Statistical Methods for Speech Recognition . MIT Press . Jelinek, F. 1998. Statistical Methods for Speech Recognition. MIT Press."},{"volume-title":"Proceedings of the International Conference on Machine Learning (ICML). 282--289","author":"Lafferty J.","key":"e_1_2_1_31_1","unstructured":"Lafferty , J. , McCallum , A. and Pereira , F . 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data , In Proceedings of the International Conference on Machine Learning (ICML). 282--289 . Lafferty, J., McCallum, A. and Pereira, F. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data, In Proceedings of the International Conference on Machine Learning (ICML). 282--289."},{"volume-title":"Proceedings of the 18th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). II, 45--48","author":"Lau R.","key":"e_1_2_1_32_1","unstructured":"Lau , R. , Rosenfeld , R. , and Roukos , S . 1993. Trigger-based language models: A maximum entropy approach . In Proceedings of the 18th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). II, 45--48 . Lau, R., Rosenfeld, R., and Roukos, S. 1993. Trigger-based language models: A maximum entropy approach. In Proceedings of the 18th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). II, 45--48."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/0167-9473(93)E0056-A"},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Lauritzen S. 1996. Graphical Models. Oxford University Press. Lauritzen S. 1996. Graphical Models . Oxford University Press.","DOI":"10.1093\/oso\/9780198522195.001.0001"},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Little R. and Rubin D. 2002. Statistical Analysis with Missing Data 2nd Ed. Wiley-Interscience. Little R. and Rubin D. 2002. Statistical Analysis with Missing Data 2nd Ed. Wiley-Interscience.","DOI":"10.1002\/9781119013563"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324900000218"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.3115\/1118853.1118871"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/80.2.267"},{"key":"e_1_2_1_39_1","unstructured":"Minka T. 2003. A comparison of numerical 1169 optimizers for logistic regression. Tech. rep. http:\/\/research.microsoft.com\/en-us\/um\/people\/minka\/papers\/logreg\/minka-logreg.pdf. Minka T. 2003. A comparison of numerical 1169 optimizers for logistic regression. Tech. rep. http:\/\/research.microsoft.com\/en-us\/um\/people\/minka\/papers\/logreg\/minka-logreg.pdf."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.3115\/1075218.1075279"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1006\/csla.1996.0011"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2003.810642"},{"volume-title":"Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding.","author":"Wang S.","key":"e_1_2_1_44_1","unstructured":"Wang , S. , Rosenfeld , R. , and Zhao , Y . 2001. Latent maximum entropy principle for statistical language modeling . In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding. Wang, S., Rosenfeld, R., and Zhao, Y. 2001. Latent maximum entropy principle for statistical language modeling. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding."},{"volume-title":"Proceedings of the 20th International Conference on Machine Learning (ICML). 784--791","author":"Wang S.","key":"e_1_2_1_45_1","unstructured":"Wang , S. , Schuurmans , D. , Peng , F. , and Zhao , Y . 2003a. Learning mixture models with the latent maximum entropy principle . In Proceedings of the 20th International Conference on Machine Learning (ICML). 784--791 . Wang, S., Schuurmans, D., Peng, F., and Zhao, Y. 2003a. Learning mixture models with the latent maximum entropy principle. In Proceedings of the 20th International Conference on Machine Learning (ICML). 784--791."},{"volume-title":"Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence (UAI). 567--574","author":"Wang S.","key":"e_1_2_1_46_1","unstructured":"Wang , S. , Schuurmans , D. , Peng , F. , and Zhao , Y . 2003b. Boltzmann machine learning with the latent maximum entropy principle . In Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence (UAI). 567--574 . Wang, S., Schuurmans, D., Peng, F., and Zhao, Y. 2003b. Boltzmann machine learning with the latent maximum entropy principle. In Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence (UAI). 567--574."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-005-0928-7"},{"key":"e_1_2_1_48_1","unstructured":"Wang S. Greiner R. and Wang S. 2009. Consistency and generalization bounds for maximum entropy density. Tech. rep. Department of Computer Science Wright State University. Wang S. Greiner R. and Wang S. 2009. Consistency and generalization bounds for maximum entropy density. Tech. rep. Department of Computer Science Wright State University."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1176346060"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2005.850085"},{"key":"e_1_2_1_51_1","unstructured":"Zhu J. Xing E. and Zhang B. 2008. Partially observed maximum entropy discrimination Markov networks. In Advances in Neural Information Processing Systems (NIPS). 1977--1984. Zhu J. Xing E. and Zhang B. 2008. Partially observed maximum entropy discrimination Markov networks. In Advances in Neural Information Processing Systems (NIPS) . 1977--1984."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.5555\/1577069.1755871"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2297456.2297460","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2297456.2297460","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:15:09Z","timestamp":1750281309000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2297456.2297460"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,7]]},"references-count":50,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2012,7]]}},"alternative-id":["10.1145\/2297456.2297460"],"URL":"https:\/\/doi.org\/10.1145\/2297456.2297460","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"type":"print","value":"1556-4681"},{"type":"electronic","value":"1556-472X"}],"subject":[],"published":{"date-parts":[[2012,7]]},"assertion":[{"value":"2011-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-07-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}