{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:17:32Z","timestamp":1760242652508,"version":"build-2065373602"},"reference-count":26,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2016,1,21]],"date-time":"2016-01-21T00:00:00Z","timestamp":1453334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The National Science Foundation of China","award":["61273365","11407776"],"award-info":[{"award-number":["61273365","11407776"]}]},{"name":"National High Technology Research and Development Program of China","award":["2012AA011103"],"award-info":[{"award-number":["2012AA011103"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>This paper studies contrastive divergence (CD) learning algorithm and proposes a new algorithm for training restricted Boltzmann machines (RBMs). We derive that CD is a biased estimator of the log-likelihood gradient method and make an analysis of the bias. Meanwhile, we propose a new learning algorithm called average contrastive divergence (ACD) for training RBMs. It is an improved CD algorithm, and it is different from the traditional CD algorithm. Finally, we obtain some experimental results. The results show that the new algorithm is a better approximation of the log-likelihood gradient method and outperforms the traditional CD algorithm.<\/jats:p>","DOI":"10.3390\/e18010035","type":"journal-article","created":{"date-parts":[[2016,1,22]],"date-time":"2016-01-22T11:36:13Z","timestamp":1453462573000},"page":"35","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Average Contrastive Divergence for Training Restricted Boltzmann Machines"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7150-6285","authenticated-orcid":false,"given":"Xuesi","family":"Ma","sequence":"first","affiliation":[{"name":"Center for Intelligence Science and Technology, School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China"},{"name":"School of Mathematic and Information Science, Henan Polytechnic University, Jiaozuo 454000, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaojie","family":"Wang","sequence":"additional","affiliation":[{"name":"Center for Intelligence Science and Technology, School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2016,1,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1771","DOI":"10.1162\/089976602760128018","article-title":"Training products of experts by minimizing Contrastive Divergence","volume":"14","author":"Hinton","year":"2002","journal-title":"Neural Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"428","DOI":"10.1016\/j.tics.2007.09.004","article-title":"Learning multiple layers of representation","volume":"11","author":"Hinton","year":"2007","journal-title":"Trends Cognit. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1527","DOI":"10.1162\/neco.2006.18.7.1527","article-title":"A Fast Learning Algorithm for Deep Belief Nets","volume":"18","author":"Hinton","year":"2006","journal-title":"Neural Comput."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.neucom.2014.12.020","article-title":"Deep correspondence restricted Boltzmann machine for cross-modalretrieval","volume":"154","author":"Feng","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Scho\u0151lkopf, B., Platt, J., and Hoffman, T. (2007). Advances in Neural Information Processing (NIPS19), MIT Press.","DOI":"10.7551\/mitpress\/7503.001.0001"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1601","DOI":"10.1162\/neco.2008.11-07-647","article-title":"Justifying and generalizing contrastive divergence","volume":"21","author":"Bengio","year":"2009","journal-title":"Neural Comput."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Fischer, A., and Igel, C. (2012). An Introduction to Restricted Boltzmann Machines, Springer. CIARP2012.","DOI":"10.1007\/978-3-642-33275-3_2"},{"key":"ref_8","unstructured":"Akaho, S., and Takabatake, K. (2008). Information Geometry of Contrastive Divergence, CSREA Press. ITSL2008."},{"key":"ref_9","first-page":"789","article-title":"On the convergence properties of Contrastive Divergence","volume":"9","author":"Sutskever","year":"2010","journal-title":"J. Mach. Learn. Res. Proc. Track"},{"key":"ref_10","unstructured":"Saul, L., Weiss, Y., and Bottou, L. (2005). Advances in Neural Processing Systems, NIPS17, MIT Press."},{"key":"ref_11","unstructured":"Carreira-Perpin\u00e1n, M.\u00c1., and Hinton, G.E. (2005, January 6\u20138). On contrastive divergence learning. Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics (AISTATS), The Society for Artificial Intelligence and Statistics, The Savannah Hotel, Barbados."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"350102","DOI":"10.1155\/2015\/350102","article-title":"Convergence analysis of contrastive divergence algorithm based on gradient method with errors","volume":"2015","author":"Ma","year":"2015","journal-title":"Math. Probl. Eng."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1162\/NECO_a_00085","article-title":"Bounding the bias of contrastive divergence learning","volume":"23","author":"Fischer","year":"2011","journal-title":"Neural Comput."},{"key":"ref_14","unstructured":"Cohen, W.W., McCallum, A., and Roweis, S.T. (2008, January 5\u20139). Training restricted Boltzmann machines using approximations to the likelihood gradient. Proceedings of the International Conference on Machine learning (ICML), Helsinki, Finland."},{"key":"ref_15","unstructured":"Pohoreckyj Danyluk, A., Bottou, L., and Littman, M.L. (2009, January 14\u201318). Using fast weights to improve persistent contrastive divergence. Proceedings of the International Conference on Machine Learning (ICML), Montreal, QC, Canada."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Cho, K., Raiko, T., and Ilin, A. (2010, January 18\u201323). Parallel tempering is efficient for learning restricted Boltzmann machines. Proceedings of the International Joint Conference on Neural Networks (IJCNN), Barcelona, Spain.","DOI":"10.1109\/IJCNN.2010.5596837"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1162\/NECO_a_00397","article-title":"Enhanced Gradient for Training Restricted Boltzmann Machines","volume":"25","author":"Cho","year":"2013","journal-title":"Neural Comput."},{"key":"ref_18","unstructured":"Desjardins, U., Courville, A., Bengio, Y., Vincent, P., and Delalleau, O. (2010). Tempered Markov Chain Monte Carlo for Training of Restricted Boltzmann Machines, AISTATS."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1007\/BF00341287","article-title":"Parametric inference for imperfectly observed gibbsian fields","volume":"82","author":"Younes","year":"1989","journal-title":"Probab.Theory Relat. Fields"},{"key":"ref_20","unstructured":"LeCun, Y., Cortes, C., and Burges, C. The MNIST database of handwritten digits. Available online: http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1016\/j.neucom.2014.02.024","article-title":"Improving mixing rate with tempered transition for learning restricted Boltzmann machines","volume":"139","author":"Xu","year":"2014","journal-title":"Neurcomputing"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.patcog.2013.05.025","article-title":"Training restricted Boltzmann machines: An introduction","volume":"47","author":"Fischer","year":"2014","journal-title":"Pattern Recognit."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Salakhutdinov, R., and Murray, I. (2008, January 5\u20139). On the quantitative analysis of deep belief networks. Proceedings of the International Conference on Machine Learning, Helsinki, Finland.","DOI":"10.1145\/1390156.1390266"},{"key":"ref_24","unstructured":"Salakhutdinov, R., and Larochelle, H. (2010). Efficient Learning of Deep Boltzmann Machines, AISTATS."},{"key":"ref_25","unstructured":"Salakhutdinov, R. (2010). Learning Deep Boltzmann Machines Using Adaptive MCMC, ICML2010, Omnipress."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1967","DOI":"10.1162\/NECO_a_00311","article-title":"An efficient learning procedure for deep Boltzmann machines","volume":"24","author":"Salakhutdinov","year":"2012","journal-title":"Neural Comput."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/18\/1\/35\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:18:02Z","timestamp":1760210282000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/18\/1\/35"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,1,21]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2016,1]]}},"alternative-id":["e18010035"],"URL":"https:\/\/doi.org\/10.3390\/e18010035","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2016,1,21]]}}}