{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T22:56:29Z","timestamp":1773183389591,"version":"3.50.1"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,10,21]],"date-time":"2019-10-21T00:00:00Z","timestamp":1571616000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,10,21]],"date-time":"2019-10-21T00:00:00Z","timestamp":1571616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Labex Cominlabs","award":["ANR-10-LABX-07-01"],"award-info":[{"award-number":["ANR-10-LABX-07-01"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["EURASIP J. Adv. Signal Process."],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n              <jats:p>This paper proposes a learning policy to improve the energy efficiency (EE) of heterogeneous cellular networks. The combination of active and inactive base stations (BS) that allows for maximizing EE is identified as a combinatorial learning problem and requires high computational complexity as well as a large signaling overhead. This paper aims at presenting a learning policy that dynamically switches a BS to ON or OFF status in order to follow the traffic load variation during the day. The network traffic load is represented as a Markov decision process, and we propose a modified upper confidence bound algorithm based on restless Markov multi-armed bandit framework for the BS switching operation. Moreover, to cope with initial reward loss and to speed up the convergence of the learning algorithm, the transfer learning concept is adapted to our algorithm in order to benefit from the transferred knowledge observed in historical periods from the same region. Based on our previous work, a convergence theorem is provided for the proposed policy. Extensive simulations demonstrate that the proposed algorithms follow the traffic load variation during the day and contribute to a performance jump-start in EE improvement under various practical traffic load profiles. It also demonstrates that proposed schemes can significantly reduce the total energy consumption of cellular network, e.g., up to 70% potential energy savings based on a real traffic profile.<\/jats:p>","DOI":"10.1186\/s13634-019-0637-1","type":"journal-article","created":{"date-parts":[[2019,10,21]],"date-time":"2019-10-21T12:41:35Z","timestamp":1571661695000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Transfer restless multi-armed bandit policy for energy-efficient heterogeneous cellular network"],"prefix":"10.1186","volume":"2019","author":[{"given":"Navikkumar","family":"Modi","sequence":"first","affiliation":[]},{"given":"Philippe","family":"Mary","sequence":"additional","affiliation":[]},{"given":"Christophe","family":"Moy","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,10,21]]},"reference":[{"key":"637_CR1","unstructured":"N. Modi, Machine learning and statistical decision making for green radio (2017). PhD thesis, CentraleSupelec, Rennes."},{"key":"637_CR2","doi-asserted-by":"publisher","unstructured":"M. A. Marsan, L. Chiaraviglio, D. Ciullo, M. Meo, in IEEE International Conference on Communications Workshops (ICCW). Optimal energy savings in cellular access networks, (2009), pp. 1\u20135. \n                    https:\/\/doi.org\/10.1109\/iccw.2009.5208045\n                    \n                  .","DOI":"10.1109\/iccw.2009.5208045"},{"key":"637_CR3","unstructured":"G. P. Fettweis, E. Zimmermann, in The 11th International Symposium on Wireless Personal Multimedia Communications (WPMC). ICT energy consumption-trends and challenges, (2009)."},{"issue":"8","key":"637_CR4","doi-asserted-by":"publisher","first-page":"1525","DOI":"10.1109\/JSAC.2011.110903","volume":"29","author":"K. Son","year":"2011","unstructured":"K. Son, H. Kim, Y. Yi, B. Krishnamachari, Base station operation and user association mechanisms for energy-delay tradeoffs in green cellular networks. IEEE J. Sel. Areas Commun.29(8), 1525\u20131536 (2011).","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"637_CR5","first-page":"121","volume-title":"The 17th Annual International Conference on Mobile Computing and Networking (MobiCom)","author":"C. Peng","year":"2011","unstructured":"C. Peng, S. -B. Lee, S. Lu, H. Luo, H. Li, in The 17th Annual International Conference on Mobile Computing and Networking (MobiCom). Traffic-driven power saving in operational 3G cellular networks (ACMNew York, 2011), pp. 121\u2013132."},{"key":"637_CR6","unstructured":"H. Karl, An overview of energy-efficiency techniques for mobile communication systems. 2003."},{"issue":"6","key":"637_CR7","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1109\/MCOM.2011.5783985","volume":"49","author":"E. Oh","year":"2011","unstructured":"E. Oh, B. Krishnamachari, X. Liu, Z. Niu, Toward dynamic energy-efficient operation of cellular network infrastructure. IEEE Commun. Mag.49(6), 56\u201361 (2011).","journal-title":"IEEE Commun. Mag."},{"key":"637_CR8","unstructured":"N. Modi, P. Mary, C. Moy, in 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR). A sensing policy based on confidence bounds and a restless multi-armed bandit model (San Diego, 2012), pp. 318\u2013323."},{"key":"637_CR9","doi-asserted-by":"crossref","unstructured":"C. Robert, C. Moy, C. -X. Wang, in IEEE International Conference on Communications (ICC). Reinforcement learning approaches and evaluation criteria for opportunistic spectrum access (Sydney, 2014), pp. 1508\u20131513.","DOI":"10.1109\/ICC.2014.6883535"},{"issue":"1","key":"637_CR10","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1109\/TCCN.2017.2675901","volume":"3","author":"N. Modi","year":"2017","unstructured":"N. Modi, P. Mary, C. Moy, QoS driven channel selection algorithm for cognitive radio network: Multi-user multi-armed bandit approach. IEEE Trans. Cogn. Commun. Netw.3(1), 49\u201366 (2017).","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"637_CR11","first-page":"1633","volume":"10","author":"M. E. Taylor","year":"2009","unstructured":"M. E. Taylor, P. Stone, Transfer learning for reinforcement learning domains: A survey. J. Mach. Learn. Res.10:, 1633\u20131685 (2009).","journal-title":"J. Mach. Learn. Res."},{"issue":"4","key":"637_CR12","doi-asserted-by":"publisher","first-page":"2000","DOI":"10.1109\/TWC.2014.022014.130840","volume":"13","author":"R. Li","year":"2014","unstructured":"R. Li, Z. Zhao, X. Chen, J. Palicot, H. Zhang, TACT: A transfer actor-critic learning framework for energy saving in cellular radio access networks. IEEE Trans. Wirel. Commun.13(4), 2000\u20132011 (2014).","journal-title":"IEEE Trans. Wirel. Commun."},{"issue":"5","key":"637_CR13","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1109\/MWC.2011.6056689","volume":"18","author":"Z. Niu","year":"2011","unstructured":"Z. Niu, TANGO: Traffic-aware network planning and green operation. IEEE Wirel. Commun.18(5), 25\u201329 (2011). \n                    https:\/\/doi.org\/10.1109\/MWC.2011.6056689\n                    \n                  .","journal-title":"IEEE Wirel. Commun."},{"key":"637_CR14","unstructured":"L. Chiaraviglio, D. Ciullo, M. Meo, M Ajmone Marsan, in The 11th International Symposium on Wireless Personal Multimedia Communications (WPMC). Energy-aware UMTS access networks, (2008), pp. 8\u201311."},{"issue":"11","key":"637_CR15","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1109\/MCOM.2010.5621970","volume":"48","author":"Z. Niu","year":"2010","unstructured":"Z. Niu, Y. Wu, J. Gong, Z. Yang, Cell zooming for cost-efficient green cellular networks. IEEE Commun. Mag.48(11), 74\u201379 (2010). \n                    https:\/\/doi.org\/10.1109\/MCOM.2010.5621970\n                    \n                  .","journal-title":"IEEE Commun. Mag."},{"key":"637_CR16","doi-asserted-by":"publisher","unstructured":"R. Li, Z. Zhao, Y. Wei, X. Zhou, H. Zhang, in IEEE International Conference on Communications (ICC). GM-PAB: A grid-based energy saving scheme with predicted traffic load guidance for cellular networks, (2012), pp. 1160\u20131164. \n                    https:\/\/doi.org\/10.1109\/ICC.2012.6364637\n                    \n                  .","DOI":"10.1109\/ICC.2012.6364637"},{"key":"637_CR17","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1587\/transcom.E95.B.551","volume":"95","author":"J. Gong","year":"2012","unstructured":"J. Gong, S. Zhou, Z. Niu, A dynamic programming approach for base station sleeping in cellular networks. IEICE Trans. Commun.95:, 551\u2013562 (2012). \n                    https:\/\/doi.org\/10.1587\/transcom.E95.B.551\n                    \n                  .","journal-title":"IEICE Trans. Commun."},{"key":"637_CR18","doi-asserted-by":"publisher","unstructured":"M. A. Marsan, L. Chiaraviglio, D. Ciullo, M. Meo, in IEEE International Conference on Communications Workshops (ICCW). Optimal energy savings in cellular access networks, (2009), pp. 1\u20135. \n                    https:\/\/doi.org\/10.1109\/ICCW.2009.5208045\n                    \n                  .","DOI":"10.1109\/ICCW.2009.5208045"},{"issue":"4","key":"637_CR19","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1145\/1773394.1773406","volume":"37","author":"M. A. Marsan","year":"2010","unstructured":"M. A. Marsan, M. Meo, Energy efficient management of two cellular access networks. SIGMETRICS Perform. Eval. Rev.37(4), 69\u201373 (2010). \n                    https:\/\/doi.org\/10.1145\/1773394.1773406\n                    \n                  .","journal-title":"SIGMETRICS Perform. Eval. Rev."},{"key":"637_CR20","doi-asserted-by":"crossref","unstructured":"A. S. Alam, L. S. Dooley, A. S. Poulton, in 2013 IEEE 18th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD). Traffic-and-interference aware base station switching for green cellular networks (Berlin, 2013), pp. 63\u201367.","DOI":"10.1109\/CAMAD.2013.6708090"},{"key":"637_CR21","doi-asserted-by":"crossref","unstructured":"E. Oh, B. Krishnamachari, in IEEE Global Telecommunications Conference (GLOBECOM). Energy savings through dynamic base station switching in cellular wireless access networks (Miami, 2010), pp. 1\u20135.","DOI":"10.1109\/GLOCOM.2010.5683654"},{"key":"637_CR22","doi-asserted-by":"crossref","unstructured":"R. M. Karp, in Reducibility among Combinatorial Problems, ed. by R. E. Miller, J. W. Thatcher, and J. D. Bohlinger. Complexity of Computer Computations (SpringerBoston, 1972), pp. 85\u2013103.","DOI":"10.1007\/978-1-4684-2001-2_9"},{"key":"637_CR23","volume-title":"Computers and intractability: A guide to the theory of NP-completeness","author":"M. R. Garey","year":"1979","unstructured":"M. R. Garey, D. S. Johnson, Computers and intractability: A guide to the theory of NP-completeness (W. H. Freeman & Co., New York, 1979)."},{"issue":"8","key":"637_CR24","doi-asserted-by":"publisher","first-page":"3505","DOI":"10.1109\/TCOMM.2013.061913.120743","volume":"61","author":"F. Han","year":"2013","unstructured":"F. Han, Z. Safar, K. J. R. Liu, Energy-efficient base-station cooperative operation with guaranteed QoS. IEEE Trans. Commun.61(8), 3505\u20133517 (2013). \n                    https:\/\/doi.org\/10.1109\/TCOMM.2013.061913.120743\n                    \n                  .","journal-title":"IEEE Trans. Commun."},{"issue":"5","key":"637_CR25","doi-asserted-by":"publisher","first-page":"840","DOI":"10.1109\/JSAC.2013.130503","volume":"31","author":"Y. S. Soh","year":"2013","unstructured":"Y. S. Soh, T. Q. S. Quek, M. Kountouris, H. Shin, Energy efficient heterogeneous cellular networks. IEEE J. Sel. Areas Commun.31(5), 840\u2013850 (2013).","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"637_CR26","doi-asserted-by":"crossref","unstructured":"J. Kim, H. W. Lee, S. Chong, in 13th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt). TAES: Traffic-aware energy-saving base station sleeping and clustering in cooperative networks (Mumbai, 2015), pp. 259\u2013266.","DOI":"10.1109\/WIOPT.2015.7151081"},{"issue":"1","key":"637_CR27","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1137\/S036301299731669X","volume":"38","author":"V. Konda","year":"2013","unstructured":"V. Konda, V. Borkar, Energy-efficient base-station cooperative operation with guaranteed QoS. SIAM J. Contr. Optim.38(1), 94\u2013123 (2013).","journal-title":"SIAM J. Contr. Optim."},{"key":"637_CR28","doi-asserted-by":"crossref","unstructured":"W. -T. Wong, Y. -J. Yu, A. -C. Pang, in IEEE Global Communications Conference (GLOBECOM). Decentralized energy-efficient base station operation for green cellular networks (Anaheim, 2012), pp. 5194\u20135200.","DOI":"10.1109\/GLOCOM.2012.6503945"},{"issue":"5","key":"637_CR29","doi-asserted-by":"publisher","first-page":"2126","DOI":"10.1109\/TWC.2013.032013.120494","volume":"12","author":"E. Oh","year":"2013","unstructured":"E. Oh, K. Son, B. Krishnamachari, Dynamic base station switching-on\/off strategies for green cellular networks. IEEE Trans. Wirel. Commun.12(5), 2126\u20132136 (2013).","journal-title":"IEEE Trans. Wirel. Commun."},{"issue":"262","key":"637_CR30","first-page":"10","volume":"9","author":"S. Zhou","year":"2009","unstructured":"S. Zhou, J. Gong, Z. Yang, Z. Niu, P. Yang, Green mobile access network with dynamic base station energy saving. ACM MobiCom. 9(262), 10\u201312 (2009).","journal-title":"ACM MobiCom"},{"issue":"5","key":"637_CR31","doi-asserted-by":"publisher","first-page":"851","DOI":"10.1109\/JSAC.2013.130504","volume":"31","author":"W. Guo","year":"2013","unstructured":"W. Guo, T O\u2019Farrell, Dynamic cell expansion with self-organizing cooperation. IEEE J. Sel. Areas Commun.31(5), 851\u2013860 (2013). \n                    https:\/\/doi.org\/10.1109\/JSAC.2013.130504\n                    \n                  .","journal-title":"IEEE J. Sel. Areas Commun."},{"issue":"8","key":"637_CR32","doi-asserted-by":"publisher","first-page":"5588","DOI":"10.1109\/TIT.2012.2198613","volume":"58","author":"C. Tekin","year":"2012","unstructured":"C. Tekin, M. Liu, Online learning of rested and restless bandits. IEEE Trans. Inf. Theory. 58(8), 5588\u20135611 (2012). \n                    https:\/\/doi.org\/10.1109\/TIT.2012.2198613\n                    \n                  .","journal-title":"IEEE Trans. Inf. Theory"},{"key":"637_CR33","doi-asserted-by":"crossref","unstructured":"J. Oksanen, V. Koivunen, H. V. Poor, A sensing policy based on confidence bounds and a restless multi-armed bandit model. CoRR abs\/1211.4384 (2012).","DOI":"10.1109\/ACSSC.2012.6489015"},{"issue":"5","key":"637_CR34","doi-asserted-by":"publisher","first-page":"1214","DOI":"10.1109\/TSP.2015.2391072","volume":"63","author":"J. Oksanen","year":"2015","unstructured":"J. Oksanen, V. Koivunen, An order optimal policy for exploiting idle spectrum in cognitive radio networks. IEEE Trans. Signal Process.63(5), 1214\u20131227 (2015). \n                    https:\/\/doi.org\/10.1109\/TSP.2015.2391072\n                    \n                  .","journal-title":"IEEE Trans. Signal Process."},{"key":"637_CR35","unstructured":"W. Zhang, in Proceedings of the 19th International Teletraffic Congress. Performance of real-time and data traffic in heterogeneous overlay wireless networks, (2005)."},{"issue":"11","key":"637_CR36","doi-asserted-by":"publisher","first-page":"5929","DOI":"10.1109\/TWC.2013.100213.130672","volume":"12","author":"M. F. Hossain","year":"2013","unstructured":"M. F. Hossain, K. S. Munasinghe, A. Jamalipour, Distributed inter-BS cooperation aided energy efficient load balancing for cellular networks. IEEE Trans. Wirel. Commun.12(11), 5929\u20135939 (2013).","journal-title":"IEEE Trans. Wirel. Commun."},{"issue":"1","key":"637_CR37","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1109\/TNET.2011.2157937","volume":"20","author":"H. Kim","year":"2012","unstructured":"H. Kim, G. de Veciana, X. Yang, M. Venkatachalam, Distributed \u03b1-optimal user association and cell load balancing in wireless networks. IEEE\/ACM Trans. Netw.20(1), 177\u2013190 (2012).","journal-title":"IEEE\/ACM Trans. Netw."},{"issue":"7","key":"637_CR38","doi-asserted-by":"publisher","first-page":"3566","DOI":"10.1109\/TWC.2009.071140","volume":"8","author":"K. Son","year":"2009","unstructured":"K. Son, S. Chong, G. D. Veciana, Dynamic association for load balancing and interference avoidance in multi-cell networks. IEEE Trans. Wirel. Commun.8(7), 3566\u20133576 (2009).","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"637_CR39","doi-asserted-by":"crossref","unstructured":"A. J. Fehske, F. Richter, G. P. Fettweis, in IEEE Globecom Workshops. Energy efficiency improvements through micro sites in cellular mobile radio networks (Honolulu, 2009), pp. 1\u20135.","DOI":"10.1109\/GLOCOMW.2009.5360741"},{"key":"637_CR40","doi-asserted-by":"crossref","unstructured":"A. Alam, L. Dooley, in IEEE Wireless Communications and Networking Conference. A scalable multimode base station switching model for green cellular networks (New Orleans, 2015).","DOI":"10.1109\/WCNC.2015.7127585"},{"key":"637_CR41","first-page":"1675","volume-title":"The 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton)","author":"C. Tekin","year":"2010","unstructured":"C. Tekin, M. Liu, in The 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton). Online algorithms for the multi-armed bandit problem with Markovian rewards (IEEEAllerton, 2010), pp. 1675\u20131682."},{"key":"637_CR42","doi-asserted-by":"crossref","unstructured":"C. Tekin, M. Liu, in IEEE INFOCOM. Online learning in opportunistic spectrum access: A restless bandit approach (Shanghai, 2011), pp. 2462\u20132470.","DOI":"10.1109\/INFCOM.2011.5935068"},{"issue":"3","key":"637_CR43","doi-asserted-by":"publisher","first-page":"338","DOI":"10.1109\/TAC.2005.844079","volume":"50","author":"C. -C. Wang","year":"2005","unstructured":"C. -C. Wang, S. R. Kulkarni, H. V. Poor, Bandit problems with side observations. IEEE Trans. Autom. Control.50(3), 338\u2013355 (2005). \n                    https:\/\/doi.org\/10.1109\/TAC.2005.844079\n                    \n                  .","journal-title":"IEEE Trans. Autom. Control."},{"key":"637_CR44","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511546921","volume-title":"Prediction, learning and games","author":"N. Cesa-Bianchi","year":"2006","unstructured":"N. Cesa-Bianchi, G. Lugosi, Prediction, learning and games (Cambridge University Press, New York, 2006)."},{"key":"637_CR45","doi-asserted-by":"crossref","unstructured":"R. Li, Z. Zhao, X. Chen, H. Zhang, in IEEE Global Communications Conference (GLOBECOM). Energy saving through a learning framework in greener cellular radio access networks (Anaheim, 2012), pp. 1556\u20131561.","DOI":"10.1109\/GLOCOM.2012.6503335"},{"key":"637_CR46","doi-asserted-by":"publisher","first-page":"849","DOI":"10.1214\/aoap\/1028903453","volume":"8","author":"P. Lezaud","year":"1998","unstructured":"P. Lezaud, Chernoff-type bound for finite markov chains. Ann. Appl. Probab.8:, 849\u2013867 (1998).","journal-title":"Ann. Appl. Probab."},{"issue":"11","key":"637_CR47","doi-asserted-by":"publisher","first-page":"977","DOI":"10.1109\/TAC.1987.1104485","volume":"32","author":"V. Anantharam","year":"1987","unstructured":"V. Anantharam, P. Varaiya, J. Walrand, Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-part II: Markovian rewards. IEEE Trans. Autom. Control. 32(11), 977\u2013982 (1987).","journal-title":"IEEE Trans. Autom. Control"}],"container-title":["EURASIP Journal on Advances in Signal Processing"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-019-0637-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s13634-019-0637-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13634-019-0637-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T23:16:23Z","timestamp":1603149383000},"score":1,"resource":{"primary":{"URL":"https:\/\/asp-eurasipjournals.springeropen.com\/articles\/10.1186\/s13634-019-0637-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,21]]},"references-count":47,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["637"],"URL":"https:\/\/doi.org\/10.1186\/s13634-019-0637-1","relation":{},"ISSN":["1687-6180"],"issn-type":[{"value":"1687-6180","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,21]]},"assertion":[{"value":"26 July 2018","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 August 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 October 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"46"}}