{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T19:29:20Z","timestamp":1775849360434,"version":"3.50.1"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,4,28]],"date-time":"2024-04-28T00:00:00Z","timestamp":1714262400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,28]],"date-time":"2024-04-28T00:00:00Z","timestamp":1714262400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002386","name":"Cairo University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002386","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Big Data"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Using recent information and communication technologies for monitoring and management initiates a revolution in the smart grid. These technologies generate massive data that can only be processed using big data tools. This paper emphasizes the role of big data in resolving load forecasting, renewable energy sources integration, and demand response as significant aspects of smart grids. Meters data from the Low Carbon London Project is investigated as a case study. Because of the immense stream of meters' readings and exogenous data added to load forecasting models, addressing the problem is in the context of big data. Descriptive analytics are developed using Spark SQL to get insights regarding household energy consumption. Spark MLlib is utilized for predictive analytics by building scalable machine learning models accommodating meters' data streams. Multivariate polynomial regression and decision tree models are preferred here based on the big data point of view and the literature that ensures they are accurate and interpretable. The results confirmed the descriptive analytics and data visualization capabilities to provide valuable insights, guide the feature selection process, and enhance load forecasting models' accuracy. Accordingly, proper evaluation of demand response programs and integration of renewable energy resources is accomplished using achieved load forecasting results.<\/jats:p>","DOI":"10.1186\/s40537-024-00909-6","type":"journal-article","created":{"date-parts":[[2024,4,28]],"date-time":"2024-04-28T16:01:14Z","timestamp":1714320074000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Big data resolving using Apache Spark for load forecasting and demand response in smart grid: a case study of Low Carbon London Project"],"prefix":"10.1186","volume":"11","author":[{"given":"Hussien","family":"Ali El-Sayed Ali","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8943-6614","authenticated-orcid":false,"given":"M. H.","family":"Alham","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0177-5089","authenticated-orcid":false,"given":"Doaa Khalil","family":"Ibrahim","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,4,28]]},"reference":[{"key":"909_CR1","doi-asserted-by":"publisher","first-page":"3289","DOI":"10.3390\/s20113289","volume":"11","author":"JS Guerrero-Prado","year":"2020","unstructured":"Guerrero-Prado JS, Alfonso-Morales W, Caicedo-Bravo E, Zayas-P\u00e9rez B, Espinosa-Reza A. The power of big data and data analytics for AMI data: a case study. Sensors. 2020;11:3289.","journal-title":"Sensors"},{"key":"909_CR2","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/j.rser.2015.11.050","volume":"56","author":"K Zhou","year":"2016","unstructured":"Zhou K, Fu C, Yang S. Big data driven smart energy management: From big data to big insights. Renew Sustain Energy Rev. 2016;56:215\u201325.","journal-title":"Renew Sustain Energy Rev"},{"issue":"3","key":"909_CR3","doi-asserted-by":"publisher","first-page":"1404","DOI":"10.3390\/en16031404","volume":"16","author":"AY Abdelaziz","year":"2023","unstructured":"Abdelaziz AY, Biswal M, Dewangan F. Load forecasting models in smart grid using smart meter information: a review. Energies. 2023;16(3):1404.","journal-title":"Energies"},{"key":"909_CR4","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1016\/j.compind.2018.12.010","volume":"105","author":"T Wilcox","year":"2019","unstructured":"Wilcox T, Jin N, Flach P, Thumim J. A big data platform for smart meter data analytics. Comput Ind. 2019;105:250\u20139.","journal-title":"Comput Ind"},{"key":"909_CR5","doi-asserted-by":"publisher","first-page":"4266","DOI":"10.3390\/en13164266","volume":"13","author":"A Mokhade","year":"2020","unstructured":"Mokhade A, Fund N, Bokde ND, Shewale A. An overview of demand response in smart grid and optimization techniques for efficient residential appliance scheduling problem. Energies. 2020;13:4266.","journal-title":"Energies"},{"key":"909_CR6","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1186\/s40537-018-0119-6","volume":"5","author":"A Bartschat","year":"2018","unstructured":"Bartschat A, Ludwig N, Braun E, Waczowicz S, Renkamp N, Peter N, D\u00fcpmeier CD, Mikut R, Hagenmeyer VH, Ordiano J\u00c1G. Concept and benchmark results for Big Data energy forecasting based on Apache Spark. J Big Data. 2018;5:11.","journal-title":"J Big Data"},{"key":"909_CR7","doi-asserted-by":"publisher","first-page":"810","DOI":"10.1016\/j.rser.2015.12.001","volume":"56","author":"K Zhou","year":"2016","unstructured":"Zhou K, Yang S. Understanding household energy consumption behavior: the contribution of energy big data analytics. Renew Sustain Energy Rev. 2016;56:810\u20139.","journal-title":"Renew Sustain Energy Rev"},{"key":"909_CR8","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1186\/s40537-017-0070-y","volume":"4","author":"A El Hannani","year":"2017","unstructured":"El Hannani A, Aqqal A, Haidine A, Dahbi A, Daki H. Big data management in smart grid: concepts, requirements, and implementation. J Big Data. 2017;4:13.","journal-title":"J Big Data"},{"key":"909_CR9","doi-asserted-by":"publisher","first-page":"59564","DOI":"10.1109\/ACCESS.2020.3041178","volume":"9","author":"D Syed","year":"2020","unstructured":"Syed D, Zainab A, Ghrayeb A, Refaat SS, Abu-Rub H, Bouhali O. Smart grid big data analytics: survey of technologies, techniques, and applications. IEEE Access. 2020;9:59564\u201385.","journal-title":"IEEE Access"},{"key":"909_CR10","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1016\/j.egypro.2016.12.128","volume":"107","author":"M Martinez-Pabon","year":"2017","unstructured":"Martinez-Pabon M, Eveleigh T, Tanju B. Smart meter data analytics for optimal customer selection in demand response programs. Energy Procedia. 2017;107:49\u201359.","journal-title":"Energy Procedia"},{"key":"909_CR11","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1186\/s40537-019-0270-8","volume":"6","author":"VT Vakili","year":"2019","unstructured":"Vakili VT, Bahrak B, Ansari MH. Evaluation of big data frameworks for analysis of smart grids. J Big Data. 2019;6:109.","journal-title":"J Big Data"},{"key":"909_CR12","doi-asserted-by":"publisher","first-page":"100945","DOI":"10.1016\/j.aei.2019.100945","volume":"42","author":"K Moharm","year":"2019","unstructured":"Moharm K. State of the art in big data applications in microgrid: a review. Adv Eng Inf. 2019;42:100945.","journal-title":"Adv Eng Inf"},{"key":"909_CR13","doi-asserted-by":"crossref","unstructured":"R. J. Bessa, Chapter 10\u2014future trends for big data application in power systems, In: Y. Z. Reza Arghandeh, Eds., Big data application in power systems, 2018, pp. 223\u2013242.","DOI":"10.1016\/B978-0-12-811968-6.00010-3"},{"key":"909_CR14","doi-asserted-by":"crossref","unstructured":"Kwac J, Rajagopal R, Demand response targeting using big data analytics. In: IEEE International Conference on Big Data, Silicon Valley, CA, USA, 2013.","DOI":"10.1109\/BigData.2013.6691643"},{"key":"909_CR15","doi-asserted-by":"crossref","unstructured":"Vajjala VAH, A novel solution to use Big Data technologies and improve demand response program in aggregated residential houses, In: IEEE Conference on Technologies for Sustainability (SusTech), Phoenix, AZ, 2016.","DOI":"10.1109\/SusTech.2016.7897176"},{"key":"909_CR16","doi-asserted-by":"publisher","first-page":"4158","DOI":"10.1109\/JSYST.2019.2931879","volume":"13","author":"M Ghorbanian","year":"2019","unstructured":"Ghorbanian M, Dolatabadi SH, Siano P. Big data issues in smart grids: a survey. IEEE Syst J. 2019;13:4158\u201368.","journal-title":"IEEE Syst J"},{"key":"909_CR17","doi-asserted-by":"crossref","unstructured":"Safhi HM, Frikh B, Ouhbi B. Energy load forecasting in big data context, In: 5th International Conference on Renewable Energies for Developing Countries (REDEC), 2020, Ben Guerir &Marrakech, Morocco.","DOI":"10.1109\/REDEC49234.2020.9163901"},{"key":"909_CR18","doi-asserted-by":"crossref","unstructured":"Saber AY, Alam AKMR. Short-term load forecasting using multiple linear regression for big data. In: IEEE Symposium Series on Computational Intelligence, Honolulu, Hawaii, USA, pp. 1-6, 2017,","DOI":"10.1109\/SSCI.2017.8285261"},{"key":"909_CR19","doi-asserted-by":"publisher","first-page":"10163","DOI":"10.1007\/s10586-017-1198-4","volume":"22","author":"J Lei","year":"2019","unstructured":"Lei J, Jin T, Hao J, Li F. Short-term load forecasting with clustering\u2013regression model in distributed cluster. Clust Comput. 2019;22:10163\u201373.","journal-title":"Clust Comput"},{"key":"909_CR20","doi-asserted-by":"publisher","first-page":"134911","DOI":"10.1109\/ACCESS.2020.3010702","volume":"8","author":"AA Mamun","year":"2020","unstructured":"Mamun AA, Sohel M, Mohammad N, Sunny MSH, Dipta DR. A comprehensive review of the load forecasting techniques using single and hybrid predictive model. IEEE Access. 2020;8:134911\u201339.","journal-title":"IEEE Access"},{"key":"909_CR21","doi-asserted-by":"publisher","first-page":"177874","DOI":"10.1109\/ACCESS.2019.2958383","volume":"7","author":"S-V Oprea","year":"2019","unstructured":"Oprea S-V, B\u00e2ra A. Machine learning algorithms for short-term load forecast in residential buildings using smart meters, sensors and big data solutions. IEEE Access. 2019;7:177874\u201389.","journal-title":"IEEE Access"},{"key":"909_CR22","doi-asserted-by":"crossref","unstructured":"Syed D, Refaat SS, Abu-Rub H. Performance evaluation of distributed machine learning for load forecasting in smart grids. In: 2020 Cybernetics & Informatics (K&I), Velke Karlovice, Czech Republic, 2020,","DOI":"10.1109\/KI48306.2020.9039797"},{"key":"909_CR23","volume-title":"Residential consumer responsiveness to time-varying pricing","author":"J Schofield","year":"2014","unstructured":"Schofield J, Carmichael R, Tindemans S, Woolf M, Bilton M, Strbac G. Residential consumer responsiveness to time-varying pricing. London: Imperial College London; 2014."},{"key":"909_CR24","unstructured":"UK Power Networks, Low Carbon London. UK Power Networks, 2015. [Online]. Available: https:\/\/libguides.sccsc.edu\/findcitationinfo\/websites. Accessed Jan 2023."},{"key":"909_CR25","doi-asserted-by":"publisher","first-page":"103484","DOI":"10.1016\/j.scs.2021.103484","volume":"76","author":"R Chen","year":"2022","unstructured":"Chen R, Lai CS, Zhong C, Pan K, Ng WW, Li Z. MultiCycleNet: multiple cycles self-boosted neural network for short-term electric household load forecasting. Sustain Cities Soc. 2022;76:103484.","journal-title":"Sustain Cities Soc"},{"key":"909_CR26","unstructured":"A. C. M\u00fcller and S. Guido, Introduction to machine learning with Python: a guide for data scientists, O'Reilly Media, Inc., 2016."},{"issue":"3","key":"909_CR27","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.bdr.2015.04.001","volume":"2","author":"OY Al-Jarrah","year":"2015","unstructured":"Al-Jarrah OY, Yoo PD, Muhaidat S, Karagiannidis GK, Taha K. Efficient machine learning for big data: a review. Big Data Res. 2015;2(3):87\u201393.","journal-title":"Big Data Res"},{"key":"909_CR28","volume-title":"The elements of statistical learning: data mining, inference, and prediction","author":"JH Friedman","year":"2009","unstructured":"Friedman JH, Tibshirani R, Hastie T. The elements of statistical learning: data mining, inference, and prediction. New York: Springer; 2009."},{"key":"909_CR29","unstructured":"Apache Spark Foundation, \u201cMLlib - Decision Tree,\u201d [Online]. https:\/\/spark.apache.org\/docs\/1.1.0\/mllib-decision-tree.html. Accessed Jan 2023."},{"key":"909_CR30","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/s11750-021-00594-1","volume":"29","author":"CMADRM EmilioCarrizosa","year":"2021","unstructured":"EmilioCarrizosa CMADRM. Mathematical optimization in classification and regression trees. TOP. 2021;29:5\u201333.","journal-title":"TOP"},{"key":"909_CR31","unstructured":"Apache Spark Foundation, \u201cExtracting, transforming and selecting features,\u201d [Online]. https:\/\/spark.apache.org\/docs\/latest\/ml-features. Accessed Jan 2023."},{"key":"909_CR32","unstructured":"Apache Spark Foundation, \u201cML Tuning: model selection and hyperparameter tuning,\u201d [Online]. https:\/\/spark.apache.org\/docs\/latest\/ml-tuning.html. Accessed Jan 2023."}],"container-title":["Journal of Big Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-024-00909-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s40537-024-00909-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-024-00909-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,28]],"date-time":"2024-04-28T16:01:37Z","timestamp":1714320097000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-024-00909-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,28]]},"references-count":32,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["909"],"URL":"https:\/\/doi.org\/10.1186\/s40537-024-00909-6","relation":{},"ISSN":["2196-1115"],"issn-type":[{"value":"2196-1115","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,28]]},"assertion":[{"value":"26 August 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 April 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"59"}}