{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T11:03:10Z","timestamp":1781089390491,"version":"3.54.1"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,12,1]],"date-time":"2019-12-01T00:00:00Z","timestamp":1575158400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,12,3]],"date-time":"2019-12-03T00:00:00Z","timestamp":1575331200000},"content-version":"vor","delay-in-days":2,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Big Data"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>With the rapid development of smart grids and increasing data collected in these networks, analyzing this massive data for applications such as marketing, cyber-security, and performance analysis, has gained popularity. This paper focuses on analysis and performance evaluation of big data frameworks that are proposed for handling smart grid data. Since obtaining large amounts of smart grid data is difficult due to privacy concerns, we propose and implement a large scale smart grid data generator to produce massive data under conditions similar to those in real smart grids. We use four open source big data frameworks namely Hadoop-Hbase, Cassandra, Elasticsearch, and MongoDB, in our implementation. Finally, we evaluate the performance of different frameworks on smart grid big data and present a performance benchmark that includes common data analysis techniques on smart grid data.<\/jats:p>","DOI":"10.1186\/s40537-019-0270-8","type":"journal-article","created":{"date-parts":[[2019,12,3]],"date-time":"2019-12-03T17:03:27Z","timestamp":1575392607000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Evaluation of big data frameworks for analysis of smart grids"],"prefix":"10.1186","volume":"6","author":[{"given":"Mohammad Hasan","family":"Ansari","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vahid","family":"Tabatab Vakili","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Behnam","family":"Bahrak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,12,3]]},"reference":[{"issue":"5","key":"270_CR1","doi-asserted-by":"publisher","first-page":"860","DOI":"10.1007\/s40565-018-0432-2","volume":"6","author":"MH Ansari","year":"2018","unstructured":"Ansari MH, Vakili VT, Bahrak B, Tavassoli P. Graph theoretical defense mechanisms against false data injection attacks in smart grids. J Mod Power Syst Clean Energy. 2018;6(5):860\u201371.","journal-title":"J Mod Power Syst Clean Energy"},{"key":"270_CR2","doi-asserted-by":"publisher","unstructured":"Ye F, Qian Y, Hu RQ. Big data analytics and cloud computing in the smart grid. Smart Grid Commun Infrastruct. 2018;171\u2013185. https:\/\/doi.org\/10.1002\/9781119240136.ch8. http:\/\/doi.wiley.com\/10.1002\/9781119240136.ch8.","DOI":"10.1002\/9781119240136.ch8"},{"key":"270_CR3","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/j.rser.2015.11.050","volume":"56","author":"K Zhou","year":"2016","unstructured":"Zhou K, Fu C, Yang S. Big data driven smart energy management: from big data to big insights. Renew Sustain Energy Rev. 2016;56:215\u201325. https:\/\/doi.org\/10.1016\/j.rser.2015.11.050.","journal-title":"Renew Sustain Energy Rev"},{"issue":"99","key":"270_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TSG.2016.2563461","volume":"PP","author":"A Vasilakos","year":"2016","unstructured":"Vasilakos A, Hu J. Energy big data analytics and security: challenges and opportunities. IEEE Trans Smart Grid. 2016;PP(99):1. https:\/\/doi.org\/10.1109\/TSG.2016.2563461.","journal-title":"IEEE Trans Smart Grid"},{"key":"270_CR5","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1016\/j.epsr.2017.06.006","volume":"151","author":"AA Munshi","year":"2017","unstructured":"Munshi AA, Mohamed YA. Big data framework for analytics in smart grids. Electric Power Syst Res. 2017;151:369\u201380. https:\/\/doi.org\/10.1016\/j.epsr.2017.06.006.","journal-title":"Electric Power Syst Res"},{"issue":"May","key":"270_CR6","doi-asserted-by":"publisher","first-page":"1099","DOI":"10.1016\/j.rser.2017.05.134","volume":"79","author":"C Tu","year":"2017","unstructured":"Tu C, He X, Shuai Z, Jiang F. Big data issues in smart grid\u2014a review. Renew Sustain Energy Rev. 2017;79(May):1099\u2013107. https:\/\/doi.org\/10.1016\/j.rser.2017.05.134.","journal-title":"Renew Sustain Energy Rev"},{"key":"270_CR7","doi-asserted-by":"publisher","first-page":"15132","DOI":"10.1109\/ACCESS.2018.2806881","volume":"6","author":"F Liang","year":"2018","unstructured":"Liang F, Yu W, An D, Yang Q, Fu X, Zhao W. A survey on big data market: pricing trading and protection. IEEE Access. 2018;6:15132\u201354. https:\/\/doi.org\/10.1109\/ACCESS.2018.2806881.","journal-title":"IEEE Access"},{"issue":"3","key":"270_CR8","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1109\/MPE.2018.2790819","volume":"16","author":"C Kang","year":"2018","unstructured":"Kang C, Wang Y, Xue Y, Mu G, Liao R. Big data analytics in china\u2019s electric power industry: modern information, communication technologies, and millions of smart meters. IEEE Power Energy Mag. 2018;16(3):54\u201365.","journal-title":"IEEE Power Energy Mag"},{"issue":"April","key":"270_CR9","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1016\/j.rser.2018.03.088","volume":"91","author":"L Wen","year":"2018","unstructured":"Wen L, Zhou K, Yang S, Li L. Compression of smart meter big data: a survey. Renew Sustain Energy Rev. 2018;91(April):59\u201369. https:\/\/doi.org\/10.1016\/j.rser.2018.03.088.","journal-title":"Renew Sustain Energy Rev"},{"key":"270_CR10","doi-asserted-by":"publisher","first-page":"40463","DOI":"10.1109\/ACCESS.2018.2858256","volume":"6","author":"AA Munshi","year":"2018","unstructured":"Munshi AA, Mohamed YARI. Data lake lambda architecture for smart grids big data analytics. IEEE Access. 2018;6:40463\u201371. https:\/\/doi.org\/10.1109\/ACCESS.2018.2858256.","journal-title":"IEEE Access"},{"key":"270_CR11","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1016\/j.future.2016.05.005.","volume":"78","author":"R Jiang","year":"2018","unstructured":"Jiang R, Lu R, Choo KKR. Achieving high performance and privacy-preserving query over encrypted multidimensional big metering data. Future Gener Comput Syst. 2018;78:392\u2013401. https:\/\/doi.org\/10.1016\/j.future.2016.05.005..","journal-title":"Future Gener Comput Syst"},{"key":"270_CR12","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1016\/j.compeleceng.2018.03.024","volume":"67","author":"YC Chang","year":"2018","unstructured":"Chang YC, Huang TC. An interactive smart grid communication approach for big data traffic. Comput Electr Eng. 2018;67:170\u201381. https:\/\/doi.org\/10.1016\/j.compeleceng.2018.03.024.","journal-title":"Comput Electr Eng"},{"key":"270_CR13","doi-asserted-by":"crossref","unstructured":"Chin Wl, Li W, Chen Hh. Energy Big Data Security Threats in IoT-Based Smart Grid Communications (October), 2017;70\u201375. https:\/\/ieeexplore-ieee-org.udel.idm.oclc.org\/document\/8067688\/.","DOI":"10.1109\/MCOM.2017.1700154"},{"issue":"3","key":"270_CR14","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1109\/TETC.2017.2681113","volume":"7","author":"Weigang Hou","year":"2019","unstructured":"Hou W, Ning Z, Guo L, Zhang X. Temporal, functional and spatial big data computing framework for large-scale smart grid. IEEE Trans Emerg Topics Comput. 2017;6750(1):1\u20131. https:\/\/doi.org\/10.1109\/TETC.2017.2681113. http:\/\/ieeexplore.ieee.org\/document\/7876814\/.","journal-title":"IEEE Transactions on Emerging Topics in Computing"},{"key":"270_CR15","doi-asserted-by":"crossref","unstructured":"Junaidi N. Big Data Applications in Electric Energy Systems. In: 2018 international conference on computational approach in smart systems design and applications (ICASSDA). 2017; p. 1\u20135.","DOI":"10.1109\/ICASSDA.2018.8477607"},{"key":"270_CR16","doi-asserted-by":"publisher","first-page":"19488","DOI":"10.1109\/ACCESS.2018.2886551","volume":"7","author":"Chunyan Shuai","year":"2019","unstructured":"Shuai C, Yang H, Ouyang X, He M, Gong Z, Shu W. Analysis and identification of power blackout-sensitive users by using big data in the energy system. IEEE Access. 2018; PP(c):1\u20131 . https:\/\/doi.org\/10.1109\/ACCESS.2018.2886551. https:\/\/ieeexplore.ieee.org\/document\/8576506\/.","journal-title":"IEEE Access"},{"key":"270_CR17","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.is.2018.05.007","volume":"77","author":"X Liu","year":"2018","unstructured":"Liu X, Nielsen PS. Scalable prediction-based online anomaly detection for smart meter data. Inf Syst. 2018;77:34\u201347. https:\/\/doi.org\/10.1016\/j.is.2018.05.007.","journal-title":"Inf Syst"},{"issue":"6","key":"270_CR18","doi-asserted-by":"publisher","first-page":"5820","DOI":"10.1109\/TSG.2017.2697440","volume":"9","author":"R Moghaddass","year":"2018","unstructured":"Moghaddass R, Wang J. A hierarchical framework for smart grid anomaly detection using large-scale smart meter data. IEEE Trans Smart Grid. 2018;9(6):5820\u201330. https:\/\/doi.org\/10.1109\/TSG.2017.2697440.","journal-title":"IEEE Trans Smart Grid"},{"key":"270_CR19","unstructured":"Vimalkumar K, Radhika N. A big data framework for intrusion detection in smart grids using Apache Spark. 2017 International conference on advances in computing, communications and informatics (ICACCI). IEEE, p. 198\u2013204, 2017."},{"issue":"3","key":"270_CR20","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1109\/TETC.2017.2692098","volume":"7","author":"Shailendra Singh","year":"2019","unstructured":"Singh S, Yassine A. Mining energy consumption behavior patterns for households in smart grid. IEEE Trans Emerg Topics Comput. 2017;6750(c):1\u20131 . https:\/\/doi.org\/10.1109\/TETC.2017.2692098. http:\/\/ieeexplore.ieee.org\/document\/7894203\/.","journal-title":"IEEE Transactions on Emerging Topics in Computing"},{"issue":"3","key":"270_CR21","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1109\/TBDATA.2016.2616146","volume":"4","author":"Jun Wu","year":"2018","unstructured":"Wu J, Ota K, Dong M, Li J, Wang H. Big data analysis based security situational awareness for smart grid. IEEE Trans Big Data 2016;7790(c):1\u20131 . https:\/\/doi.org\/10.1109\/TBDATA.2016.2616146. http:\/\/ieeexplore.ieee.org\/document\/7587350\/.","journal-title":"IEEE Transactions on Big Data"},{"issue":"1","key":"270_CR22","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1109\/TBDATA.2017.2723563","volume":"5","author":"Kun Wang","year":"2019","unstructured":"Wang K, Xu C, Zhang Y, Guo S, Zomaya A. Robust big data analytics for electricity price forecasting in the smart grid. IEEE Trans Big Data 2017;7790(c):1\u20131 . https:\/\/doi.org\/10.1109\/TBDATA.2017.2723563. http:\/\/ieeexplore.ieee.org\/document\/7968454\/.","journal-title":"IEEE Transactions on Big Data"},{"key":"270_CR23","unstructured":"Ye F, Qian Y, Hu RQ. A secure data learning scheme for big data applications in the smart grid. 2017."},{"issue":"1","key":"270_CR24","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1109\/TSG.2016.2550422","volume":"9","author":"HK Nguyen","year":"2018","unstructured":"Nguyen HK, Khodaei A, Han Z. A big data scale algorithm for optimal scheduling of integrated microgrids. IEEE Trans Smart Grid. 2018;9(1):274\u201382. https:\/\/doi.org\/10.1109\/TSG.2016.2550422.","journal-title":"IEEE Trans Smart Grid"},{"issue":"2","key":"270_CR25","doi-asserted-by":"publisher","first-page":"695","DOI":"10.1109\/TSG.2016.2562123","volume":"9","author":"G Sheng","year":"2018","unstructured":"Sheng G, Hou H, Jiang X, Chen Y. A novel association rule mining method of big data for power transformers state parameters based on probabilistic graph model. IEEE Trans Smart Grid. 2018;9(2):695\u2013702. https:\/\/doi.org\/10.1109\/TSG.2016.2562123.","journal-title":"IEEE Trans Smart Grid"},{"issue":"1","key":"270_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3004295","volume":"42","author":"Xiufeng Liu","year":"2016","unstructured":"Liu X, Golab L, Golab W, Ilyas IF, Jin S. Smart meter data analytics. ACM Trans Database Syst. 2016;42(1):1\u201339. https:\/\/doi.org\/10.1145\/3004295. http:\/\/dl.acm.org\/citation.cfm?doid=3015779.3004295.","journal-title":"ACM Transactions on Database Systems"},{"key":"270_CR27","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1007\/978-3-319-69462-7_2","volume-title":"On the Move to Meaningful Internet Systems. OTM 2017 Conferences","author":"Nadeem Iftikhar","year":"2017","unstructured":"Iftikhar N, Liu X, Danalachi S, Nordbjerg FE, Vollesen JH. A scalable smart meter data generator using spark. In: OTM Confederated International Conferences \u201cOn the move to meaningful internet systems\u201d. Cham: Springer; 2017. p. 21\u201336."},{"key":"270_CR28","unstructured":"Electricity customer behaviour trial. http:\/\/www.ucd.ie\/issda\/data\/commissionforenergyregulationcer\/."},{"key":"270_CR29","doi-asserted-by":"publisher","unstructured":"Iftikhar N, Liu X, Nordbjerg FE, Danalachi S. A Prediction-Based Smart Meter Data Generator. In: NBiS 2016\u201419th international conference on network-based information systems. 2016; p. 173\u201380. https:\/\/doi.org\/10.1109\/NBiS.2016.15.","DOI":"10.1109\/NBiS.2016.15"},{"key":"270_CR30","doi-asserted-by":"publisher","unstructured":"Kafka, Walkenbach J. The definitive guide. 2010;. https:\/\/doi.org\/10.1017\/CBO9781107415324.004.","DOI":"10.1017\/CBO9781107415324.004"},{"key":"270_CR31","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4842-1329-2","volume-title":"Next Generation Databases","author":"Guy Harrison","year":"2015","unstructured":"Harrison G. Next generation databases. 2015;. https:\/\/doi.org\/10.1007\/978-1-4842-1329-2. http:\/\/link.springer.com\/10.1007\/978-1-4842-1329-2."},{"key":"270_CR32","unstructured":"Vaish G. Getting Started with NoSQL. Packt Publishing Ltd, 2013."},{"key":"270_CR33","volume-title":"Cassandra: the definitive guide, first","author":"E Hewitt","year":"2011","unstructured":"Hewitt E. Cassandra: the definitive guide, first. edit ed. Sebastopol: O\u2019Reilly; 2011.","edition":"edit"},{"key":"270_CR34","volume-title":"Cassandra data modeling and analysis","author":"C Kan","year":"2014","unstructured":"Kan C. Cassandra data modeling and analysis. Birmingham: Packt Publishing Ltd; 2014."},{"key":"270_CR35","unstructured":"Kristina Chodorow and Michael Dirolf: MongoDB: The Definitive Guide (2010)."},{"key":"270_CR36","volume-title":"Elasticsearch: the definitive guide: a distributed real-time search and analytics engine","author":"C Gormley","year":"2015","unstructured":"Gormley C, Tong Z. Elasticsearch: the definitive guide: a distributed real-time search and analytics engine. Sebastopol: O\u2019Reilly Media, Inc.; 2015."}],"container-title":["Journal of Big Data"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-019-0270-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s40537-019-0270-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-019-0270-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,2]],"date-time":"2020-12-02T00:15:38Z","timestamp":1606868138000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-019-0270-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12]]},"references-count":36,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["270"],"URL":"https:\/\/doi.org\/10.1186\/s40537-019-0270-8","relation":{},"ISSN":["2196-1115"],"issn-type":[{"value":"2196-1115","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,12]]},"assertion":[{"value":"25 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 November 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 December 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare that they have no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"109"}}