{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,30]],"date-time":"2026-03-30T19:04:06Z","timestamp":1774897446034,"version":"3.50.1"},"reference-count":31,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,3,13]],"date-time":"2024-03-13T00:00:00Z","timestamp":1710288000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2022R1F1A1067008"],"award-info":[{"award-number":["2022R1F1A1067008"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Big Data"],"abstract":"<jats:p>With the increasing utilization of data in various industries and applications, constructing an efficient data pipeline has become crucial. In this study, we propose a machine learning operations-centric data pipeline specifically designed for an energy consumption management system. This pipeline seamlessly integrates the machine learning model with real-time data management and prediction capabilities. The overall architecture of our proposed pipeline comprises several key components, including Kafka, InfluxDB, Telegraf, Zookeeper, and Grafana. To enable accurate energy consumption predictions, we adopt two time-series prediction models, long short-term memory (LSTM), and seasonal autoregressive integrated moving average (SARIMA). Our analysis reveals a clear trade-off between speed and accuracy, where SARIMA exhibits faster model learning time while LSTM outperforms SARIMA in prediction accuracy. To validate the effectiveness of our pipeline, we measure the overall processing time by optimizing the configuration of Telegraf, which directly impacts the load in the pipeline. The results are promising, as our pipeline achieves an average end-to-end processing time of only 0.39 s for handling 10,000 data records and an impressive 1.26 s when scaling up to 100,000 records. This indicates 30.69\u201390.88 times faster processing compared to the existing Python-based approach. Additionally, when the number of records increases by ten times, the increased overhead is reduced by 3.07 times. This verifies that the proposed pipeline exhibits an efficient and scalable structure suitable for real-time environments.<\/jats:p>","DOI":"10.3389\/fdata.2024.1308236","type":"journal-article","created":{"date-parts":[[2024,3,13]],"date-time":"2024-03-13T04:26:55Z","timestamp":1710304015000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Data pipeline for real-time energy consumption data management and prediction"],"prefix":"10.3389","volume":"7","author":[{"given":"Jeonghwan","family":"Im","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jaekyu","family":"Lee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Somin","family":"Lee","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hyuk-Yoon","family":"Kwon","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,3,13]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"1483","DOI":"10.1109\/ISIE.2017.8001465","article-title":"\u201cDeep neural networks for energy load forecasting,\u201d","volume-title":"2017 IEEE 26th International Symposium on Industrial Electronics (ISIE)","author":"Amarasinghe","year":"2017"},{"key":"B2","doi-asserted-by":"publisher","first-page":"498","DOI":"10.1109\/59.932287","article-title":"Short-term hourly load forecasting using time-series modeling with peak load estimation capability","volume":"16","author":"Amjady","year":"2001","journal-title":"IEEE Transact. Power Syst"},{"key":"B3","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1007\/b97636","volume-title":"Model Selection and Multimodel Inference. A Practical information-Theoretic Approach","author":"Burnham","year":"2004"},{"key":"B4","first-page":"295","article-title":"\u201cTime series analysis of household electric consumption with arima and arma models,\u201d","volume-title":"Proceedings of the International Multiconference of Engineers and Computer Scientists, Vol. 1","author":"Chujai","year":"2013"},{"key":"B5","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1016\/j.energy.2009.10.018","article-title":"Greek long-term energy consumption prediction using artificial neural networks","volume":"35","author":"Ekonomou","year":"2010","journal-title":"Energy"},{"key":"B6","doi-asserted-by":"publisher","first-page":"958","DOI":"10.1016\/j.neucom.2015.08.051","article-title":"Electric load forecasting by the svr model with differential empirical mode decomposition and auto regression","volume":"173","author":"Fan","year":"2016","journal-title":"Neurocomputing"},{"key":"B7","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1080\/0952813X.2013.813976","article-title":"A hybrid method based on wavelet, ann and arima model for short-term load forecasting","volume":"26","author":"Fard","year":"2014","journal-title":"J. Exp. Theoret. Artif. Intell"},{"key":"B8","doi-asserted-by":"crossref","first-page":"2503","DOI":"10.1145\/3448016.3457552","article-title":"\u201cReal-time data infrastructure at uber,\u201d","volume-title":"Proceedings of the 2021 International Conference on Management of Data","author":"Fu","year":"2021"},{"key":"B9","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1016\/j.procs.2014.08.140","article-title":"Short term electricity forecasting using individual smart meter data","volume":"35","author":"Gajowniczek","year":"2014","journal-title":"Proc. Comput. Sci"},{"key":"B10","doi-asserted-by":"publisher","first-page":"685","DOI":"10.11591\/ijece.v5i4.pp685-694","article-title":"Real time domestic power consumption monitoring using wireless sensor networks","volume":"5","author":"Gogineni","year":"2015","journal-title":"Int. J. Elect. Comp. Eng"},{"key":"B11","author":"Hebrail","year":"2012","journal-title":"Individual Household Electric Power Consumption"},{"key":"B12","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"B13","doi-asserted-by":"publisher","first-page":"767","DOI":"10.3390\/en9100767","article-title":"A permutation importance-based feature selection method for short-term electricity load forecasting using random forest","volume":"9","author":"Huang","year":"2016","journal-title":"Energies"},{"key":"B14","doi-asserted-by":"publisher","first-page":"6783","DOI":"10.3390\/s20236783","article-title":"Advances in sensor technologies in the era of smart factory and industry 4.0","volume":"20","author":"Kalsoom","year":"2020","journal-title":"Sensors"},{"key":"B15","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1109\/GCCE.2018.8574484","article-title":"\u201cLstm based short-term electricity consumption forecast with daily load profile sequences,\u201d","volume-title":"2018 IEEE 7th Global Conference on Consumer Electronics (GCCE)","author":"Kim","year":"2018"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1412.6980","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"arXiv"},{"key":"B17","doi-asserted-by":"publisher","first-page":"841","DOI":"10.1109\/TSG.2017.2753802","article-title":"Short-term residential load forecasting based on lstm recurrent neural network","volume":"10","author":"Kong","year":"2017","journal-title":"IEEE Trans. Smart Grid"},{"key":"B18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/IJCNN.2018.8489399","article-title":"\u201cConvolutional neural networks for energy time series forecasting,\u201d","volume-title":"2018 International Joint Conference on Neural Networks (IJCNN)","author":"Koprinska","year":"2018"},{"key":"B19","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1016\/j.energy.2019.04.075","article-title":"Toward efficient energy systems based on natural gas consumption prediction with lstm recurrent neural networks","volume":"177","author":"Laib","year":"2019","journal-title":"Energy"},{"key":"B20","doi-asserted-by":"publisher","first-page":"1748","DOI":"10.1016\/j.ijforecast.2021.03.012","article-title":"Temporal fusion transformers for interpretable multi-horizon time series forecasting","volume":"37","author":"Lim","year":"2021","journal-title":"Int. J. Forecast"},{"key":"B21","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1016\/j.enbuild.2015.02.007","article-title":"Short-term load forecasting in a non-residential building contrasting models and attributes","volume":"92","author":"Massana","year":"2015","journal-title":"Energy Build"},{"key":"B22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/LA-CCI.2016.7885697","article-title":"\u201cA hybrid method using exponential smoothing and gradient boosting for electrical short-term load forecasting,\u201d","volume-title":"2016 IEEE Latin American Conference on Computational Intelligence (LA-CCI)","author":"Mayrink","year":"2016"},{"key":"B23","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1002\/er.787","article-title":"Support vector machines for short-term electrical load forecasting","volume":"26","author":"Mohandes","year":"2002","journal-title":"Int. J. Energy Res"},{"key":"B24","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/ICCCNT.2015.7395242","article-title":"\u201cHadoop based real-time big data architecture for remote sensing earth observatory system,\u201d","volume-title":"2015 6th International Conference on Computing, Communication and Networking Technologies (ICCCNT)","author":"Rathore","year":"2015"},{"key":"B25","volume-title":"Machine Learning: The High Interest Credit Card of Technical Debt","author":"Sculley","year":"2014"},{"key":"B26","doi-asserted-by":"publisher","first-page":"100037","DOI":"10.1016\/j.dibe.2020.100037","article-title":"Energy consumption prediction by using machine learning for smart building: case study in malaysia","volume":"5","author":"Shapi","year":"2021","journal-title":"Dev. Built Environ"},{"key":"B27","doi-asserted-by":"publisher","first-page":"2139","DOI":"10.3390\/su9112139","article-title":"An open source-based real-time data processing architecture framework for manufacturing sustainability","volume":"9","author":"Syafrudin","year":"2017","journal-title":"Sustainability"},{"key":"B28","doi-asserted-by":"publisher","first-page":"592","DOI":"10.1002\/met.1491","article-title":"Long-term runoff study using sarima and arima models in the united states","volume":"22","author":"Valipour","year":"2015","journal-title":"Meteorol. Appl"},{"key":"B29","doi-asserted-by":"publisher","first-page":"117197","DOI":"10.1016\/j.energy.2020.117197","article-title":"Lstm based long-term energy consumption prediction with periodicity","volume":"197","author":"Wang","year":"2020","journal-title":"Energy"},{"key":"B30","doi-asserted-by":"publisher","first-page":"157633","DOI":"10.1109\/ACCESS.2019.2949065","article-title":"A hybrid lstm neural network for energy consumption forecasting of individual households","volume":"7","author":"Yan","year":"2019","journal-title":"IEEE Access"},{"key":"B31","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1109\/ICDM50108.2020.00093","article-title":"\u201cMultivariate time-series anomaly detection via graph attention network,\u201d","volume-title":"2020 IEEE International Conference on Data Mining (ICDM)","author":"Zhao","year":"2020"}],"container-title":["Frontiers in Big Data"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2024.1308236\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,18]],"date-time":"2024-03-18T13:26:52Z","timestamp":1710768412000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2024.1308236\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,13]]},"references-count":31,"alternative-id":["10.3389\/fdata.2024.1308236"],"URL":"https:\/\/doi.org\/10.3389\/fdata.2024.1308236","relation":{},"ISSN":["2624-909X"],"issn-type":[{"value":"2624-909X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,13]]},"article-number":"1308236"}}