{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T18:42:38Z","timestamp":1773513758393,"version":"3.50.1"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T00:00:00Z","timestamp":1663891200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T00:00:00Z","timestamp":1663891200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The integration of small-scale Photovoltaics (PV) systems (such as rooftop PVs) decreases the visibility of power systems, since the real demand load is masked. Most rooftop systems are behind the metre and cannot be measured by household smart meters. To overcome the challenges mentioned above, this paper proposes an online solar energy disaggregation system to decouple the solar energy generated by rooftop PV systems and the ground truth demand load from net measurements. A 1D Convolutional Neural Network (CNN) Bidirectional Long Short-Term Memory (BiLSTM) deep learning method is used as the core algorithm of the proposed system. The system takes a wide range of online information (Advanced Metering Infrastructure (AMI) data, meteorological data, satellite-driven irradiance, and temporal information) as inputs to evaluate PV generation, and the system also enables online and offline modes. The effectiveness of the proposed algorithm is evaluated by comparing it to baselines. The results show that the proposed method achieves good performance under different penetration rates and different feeder levels. Finally, a transfer learning process is introduced to verify that the proposed system has good robustness and can be applied to other feeders.<\/jats:p>","DOI":"10.1007\/s40747-022-00842-2","type":"journal-article","created":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T07:16:06Z","timestamp":1663917366000},"page":"3695-3716","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["A hybrid data-driven online solar energy disaggregation system from the grid supply point"],"prefix":"10.1007","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0209-531X","authenticated-orcid":false,"given":"Xiao-Yu","family":"Zhang","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4979-2198","authenticated-orcid":false,"given":"Stefanie","family":"Kuenzel","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6555-351X","authenticated-orcid":false,"given":"Peiqian","family":"Guo","sequence":"additional","affiliation":[]},{"given":"Lei","family":"Yin","sequence":"additional","affiliation":[]},{"given":"Chris","family":"Watkins","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,9,23]]},"reference":[{"key":"842_CR1","unstructured":"Irena GEC (2020) Renewable capacity statistics 2020. International renewable energy agency"},{"key":"842_CR2","unstructured":"DrIncecco M, Squartini S and Zhong M (2019) Transfer learning for non-intrusive load monitoring. IEEE Trans Smart Grid 1\u20131 (08\/28)"},{"key":"842_CR3","doi-asserted-by":"crossref","unstructured":"Wytock M and Kolter J (2014) Contextually supervised source separation with application to energy disaggregation. In Proceedings of the AAAI Conference on Artificial Intelligence","DOI":"10.1609\/aaai.v28i1.8769"},{"key":"842_CR4","unstructured":"\"SMART METERING IMPLEMENTATION PROGRAMME: review of the data access and privacy framework,\" E. I. S. Department for Business, UK, ed., 2018"},{"key":"842_CR5","doi-asserted-by":"crossref","unstructured":"Zhang XY, Kuenzel S, C\u00f3rdoba-Pach\u00f3n J-R and Watkins C (2020) Privacy-functionality trade-off: a privacy-preserving multi-channel smart metering system. Energies 13","DOI":"10.3390\/en13123221"},{"key":"842_CR6","unstructured":"Street P (2015) Dataport: the world's largest energy data resource. Pecan Street Inc"},{"key":"842_CR7","doi-asserted-by":"crossref","unstructured":"Angelis G-F, Timplalexis C, Krinidis S, Ioannidis D and Tzovaras D (2022) NILM applications: literature review of learning approaches, recent developments and challenges. Energy Build 111951","DOI":"10.1016\/j.enbuild.2022.111951"},{"issue":"3","key":"842_CR8","doi-asserted-by":"publisher","first-page":"917","DOI":"10.1109\/TSTE.2014.2313600","volume":"5","author":"H-T Yang","year":"2014","unstructured":"Yang H-T, Huang C-M, Huang Y-C, Pai Y-S (2014) A weather-based hybrid method for 1-day ahead hourly forecasting of PV power output. IEEE Trans Sustain Energy 5(3):917\u2013926","journal-title":"IEEE Trans Sustain Energy"},{"issue":"2","key":"842_CR9","doi-asserted-by":"publisher","DOI":"10.1063\/1.5094161","volume":"11","author":"E Vrettos","year":"2019","unstructured":"Vrettos E, Kara EC, Stewart EM, Roberts C (2019) Estimating PV power from aggregate power measurements within the distribution grid. J Renew Sustain Energy 11(2):023707","journal-title":"J Renew Sustain Energy"},{"key":"842_CR10","doi-asserted-by":"crossref","unstructured":"Cheung CM, Zhong W, Xiong C, Srivastava A, Kannan R and Prasanna VK (2018) Behind-the-meter solar generation disaggregation using consumer mixture models. In: 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), pp 1\u20136","DOI":"10.1109\/SmartGridComm.2018.8587539"},{"key":"842_CR11","unstructured":"Sossan F, Nespoli L, Medici V and Paolone M (2017) Unsupervised disaggregation of photovoltaic production from composite power flow measurements of heterogeneous prosumers. IEEE Trans Indust Inform 06\/15"},{"issue":"5","key":"842_CR12","doi-asserted-by":"publisher","first-page":"2456","DOI":"10.1109\/TSG.2016.2533164","volume":"7","author":"H Shaker","year":"2016","unstructured":"Shaker H, Zareipour H, Wood D (2016) Estimating power generation of invisible solar sites using publicly available data. IEEE Trans Smart Grid 7(5):2456\u20132465","journal-title":"IEEE Trans Smart Grid"},{"issue":"5","key":"842_CR13","doi-asserted-by":"publisher","first-page":"2466","DOI":"10.1109\/TSG.2015.2502140","volume":"7","author":"H Shaker","year":"2015","unstructured":"Shaker H, Zareipour H, Wood D (2015) A data-driven approach for estimating the power generation of invisible solar sites. IEEE Trans Smart Grid 7(5):2466\u20132476","journal-title":"IEEE Trans Smart Grid"},{"key":"842_CR14","doi-asserted-by":"crossref","unstructured":"Bright J, Killinger S, Lingfors D and Engerer N (2018) Improved satellite-derived PV power nowcasting using real-time power data from reference PV systems. Solar Energy 168:118\u2013139 (07\/01)","DOI":"10.1016\/j.solener.2017.10.091"},{"key":"842_CR15","doi-asserted-by":"publisher","first-page":"1026","DOI":"10.1016\/j.solener.2017.09.068","volume":"158","author":"M Pierro","year":"2017","unstructured":"Pierro M, De Felice M, Maggioni E, Moser D, Perotto A, Spada F, Cornaro C (2017) Data-driven upscaling methods for regional photovoltaic power estimation and forecast using satellite and numerical weather prediction data. Sol Energy 158:1026\u20131038","journal-title":"Sol Energy"},{"key":"842_CR16","doi-asserted-by":"crossref","unstructured":"Zhang X-Y, Watkins C and Kuenzel S (2022) Multi-quantile recurrent neural network for feeder-level probabilistic energy disaggregation considering roof-top solar energy. Eng Appl Artif Intell 110:104707 (2022\/04\/01\/)","DOI":"10.1016\/j.engappai.2022.104707"},{"key":"842_CR17","unstructured":"\"NOAA National Centers for Environmental Information (NCEI),\" N. S. US Department of Commerce and N. C. f. E. I. Information Service, eds."},{"issue":"5","key":"842_CR18","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1002\/wea.3703","volume":"75","author":"S Burt","year":"2020","unstructured":"Burt S (2020) The climatological observers link (COL) at 50. Weather 75(5):137\u2013144","journal-title":"Weather"},{"key":"842_CR19","doi-asserted-by":"crossref","unstructured":"Jaramillo AFM, Laverty DM, Del Rinc\u00f3n JM, Brogan P and Morrow DJ (2020) Non-intrusive load monitoring algorithm for PV identification in the residential sector. In: 2020 31st Irish Signals and Systems Conference (ISSC), pp 1\u20136","DOI":"10.1109\/ISSC49989.2020.9180192"},{"key":"842_CR20","doi-asserted-by":"crossref","unstructured":"Moreno Jaramillo AF, Lopez\u2010Lorente J, Laverty DM, Martinez\u2010del\u2010Rincon J, Morrow DJ and Foley AM (2022) Effective identification of distributed energy resources using smart meter net\u2010demand data. IET Smart Grid 5(2):120\u2013135","DOI":"10.1049\/stg2.12056"},{"key":"842_CR21","doi-asserted-by":"crossref","unstructured":"Kabir F, Yu N, Yao W, Yang R and Zhang Y (2019) Estimation of behind-the-meter solar generation by integrating physical with statistical models. In: 2019 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), pp 1\u20136","DOI":"10.1109\/SmartGridComm.2019.8909743"},{"issue":"3","key":"842_CR22","doi-asserted-by":"publisher","first-page":"3255","DOI":"10.1109\/TPWRS.2017.2762599","volume":"33","author":"Y Wang","year":"2018","unstructured":"Wang Y, Zhang N, Chen Q, Kirschen DS, Li P, Xia Q (2018) Data-driven probabilistic net load forecasting with high penetration of behind-the-meter PV. IEEE Trans Power Syst 33(3):3255\u20133264","journal-title":"IEEE Trans Power Syst"},{"key":"842_CR23","doi-asserted-by":"crossref","unstructured":"Zhang XY, Kuenzel S and Watkins C (2020) Feeder-level deep learning-based photovoltaic penetration estimation scheme. In: 2020\u00a012th IEEE\u00a0PES Asia-Pacific Power and Energy Engineering Conference (APPEEC), pp 1\u20135","DOI":"10.1109\/APPEEC48164.2020.9220536"},{"key":"842_CR24","first-page":"1","volume":"2021","author":"S Vejdan","year":"2021","unstructured":"Vejdan S, Mason K, Grijalva S (2021) Detecting behind-the-meter PV installation using convolutional neural networks. IEEE Texas Power Energy Conf (TPEC) 2021:1\u20136","journal-title":"IEEE Texas Power Energy Conf (TPEC)"},{"key":"842_CR25","doi-asserted-by":"crossref","unstructured":"Cheung CM, Kuppannagari SR, Kannan R and Prasanna VK (2019) Towards improved real-time observability of behind-meter photovoltaic systems: a data-driven approach. In: Proceedings of the Tenth ACM International Conference on Future Energy Systems, pp 447\u2013455","DOI":"10.1145\/3307772.3331019"},{"key":"842_CR26","unstructured":"P. S. Incorporation (2015) \u201cDataport,\u201d Pecan Street Inc., Austin, TX, 11:2018"},{"key":"842_CR27","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1016\/j.segan.2017.11.001","volume":"13","author":"EC Kara","year":"2018","unstructured":"Kara EC, Roberts CM, Tabone M, Alvarez L, Callaway DS, Stewart EM (2018) Disaggregating solar generation from feeder-level measurements. Sustain Energy Grids Netw 13:112\u2013121","journal-title":"Sustain Energy Grids Netw"},{"key":"842_CR28","doi-asserted-by":"crossref","unstructured":"Vrettos E, Kara EC, Stewart EM and Roberts C (2019) Estimating PV power from aggregate power measurements within the distribution grid. J Renew Sustain Energy 11(2):023707","DOI":"10.1063\/1.5094161"},{"key":"842_CR29","unstructured":"I. N. England. ISO New England Zonal information. https:\/\/www.iso-ne.com\/isoexpress\/web\/reports\/pricing\/-\/tree\/zone-info."},{"key":"842_CR30","doi-asserted-by":"crossref","unstructured":"Blair N, Dobos AP, Freeman J, Neises T, Wagner M, Ferguson T, Gilman P and Janzou S (2014) System advisor model, sam 2014.1. 14: General description, National Renewable Energy Lab.(NREL), Golden, CO (United States)","DOI":"10.2172\/1126294"},{"key":"842_CR31","unstructured":"\"Smart* Data Set for Sustainability,\" 2019."},{"key":"842_CR32","doi-asserted-by":"crossref","unstructured":"Saffari M, Khodayar M and Khodayar ME (2022) Deep recurrent extreme learning machine for behind-the-meter photovoltaic disaggregation. Electricity J 35(5):107137 (2022\/06\/01\/)","DOI":"10.1016\/j.tej.2022.107137"},{"key":"842_CR33","doi-asserted-by":"crossref","unstructured":"Schneider KP, Chen Y, Chassin DP, Pratt RG, Engel DW and Thompson SE (2008) Modern grid initiative distribution taxonomy final report. In Pacific Northwest National Lab.(PNNL), Richland, WA (United States)","DOI":"10.2172\/1040684"},{"key":"842_CR34","unstructured":"Oceanic N and Administration A (2016) National Centers for Environmental Information (NCEI). Historical Palmer Drought Indices"},{"key":"842_CR35","unstructured":"Data IS (2001) National climatic data center (NCDC). Asheville"},{"key":"842_CR36","unstructured":"C. Seger, \"An investigation of categorical variable encoding techniques in machine learning: binary versus one-hot and feature hashing,\" 2018."},{"key":"842_CR37","unstructured":"G\u00e9ron A (2019) Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: concepts, tools, and techniques to build intelligent systems: O'Reilly Media"},{"key":"842_CR38","doi-asserted-by":"crossref","unstructured":"Zeyer A, Schl\u00fcter R and Ney H (2016) Towards online-recognition with deep bidirectional LSTM acoustic models. Interspeech 2016","DOI":"10.21437\/Interspeech.2016-759"},{"key":"842_CR39","doi-asserted-by":"publisher","first-page":"180544","DOI":"10.1109\/ACCESS.2020.3028281","volume":"8","author":"M Alhussein","year":"2020","unstructured":"Alhussein M, Aurangzeb K, Haider SI (2020) Hybrid CNN-LSTM model for short-term individual household load forecasting. IEEE Access 8:180544\u2013180557","journal-title":"IEEE Access"},{"key":"842_CR40","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1016\/j.energy.2019.05.230","volume":"182","author":"T-Y Kim","year":"2019","unstructured":"Kim T-Y, Cho S-B (2019) Predicting residential energy consumption using CNN-LSTM neural networks. Energy 182:72\u201381","journal-title":"Energy"},{"key":"842_CR41","doi-asserted-by":"crossref","unstructured":"Kim P (2017) Convolutional neural network. In: MATLAB deep learning. Springer, pp 121\u2013147","DOI":"10.1007\/978-1-4842-2845-6_6"},{"key":"842_CR42","first-page":"1","volume":"2017","author":"S Albawi","year":"2017","unstructured":"Albawi S, Mohammed TA, Al-Zawi S (2017) Understanding of a convolutional neural network. Int Conf Eng Technol (ICET) 2017:1\u20136","journal-title":"Int Conf Eng Technol (ICET)"},{"key":"842_CR43","doi-asserted-by":"publisher","first-page":"312","DOI":"10.1016\/j.bspc.2018.08.035","volume":"47","author":"J Zhao","year":"2019","unstructured":"Zhao J, Mao X, Chen L (2019) Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomed Signal Process Control 47:312\u2013323","journal-title":"Biomed Signal Process Control"},{"issue":"10","key":"842_CR44","doi-asserted-by":"publisher","first-page":"2196","DOI":"10.1109\/LCOMM.2020.3002073","volume":"24","author":"J Xie","year":"2020","unstructured":"Xie J, Fang J, Liu C, Li X (2020) Deep learning-based spectrum sensing in cognitive radio: A CNN-LSTM approach. IEEE Commun Lett 24(10):2196\u20132200","journal-title":"IEEE Commun Lett"},{"key":"842_CR45","doi-asserted-by":"crossref","unstructured":"Meng L, Sanseverino ER, Luna A, Dragicevic T, Vasquez JC and Guerrero JM (2016) Microgrid supervisory controllers and energy management systems: a literature review. Renew Sustain Energy Rev 60:1263\u20131273 (2016\/07\/01\/)","DOI":"10.1016\/j.rser.2016.03.003"},{"issue":"10","key":"842_CR46","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","volume":"22","author":"SJ Pan","year":"2010","unstructured":"Pan SJ, Qiang Y (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345\u20131359","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"842_CR47","doi-asserted-by":"crossref","unstructured":"Arnold A, Nallapati R and Cohen WW (2007) A comparative study of methods for transductive transfer learning. In Workshops Proceedings of the 7th IEEE International Conference on Data Mining (ICDM 2007), October 28\u201331, 2007, Omaha, Nebraska, USA","DOI":"10.1109\/ICDMW.2007.109"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00842-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-022-00842-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00842-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,27]],"date-time":"2023-07-27T09:15:53Z","timestamp":1690449353000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-022-00842-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,23]]},"references-count":47,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["842"],"URL":"https:\/\/doi.org\/10.1007\/s40747-022-00842-2","relation":{"has-preprint":[{"id-type":"doi","id":"10.36227\/techrxiv.16569744.v1","asserted-by":"object"},{"id-type":"doi","id":"10.36227\/techrxiv.16569744","asserted-by":"object"}]},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,23]]},"assertion":[{"value":"27 November 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 July 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 September 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}