{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T20:47:04Z","timestamp":1781815624430,"version":"3.54.5"},"reference-count":28,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T00:00:00Z","timestamp":1759968000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Accurate estimation of the Remaining Useful Life (RUL) of lithium-ion batteries is essential for enhancing the reliability and efficiency of energy storage systems. This study explores custom deep learning models to predict RUL using a dataset from the Hawaii Natural Energy Institute (HNEI). Three approaches are investigated: an Encoder-only Transformer model, its enhancement with SimSiam transfer learning, and a CNN\u2013Encoder hybrid model. These models leverage advanced mechanisms such as multi-head attention, robust feedforward networks, and self-supervised learning to capture complex degradation patterns in the data. Rigorous preprocessing and optimisation ensure optimal performance, reducing key metrics such as mean squared error (MSE) and mean absolute error (MAE). Experimental results demonstrated that Transformer\u2013CNN with Noise Augmentation outperforms other methods, highlighting its potential for battery health monitoring and predictive maintenance.<\/jats:p>","DOI":"10.3390\/info16100878","type":"journal-article","created":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:50:16Z","timestamp":1760107816000},"page":"878","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A Comparison of Different Transformer Models for Time Series Prediction"],"prefix":"10.3390","volume":"16","author":[{"given":"Emek Utku","family":"Capoglu","sequence":"first","affiliation":[{"name":"School of Computer Science and Informatics, De Montfort University, Leicester LE1 9BH, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3627-6362","authenticated-orcid":false,"given":"Aboozar","family":"Taherkhani","sequence":"additional","affiliation":[{"name":"School of Computer Science and Informatics, De Montfort University, Leicester LE1 9BH, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,9]]},"reference":[{"key":"ref_1","unstructured":"Adithya, C.H., Hegde, A.R., and Prasad, S. (2024, January 5\u20137). Early Prediction of Remaining Useful Life for Li-ion Batteries Using Transformer Model with Dual Auto-Encoder and Ensemble Techniques. Proceedings of the 2024 IEEE 9th International Conference for Convergence in Technology (I2CT), Pune, India."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"125501","DOI":"10.1016\/j.energy.2022.125501","article-title":"A novel state-of-health estimation for the lithium-ion battery using a convolutional neural network and transformer model","volume":"262","author":"Gu","year":"2023","journal-title":"Energy"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"112703","DOI":"10.1016\/j.est.2024.112703","article-title":"Review of battery state estimation methods for electric vehicles-Part II: SOH estimation","volume":"96","author":"Demirci","year":"2024","journal-title":"J. Energy Storage"},{"key":"ref_4","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1830","DOI":"10.1016\/j.procs.2022.12.383","article-title":"A remaining useful life prediction method for lithium-ion battery based on temporal transformer network","volume":"217","author":"Song","year":"2023","journal-title":"Procedia Comput. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"108475","DOI":"10.1016\/j.engappai.2024.108475","article-title":"Using transformer and a reweighting technique to develop a remaining useful life estimation method for turbofan engines","volume":"133","author":"Kim","year":"2024","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Wang, S., Li, Y., Zhou, S., Chen, L., and Michael, P. (2025). Remaining useful life prediction of lithium-ion batteries using a novel particle flow filter framework with grey model. Sci. Rep., 15.","DOI":"10.1038\/s41598-025-86511-z"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"8831","DOI":"10.1109\/TIM.2020.2996004","article-title":"Remaining useful life prediction of lithium-ion batteries based on conditional variational autoencoders-particle filter","volume":"69","author":"Jiao","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1016\/j.microrel.2012.12.004","article-title":"Remaining Useful Life Prediction of Lithium-Ion Battery with Unscented Particle Filter Technique","volume":"53","author":"Miao","year":"2013","journal-title":"Microelectron. Reliab."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"227700","DOI":"10.1016\/j.jpowsour.2020.227700","article-title":"Remaining Useful Life Prediction of Lithium-Ion Battery Based on Improved Cuckoo Search Particle Filter and a Novel State of Charge Estimation Method","volume":"450","author":"Qiu","year":"2020","journal-title":"J. Power Sources"},{"key":"ref_11","first-page":"01423312","article-title":"Experimental Verification of Lithium-Ion Battery Prognostics Based on an Interacting Multiple Model Particle Filter","volume":"42","author":"Wang","year":"2020","journal-title":"Trans. Inst. Meas. Control"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.renene.2015.06.034","article-title":"Transfer learning for short-term wind speed prediction with deep neural networks","volume":"85","author":"Hu","year":"2016","journal-title":"Renew. Energy"},{"key":"ref_13","unstructured":"Arai, K., Kapoor, S., and Bhatia, R. (2019). Activity Recognition from Multi-modal Sensor Data Using a Deep Convolutional Neural Network. Intelligent Computing, Springer. Advances in Intelligent Systems and Computing."},{"key":"ref_14","unstructured":"Shi, Z., Torresen, J., and Yang, S. (2024). Hand Gesture Recognition Using a Multi-modal Deep Neural Network. Intelligent Information Processing XII, Springer. IFIP Advances in Information and Communication Technology."},{"key":"ref_15","unstructured":"Malhotra, P., Vig, L., Agarwal, P., and Shroff, G. (2017, January 26\u201328). TimeNet: Pre-trained deep recurrent neural network for time series classification. Proceedings of the 25th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1016\/j.procs.2017.09.066","article-title":"ScienceDirect ScienceDirect time series classification using deep learning for process planning: A case from the process industry","volume":"114","author":"Mehdiyev","year":"2017","journal-title":"Proc. Comput. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Rajan, D., and Thiagarajan, J.J. (2018, January 18\u201321). A Generative Modeling Approach to Limited Channel ECG Classification. Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.","DOI":"10.1109\/EMBC.2018.8512757"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Schwenker, F., Abbas, H.M., El Gayar, N., and Trentin, E. (2016). Time series classification in reservoir- and model-space: A comparison. Artificial Neural Networks in Pattern Recognition, Springer International Publishing.","DOI":"10.1007\/978-3-319-46182-3"},{"key":"ref_19","unstructured":"Bianchi, F.M., Scardapane, S., and Jenssen, R. (2018). Reservoir computing approaches for representation and classification of multivariate time series. arXiv."},{"key":"ref_20","unstructured":"Chouikhi, N., Ammar, B., Alimi, A.M., and Member, S. (2018). Genesis of basic and multi-layer echo state network recurrent autoencoder for efficient data representations. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ins.2016.08.081","article-title":"Functional echo state network for time series classification","volume":"373","author":"Ma","year":"2016","journal-title":"Inf. Sci."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"832","DOI":"10.1007\/s42979-023-02159-4","article-title":"A Deep Convolutional Neural Network for Time Series Classification with Intermediate Targets","volume":"4","author":"Taherkhani","year":"2023","journal-title":"SN Comput. Sci."},{"key":"ref_23","first-page":"1","article-title":"Detection of interictal discharges with convolutional neural networks using discrete ordered multichannel intracranial EEG","volume":"4320","author":"Antoniades","year":"2017","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"19621","DOI":"10.1109\/ACCESS.2022.3151975","article-title":"Transformer network for remaining useful life prediction of lithium-ion batteries","volume":"10","author":"Chen","year":"2022","journal-title":"IEEE Access"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"105558","DOI":"10.1016\/j.est.2022.105558","article-title":"Li-ion battery degradation modes diagnosis via Convolutional Neural Networks","volume":"55","author":"Costa","year":"2022","journal-title":"J. Energy Storage"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Gui, X., Du, J., Song, L., Yan, Z., and Guo, L. (2023, January 12\u201315). A Transformer-CNN Based Transferable Model for State-of-Health Prediction of Lithium-ion Batteries. Proceedings of the 2023 Global Reliability and Prognostics and Health Management Conference (PHM-Hangzhou), Hangzhou, China.","DOI":"10.1109\/PHM-Hangzhou58797.2023.10482710"},{"key":"ref_27","unstructured":"GitHub (2025, August 07). Battery RUL Prediction Dataset. Available online: https:\/\/github.com\/ignavinuales\/Battery_RUL_Prediction\/blob\/main\/Datasets\/HNEI_Processed\/Final%20Database.csv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chen, X., and He, K. (2021, January 20\u201325). Exploring simple siamese representation learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01549"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/10\/878\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:19:23Z","timestamp":1760109563000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/10\/878"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,9]]},"references-count":28,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["info16100878"],"URL":"https:\/\/doi.org\/10.3390\/info16100878","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,9]]}}}