{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:24:33Z","timestamp":1760235873632,"version":"build-2065373602"},"reference-count":57,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T00:00:00Z","timestamp":1632960000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100013276","name":"Interreg","doi-asserted-by":"publisher","award":["Upper Rhine Offensive Science HALFBACK project"],"award-info":[{"award-number":["Upper Rhine Offensive Science HALFBACK project"]}],"id":[{"id":"10.13039\/100013276","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Ministries for Research of Baden-Wurttemberg, Rheinland-Pfalz (Germany) and from the Grand Est French Region","award":["Upper Rhine Offensive Science HALFBACK project"],"award-info":[{"award-number":["Upper Rhine Offensive Science HALFBACK project"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Industry 4.0 is characterized by the availability of sensors to operate the so-called intelligent factory. Predictive maintenance, in particular, failure prediction, is an important issue to cut the costs associated with production breaks. We studied more than 40 publications on predictive maintenance. We point out that they focus on various machine learning algorithms rather than on the selection of suitable datasets. In fact, most publications consider a single, usually non-public, benchmark. More benchmarks are needed to design and test the generality of the proposed approaches. This paper is the first to define the requirements on these benchmarks. It highlights that there are only two benchmarks that can be used for supervised learning among the six publicly available ones we found in the literature. We also illustrate how such a benchmark can be used with deep learning to successfully train and evaluate a failure prediction model. We raise several perspectives for research.<\/jats:p>","DOI":"10.3390\/informatics8040068","type":"journal-article","created":{"date-parts":[[2021,10,11]],"date-time":"2021-10-11T21:47:34Z","timestamp":1633988854000},"page":"68","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Identifying Benchmarks for Failure Prediction in Industry 4.0"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0786-8008","authenticated-orcid":false,"given":"Mouhamadou Saliou","family":"Diallo","sequence":"first","affiliation":[{"name":"ICube, University of Strasbourg, 300 Bd S\u00e9bastien Brant, 67400 Illkirch-Graffenstaden, France"}]},{"given":"Sid Ahmed","family":"Mokeddem","sequence":"additional","affiliation":[{"name":"ICube, University of Strasbourg, 300 Bd S\u00e9bastien Brant, 67400 Illkirch-Graffenstaden, France"}]},{"given":"Agn\u00e8s","family":"Braud","sequence":"additional","affiliation":[{"name":"ICube, University of Strasbourg, 300 Bd S\u00e9bastien Brant, 67400 Illkirch-Graffenstaden, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2022-9729","authenticated-orcid":false,"given":"Gabriel","family":"Frey","sequence":"additional","affiliation":[{"name":"ICube, University of Strasbourg, 300 Bd S\u00e9bastien Brant, 67400 Illkirch-Graffenstaden, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4318-4252","authenticated-orcid":false,"given":"Nicolas","family":"Lachiche","sequence":"additional","affiliation":[{"name":"ICube, University of Strasbourg, 300 Bd S\u00e9bastien Brant, 67400 Illkirch-Graffenstaden, France"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2941","DOI":"10.1080\/00207543.2018.1444806","article-title":"Industry 4.0: State of the art and future trends","volume":"56","author":"Xu","year":"2018","journal-title":"Int. J. Prod. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1531","DOI":"10.1007\/s10845-019-01531-7","article-title":"Machine learning applied in production planning and control: A state-of-the-art in the era of industry 4.0","volume":"31","author":"Lamouri","year":"2020","journal-title":"J. Intell. Manuf."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1016\/J.ENG.2017.05.015","article-title":"Intelligent Manufacturing in the Context of Industry 4.0: A Review","volume":"3","author":"Zhong","year":"2017","journal-title":"Engineering"},{"key":"ref_4","first-page":"899","article-title":"Scanning the Industry 4.0: A Literature Review on Technologies for Manufacturing Systems","volume":"22","year":"2019","journal-title":"Eng. Sci. Technol. Int. J."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"812","DOI":"10.1109\/TII.2014.2349359","article-title":"Machine Learning for Predictive Maintenance: A Multiple Classifier Approach","volume":"11","author":"Susto","year":"2015","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_6","unstructured":"Krishnamurthy, L., Adler, R., Buonadonna, P., Chhabra, J., Flanigan, M., Kushalnagar, N., Nachman, L., and Yarvis, M. (2015, January 2\u20134). Design and Deployment of Industrial Sensor Networks: Experiences from a Semiconductor Plant and the North Sea. Proceedings of the 3rd International Conference on Embedded Networked Sensor Systems, New York, NY, USA."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mangal, A., and Kumar, N. (2017). Using Big Data to Enhance the Bosch Production Line Performance: A Kaggle Challenge. arXiv.","DOI":"10.1109\/BigData.2016.7840826"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1007\/s11465-018-0499-5","article-title":"Smart manufacturing systems for Industry 4.0: Conceptual framework, scenarios, and future perspectives","volume":"13","author":"Zheng","year":"2018","journal-title":"Front. Mech. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Karampidis, K., Panagiotakis, S., Vasilakis, M., Markakis, E.K., and Papadourakis, G. (2019, January 11\u201313). Industrial CyberSecurity 4.0: Preparing the Operational Technicians for Industry 4.0. Proceedings of the 2019 IEEE 24th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), Limassol, Cyprus.","DOI":"10.1109\/CAMAD.2019.8858454"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"113653","DOI":"10.1016\/j.eswa.2020.113653","article-title":"Cost-sensitive learning classification strategy for predicting product failures","volume":"161","author":"Frumosu","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1007\/s10845-016-1244-8","article-title":"Prognostics of multiple failure modes in rotating machinery using a pattern-based classifier and cumulative incidence functions","volume":"30","author":"Ragab","year":"2019","journal-title":"J. Intell. Manuf."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1007\/s10845-018-1453-4","article-title":"Failure time prediction using adaptive logical analysis of survival curves and multiple machining signals","volume":"31","author":"Elsheikh","year":"2020","journal-title":"J. Intell. Manuf."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1007\/s10845-018-1456-1","article-title":"Intelligent rotating machinery fault diagnosis based on deep learning using data augmentation","volume":"31","author":"Li","year":"2020","journal-title":"J. Intell. Manuf."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"953","DOI":"10.1007\/s10845-019-01488-7","article-title":"Tool wear predicting based on multi-domain feature fusion by deep convolutional neural network in milling operations","volume":"31","author":"Huang","year":"2019","journal-title":"J. Intell. Manuf."},{"key":"ref_15","unstructured":"Li, X., Lim, B.S., Zhou, J., Huang, S.G., Phua, S.J., Shaw, K.C., and Er, M.J. (October, January 27). Fuzzy Neural Network Modelling for Tool Wear Estimation in Dry Milling Operation. Proceedings of the Annual Conference of the Prognostics and Health Management Society, San Diego, CA, USA."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1369","DOI":"10.1007\/s00170-019-04020-6","article-title":"Tool condition monitoring in CNC end milling using wavelet neural network based on machine vision","volume":"104","author":"Ong","year":"2019","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1016\/j.ymssp.2011.09.029","article-title":"Application of a state space modeling technique to system prognostics based on a health index for condition-based maintenance","volume":"28","author":"Sun","year":"2012","journal-title":"Mech. Syst. Signal Process."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s10845-016-1249-3","article-title":"Two improvements of similarity-based residual life prediction methods","volume":"30","author":"Gu","year":"2019","journal-title":"J. Intell. Manuf."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1007\/s10845-016-1264-4","article-title":"Damage forecasting based on multi-factor fuzzy time series and cloud model","volume":"30","author":"Dong","year":"2019","journal-title":"J. Intell. Manuf."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Paolanti, M., Romeo, L., Felicetti, A., Mancini, A., Frontoni, E., and Loncarski, J. (2018, January 2\u20134). Machine Learning approach for Predictive Maintenance in Industry 4.0. Proceedings of the 14th IEEE\/ASME International Conference on Mechatronic and Embedded Systems and Applications, MESA 2018, Oulu, Finland.","DOI":"10.1109\/MESA.2018.8449150"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"114598","DOI":"10.1016\/j.eswa.2021.114598","article-title":"Predictive maintenance system for production lines in manufacturing: A machine learning approach using IoT data in real-time","volume":"173","author":"Ayvaz","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s00170-019-03822-y","article-title":"A predictive association rule-based maintenance policy to minimize the probability of breakages: Application to an oil refinery","volume":"105","author":"Antomarioni","year":"2019","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"3647","DOI":"10.1007\/s00170-019-04090-6","article-title":"Tool wear classification using time series imaging and deep learning","volume":"104","author":"Terrazas","year":"2019","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3761","DOI":"10.1007\/s00170-019-03819-7","article-title":"Automatic detection of defective crankshafts by image analysis and supervised classification","volume":"105","author":"Remeseiro","year":"2019","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"106458","DOI":"10.1016\/j.compeleceng.2019.106458","article-title":"Real-time anomaly detection based on long short-Term memory and Gaussian Mixture Model","volume":"79","author":"Ding","year":"2019","journal-title":"Comput. Electr. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5695","DOI":"10.1109\/TVT.2018.2805189","article-title":"Long Short-Term Memory Recurrent Neural Network for Remaining Useful Life Prediction of Lithium-Ion Batteries","volume":"67","author":"Zhang","year":"2018","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_27","unstructured":"Malhotra, P., Vig, L., Shroff, G., and Agarwal, P. (2015, January 22\u201324). Long Short Term Memory Networks for Anomaly Detection in Time Series. Proceedings of the 23rd European Symposium on Artificial Neural Networks, ESANN 2015, Bruges, Belgium."},{"key":"ref_28","unstructured":"Bondu, A., Gay, D., Lemaire, V., Boull\u00e9, M., and Cervenka, E. (2019, January 17\u201319). FEARS: A Feature and Representation Selection approach for Time Series Classification. In Proceedings of The 11th Asian Conference on Machine Learning, ACML 2019, Nagoya, Japan."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Appice, A., Ceci, M., Loglisci, C., Manco, G., Masciari, E., and Ras, Z.W. (2013, January 27). New Frontiers in Mining Complex Patterns\u2014Second International Workshop, NFMCP 2013. Proceedings of the Conjunction with ECML-PKDD 2013, Prague, Czech Republic. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-642-37382-4"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Grabocka, J., Schilling, N., Wistuba, M., and Schmidt-Thieme, L. (2014, January 24\u201327). Learning Time-series Shapelets. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA.","DOI":"10.1145\/2623330.2623613"},{"key":"ref_31","unstructured":"Ye, L., and Keogh, E. (July, January 28). Time Series Shapelets: A New Primitive for Data Mining. Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Teng, W., Zhang, X., Liu, Y., Kusiak, A., and Ma, Z. (2017). Prognosis of the Remaining Useful Life of Bearings in a Wind Turbine Gearbox. Energies, 10.","DOI":"10.3390\/en10010032"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Yoo, Y., and Baek, J.G. (2018). A Novel Image Feature for the Remaining Useful Lifetime Prediction of Bearings Based on Continuous Wavelet Transform and Convolutional Neural Network. Appl. Sci., 8.","DOI":"10.3390\/app8071102"},{"key":"ref_34","unstructured":"Gay, D., and Lemaire, V. (2019). Should we Reload Time Series Classification Performance Evaluation ? (a position paper). arXiv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1066","DOI":"10.1016\/j.jsv.2005.03.007","article-title":"Wavelet filter-based weak signature detection method and its application on rolling element bearing prognostics","volume":"289","author":"Qiu","year":"2006","journal-title":"J. Sound Vib."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Duong, B.P., Khan, S.A., Shon, D., Im, K., Park, J., Lim, D.S., Jang, B., and Kim, J.M. (2018). A Reliable Health Indicator for Fault Prognosis of Bearings. Sensors, 18.","DOI":"10.3390\/s18113740"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Zhang, N., Wu, L., Wang, Z., and Guan, Y. (2018). Bearing Remaining Useful Life Prediction Based on Naive Bayes and Weibull Distributions. Entropy, 20.","DOI":"10.3390\/e20120944"},{"key":"ref_38","unstructured":"Lee, J., Qiu, H., Yuand, G., and J, L. (2007). Bearing data set. IMS, University of Cincinnati, NASA Ames Prognostics Data Repository, Rexnord Technical Services, NASA AMES, Moffett Field."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Khan, F., Eker, O.F., Khan, A., and Orfali, W. (2018). Adaptive Degradation Prognostic Reasoning by Particle Filter with a Neural Network Degradation Model for Turbofan Jet Engine. Data, 3.","DOI":"10.3390\/data3040049"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Saxena, A., Goebel, K., Simon, D., and Eklund, N. (2008, January 25). Damage propagation modeling for aircraft engine run-to-failure simulation. Proceedings of the 2008 International Conference on Prognostics and Health Management, Denver, CO, USA.","DOI":"10.1109\/PHM.2008.4711414"},{"key":"ref_41","unstructured":"McCann, M., and Johnston, A. (2021, August 20). SECOM Data Set. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/secom."},{"key":"ref_42","unstructured":"Dashlink (2021, August 20). Li-ion Battery Aging Datasets, Available online: https:\/\/data.nasa.gov\/dataset\/Li-ion-Battery-Aging-Datasets\/uj5r-zjdb."},{"key":"ref_43","unstructured":"Bosch (2021, August 20). Kaggle: Bosch Production Line Performance. Available online: https:\/\/www.kaggle.com\/c\/bosch-production-line-performance."},{"key":"ref_44","unstructured":"Backblaze (2021, August 20). Hard Drive Data and Stats. Available online: https:\/\/www.backblaze.com\/b2\/hard-drive-test-data.html."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Basak, S., Sengupta, S., and Dubey, A. (2019, January 12\u201315). Mechanisms for Integrated Feature Normalization and Remaining Useful Life Estimation Using LSTMs Applied to Hard-Disks. Proceedings of the 2019 IEEE International Conference on Smart Computing (SMARTCOMP), Washington, DC, USA.","DOI":"10.1109\/SMARTCOMP.2019.00055"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Anantharaman, P., Qiao, M., and Jadav, D. (2018, January 11\u201314). Large Scale Predictive Analytics for Hard Disk Remaining Useful Life Estimation. Proceedings of the 2018 IEEE International Congress on Big Data (BigData Congress), Boston, MA, USA.","DOI":"10.1109\/BigDataCongress.2018.00044"},{"key":"ref_47","unstructured":"Basak, S., Sengupta, S., and Dubey, A. (2018). A Data-driven Prognostic Architecture for Online Monitoring of Hard Disks Using Deep LSTM Networks. arXiv."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Su, C.J., and Li, Y. (2019). Recurrent neural network based real-time failure detection of storage devices. Microsyst. Technol.","DOI":"10.1007\/s00542-019-04454-8"},{"key":"ref_49","unstructured":"Dashlink (2021, August 20). Turbofan Engine Degradation Simulation Data Set, Available online: https:\/\/data.nasa.gov\/dataset\/Turbofan-engine-degradation-simulation-data-set\/vrks-gjie."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_51","unstructured":"Witten, I.H., and Frank, E. (2005). Data Mining\u2014Practical Machine Learning Tools and Techniques, Morgan Kaufmann. [2nd ed.]."},{"key":"ref_52","unstructured":"Narasimhan, H., and Agarwal, S. (2013, January 16\u201321). A Structural SVM Based Approach for Optimizing Partial AUC. Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA."},{"key":"ref_53","unstructured":"Dhillon, I.S., Koren, Y., Ghani, R., Senator, T.E., Bradley, P., Parekh, R., He, J., Grossman, R.L., and Uthurusamy, R. (2013, January 11\u201314). SVMpAUCtight: A new support vector method for optimizing partial AUC based on a tight convex upper bound. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1111\/1541-0420.00071","article-title":"Partial AUC estimation and regression","volume":"59 3","author":"Dodd","year":"2003","journal-title":"Biometrics"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1093\/biostatistics\/kxq052","article-title":"Marker selection via maximizing the partial area under the ROC curve of linear risk scores","volume":"12","author":"Wang","year":"2011","journal-title":"Biostatistics"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Ye, W., Lin, Y., Li, M., Liu, Q., and Pan, D.Z. (2019, January 21\u201324). LithoROC: Lithography hotspot detection with explicit ROC optimization. Proceedings of the 24th Annual International Conference on VLSI Design Automation, Tokyo, Japan.","DOI":"10.1145\/3287624.3288746"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"3395","DOI":"10.1016\/j.patcog.2013.06.014","article-title":"ROC curves for regression","volume":"46","year":"2013","journal-title":"Pattern Recognit."}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/8\/4\/68\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:08:18Z","timestamp":1760166498000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/8\/4\/68"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,30]]},"references-count":57,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["informatics8040068"],"URL":"https:\/\/doi.org\/10.3390\/informatics8040068","relation":{},"ISSN":["2227-9709"],"issn-type":[{"type":"electronic","value":"2227-9709"}],"subject":[],"published":{"date-parts":[[2021,9,30]]}}}