{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T03:47:51Z","timestamp":1781754471028,"version":"3.54.5"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T00:00:00Z","timestamp":1597190400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T00:00:00Z","timestamp":1597190400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001808","name":"Kommission f\u00fcr Technologie und Innovation","doi-asserted-by":"publisher","award":["25728.1 PFES-ES"],"award-info":[{"award-number":["25728.1 PFES-ES"]}],"id":[{"id":"10.13039\/501100001808","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Big Data"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Exploiting available condition monitoring data of industrial machines for intelligent maintenance purposes has been attracting attention in various application fields. Machine learning algorithms for fault detection, diagnosis and prognosis are popular and easily accessible. However, our experience in working at the intersection of academia and industry showed that the major challenges of building an end-to-end system in a real-world industrial setting go beyond the design of machine learning algorithms. One of the major challenges is the design of an end-to-end data management solution that is able to efficiently store and process large amounts of heterogeneous data streams resulting from a variety of physical machines. In this paper we present the design of an end-to-end Big Data architecture that enables intelligent maintenance in a real-world industrial setting. In particular, we will discuss various physical design choices for optimizing high-dimensional queries, such as partitioning and Z-ordering, that serve as the basis for health analytics. Finally, we describe a concrete fault detection use case with two different health monitoring algorithms based on machine learning and classical statistics and discuss their advantages and disadvantages. The paper covers some of the most important aspects of the practical implementation of such an end-to-end solution and demonstrates the challenges and their mitigation for the specific application of laser\u00a0cutting machines.<\/jats:p>","DOI":"10.1186\/s40537-020-00340-7","type":"journal-article","created":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T13:03:17Z","timestamp":1597237397000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Big Data architecture for intelligent maintenance: a focus on query processing and machine learning algorithms"],"prefix":"10.1186","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4693-0444","authenticated-orcid":false,"given":"Claude","family":"Lehmann","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lilach","family":"Goren Huber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Horisberger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Georg","family":"Scheiba","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ana Claudia","family":"Sima","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kurt","family":"Stockinger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,8,12]]},"reference":[{"key":"340_CR1","unstructured":"Sima A-C, Stockinger K, Affolter K, Braschler M, Monte P, Kaiser L. A hybrid approach for alarm verification using stream processing, machine learning and text analytics. In: EDBT 2018, Vienna, Austria, 26-29 March 2018. ACM; 2018."},{"key":"340_CR2","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.procir.2015.08.026","volume":"38","author":"J Lee","year":"2015","unstructured":"Lee J, Ardakani HD, Yang S, Bagheri B. Industrial big data analytics and cyber-physical systems for future maintenance & service innovation. Procedia Cirp. 2015;38:3\u20137.","journal-title":"Procedia Cirp"},{"key":"340_CR3","doi-asserted-by":"crossref","unstructured":"Canizo M, Onieva E, Conde A, Charramendieta S, Trujillo S. Real-time predictive maintenance for wind turbines using big data frameworks. In: 2017 IEEE International Conference on Prognostics and Health Management (ICPHM). IEEE; 2017. p. 70\u20137.","DOI":"10.1109\/ICPHM.2017.7998308"},{"issue":"9","key":"340_CR4","doi-asserted-by":"publisher","first-page":"2946","DOI":"10.3390\/s18092946","volume":"18","author":"M Syafrudin","year":"2018","unstructured":"Syafrudin M, Alfian G, Fitriyani NL, Rhee J. Performance analysis of iot-based sensor, big data processing, and machine learning model for real-time monitoring system in automotive manufacturing. Sensors. 2018;18(9):2946.","journal-title":"Sensors"},{"issue":"2","key":"340_CR5","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1145\/356770.356776","volume":"11","author":"D Comer","year":"1979","unstructured":"Comer D. Ubiquitous b-tree. ACM Comput Surv. 1979;11(2):121\u201337.","journal-title":"ACM Comput Surv"},{"issue":"2","key":"340_CR6","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1145\/152610.152611","volume":"25","author":"G Graefe","year":"1993","unstructured":"Graefe G. Query evaluation techniques for large databases. ACM Comput Surv. 1993;25(2):73\u2013169.","journal-title":"ACM Comput Surv"},{"issue":"1","key":"340_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/BF00288933","volume":"4","author":"RA Finkel","year":"1974","unstructured":"Finkel RA, Bentley JL. Quad trees a data structure for retrieval on composite keys. Acta Informatica. 1974;4(1):1\u20139.","journal-title":"Acta Informatica"},{"issue":"9","key":"340_CR8","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1145\/361002.361007","volume":"18","author":"JL Bentley","year":"1975","unstructured":"Bentley JL. Multidimensional binary search trees used for associative searching. Commun ACM. 1975;18(9):509\u201317.","journal-title":"Commun ACM"},{"key":"340_CR9","unstructured":"Guttman A. R-trees: A dynamic index structure for spatial searching. In: Proceedings of the 1984 ACM SIGMOD International Conference on Management of Data, 1984, pp. 47\u201357."},{"issue":"1","key":"340_CR10","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1007\/BF01212696","volume":"38","author":"D Hilbert","year":"1891","unstructured":"Hilbert D. Ueber die reellen z\u00fcge algebraischer curven. Mathematische Annalen. 1891;38(1):115\u201338.","journal-title":"Mathematische Annalen"},{"key":"340_CR11","unstructured":"Morton GM. A computer oriented geodetic data base and a new technique in file sequencing, 1966."},{"issue":"3","key":"340_CR12","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1016\/0306-4573(85)90108-6","volume":"21","author":"I Spiegler","year":"1985","unstructured":"Spiegler I, Maayan R. Storage and retrieval considerations of binary data bases. Inform Process Manag. 1985;21(3):233\u201354.","journal-title":"Inform Process Manag"},{"issue":"1","key":"340_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1670243.1670245","volume":"35","author":"K Wu","year":"2008","unstructured":"Wu K, Shoshani A, Stockinger K. Analyses of multi-level and multi-component compressed bitmap indexes. ACM Trans Database Syst. 2008;35(1):1\u201352.","journal-title":"ACM Trans Database Syst"},{"key":"340_CR14","doi-asserted-by":"crossref","unstructured":"Nathan V, Ding J, Alizadeh M, Kraska T. Learning Multi-dimensional Indexes 2019. arxiv:1912.01668.","DOI":"10.1145\/3318464.3380579"},{"key":"340_CR15","unstructured":"MongoDB Documentation: Indexes. https:\/\/docs.mongodb.com\/manual\/indexes\/. Accessed 13 Feb 2020."},{"key":"340_CR16","unstructured":"Neo4j Documentation: Index configuration. https:\/\/neo4j.com\/docs\/operations-manual\/current\/performance\/index-configuration\/#index-configuration-btree. Accessed 13 Feb 2020."},{"key":"340_CR17","unstructured":"Apache Hive Confluence: LanguageManual Indexing. https:\/\/cwiki.apache.org\/confluence\/display\/Hive\/LanguageManual+Indexing. Accessed 13 Feb 2020."},{"key":"340_CR18","unstructured":"PostgreSQL Documentation: Combining Multiple Indexes. https:\/\/www.postgresql.org\/docs\/10\/indexes-bitmap-scans.html. Accessed 13 Feb 2020."},{"key":"340_CR19","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1016\/j.datak.2017.09.004","volume":"112","author":"K Stockinger","year":"2017","unstructured":"Stockinger K, B\u00f6di R, Heitz J, Weinmann T. Zns-efficient query processing with zurichnosql. Data Knowl Eng. 2017;112:38\u201354.","journal-title":"Data Knowl Eng"},{"key":"340_CR20","unstructured":"Amazon AWS: Amazon Redshift Engineering\u2019s Advanced Table Design Playbook: Compound and Interleaved Sort Keys. https:\/\/aws.amazon.com\/de\/blogs\/big-data\/amazon-redshift-engineerings-advanced-table-design-playbook-compound-and-interleaved-sort-keys. Accessed 30 Feb 2020."},{"key":"340_CR21","unstructured":"Amazon AWS Database Blog: Z-Order Indexing for Multifaceted Queries in Amazon DynamoDB: Part 1. https:\/\/aws.amazon.com\/de\/blogs\/database\/z-order-indexing-for-multifaceted-queries-in-amazon-dynamodb-part-1. Accessed 30 Jan 2020."},{"key":"340_CR22","unstructured":"Databricks Engineering Blog: Optimize Performance with File Management. https:\/\/docs.databricks.com\/delta\/optimizations\/file-mgmt.html. Accessed 28 Nov 2019."},{"key":"340_CR23","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1016\/j.ymssp.2017.11.024","volume":"107","author":"S Khan","year":"2018","unstructured":"Khan S, Yairi T. A review on the application of deep learning in system health management. Mech Syst Sign Process. 2018;107:241\u201365.","journal-title":"Mech Syst Sign Process"},{"key":"340_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.measurement.2017.07.017","volume":"111","author":"L Jing","year":"2017","unstructured":"Jing L, Zhao M, Li P, Xu X. A convolutional neural network based feature learning and fault diagnosis method for the condition monitoring of gearbox. Measurement. 2017;111:1\u201310.","journal-title":"Measurement"},{"key":"340_CR25","doi-asserted-by":"crossref","unstructured":"Chen Z, Li C, Sanchez R-V. Gearbox fault identification and classification with convolutional neural networks. Shock Vibr. 2015;2015.","DOI":"10.1155\/2015\/390134"},{"key":"340_CR26","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1016\/j.ymssp.2017.06.022","volume":"100","author":"W Zhang","year":"2018","unstructured":"Zhang W, Li C, Peng G, Chen Y, Zhang Z. A deep convolutional neural network with new training methods for bearing fault diagnosis under noisy environment and different working load. Mech Syst Sign Process. 2018;100:439\u201353.","journal-title":"Mech Syst Sign Process"},{"issue":"6","key":"340_CR27","doi-asserted-by":"publisher","first-page":"4788","DOI":"10.1109\/TIE.2018.2864702","volume":"66","author":"C-L Liu","year":"2018","unstructured":"Liu C-L, Hsaio W-H, Tu Y-C. Time series classification with multivariate convolutional neural network. IEEE Trans Ind Electr. 2018;66(6):4788\u201397.","journal-title":"IEEE Trans Ind Electr"},{"key":"340_CR28","doi-asserted-by":"crossref","unstructured":"Bellatreche L, Boukhalfa K, Richard P. Data partitioning in data warehouses: Hardness study, heuristics and oracle validation. In: International Conference on Data Warehousing and Knowledge Discovery. Springer: New York; 2008. pp. 87\u201396.","DOI":"10.1007\/978-3-540-85836-2_9"},{"key":"340_CR29","doi-asserted-by":"crossref","unstructured":"Stockinger K, Wu K. Bitmap indices for data warehouses. In: Data Warehouses and OLAP: Concepts, Architectures and Solutions. IGI Global, 2007. p. 157\u201378.","DOI":"10.4018\/987-1-59904-364-7.ch007"},{"key":"340_CR30","unstructured":"Ionescu A. Processing Petabytes of Data in Seconds with Databricks Delta. https:\/\/databricks.com\/blog\/2018\/07\/31\/processing-petabytes-of-data-in-seconds-with-databricks-delta.html. Accessed 28 Nov 2019."},{"key":"340_CR31","unstructured":"Bai S, Kolter JZ, Koltun V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 2018."}],"container-title":["Journal of Big Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-020-00340-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s40537-020-00340-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-020-00340-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,6]],"date-time":"2022-11-06T18:01:42Z","timestamp":1667757702000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-020-00340-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,12]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["340"],"URL":"https:\/\/doi.org\/10.1186\/s40537-020-00340-7","relation":{},"ISSN":["2196-1115"],"issn-type":[{"value":"2196-1115","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,12]]},"assertion":[{"value":"15 April 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 July 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 August 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare that they have no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"61"}}