{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T11:06:44Z","timestamp":1777028804820,"version":"3.51.4"},"reference-count":98,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T00:00:00Z","timestamp":1730764800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Big Data"],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>The ubiquity of digital devices, the infrastructure of today, and the ever-increasing proliferation of digital products have dawned a new era, the era of big data (BD). This era began when the volume, variety, and velocity of data overwhelmed traditional systems that used to analyze and store that data. This precipitated a new class of software systems, namely, BD systems. Whereas BD systems provide a competitive advantage to businesses, many have failed to harness the power of them. It has been estimated that only 20% of companies have successfully implemented a BD project.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>This study aims to facilitate BD system development by introducing Cybermycelium, a domain-driven decentralized BD reference architecture (RA). The artifact was developed following the guidelines of empirically grounded RAs and evaluated through implementation in a real-world scenario using the Architecture Tradeoff Analysis Method (ATAM).<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The evaluation revealed that Cybermycelium successfully addressed key architectural qualities: performance (achieving &amp;lt;1,000 ms response times), availability (through event brokers and circuit breaking), and modifiability (enabling rapid service deployment and configuration). The prototype demonstrated effective handling of data processing, scalability challenges, and domain-specific requirements in a large-scale international company setting.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>The results highlight important architectural trade-offs between event backbone implementation and service mesh design. While the domain-driven distributed approach improved scalability and maintainability compared to traditional monolithic architectures, it requires significant technical expertise for implementation. This contribution advances the field by providing a validated reference architecture that addresses the challenges of adopting BD in modern enterprises.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fdata.2024.1448481","type":"journal-article","created":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T04:42:23Z","timestamp":1730781743000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Cybermycelium: a reference architecture for domain-driven distributed big data systems"],"prefix":"10.3389","volume":"7","author":[{"given":"Pouya","family":"Ataei","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,11,5]]},"reference":[{"key":"B1","article-title":"Software engineering body of knowledge","volume-title":"IEEE Computer Society","author":"Abran","year":"2004"},{"key":"B2","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511814105","volume-title":"Reactive Systems: Modelling, Specification and Verification","author":"Aceto","year":"2007"},{"key":"B3","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1109\/SOCA.2016.15","article-title":"\u201cA systematic mapping study in microservice architecture,\u201d","volume-title":"2016 IEEE 9th International Conference on Service-Oriented Computing and Applications (SOCA)","author":"Alshuqayran","year":"2016"},{"key":"B4","volume-title":"The Age of Analytics: Competing in a Data-driven World","author":"Analytics","year":"2016"},{"key":"B5","doi-asserted-by":"publisher","first-page":"1816","DOI":"10.1016\/j.jss.2008.02.023","article-title":"An e-contracting reference architecture","volume":"81","author":"Angelov","year":"2008","journal-title":"J. Syst. Softw"},{"key":"B6","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1109\/WICSA.2009.5290800","article-title":"\u201cA classification of software reference architectures: analyzing their success and effectiveness,\u201d","author":"Angelov","year":"2009","journal-title":"2009 Joint Working IEEE\/IFIP Conference on Software Architecture"},{"key":"B7","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1007\/978-3-540-88030-1_17","article-title":"\u201cTowards a method for the evaluation of reference architectures: experiences from a case,\u201d","volume-title":"European Conference on Software Architecture","author":"Angelov","year":"2008"},{"key":"B8","unstructured":"Apache Beam Programming Guide2022"},{"key":"B9","unstructured":"AtaeiP.\n          Cybermycelium Templates and Scripts2023"},{"key":"B10","doi-asserted-by":"crossref","first-page":"452","DOI":"10.1109\/APSEC53868.2021.00052","article-title":"\u201cNeomycelia: a software reference architecturefor big data systems,\u201d","volume-title":"2021 28th Asia-Pacific Software Engineering Conference (APSEC)","author":"Ataei","year":"2021"},{"key":"B11","doi-asserted-by":"publisher","first-page":"113789","DOI":"10.1109\/ACCESS.2022.3217557","article-title":"The state of big data reference architectures: a systematic literature review","volume":"10","author":"Ataei","year":"2022","journal-title":"IEEE Access"},{"key":"B12","article-title":"\u201cTowards a domain-driven distributed reference architecture for big data systems,\u201d","author":"Ataei","year":"2023","journal-title":"Proceedings of the 29th Americas Conference on Information Systems (AMCIS 2023)"},{"key":"B13","article-title":"\u201cBig data reference architectures, a systematic literature review,\u201d","volume-title":"Proceedings of the Australasian Conference on Information Systems","author":"Ataei","year":"2020"},{"key":"B14","unstructured":"\u201cImpact of big data analytics on business performance: A systematic literature review,\u201d\n            AtaeiP.\n            RegulaS.\n            HaertelC.\n            StaegemannD.\n          MCIS 2024 Proceedings2024"},{"key":"B15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-023-00733-4","article-title":"Application of microservices patterns to big data systems","volume":"10","author":"Ataei","year":"2023","journal-title":"J. Big Data"},{"key":"B16","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1016\/j.intcom.2010.07.003","article-title":"Socio-technical systems: From design methods to systems engineering","volume":"23","author":"Baxter","year":"2011","journal-title":"Interact. Comput"},{"key":"B17","volume-title":"Definition of reference architectures based on existing systems","author":"Bayer","year":"2004"},{"key":"B18","volume-title":"Building Event-Driven Microservices: Leveraging Organizational Data at Scale","author":"Bellemare","year":"2020"},{"key":"B19","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1016\/S0164-1212(03)00080-3","article-title":"Architecture-level modifiability analysis (ALMA)","volume":"69","author":"Bengtsson","year":"2004","journal-title":"J. Syst. Softw"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1145\/3209900.3209911","article-title":"Draining the data swamp: A similarity-based approach","author":"Brackenbury","year":"2018","journal-title":"In Proceedings of the workshop on human-in-the-loop data analytics, pages"},{"key":"B21","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1186\/s40537-015-0014-3","article-title":"Big data, big bang?","volume":"3","author":"Bughin","year":"2016","journal-title":"J. Big Data"},{"key":"B22","volume-title":"Information Management and Big Data, A Reference Architecture","author":"Cackett","year":"2013"},{"key":"B23","author":"Chang","year":"2018","journal-title":"NIST Big Data Interoperability Framework: Volume 6, Big Data Reference Architecture (NIST SP 1500-6r2)."},{"key":"B24","author":"Chang","year":"2015","journal-title":"Nist Big Data Interoperability Framework: Volume 5, Architectures White Paper Survey"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.2307\/41703503","article-title":"Business intelligence and analytics: from big data to big impact","author":"Chen","year":"2012","journal-title":"MIS Q. 36"},{"key":"B26","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1145\/3344948.3344973","article-title":"\u201cTowards creation of a reference architecture for trust-based digital ecosystems,\u201d","volume-title":"Proceedings of the 13th European Conference on Software Architecture-Volume 2","author":"Cioroaica","year":"2019"},{"key":"B27","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1002\/sys.20129","article-title":"The concept of reference architectures","volume":"13","author":"Cloutier","year":"2010","journal-title":"Syst. Eng"},{"key":"B28","unstructured":"Databricks Platform Documentation2022"},{"key":"B29","unstructured":"DehghaniZ.\n          How to move beyond a monolithic data lake to a distributed data mesh2019"},{"key":"B30","unstructured":"DehghaniZ.\n          Data mesh principles and logical architecture2020"},{"key":"B31","volume-title":"Data Mesh: Delivering Data-Driven Value at Scale","author":"Dehghani","year":"2022"},{"key":"B32","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1109\/ICSAW.2017.65","article-title":"\u201cArchitecting microservices,\u201d","volume-title":"2017 IEEE International Conference on Software Architecture Workshops (ICSAW)","author":"Di Francesco","year":"2017"},{"key":"B33","volume-title":"Augmented data catalogs: Now an enterprise must-have for data and analytics leaders","author":"Ehtisham Zaidi","year":"2019"},{"key":"B34","volume-title":"Domain-driven design: tackling complexity in the heart of software","author":"Evans","year":"2004"},{"key":"B35","first-page":"153","article-title":"\u201cEmpirically-grounded reference architectures: a proposal,\u201d","volume-title":"Proceedings of the joint ACM SIGSOFT conference-QoSA and ACM SIGSOFT symposium-ISARCS on Quality of software architectures-QoSA and architecting critical systems-ISARCS","author":"Galster","year":"2011"},{"key":"B36","year":"2021","journal-title":"Top trends in data and analytics for 2021"},{"key":"B37","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1109\/MS.2014.51","article-title":"Distribution, data, deployment: Software architecture convergence in big data systems","volume":"32","author":"Gorton","year":"2015","journal-title":"IEEE Software"},{"key":"B38","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1109\/CSMR.2005.28","article-title":"\u201cEvaluating an embedded software reference architecture-industrial experience report,\u201d","volume-title":"Ninth European Conference on Software Maintenance and Reengineering","author":"Graaf","year":"2005"},{"key":"B39","article-title":"The hunt is on for the renaissance man of computing","author":"Guest","year":"1991","journal-title":"The Independent"},{"key":"B40","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1002\/smr.491","article-title":"Capturing variability in business process models: the provop approach","volume":"22","author":"Hallerbach","year":"2010","journal-title":"J. Soft. Maint. Evol. Res. Pract"},{"key":"B41","unstructured":"Terraform Documentation2022"},{"key":"B42","unstructured":"36515465The package manager for kubernetes2022"},{"key":"B43","year":"2017","journal-title":"ISO\/IEC\/IEEE 42010:2011"},{"key":"B44","volume-title":"ISO\/IEC 25000:2005 Software engineering\u2014Software product Quality Requirements and Evaluation (SQuaRE).","year":"2014"},{"key":"B45","volume-title":"ISO 19115-1:2014 Geographic information\u2014Metadata.","year":"2019"},{"key":"B46","year":"2018","journal-title":"Systems and software engineering\u2014Life cycle processes\u2014Requirements engineering."},{"key":"B47","volume-title":"TOGAF","author":"Josey","year":"2016"},{"key":"B48","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1007\/s11334-014-0232-4","article-title":"State of practice in requirements engineering: contemporary data","volume":"10","author":"Kassab","year":"2014","journal-title":"Innov. Syst. Softw. Eng"},{"key":"B49","first-page":"81","article-title":"\u201cSaam: a method for analyzing the properties of software architectures,\u201d","volume-title":"Proceedings of 16th International Conference on Software Engineering","author":"Kazman","year":"1994"},{"key":"B50","doi-asserted-by":"crossref","DOI":"10.21236\/ADA350761","author":"Kazman","year":"1998","journal-title":"The architecture tradeoff analysis method"},{"key":"B51","volume-title":"Learning Domain-Driven Design","author":"Khononov","year":"2021"},{"key":"B52","doi-asserted-by":"crossref","first-page":"2785","DOI":"10.1109\/BigData.2015.7364082","article-title":"\u201cLambda architecture for cost-effective batch and speed big data processing,\u201d","volume-title":"2015 IEEE International Conference on Big Data (Big Data)","author":"Kiran","year":"2015"},{"key":"B53","first-page":"51","article-title":"\u201cA reference architecture for big data systems in the national security domain,\u201d","volume-title":"2016 IEEE\/ACM 2nd International Workshop on Big Data Software Engineering (BIGDSE)","author":"Klein","year":"2016"},{"key":"B54","doi-asserted-by":"crossref","first-page":"38","DOI":"10.4018\/978-1-5225-6207-8.ch003","article-title":"\u201cTowards a secure, distributed, and reliable cloud-based reference architecture for big data in smart cities,\u201d","volume-title":"Big Data Analytics for Smart and Connected Cities","author":"Kohler","year":"2019"},{"key":"B55","volume-title":"Leading Change","author":"Kotter","year":"1996"},{"key":"B56","article-title":"Questioning the lambda architecture","author":"Kreps","year":"2014","journal-title":"Online article, July"},{"key":"B57","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1007\/s10270-008-0090-3","article-title":"Questionnaire-based variability modeling for system configuration","volume":"8","author":"La Rosa","year":"2009","journal-title":"Softw. Syst. Model"},{"key":"B58","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2103.00170","article-title":"Data management in microservices: state of the practice, challenges, and research directions","author":"Laigner","year":"2021","journal-title":"arXiv"},{"key":"B59","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1007\/978-3-642-29651-2_5","article-title":"\u201cA language for enterprise modelling,\u201d","volume-title":"Enterprise Architecture at Work","author":"Lankhorst","year":"2013"},{"key":"B60","volume-title":"Requirements engineering for software and systems","author":"Laplante","year":"2017"},{"key":"B61","year":"2021","journal-title":"Software Architecture in Practice (SEI Series in Software Engineering)"},{"key":"B62","volume-title":"Design and Implementation of an Enterprise Data Warehouse","author":"Leonard","year":"2011"},{"key":"B63","volume-title":"Big data ecosystem reference architecture","author":"Levin","year":"2013"},{"key":"B64","volume-title":"Data as a product vs data as a service","author":"Levy","year":"2021"},{"key":"B65","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1109\/MIC.2017.3481351","article-title":"The lambda and the kappa","volume":"21","author":"Lin","year":"2017","journal-title":"IEEE Internet Comput"},{"key":"B66","first-page":"3","article-title":"\u201cPhi: a generic microservices-based big data architecture,\u201d","volume-title":"European, Mediterranean, and Middle Eastern Conference on Information Systems","author":"Maamouri","year":"2021"},{"key":"B67","volume-title":"Towards a big data reference architecture","author":"Maier","year":"2013"},{"key":"B68","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1016\/j.infsof.2017.06.001","article-title":"A software reference architecture for semantic-aware big data systems","volume":"90","author":"Nadal","year":"2017","journal-title":"Inf. Softw. Technol"},{"key":"B69","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1109\/WICSA.2014.25","article-title":"\u201cConsolidating a process for the design, representation, and evaluation of reference architectures,\u201d","volume-title":"2014 IEEE\/IFIP Conference on Software Architecture","author":"Nakagawa","year":"2014"},{"key":"B70","article-title":"\u201cTowards a process to design aspect-oriented reference architectures,\u201d","volume-title":"Proceedings of CLEI 2009","author":"Nakagawa","year":"2009"},{"key":"B71","volume-title":"Cio Survey 2015","author":"Nash","year":"2015"},{"key":"B72","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1016\/j.bdr.2015.01.001","article-title":"Reference architecture and classification of technologies, products and services for big data systems","volume":"2","author":"P\u00e4\u00e4kk\u00f6nen","year":"2015","journal-title":"Big Data Res"},{"key":"B73","volume-title":"Big data and AI executive survey 2021","author":"Partners","year":"2021"},{"key":"B74","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1007\/s10796-016-9720-4","article-title":"The impact of big data analytics on firms' high value business performance","volume":"20","author":"Popovi\u010d","year":"2018","journal-title":"Inf. Syst. Front"},{"key":"B75","author":"Quintero","year":"2019","journal-title":"IBM Reference Architecture for High Performance Data and AI in Healthcare and Life Sciences"},{"key":"B76","first-page":"38","article-title":"The big data ecosystem and its environs","volume":"17","author":"Rad","year":"2017","journal-title":"Int. J. Comput. Sci. Netw. Secur"},{"key":"B77","first-page":"525","article-title":"The hype of emerging technologies: Big data as a service","volume":"9","author":"Rada","year":"2017","journal-title":"Int. J. Control Theory Appl"},{"key":"B78","volume-title":"Microservices Patterns: With examples in Java, 1st Edn","author":"Richardson","year":"2018"},{"key":"B79","author":"Rogers","year":"2003","journal-title":"Diffusion of Innovations"},{"key":"B80","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1007\/s11334-019-00322-w","article-title":"A reference architecture for satellite control systems","volume":"15","author":"Rohling","year":"2019","journal-title":"Innov. Syst. Softw. Eng"},{"key":"B81","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.is.2005.05.003","article-title":"A configurable reference modelling language","volume":"32","author":"Rosemann","year":"2007","journal-title":"Inf. Syst"},{"key":"B82","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1007\/s12599-017-0511-3","article-title":"Representing variability in enterprise architecture","volume":"61","author":"Rurua","year":"2019","journal-title":"Bus. Inf. Syst. Eng"},{"key":"B83","unstructured":"The leading customer data platform2022"},{"key":"B84","volume-title":"Deciphering Data Architectures: From Data Warehouse to Data Lakehouse","author":"Serra","year":"2024"},{"key":"B85","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.compind.2019.02.007","article-title":"An industrial evaluation of an industry 4.0 reference architecture demonstrating the need for the inclusion of security and human components","volume":"108","author":"Sharpe","year":"2019","journal-title":"Comput. Ind"},{"key":"B86","volume-title":"Accelerating Secondary Genome Analysis Using Intel Big Data Reference Architecture","author":"Sikora-Wohlfeld","year":"2014"},{"key":"B87","volume-title":"Software Engineering, 9\/E","author":"Sommerville","year":"2011"},{"key":"B88","first-page":"149","article-title":"\u201cCreating a reference architecture for service-based systems-a pattern-based approach,\u201d","volume-title":"Towards the Future Internet","author":"Stricker","year":""},{"key":"B89","first-page":"149","article-title":"\u201cCreating a reference architecture for service-based systems-a pattern-based approach,\u201d","author":"Stricker","year":"","journal-title":"Future Internet Assembly"},{"key":"B90","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1108\/17410391211272829","article-title":"A model of critical success factors for software projects","volume":"25","author":"Sudhakar","year":"2012","journal-title":"J. Enterp. Inf. Manag"},{"key":"B91","author":"Suthakar","year":"2017","journal-title":"A scalable data store and analytic platform for real-time monitoring of data-intensive scientific infrastructure"},{"key":"B92","author":"Technology Review Insights in Partnership with Databricks","year":"2021","journal-title":"Building a high-performance data organization"},{"key":"B93","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110798","article-title":"A systematic mapping study on microservices architecture in devops","author":"Waseem","year":"2020","journal-title":"J. Syst. Softw"},{"key":"B94","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1109\/MS.2016.20","article-title":"Reference architectures for the internet of things","volume":"33","author":"Weyrich","year":"2015","journal-title":"IEEE Softw"},{"key":"B95","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-43839-8","author":"Wieringa","year":"2014","journal-title":"Design science Methodology for Information Systems and Software Engineering"},{"key":"B96","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1145\/584369.584397","article-title":"\u201cPasasm: a method for the performance assessment of software architectures,\u201d","volume-title":"Proceedings of the 3rd international workshop on Software and performance","author":"Williams","year":"2002"},{"key":"B97","first-page":"495","article-title":"\u201cSalt: Combining ACID and BASE in a distributed database,\u201d","author":"Xie","year":"2014","journal-title":"Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI '14)"},{"key":"B98","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1007\/s00450-016-0337-0","article-title":"Microservices tenets","volume":"32","author":"Zimmermann","year":"2017","journal-title":"Comput. Sci.-Res. Dev"}],"container-title":["Frontiers in Big Data"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2024.1448481\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T04:42:49Z","timestamp":1730781769000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2024.1448481\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,5]]},"references-count":98,"alternative-id":["10.3389\/fdata.2024.1448481"],"URL":"https:\/\/doi.org\/10.3389\/fdata.2024.1448481","relation":{},"ISSN":["2624-909X"],"issn-type":[{"value":"2624-909X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,5]]},"article-number":"1448481"}}