{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:07:35Z","timestamp":1760242055866,"version":"build-2065373602"},"reference-count":23,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2018,12,1]],"date-time":"2018-12-01T00:00:00Z","timestamp":1543622400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100004672","name":"Accenture","doi-asserted-by":"publisher","award":["RDF 15\u201002\u201035"],"award-info":[{"award-number":["RDF 15\u201002\u201035"]}],"id":[{"id":"10.13039\/100004672","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Data"],"abstract":"<jats:p>We study big-data hybrid-data-processing lambda architecture, which consolidates low-latency real-time frameworks with high-throughput Hadoop-batch frameworks over a massively distributed setup. In particular, real-time and batch-processing engines act as autonomous multi-agent systems in collaboration. We propose a Multi-Agent Lambda Architecture (MALA) for e-commerce data analytics. We address the high-latency problem of Hadoop MapReduce jobs by simultaneous processing at the speed layer to the requests which require a quick turnaround time. At the same time, the batch layer in parallel provides comprehensive coverage of data by intelligent blending of stream and historical data through the weighted voting method. The cold-start problem of streaming services is addressed through the initial offset from historical batch data. Challenges of high-velocity data ingestion is resolved with distributed message queues. A proposed multi-agent decision-maker component is placed at the MALA stack as the gateway of the data pipeline. We prove efficiency of our batch model by implementing an array of features for an e-commerce site. The novelty of the model and its key significance is a scheme for multi-agent interaction between batch and real-time agents to produce deeper insights at low latency and at significantly lower costs. Hence, the proposed system is highly appealing for applications involving big data and caters to high-velocity streaming ingestion and a massive data pool.<\/jats:p>","DOI":"10.3390\/data3040058","type":"journal-article","created":{"date-parts":[[2018,12,3]],"date-time":"2018-12-03T06:02:09Z","timestamp":1543816929000},"page":"58","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Multi-Agent Big-Data Lambda Architecture Model for E-Commerce Analytics"],"prefix":"10.3390","volume":"3","author":[{"given":"Gautam","family":"Pal","sequence":"first","affiliation":[{"name":"Department of Computer Science, The University of Liverpool, Liverpool L69 7ZX, UK"}]},{"given":"Gangmin","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Xi\u2019an Jiaotong-Liverpool University, Wuzhong 215123, China"}]},{"given":"Katie","family":"Atkinson","sequence":"additional","affiliation":[{"name":"Department of Computer Science, The University of Liverpool, Liverpool L69 7ZX, UK"}]}],"member":"1968","published-online":{"date-parts":[[2018,12,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Xiang, D., Wu, Y., Shang, P., Jiang, J., Wu, J., and Yu, K. (2017, January 9\u201313). RB-storm: Resource balance scheduling in apache storm. Proceedings of the 6th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), Hamamatsu, Japan.","DOI":"10.1109\/IIAI-AAI.2017.63"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Yamato, Y., Kumazaki, H., and Fukumoto, Y. (2015, January 22\u201325). Proposal of lambda architecture adoption for real time predictive maintenance. Proceedings of the Fourth International Symposium on Computing and Networking (CANDAR), Hiroshima, Japan.","DOI":"10.1109\/CANDAR.2016.0130"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kim, H., Madhvanath, S., and Sun, T. (November, January 29). Hybrid active learning for non-stationary streaming data with asynchronous labeling. Proceedings of the 2015 IEEE International Conference on Big Data (Big Data), Santa Clara, CA, USA.","DOI":"10.1109\/BigData.2015.7363766"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Pal, G., Li, G., and Atkinson, K. (2018, January 21\u201325). Big data real time ingestion and machine learning. Proceedings of the 2018 IEEE Second International Conference on Data Stream Mining Processing (DSMP), Lviv, Ukraine.","DOI":"10.1109\/DSMP.2018.8478598"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Lee, C.H., and Lin, C.Y. (2017, January 27\u201329). Implementation of lambda architecture: A restaurant recommender system over apache mesos. Proceedings of the 31st International Conference on Advanced Information Networking and Applications (AINA), Taipei, Taiwan.","DOI":"10.1109\/AINA.2017.63"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Batyuk, A., and Voityshyn, V. (2016, January 23\u201327). Apache storm based on topology for real-time processing of streaming data from social networks. Proceedings of the 2016 IEEE First International Conference on Data Stream Mining & Processing (DSMP), Lviv, Ukraine.","DOI":"10.1109\/DSMP.2016.7583573"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Hanif, M., Yoon, H., Jang, S., and Lee, C. (2017, January 18\u201320). An adaptive SLA-based data flow mechanism for stream processing engines. Proceedings of the 2017 International Conference on Information and Communication Technology Convergence (ICTC), Jeju Island, Korea.","DOI":"10.1109\/ICTC.2017.8190947"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Hu, Y., Koren, Y., and Volinsky, C. (2008, January 15\u201319). Collaborative filtering for implicit feedback datasets. Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy.","DOI":"10.1109\/ICDM.2008.22"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, J., Peng, X., Xing, Z., Fu, K., and Zhao, W. (2017, January 17\u201318). Contextual recommendation of relevant program elements in an interactive feature location process. Proceedings of the 2017 IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM), Shanghai, China.","DOI":"10.1109\/SCAM.2017.14"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1109\/TKDE.2017.2766059","article-title":"A location-query-browse graph for contextual recommendation","volume":"30","author":"Ren","year":"2018","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Rahman, M.M. (2013, January 17\u201318). Contextual recommendation system. Proceedings of the 2013 International Conference on Informatics, Electronics and Vision (ICIEV), Dhaka, Bangladesh.","DOI":"10.1109\/ICIEV.2013.6572542"},{"key":"ref_12","unstructured":"Kharrat, F.B., Elkhleifi, A., and Faiz, R. (December, January 29). Recommendation system based contextual analysis of Facebook comment. Proceedings of the 2016 IEEE\/ACS 13th International Conference of Computer Systems and Applications (AICCSA), Agadir, Morocco."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Domingues, M.A., Sundermann, C.V., Manzato, M.G., Marcacini, R.M., and Rezende, S.O. (2017, January 23\u201326). Exploiting text mining techniques for contextual recommendations. Proceedings of the 2014 IEEE\/WIC\/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), Leipzig, Germany.","DOI":"10.1109\/WI-IAT.2014.100"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1109\/MS.2016.99","article-title":"Exploiting big data\u2019s benefits","volume":"33","author":"Heidrich","year":"2016","journal-title":"IEEE Softw."},{"key":"ref_15","unstructured":"Tyler, A., Reuven, L., and Slava, C. (2018). The What, Where, When, and How of Data Processing. Streaming Systems, O\u2019reilly Media."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Thrun, S. (1996). Explanation-based Neural Network Learning: A Lifelong Learning Approach, Kluwer Academic Publishers.","DOI":"10.1007\/978-1-4613-1381-6"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1109\/TAFFC.2017.2771234","article-title":"Distantly supervised lifelong learning for large-scale social media sentiment analysis","volume":"8","author":"Xia","year":"2017","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Agarwal, K., and Chen, B. (2015). Personalization through feature-based regression. Statistical Methods for Recommender Systems, Cambridge University Press.","DOI":"10.1017\/CBO9781139565868.008"},{"key":"ref_19","unstructured":"(2018, October 27). Clustering\u2014RDD-Based API. Available online: https:\/\/spark.apache.org\/docs\/latest\/mllib-clustering.html."},{"key":"ref_20","first-page":"58","article-title":"Multi-agent Item to Item Contextual Big Data Recommender System","volume":"6","author":"Pal","year":"2017","journal-title":"Int. J. Des. Analysis Tools Integr. Circuit Syst."},{"key":"ref_21","unstructured":"(2018, October 27). Splunk Operational Intelligence Cookbook Second Edition by Packt Publishing. Available online: https:\/\/github.com\/PacktPublishing\/Splunk-Operational-Intelligence-Cookbook-Second-Edition\/find\/master."},{"key":"ref_22","unstructured":"(2018, October 27). Product Details on Flipkart. Available online: https:\/\/data.world\/promptcloud\/product-details-on-flipkart-com."},{"key":"ref_23","unstructured":"Maas, G., and Garillot, F. (2018). Streaming Application Design. Learning Spark Streaming, O\u2019Reilly Media."}],"container-title":["Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5729\/3\/4\/58\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:33:52Z","timestamp":1760196832000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5729\/3\/4\/58"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,12,1]]},"references-count":23,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2018,12]]}},"alternative-id":["data3040058"],"URL":"https:\/\/doi.org\/10.3390\/data3040058","relation":{},"ISSN":["2306-5729"],"issn-type":[{"type":"electronic","value":"2306-5729"}],"subject":[],"published":{"date-parts":[[2018,12,1]]}}}