{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T10:59:22Z","timestamp":1777546762570,"version":"3.51.4"},"reference-count":40,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,22]],"date-time":"2023-01-22T00:00:00Z","timestamp":1674345600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"publisher","award":["957337"],"award-info":[{"award-number":["957337"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>In terms of the calibre and variety of services offered to end users, smart city management is undergoing a dramatic transformation. The parties involved in delivering pervasive applications can now solve key issues in the big data value chain, including data gathering, analysis, and processing, storage, curation, and real-world data visualisation. This trend is being driven by Industry 4.0, which calls for the servitisation of data and products across all industries, including the field of smart cities, where people, sensors, and technology work closely together. In order to implement reactive services such as situational awareness, video surveillance, and geo-localisation while constantly preserving the safety and privacy of affected persons, the data generated by omnipresent devices needs to be processed fast. This paper proposes a modular architecture to (i) leverage cutting-edge technologies for data acquisition, management, and distribution (such as Apache Kafka and Apache NiFi); (ii) develop a multi-layer engineering solution for revealing valuable and hidden societal knowledge in the context of smart cities processing multi-modal, real-time, and heterogeneous data flows; and (iii) address the key challenges in tasks involving complex data flows and offer general guidelines to solve them. In order to create an effective system for the monitoring and servitisation of smart city assets with a scalable platform that proves its usefulness in numerous smart city use cases with various needs, we deduced some guidelines from an experimental setting performed in collaboration with leading industrial technical departments. Ultimately, when deployed in production, the proposed data platform will contribute toward the goal of revealing valuable and hidden societal knowledge in the context of smart cities.<\/jats:p>","DOI":"10.3390\/fi15020043","type":"journal-article","created":{"date-parts":[[2023,1,23]],"date-time":"2023-01-23T03:48:08Z","timestamp":1674445688000},"page":"43","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Engineering Resource-Efficient Data Management for Smart Cities with Apache Kafka"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2906-584X","authenticated-orcid":false,"given":"Theofanis P.","family":"Raptis","sequence":"first","affiliation":[{"name":"Institute of Informatics and Telematics, National Research Council, 56124 Pisa, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4503-4223","authenticated-orcid":false,"given":"Claudio","family":"Cicconetti","sequence":"additional","affiliation":[{"name":"Institute of Informatics and Telematics, National Research Council, 56124 Pisa, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3620-4668","authenticated-orcid":false,"given":"Manolis","family":"Falelakis","sequence":"additional","affiliation":[{"name":"Netcompany-Intrasoft, 190 02 Athens, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Grigorios","family":"Kalogiannis","sequence":"additional","affiliation":[{"name":"Sphynx Technologies Solution AG, 6300 Zug, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tassos","family":"Kanellos","sequence":"additional","affiliation":[{"name":"ITML, 115 25 Athens, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2182-651X","authenticated-orcid":false,"given":"Tom\u00e1s Pariente","family":"Lobo","sequence":"additional","affiliation":[{"name":"Atos Spain, 28037 Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2347","DOI":"10.1109\/COMST.2015.2444095","article-title":"Internet of Things: A Survey on Enabling Technologies, Protocols, and Applications","volume":"17","author":"Guizani","year":"2015","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1109\/JIOT.2017.2683200","article-title":"A Survey on Internet of Things: Architecture, Enabling Technologies, Security and Privacy, and Applications","volume":"4","author":"Lin","year":"2017","journal-title":"IEEE Internet Things J."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2233","DOI":"10.1109\/TII.2014.2300753","article-title":"Internet of Things in Industries: A Survey","volume":"10","author":"Xu","year":"2014","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"678","DOI":"10.1109\/ACCESS.2015.2437951","article-title":"The Internet of Things for Health Care: A Comprehensive Survey","volume":"3","author":"Islam","year":"2015","journal-title":"IEEE Access"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/JIOT.2014.2306328","article-title":"Internet of Things for Smart Cities","volume":"1","author":"Zanella","year":"2014","journal-title":"IEEE Internet Things J."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"103578","DOI":"10.1016\/j.scs.2021.103578","article-title":"Where do Smart Cities grow? The spatial and socio-economic configurations of smart city development","volume":"77","author":"Duygan","year":"2022","journal-title":"Sustain. Cities Soc."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"103794","DOI":"10.1016\/j.cities.2022.103794","article-title":"Future smart cities: Requirements, emerging technologies, applications, challenges, and future aspects","volume":"129","author":"Javed","year":"2022","journal-title":"Cities"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"406","DOI":"10.1109\/JSEN.2013.2282292","article-title":"Sensor Search Techniques for Sensing as a Service Architecture for the Internet of Things","volume":"14","author":"Perera","year":"2014","journal-title":"IEEE Sens. J."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Desai, P., Sheth, A., and Anantharam, P. (July, January 27). Semantic Gateway as a Service Architecture for IoT Interoperability. Proceedings of the 2015 IEEE International Conference on Mobile Services, New York, NY, USA.","DOI":"10.1109\/MobServ.2015.51"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e12753","DOI":"10.1111\/exsy.12753","article-title":"Conceptualizing smart city applications: Requirements, architecture, security issues, and emerging trends","volume":"39","author":"Haque","year":"2022","journal-title":"Expert Syst."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bajovic, D., Bakhtiarnia, A., Bravos, G., Brutti, A., Burkhardt, F., Cauchi, D., Chazapis, A., Cianco, C., Dall\u2019Asen, N., and Delic, V. (2021, January 20\u201322). MARVEL: Multimodal Extreme Scale Data Analytics for Smart Cities Environments. Proceedings of the 2021 International Balkan Conference on Communications and Networking (BalkanCom), Novi Sad, Serbia.","DOI":"10.1109\/BalkanCom53780.2021.9593258"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"138","DOI":"10.1109\/MCOM.2017.1700120","article-title":"Bringing Computation Closer toward the User Network: Is Edge Computing the Solution?","volume":"55","author":"Ahmed","year":"2017","journal-title":"IEEE Commun. Mag."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2961","DOI":"10.1109\/COMST.2018.2849509","article-title":"Survey on Multi-Access Edge Computing for Internet of Things Realization","volume":"20","author":"Porambage","year":"2018","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Raptis, T.P., Cicconetti, C., Falelakis, M., Kanellos, T., and Lobo, T.P. (2022, January 26\u201329). Design Guidelines for Apache Kafka Driven Data Management and Distribution in Smart Cities. Proceedings of the 2022 IEEE International Smart Cities Conference (ISC2), Pafos, Cyprus.","DOI":"10.1109\/ISC255366.2022.9922546"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"4942","DOI":"10.1109\/TWC.2019.2931315","article-title":"HetMEC: Latency-optimal Task Assignment and Resource Allocation for Heterogeneous Multi-layer Mobile Edge Computing","volume":"18","author":"Wang","year":"2019","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Becker, S., Schmidt, F., and Kao, O. (2021, January 28\u201330). EdgePier: P2P-based Container Image Distribution in Edge Computing Environments. Proceedings of the 2021 IEEE International Performance, Computing, and Communications Conference (IPCCC), Austin, TX, USA.","DOI":"10.1109\/IPCCC51483.2021.9679447"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Nicolaescu, A.C., Mastorakis, S., and Psaras, I. (2021, January 10\u201313). Store edge networked data (SEND): A data and performance driven edge storage framework. Proceedings of the IEEE INFOCOM 2021\u2014IEEE Conference on Computer Communications, Vancouver, BC, Canada.","DOI":"10.1109\/INFOCOM42981.2021.9488804"},{"key":"ref_18","unstructured":"Gupta, H., Xu, Z., and Ramachandran, U. (2018, January 11\u201313). DataFog: Towards a Holistic Data Management Platform for the IoT Age at the Network Edge. Proceedings of the 2018 USENIX Annual Technical Conference, Boston, MA, USA. USENIX Workshop on Hot Topics in Edge Computing, HotEdge 2018, co-located with USENIX ATC 2018."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1654","DOI":"10.14778\/2824032.2824063","article-title":"Building a Replicated Logging System with Apache Kafka","volume":"8","author":"Wang","year":"2015","journal-title":"Proc. VLDB Endow."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"39971","DOI":"10.1109\/ACCESS.2020.2976660","article-title":"Research on a Distributed Processing Model Based on Kafka for Large-Scale Seismic Waveform Data","volume":"8","author":"Chai","year":"2020","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Langhi, S., Tommasini, R., and Valle, E.D. (2020, January 10\u201313). Extending Kafka Streams for Complex Event Recognition. Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA.","DOI":"10.1109\/BigData50022.2020.9378217"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Obaidat, M.S., Oren, T., and Rango, F.D. (2022). Simulation and Modeling Methodologies, Technologies and Applications, Springer International Publishing.","DOI":"10.1007\/978-3-030-84811-8"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Javed, M.H., Lu, X., and Panda, D.K.D. (2017, January 5\u20138). Characterization of Big Data Stream Processing Pipeline: A Case Study Using Flink and Kafka. Proceedings of the Fourth IEEE\/ACM International Conference on Big Data Computing, Applications and Technologies, Austin, TX, USA. BDCAT \u201917.","DOI":"10.1145\/3148055.3148068"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Akanbi, A. (2020, January 27\u201329). ESTemd: A Distributed Processing Framework for Environmental Monitoring Based on Apache Kafka Streaming Engine. Proceedings of the 2020 the 4th International Conference on Big Data Research (ICBDR\u201920), Tokyo, Japan. ICBDR 2020.","DOI":"10.1145\/3445945.3445949"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1646","DOI":"10.14778\/3137765.3137771","article-title":"Query-Able Kafka: An Agile Data Analytics Pipeline for Mobile Wireless Networks","volume":"10","author":"Falk","year":"2017","journal-title":"Proc. VLDB Endow."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Raptis, T.P., and Passarella, A. (2022, January 13\u201315). On Efficiently Partitioning a Topic in Apache Kafka. Proceedings of the 2022 International Conference on Computer, Information and Telecommunication Systems (CITS), Athens, Greece.","DOI":"10.1109\/CITS55221.2022.9832981"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sivaraman, E., and Manickachezian, R. (2014, January 6\u20137). High Performance and Fault Tolerant Distributed File System for Big Data Storage and Processing Using Hadoop. Proceedings of the 2014 International Conference on Intelligent Computing Applications, Coimbatore, India.","DOI":"10.1109\/ICICA.2014.16"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"596","DOI":"10.1016\/j.procs.2015.04.091","article-title":"Big Data and Hadoop\u2014A Study in Security Perspective","volume":"50","author":"Saraladevi","year":"2015","journal-title":"Procedia Comput. Sci."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Park, J.J.J.H., Arabnia, H.R., Kim, C., Shi, W., and Gil, J.M. (2013). Grid and Pervasive Computing, Springer.","DOI":"10.1007\/978-3-642-38027-3"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1109\/35.312841","article-title":"Kerberos: An authentication service for computer networks","volume":"32","author":"Neuman","year":"1994","journal-title":"IEEE Commun. Mag."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"772","DOI":"10.18517\/ijaseit.9.3.5721","article-title":"Static Knowledge-Based Authentication Mechanism for Hadoop Distributed Platform using Kerberos","volume":"9","author":"Algaradi","year":"2019","journal-title":"Int. J. Adv. Sci. Eng. Inf. Technol."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Kanyeba, M., and Yu, L. (2016, January 24\u201325). Securing Authentication Within Hadoop. Proceedings of the 2016 International Conference on Electrical, Mechanical and Industrial Engineering, Phuket, Thailand.","DOI":"10.2991\/icemie-16.2016.25"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Smyrlis, M., Somarakis, I., Spanoudakis, G., Hatzivasilis, G., and Ioannidis, S. (2021). CYRA: A Model-Driven CYber Range Assurance Platform. Appl. Sci., 11.","DOI":"10.3390\/app11115165"},{"key":"ref_34","unstructured":"Payne, M. (2022, June 21). Processing One Billion Events per Second with NiFi. Available online: https:\/\/blog.cloudera.com\/benchmarking-nifi-performance-and-scalability\/."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"100452","DOI":"10.1016\/j.cosrev.2021.100452","article-title":"Developing future human-centered smart cities: Critical analysis of smart city security, Data management, and Ethical challenges","volume":"43","author":"Ahmad","year":"2022","journal-title":"Comput. Sci. Rev."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1608","DOI":"10.1109\/JPROC.2019.2918437","article-title":"Edge Computing Security: State of the Art and Challenges","volume":"107","author":"Xiao","year":"2019","journal-title":"Proc. IEEE"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1109\/MS.2020.3028708","article-title":"Serverless Computing-Where Are We Now, and Where Are We Heading?","volume":"38","author":"Taibi","year":"2021","journal-title":"IEEE Softw."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/MIC.2021.3093105","article-title":"Toward Sustainable Serverless Computing","volume":"25","author":"Patros","year":"2021","journal-title":"IEEE Internet Comput."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Vahidinia, P., Farahani, B., and Aliee, F.S. (September, January 31). Cold Start in Serverless Computing: Current Trends and Mitigation Strategies. Proceedings of the 2020 International Conference on Omni-layer Intelligent Systems (COINS), Barcelona, Spain.","DOI":"10.1109\/COINS49042.2020.9191377"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"101689","DOI":"10.1016\/j.pmcj.2022.101689","article-title":"FaaS execution models for edge applications","volume":"86","author":"Cicconetti","year":"2022","journal-title":"Pervasive Mob. Comput."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/15\/2\/43\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:13:46Z","timestamp":1760120026000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/15\/2\/43"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,22]]},"references-count":40,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["fi15020043"],"URL":"https:\/\/doi.org\/10.3390\/fi15020043","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,22]]}}}