{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T15:42:00Z","timestamp":1784302920322,"version":"3.55.0"},"reference-count":56,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2020,11,20]],"date-time":"2020-11-20T00:00:00Z","timestamp":1605830400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["815178"],"award-info":[{"award-number":["815178"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The high heterogeneity of 5G use cases requires the extension of the traditional per-component testing procedures provided by certification organizations, in order to devise and incorporate methodologies that cover the testing requirements from vertical applications and services. In this paper, we introduce an experimentation methodology that is defined in the context of the 5GENESIS project, which aims at enabling both the testing of network components and validation of E2E KPIs. The most important contributions of this methodology are its modularity and flexibility, as well as the open-source software that was developed for its application, which enable lightweight adoption of the methodology in any 5G testbed. We also demonstrate how the methodology can be used, by executing and analyzing different experiments in a 5G Non-Standalone (NSA) deployment at the University of Malaga. The key findings of the paper are an initial 5G performance assessment and KPI analysis and the detection of under-performance issues at the application level. Those findings highlight the need for reliable testing and validation procedures towards a fair benchmarking of generic 5G services and applications.<\/jats:p>","DOI":"10.3390\/s20226652","type":"journal-article","created":{"date-parts":[[2020,11,20]],"date-time":"2020-11-20T09:46:18Z","timestamp":1605865578000},"page":"6652","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["A Modular Experimentation Methodology for 5G Deployments: The 5GENESIS Approach"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1226-6135","authenticated-orcid":false,"given":"Almudena","family":"D\u00edaz Zayas","sequence":"first","affiliation":[{"name":"ITIS Software, Universidad de M\u00e1laga, Andaluc\u00eda Tech, 29071 M\u00e1laga, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0611-5637","authenticated-orcid":false,"given":"Giuseppe","family":"Caso","sequence":"additional","affiliation":[{"name":"Simula Metropolitan Center for Digital Engineering, Pilestredet 52, 0167 Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5800-2779","authenticated-orcid":false,"given":"\u00d6zg\u00fc","family":"Alay","sequence":"additional","affiliation":[{"name":"Simula Metropolitan Center for Digital Engineering, Pilestredet 52, 0167 Oslo, Norway"},{"name":"Deparment of Informatics, University of Oslo, 0315 Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2456-4946","authenticated-orcid":false,"given":"Pedro","family":"Merino","sequence":"additional","affiliation":[{"name":"ITIS Software, Universidad de M\u00e1laga, Andaluc\u00eda Tech, 29071 M\u00e1laga, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7311-9334","authenticated-orcid":false,"given":"Anna","family":"Brunstrom","sequence":"additional","affiliation":[{"name":"Deparment of Mathematics and Computer Science, Karlstad University, 651 88 Karlstad, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0301-3950","authenticated-orcid":false,"given":"Dimitris","family":"Tsolkas","sequence":"additional","affiliation":[{"name":"Fogus Innovations &amp; Services, 161 21 Kesariani, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0486-2700","authenticated-orcid":false,"given":"Harilaos","family":"Koumaras","sequence":"additional","affiliation":[{"name":"NCSR Demokritos, Institute of Informatics and Telecommunications, 153 41 Paraskevi, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,11,20]]},"reference":[{"key":"ref_1","unstructured":"ITU-R M.2083-0 (2015). IMT Vision-Framework and Overall Objectives of the Future Development of IMT for 2020 and Beyond, ITU."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1201","DOI":"10.1109\/JSAC.2017.2692307","article-title":"5G: A Tutorial Overview of Standards, Trials, Challenges, Deployment, and Practice","volume":"35","author":"Shafi","year":"2017","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_3","unstructured":"(2020, September 30). 5G-PPP Test, Measurement and KPIs Validation Working Group, Validating 5G Technology Performance\u2014Assessing 5G Architecture and Application Scenarios, 5G-PPP White Paper. Available online: https:\/\/5g-ppp.eu\/wp-content\/uploads\/2019\/06\/TMV-White-Paper-V1.1-25062019.pdf."},{"key":"ref_4","unstructured":"Foegelle, M.D. (April, January 31). Testing the 5G New Radio. Proceedings of the 2019 13th European Conference on Antennas and Propagation (EuCAP\u201919), Krakow, Poland."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Koumaras, H., Tsolkas, D., Gardikis, G., Gomez, P.M., Frascolla, V., Triantafyllopoulou, D., Emmelmann, M., Koumaras, V., Osma, M.L.G., and Munaretto, D. (2019, January 17\u201319). 5GENESIS: The Genesis of a flexible 5G Facility. Proceedings of the 2018 23rd International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD\u201919), Barcelona, Spain.","DOI":"10.1109\/CAMAD.2018.8514956"},{"key":"ref_6","unstructured":"5GENESIS Project (2020, September 30). Deliverable D2.4 Final Report on Facility Design and Experimentation Planning. Available online: https:\/\/5genesis.eu\/wp-content\/uploads\/2020\/07\/5GENESIS_D2.4_v1.0.pdf."},{"key":"ref_7","unstructured":"5GENESIS Project (2020, September 30). Deliverable D3.5 Monitoring and Analytics (Release A). Available online: https:\/\/5genesis.eu\/wp-content\/uploads\/2019\/10\/5GENESIS_D3.5_v1.0.pdf."},{"key":"ref_8","unstructured":"(2020, September 30). Open 5GENESIS Suite. Available online: https:\/\/github.com\/5genesis."},{"key":"ref_9","unstructured":"(2020). 3GPP TS 38.521-3; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; User Equipment (UE) Conformance Specification; Radio Transmission and Reception; Part 3: Range 1 and Range 2 Interworking Operation with Other Radios (Release 16), v16.4, 3GPP."},{"key":"ref_10","unstructured":"(2020). 3GPP TS 38.521-1; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; User Equipment (UE) Conformance Specification; Radio Transmission and Reception; Part 1: Range 1 Standalone (Release 16), v16.4, 3GPP."},{"key":"ref_11","unstructured":"(2020). 3GPP TS 38.521-2; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; User Equipment (UE) Conformance Specification; Radio Transmission and Reception; Part 2: Range 2 Standalone (Release 16), v16.4, 3GPP."},{"key":"ref_12","unstructured":"(2020). 3GPP TS 38.521-4; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; User Equipment (UE) Conformance Specification; Radio Transmission and Reception; Part 4: Performance (Release 16), v16.4, 3GPP."},{"key":"ref_13","unstructured":"(2020). 3GPP TS 38.523-1; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; 5GS; User Equipment (UE) Conformance Specification; Part 1: Protocol (Release 16), v16.4, 3GPP."},{"key":"ref_14","unstructured":"(2020). 3GPP TS 38.523-2; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; 5GS; User Equipment (UE) Conformance Specification; Part 2: Applicability of Protocol Test Cases (Release 16), v16.4, 3GPP."},{"key":"ref_15","unstructured":"(2020). 3GPP TS 38.523-3; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; 5GS; User Equipment (UE) Conformance Specification; Part 3: Protocol Test Suites (Release 15), v15.8, 3GPP."},{"key":"ref_16","unstructured":"(2020). 3GPP TS 38.533; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; User Equipment (UE) Conformance Specification; Radio Resource Management (RRM) (Release 16), 3GPP."},{"key":"ref_17","unstructured":"(2018). CTIA Battery Life Programm Management Document, CTIA."},{"key":"ref_18","unstructured":"(2020). 3GPP TS 28.552; 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Management and Orchestration; Management and Orchestration; 5G Performance Measurements (Release 16), v16.6, 3GPP."},{"key":"ref_19","unstructured":"(2020). 3GPP TS 32.425; 3rd Generation Partnership Project; Telecommunication Management; Performance Management (PM); Performance Measurements Evolved Universal Terrestrial Radio Access Network (E-UTRAN) (Release 16), v16.5, 3GPP."},{"key":"ref_20","unstructured":"(2019). 3GPP TS 32.451; 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Telecommunication Management; Key Performance Indicators (KPI) for Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Requirements (Release 16), v16.0, 3GPP."},{"key":"ref_21","unstructured":"(2020). 3GPP TS 32.404; 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Telecommunication Management; Performance Management (PM); Performance Measurements; Definitions and Template (Release 16), v16.0, 3GPP."},{"key":"ref_22","unstructured":"(2020). ETSI GS NFV-TST 010; Network Functions Virtualisation (NFV) Release 2; Testing; API Conformance Testing Specification, v2.6.1, ETSI."},{"key":"ref_23","unstructured":"(2018). 3GPP TS 37.901; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; User Equipment (UE) Application Layer Data throughput Performance (Release 15), 3GPP."},{"key":"ref_24","unstructured":"(2020). 3GPP TR 37.901-5; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Study on 5G NR User Equipment (UE) Application Layer Data throughput Performance (Release 16), v16.0, 3GPP."},{"key":"ref_25","unstructured":"(2020, September 30). 5GENESIS Project Deliverable D6.1 \u201cTrials and Experimentation (Cycle 1). Available online: https:\/\/5genesis.eu\/wp-content\/uploads\/2019\/12\/5GENESIS_D6.1_v2.00.pdf."},{"key":"ref_26","unstructured":"(2020). 3GPP TS 28.554; 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Management and Orchestration; 5G End to End Key Performance Indicators (KPI) (Release 16), v16.5, 3GPP."},{"key":"ref_27","unstructured":"NGMN Alliance (2019). Definition of the Testing Framework for the NGMN 5G Pre-Commercial Networks Trials, NGMN."},{"key":"ref_28","unstructured":"(2020). 3GPP TR 38.913; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Study on Scenarios and Requirements for Next Generation Access Technologies (Release 16), v16.5, 3GPP."},{"key":"ref_29","first-page":"2563917","article-title":"An End-to-End Automation Framework for Mobile Network Testbeds","volume":"2019","author":"Merino","year":"2019","journal-title":"Mob. Inf. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"e1699","DOI":"10.1002\/stvr.1699","article-title":"A formal approach to automatically analyse extra-functional properties in mobile applications","volume":"29","author":"Espada","year":"2019","journal-title":"Softw. Test. Verif. Reliab."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Tateishi, T.K., Kunta, D., Harada, A., Kishryama, Y., Parkvall, S., Dahlman, E., and Furuskog, J. (September, January 30). Field experiments on 5G radio access using 15-GHz band in outdoor small cell environment. Proceedings of the 2015 IEEE 26th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC\u201915), Hong Kong, China.","DOI":"10.1109\/PIMRC.2015.7343416"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Moto, K., Mikami, M., Serizawa, K., and Yoshino, H. (2019, January 22\u201325). Field Experimental Evaluation on 5G V2N Low Latency Communication for Application to Truck Platooning. Proceedings of the 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall), Honolulu, HI, USA.","DOI":"10.1109\/VTCFall.2019.8891450"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/CC.2016.7405721","article-title":"5G technology requirements and related test environments for evaluation","volume":"13","author":"Meng","year":"2016","journal-title":"China Commun."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Li, X., Deng, W., Liu, L., Tian, Y., Tong, H., Liu, J., Ma, Y., Wang, J., Horsmanh, S., and Gavras, A. (2020, January 7\u201311). Novel Test Methods for 5G Network Performance Field Trial. Proceedings of the 2020 IEEE International Conference on Communications Workshops (ICC\u201920 Workshops), Dublin, Ireland.","DOI":"10.1109\/ICCWorkshops49005.2020.9145321"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Xu, D., Zhou, A., Zhang, X., Wang, G., Liu, X., An, C., Shi, Y., Liu, L., and Ma, H. (2020, January 10\u201314). Understanding Operational 5G: A First Measurement Study on Its Coverage, Performance and Energy Consumption. Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM\u201920), Virtual Event, NY, USA.","DOI":"10.1145\/3387514.3405882"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Narayanan, A., and Ramadan, E. (2020, January 20\u201324). A First Look at Commercial 5G Performance on Smartphones. Proceedings of the Web Conference 2020 (WWW\u201920), Taipei Taiwan.","DOI":"10.1145\/3366423.3380169"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Piri, E., Ruuska, P., Kanstr\u00e9n, T., M\u00e4kel\u00e4, J., Korva, J., Hekkala, A., Pouttu, A., Liinamaa, O., Latva-aho, M., and Vierimaa, K. (2016, January 27\u201330). 5GTN: A test network for 5G application development and testing. Proceedings of the 2016 European Conference on Networks and Communications (EuCNC), Athens, Greece.","DOI":"10.1109\/EuCNC.2016.7561054"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"111522","DOI":"10.1109\/ACCESS.2020.3001985","article-title":"A Multi-Site NFV Testbed for Experimentation With SUAV-Based 5G Vertical Services","volume":"8","author":"Vidal","year":"2020","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1718","DOI":"10.1109\/JIOT.2019.2949629","article-title":"A Software-Defined IoT Device Management Framework for Edge and Cloud Computing","volume":"7","author":"Mavromatis","year":"2020","journal-title":"IEEE Internet Things J."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"79147","DOI":"10.1109\/ACCESS.2020.2990278","article-title":"Key Performance Indicators of the Reference 6TiSCH Implementation in Internet-of-Things Scenarios","volume":"8","author":"Vucinic","year":"2020","journal-title":"IEEE Access"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz, J., Rincon, F., Chang, T., Vilajosana, X., Vermeulen, B., and Walcariu, T. (May, January 29). OpenTestBed: Poor Man\u2019s IoT Testbed. Proceedings of the 2019 IEEE Conference on Computer Communications Workshops (INFOCOM\u201919 WKSHPS), Paris, France.","DOI":"10.1109\/INFCOMW.2019.8845269"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1109\/MWC.2018.1800050","article-title":"POSENS: A Practical Open Source Solution for End-to-End Network Slicing","volume":"25","author":"Gramaglia","year":"2018","journal-title":"IEEE Wirel. Commun."},{"key":"ref_43","unstructured":"5G PPP Architecture Working Group (2020, October 25). View on 5G Architecture, Version 3.0. Available online: https:\/\/5g-ppp.eu\/wp-content\/uploads\/2020\/02\/5G-PPP-5G-Architecture-White-Paper_final.pdf."},{"key":"ref_44","unstructured":"5GENESIS Project (2020, September 30). Deliverable D5.3 Documentation and Supporting Material for 5G Stakeholders (Release A). Available online: https:\/\/5genesis.eu\/wp-content\/uploads\/2020\/07\/5GENESIS-D5.3_v1.0.pdf."},{"key":"ref_45","unstructured":"EURO-5G Project (2020, September 30). Deliverable D2.6 Final Report on Programme Progress and KPIs. Available online: https:\/\/5g-ppp.eu\/wp-content\/uploads\/2017\/10\/Euro-5G-D2.6_Final-report-on-programme-progress-and-KPIs.pdf."},{"key":"ref_46","unstructured":"(2020). 3GPP TS 23.501; 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; System Architecture for the 5G System (5GS); Stage 2; (Release 16), v16.5.1, 3GPP."},{"key":"ref_47","unstructured":"(2018). 3GPP TR 28.801; 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Telecommunication Management; Study on Management and Orchestration of Network Slicing for Next Generation Network (Release 15), v15.1.0, 3GPP."},{"key":"ref_48","unstructured":"5GENESIS Project (2020, September 30). Deliverable D3.3 Slice Management (Release A). Available online: https:\/\/5genesis.eu\/wp-content\/uploads\/2019\/10\/5GENESIS_D3.3_v1.0.pdf."},{"key":"ref_49","unstructured":"(2020, September 30). Katana Slice Manager. Available online: https:\/\/github.com\/medianetlab\/katana-slice_manager."},{"key":"ref_50","unstructured":"GSM Association (2020). Official Document NG.116\u2014Generic Network Slice Template, GSM Association."},{"key":"ref_51","unstructured":"(2020, September 30). Monroe Experiments. Available online: https:\/\/github.com\/MONROE-PROJECT\/Experiments."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.comcom.2018.09.004","article-title":"Results from running an experiment as a service platform for mobile broadband networks in Europe","volume":"133","author":"Mancuso","year":"2019","journal-title":"Comput. Commun."},{"key":"ref_53","unstructured":"(2020). GPP TS 37.340; 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (E-UTRA) and NR; Multi-Connectivity; Stage 2 (Release 16), v16.2.0, 3GPP."},{"key":"ref_54","unstructured":"5GENESIS Project (2020, October 25). Deliverable D4.5 The Malaga Platform (Release B). Available online: https:\/\/5genesis.eu\/wp-content\/uploads\/2020\/02\/5GENESIS_D4.5_v1.0.pdf."},{"key":"ref_55","unstructured":"(2020). 3GPP TS 38.101-3; Technical Specification 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; User Equipment (UE) Radio Transmission and Reception; Part 3: Range 1 and Range 2 Interworking Operation with Other Radios (Release 16), v16.4.0, 3GPP."},{"key":"ref_56","unstructured":"(2020). 3GPP TS 38.214; Technical Specification 3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; Physical Layer Procedures for Data (Release 15), v15.10.0, 3GPP."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/22\/6652\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:34:59Z","timestamp":1760178899000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/22\/6652"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,20]]},"references-count":56,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2020,11]]}},"alternative-id":["s20226652"],"URL":"https:\/\/doi.org\/10.3390\/s20226652","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,20]]}}}