{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T14:56:35Z","timestamp":1773413795289,"version":"3.50.1"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"CoNEXT2","license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["CNS-2028145, CNS-2124393, CNS-2126327"],"award-info":[{"award-number":["CNS-2028145, CNS-2124393, CNS-2126327"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"France and Chicago Collaborating in the Sciences program"},{"name":"ANR","award":["N? ANR-21-CE94-0001-01 (MINT)"],"award-info":[{"award-number":["N? ANR-21-CE94-0001-01 (MINT)"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Netw."],"published-print":{"date-parts":[[2023,9,28]]},"abstract":"<jats:p>Operational networks commonly rely on machine learning models for many tasks, including detecting anomalies, inferring application performance, and forecasting demand. Yet, model accuracy can degrade due to concept drift, whereby the relationship between the features and the target to be predicted changes. Mitigating concept drift is an essential part of operationalizing machine learning models in general, but is of particular importance in networking's highly dynamic deployment environments. In this paper, we first characterize concept drift in a large cellular network for a major metropolitan area in the United States. We find that concept drift occurs across many important key performance indicators (KPIs), independently of the model, training set size, and time interval---thus necessitating practical approaches to detect, explain, and mitigate it. We then show that frequent model retraining with newly available data is not sufficient to mitigate concept drift, and can even degrade model accuracy further. Finally, we develop a new methodology for concept drift mitigation, Local Error Approximation of Features (LEAF). LEAF works by detecting drift; explaining the features and time intervals that contribute the most to drift; and mitigates it using forgetting and over-sampling. We evaluate LEAF against industry-standard mitigation approaches (notably, periodic retraining) with more than four years of cellular KPI data. Our initial tests with a major cellular provider in the US show that LEAF consistently outperforms periodic and triggered retraining on complex, real-world data while reducing costly retraining operations.<\/jats:p>","DOI":"10.1145\/3609422","type":"journal-article","created":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T19:45:17Z","timestamp":1695930317000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["LEAF: Navigating Concept Drift in Cellular Networks"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6170-2167","authenticated-orcid":false,"given":"Shinan","family":"Liu","sequence":"first","affiliation":[{"name":"University of Chicago, Chicago, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4447-960X","authenticated-orcid":false,"given":"Francesco","family":"Bronzino","sequence":"additional","affiliation":[{"name":"Univ Lyon, EnsL, UCBL, CNRS, LIP, Lyon, France"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2156-5305","authenticated-orcid":false,"given":"Paul","family":"Schmitt","sequence":"additional","affiliation":[{"name":"University of Hawaii, Manoa, Honolulu, HI, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2803-5649","authenticated-orcid":false,"given":"Arjun Nitin","family":"Bhagoji","sequence":"additional","affiliation":[{"name":"University of Chicago, Chicago, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9315-5201","authenticated-orcid":false,"given":"Nick","family":"Feamster","sequence":"additional","affiliation":[{"name":"University of Chicago, Chicago, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-1044-5261","authenticated-orcid":false,"given":"Hector Garcia","family":"Crespo","sequence":"additional","affiliation":[{"name":"Verizon, North Richland Hills, TX, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7744-0746","authenticated-orcid":false,"given":"Timothy","family":"Coyle","sequence":"additional","affiliation":[{"name":"Verizon, Chicopee, MA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0451-3147","authenticated-orcid":false,"given":"Brian","family":"Ward","sequence":"additional","affiliation":[{"name":"Verizon, Fort Worth, TX, USA"}]}],"member":"320","published-online":{"date-parts":[[2023,9,28]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"REFERENCES [1] 2023. LTE cellular network performance indicators daily measurements dataset. https:\/\/forms.gle\/ g5pbB5qRHeBsEmZJ6."},{"key":"e_1_2_1_2_1","volume-title":"accessed","author":"AutoGluon","year":"2021","unstructured":"AutoGluon AI. accessed July, 2021. AutoGluon: AutoML for Text, Image, and Tabular Data. https:\/\/auto.gluon.ai\/stable\/ index.html."},{"key":"e_1_2_1_3_1","volume-title":"On the robustness of interpretability methods. arXiv preprint arXiv:1806.08049","author":"Alvarez-Melis David","year":"2018","unstructured":"David Alvarez-Melis and Tommi S Jaakkola. 2018. On the robustness of interpretability methods. arXiv preprint arXiv:1806.08049 (2018)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12377"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3484266.3487373"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCOM.2018.1700560"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2008.119"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2018.04.014"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972771.42"},{"key":"e_1_2_1_10_1","volume-title":"Random forests. Machine learning 45, 1","author":"Breiman Leo","year":"2001","unstructured":"Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5--32."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21222-2_19"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2251352"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11339"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CNSM.2013.6727831"},{"key":"e_1_2_1_15_1","volume-title":"accessed","year":"2020","unstructured":"Comcast. accessed October, 2020. COVID-19 Network Update. https:\/\/corporate.comcast.com\/covid-19\/network\/may20--2020."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCSN.2016.7586590"},{"key":"e_1_2_1_17_1","volume-title":"CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363","author":"Dorogush Anna Veronika","year":"2018","unstructured":"Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. 2018. CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363 (2018)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2006.04.011"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419394.3423658"},{"key":"e_1_2_1_20_1","volume-title":"Greedy function approximation: a gradient boosting machine. Annals of statistics","author":"Friedman Jerome H","year":"2001","unstructured":"Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189--1232."},{"key":"e_1_2_1_21_1","volume-title":"accessed","year":"2021","unstructured":"Futuriom. accessed August, 2021. Verizon Applies Machine Learning to Operations. https:\/\/www.futuriom.com\/articles\/ news\/verizon-applies-machine-learning-to-operations\/2018\/08."},{"key":"e_1_2_1_22_1","volume-title":"Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia.","author":"Gama Jo\u00e3o","year":"2014","unstructured":"Jo\u00e3o Gama, Indre liobait e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. A survey on concept drift adaptation. ACM computing surveys (CSUR) 46, 4 (2014), 1--37."},{"key":"e_1_2_1_23_1","volume-title":"Extremely randomized trees. Machine learning 63, 1","author":"Geurts Pierre","year":"2006","unstructured":"Pierre Geurts, Damien Ernst, and Louis Wehenkel. 2006. Extremely randomized trees. Machine learning 63, 1 (2006), 3--42."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2014.07.019"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-016-9646-1"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243792"},{"key":"e_1_2_1_27_1","volume-title":"Transcend: Detecting concept drift in malware classification models. In 26th {USENIX} Security Symposium ({USENIX} Security 17). 625--642.","author":"Jordaney Roberto","year":"2017","unstructured":"Roberto Jordaney, Kumar Sharad, Santanu K Dash, Zhi Wang, Davide Papini, Ilia Nouretdinov, and Lorenzo Cavallaro. 2017. Transcend: Detecting concept drift in malware classification models. In 26th {USENIX} Security Symposium ({USENIX} Security 17). 625--642."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517312.2517320"},{"key":"e_1_2_1_29_1","first-page":"439","article-title":"Systems and methods for network performance forecasting","volume":"9","author":"Knebl Matthew Brian","year":"2016","unstructured":"Matthew Brian Knebl and Amr Albanna. 2016. Systems and methods for network performance forecasting. US Patent 9,439,081.","journal-title":"US Patent"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2017.02.004"},{"key":"e_1_2_1_31_1","volume-title":"Resampling methods for dependent data","author":"Lahiri SK","unstructured":"SK Lahiri and SN Lahiri. 2003. Resampling methods for dependent data. Springer Science & Business Media."},{"key":"e_1_2_1_32_1","volume-title":"Hector Garcia Crespo, and Brian Ward","author":"Liu Shinan","year":"2021","unstructured":"Shinan Liu, Francesco Bronzino, Paul Schmitt, Nick Feamster, Ricardo Borges, Hector Garcia Crespo, and Brian Ward. 2021. Understanding Model Drift in a Large Cellular Network. CoRR (2021)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3580818"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72582-2_2"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2016.0040"},{"key":"e_1_2_1_36_1","first-page":"2346","article-title":"Learning under concept drift: A review","volume":"31","author":"Lu Jie","year":"2018","unstructured":"Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. 2018. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering 31, 12 (2018), 2346--2363.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419394.3423655"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2018.00013"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2079296.2079309"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2535372.2535382"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1851182.1851219"},{"key":"e_1_2_1_42_1","first-page":"77","article-title":"Matchmaker: Data Drift Mitigation in Machine Learning for Large-Scale Systems","volume":"4","author":"Mallick Ankur","year":"2022","unstructured":"Ankur Mallick, Kevin Hsieh, Behnaz Arzani, and Gauri Joshi. 2022. Matchmaker: Data Drift Mitigation in Machine Learning for Large-Scale Systems. Proceedings of Machine Learning and Systems 4 (2022), 77--94.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_43_1","volume-title":"accessed","author":"McKeay Martin","year":"2020","unstructured":"Martin McKeay. accessed October, 2020. Parts of a Whole: Effect of COVID-19 on US Internet Traffic. https:\/\/blogs. akamai.com\/sitr\/2020\/04\/parts-of-a-whole-effect-of-covid-19-on-us-internet-traffic.html."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2020.107515"},{"key":"e_1_2_1_45_1","volume-title":"Notes on linear regression analysis. Fuqua School of Business","author":"Nau Robert","year":"2014","unstructured":"Robert Nau. 2014. Notes on linear regression analysis. Fuqua School of Business, Duke University Retrieved from: https:\/\/people.duke.edu\/~rnau\/Notes_on_linear_regression_analysis--Robert_Nau.pdf (2014)."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-04918-2_2"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2019.11.111"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCI.2017.2773824"},{"key":"e_1_2_1_50_1","volume-title":"Incremental learning from noisy data. Machine learning 1, 3","author":"Schlimmer Jeffrey C","year":"1986","unstructured":"Jeffrey C Schlimmer and Richard H Granger. 1986. Incremental learning from noisy data. Machine learning 1, 3 (1986), 317--354."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.85"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/IAW.2005.1495950"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2007.03.025"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSAC.1999.816048"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2010.25"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2011.25"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.3390\/computers10010013"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btr300"},{"key":"e_1_2_1_59_1","volume-title":"Proceeding of the 11th annual international conference on Mobile systems, applications, and services. 347--360.","author":"Xu Qiang","unstructured":"Qiang Xu, Sanjeev Mehrotra, Zhuoqing Mao, and Jin Li. 2013. PROTEUS: network performance forecast for real-time, interactive mobile applications. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services. 347--360."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2016.12.078"},{"key":"e_1_2_1_61_1","volume-title":"30th {USENIX} Security Symposium ({USENIX} Security 21).","author":"Yang Limin","unstructured":"Limin Yang, Wenbo Guo, Qingying Hao, Arridhana Ciptadi, Ali Ahmadzadeh, Xinyu Xing, and Gang Wang. 2021. {CADE}: Detecting and Explaining Concept Drift Samples for Security Applications. In 30th {USENIX} Security Symposium ({USENIX} Security 21)."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482271"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2716281.2836087"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCOM.2014.6957156"},{"key":"e_1_2_1_65_1","unstructured":"Indre liobait e. 2010. Adaptive training set formation. (2010)."}],"container-title":["Proceedings of the ACM on Networking"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3609422","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3609422","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3609422","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,1]],"date-time":"2025-07-01T13:53:20Z","timestamp":1751378000000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3609422"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"references-count":65,"journal-issue":{"issue":"CoNEXT2","published-print":{"date-parts":[[2023,9,28]]}},"alternative-id":["10.1145\/3609422"],"URL":"https:\/\/doi.org\/10.1145\/3609422","relation":{},"ISSN":["2834-5509"],"issn-type":[{"value":"2834-5509","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,28]]},"assertion":[{"value":"2023-09-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}