{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T03:02:38Z","timestamp":1787022158295,"version":"build-2736575974"},"reference-count":53,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T00:00:00Z","timestamp":1726444800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DMS1613202"],"award-info":[{"award-number":["DMS1613202"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Classifiers have been developed to help diagnose dengue fever in patients presenting with febrile symptoms. However, classifier predictions often rely on the assumption that new observations come from the same distribution as training data. If the population prevalence of dengue changes, as would happen with a dengue outbreak, it is important to raise an alarm as soon as possible, so that appropriate public health measures can be taken and also so that the classifier can be re-calibrated. In this paper, we consider the problem of detecting such a change in distribution in sequentially-observed, unlabeled classification data. We focus on label shift changes to the distribution, where the class priors shift but the class conditional distributions remain unchanged. We reduce this problem to the problem of detecting a change in the one-dimensional classifier scores, leading to simple nonparametric sequential changepoint detection procedures. Our procedures leverage classifier training data to estimate the detection statistic, and converge to their parametric counterparts in the size of the training data. In simulated outbreaks with real dengue data, we show that our method outperforms other detection procedures in this label shift setting.<\/jats:p>","DOI":"10.1371\/journal.pone.0310194","type":"journal-article","created":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T13:37:25Z","timestamp":1726493845000},"page":"e0310194","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":1,"title":["Sequential label shift detection in classification data: An application to dengue fever"],"prefix":"10.1371","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3256-011X","authenticated-orcid":true,"given":"Ciaran","family":"Evans","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Max","family":"G\u2019Sell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,9,16]]},"reference":[{"key":"pone.0310194.ref001","unstructured":"WHO. Dengue and severe dengue; 2020. Available from: https:\/\/www.who.int\/en\/news-room\/fact-sheets\/detail\/dengue-and-severe-dengue."},{"issue":"3","key":"pone.0310194.ref002","doi-asserted-by":"crossref","first-page":"e196","DOI":"10.1371\/journal.pntd.0000196","article-title":"Decision tree algorithms predict the diagnosis and outcome of dengue fever in the early phase of illness","volume":"2","author":"L Tanner","year":"2008","journal-title":"PLOS Neglected Tropical Diseases"},{"issue":"4","key":"pone.0310194.ref003","doi-asserted-by":"crossref","first-page":"e0003638","DOI":"10.1371\/journal.pntd.0003638","article-title":"Sensitivity and specificity of a novel classifier for the early diagnosis of dengue","volume":"9","author":"NM Tuan","year":"2015","journal-title":"PLOS Neglected Tropical Diseases"},{"issue":"5","key":"pone.0310194.ref004","doi-asserted-by":"crossref","first-page":"e0009445","DOI":"10.1371\/journal.pntd.0009445","article-title":"Rapid testing requires clinical evaluation for accurate diagnosis of dengue disease: A passive surveillance study in Southern Malaysia","volume":"15","author":"CF Ngim","year":"2021","journal-title":"PLOS Neglected Tropical Diseases"},{"issue":"2","key":"pone.0310194.ref005","first-page":"73","article-title":"An observation on correlation between rainfall and the prevalence of clinical cases of dengue in Thailand","volume":"43","author":"V Wiwanitkit","year":"2006","journal-title":"Journal of Vector Borne Diseases"},{"issue":"5","key":"pone.0310194.ref006","first-page":"85","article-title":"Prevalence of dengue among clinically suspected febrile episodes at a teaching hospital in North India","volume":"3","author":"A Garg","year":"2011","journal-title":"Journal of Infectious Diseases and Immunity"},{"key":"pone.0310194.ref007","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.ijid.2016.11.008","article-title":"Trend and geographic analysis of the prevalence of dengue in Taiwan, 2010\u20132015","volume":"54","author":"JC Hsu","year":"2017","journal-title":"International Journal of Infectious Diseases"},{"key":"pone.0310194.ref008","unstructured":"Lipton ZC, Wang YX, Smola A. Detecting and correcting for label shift with black box predictors. In: International Conferenceon Machine Learning. PMLR; 2018. p. 3122-3130."},{"key":"pone.0310194.ref009","unstructured":"Rabanser S, G\u00fcnnemann S, Lipton Z. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. Advances in Neural Information Processing Systems 32. 2019."},{"issue":"1","key":"pone.0310194.ref010","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1162\/089976602753284446","article-title":"Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure","volume":"14","author":"M Saerens","year":"2002","journal-title":"Neural Computation"},{"key":"pone.0310194.ref011","doi-asserted-by":"crossref","unstructured":"Storkey A. When training and test sets are different: characterizing learning transfer. Dataset Shift in Machine Learning. 2009; p. 3\u201328.","DOI":"10.7551\/mitpress\/7921.003.0004"},{"key":"pone.0310194.ref012","unstructured":"Ackerman S, Dube P, Farchi E. Sequential Drift Detection in Deep Learning Classifiers. arXiv preprint arXiv:200716109. 2020."},{"key":"pone.0310194.ref013","unstructured":"Azizzadenesheli K, Liu A, Yang F, Anandkumar A. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:190309734. 2019;."},{"issue":"3","key":"pone.0310194.ref014","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1007\/s11009-011-9256-5","article-title":"State-of-the-art in sequential change-point detection","volume":"14","author":"AS Polunchenko","year":"2012","journal-title":"Methodology and Computing in Applied Probability"},{"issue":"526","key":"pone.0310194.ref015","doi-asserted-by":"crossref","first-page":"514","DOI":"10.1080\/01621459.2018.1476245","article-title":"Sequential nonparametric tests for a change in distribution: an application to detecting radiological anomalies","volume":"114","author":"OH Madrid Padilla","year":"2019","journal-title":"Journal of the American Statistical Association"},{"issue":"2","key":"pone.0310194.ref016","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1080\/00224065.2012.11917887","article-title":"Two nonparametric control charts for detecting arbitrary distribution changes","volume":"44","author":"GJ Ross","year":"2012","journal-title":"Journal of Quality Technology"},{"issue":"3","key":"pone.0310194.ref017","doi-asserted-by":"crossref","first-page":"1381","DOI":"10.1214\/18-AOS1718","article-title":"Sequential change-point detection based on nearest neighbors","volume":"47","author":"H Chen","year":"2019","journal-title":"The Annals of Statistics"},{"key":"pone.0310194.ref018","doi-asserted-by":"crossref","first-page":"4498","DOI":"10.1109\/TSP.2022.3205763","article-title":"Sequential change-point detection for high-dimensional and non-Euclidean data","volume":"70","author":"L Chu","year":"2022","journal-title":"IEEE Transactions on Signal Processing"},{"key":"pone.0310194.ref019","doi-asserted-by":"crossref","DOI":"10.1007\/978-94-015-8163-9","volume-title":"Nonparametric methods in change point problems","author":"E Brodsky","year":"1993"},{"key":"pone.0310194.ref020","doi-asserted-by":"crossref","DOI":"10.1007\/978-94-015-9530-8","volume-title":"Non-parametric statistical diagnosis: problems and methods","author":"E Brodsky","year":"2000"},{"issue":"1","key":"pone.0310194.ref021","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/JSTSP.2012.2233713","article-title":"Efficient computer network anomaly detection by changepoint detection methods","volume":"7","author":"AG Tartakovsky","year":"2012","journal-title":"IEEE Journal of Selected Topics in Signal Processing"},{"issue":"3","key":"pone.0310194.ref022","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1016\/j.stamet.2005.05.003","article-title":"Detection of intrusions in information systems by sequential change-point methods","volume":"3","author":"AG Tartakovsky","year":"2006","journal-title":"Statistical Methodology"},{"issue":"9","key":"pone.0310194.ref023","doi-asserted-by":"crossref","first-page":"3372","DOI":"10.1109\/TSP.2006.879308","article-title":"A novel approach to detection of intrusions in computer networks via adaptive sequential and batch-sequential change-point detection methods","volume":"54","author":"AG Tartakovsky","year":"2006","journal-title":"IEEE Transactions on Signal Processing"},{"key":"pone.0310194.ref024","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1214\/lnms\/1215463111","article-title":"An efficient nonparametric detection scheme and its application to surveillance of a Bernoulli process with unknown baseline","author":"C Bell","year":"1994","journal-title":"Lecture Notes-Monograph Series"},{"key":"pone.0310194.ref025","first-page":"763","article-title":"An efficient sequential nonparametric scheme for detecting a change of distribution","author":"L Gordon","year":"1994","journal-title":"The Annals of Statistics"},{"key":"pone.0310194.ref026","first-page":"1350","article-title":"A robust surveillance scheme for stochastically ordered alternatives","author":"L Gordon","year":"1995","journal-title":"The Annals of Statistics"},{"issue":"5","key":"pone.0310194.ref027","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1002\/1520-6750(199010)37:5<627::AID-NAV3220370504>3.0.CO;2-F","article-title":"A CUSUM procedure based on sequential ranks","volume":"37","author":"D McDonald","year":"1990","journal-title":"Naval Research Logistics"},{"issue":"1-2","key":"pone.0310194.ref028","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/07474940008836437","article-title":"Nonparametric adaptive change point estimation and on line detection","volume":"19","author":"MI Baron","year":"2000","journal-title":"Sequential Analysis"},{"issue":"11","key":"pone.0310194.ref029","doi-asserted-by":"crossref","first-page":"5847","DOI":"10.1109\/TIT.2010.2068870","article-title":"Estimating divergence functionals and the likelihood ratio by convex risk minimization","volume":"56","author":"X Nguyen","year":"2010","journal-title":"IEEE Transactions on Information Theory"},{"issue":"Jul","key":"pone.0310194.ref030","first-page":"1391","article-title":"A least-squares approach to direct importance estimation","volume":"10","author":"T Kanamori","year":"2009","journal-title":"Journal of Machine Learning Research"},{"key":"pone.0310194.ref031","doi-asserted-by":"crossref","unstructured":"Kawahara Y, Sugiyama M. Change-point detection in time-series data by direct density-ratio estimation. In: Proceedings of the 2009 SIAM International Conference on Data Mining. SIAM; 2009. p. 389\u2013400.","DOI":"10.1137\/1.9781611972795.34"},{"issue":"4","key":"pone.0310194.ref032","doi-asserted-by":"crossref","first-page":"699","DOI":"10.1007\/s10463-008-0197-x","article-title":"Direct importance estimation for covariate shift adaptation","volume":"60","author":"M Sugiyama","year":"2008","journal-title":"Annals of the Institute of Statistical Mathematics"},{"key":"pone.0310194.ref033","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.neunet.2013.01.012","article-title":"Change-point detection in time-series data by relative density-ratio estimation","volume":"43","author":"S Liu","year":"2013","journal-title":"Neural Networks"},{"issue":"4","key":"pone.0310194.ref034","first-page":"5","article-title":"Covariate shift by kernel mean matching","volume":"3","author":"A Gretton","year":"2009","journal-title":"Dataset Shift in Machine Learning"},{"key":"pone.0310194.ref035","unstructured":"Yu Y, Szepesv\u00e1ri C. Analysis of kernel mean matching under covariate shift. arXiv preprint arXiv:12064650. 2012;."},{"issue":"6","key":"pone.0310194.ref036","doi-asserted-by":"crossref","first-page":"1897","DOI":"10.1214\/aoms\/1177693055","article-title":"Procedures for reacting to a change in distribution","volume":"42","author":"G Lorden","year":"1971","journal-title":"The Annals of Mathematical Statistics"},{"issue":"4","key":"pone.0310194.ref037","doi-asserted-by":"crossref","first-page":"1379","DOI":"10.1214\/aos\/1176350164","article-title":"Optimal stopping times for detecting changes in distributions","volume":"14","author":"GV Moustakides","year":"1986","journal-title":"The Annals of Statistics"},{"issue":"3","key":"pone.0310194.ref038","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1137\/S0040585X97985534","article-title":"Third-order Asymptotic Optimality of the Generalized Shiryaev\u2013Roberts Changepoint Detection Procedures","volume":"56","author":"AG Tartakovsky","year":"2012","journal-title":"Theory of Probability & Its Applications"},{"issue":"7","key":"pone.0310194.ref039","doi-asserted-by":"crossref","first-page":"2917","DOI":"10.1109\/18.737522","article-title":"Information bounds and quick detection of parameter changes in stochastic systems","volume":"44","author":"TL Lai","year":"1998","journal-title":"IEEE Transactions on Information Theory"},{"key":"pone.0310194.ref040","first-page":"255","article-title":"Using the generalized likelihood ratio statistic for sequential detection of a change-point","author":"D Siegmund","year":"1995","journal-title":"The Annals of Statistics"},{"key":"pone.0310194.ref041","doi-asserted-by":"crossref","DOI":"10.1201\/b17279","volume-title":"Sequential analysis: Hypothesis testing and changepoint detection","author":"A Tartakovsky","year":"2014"},{"issue":"3","key":"pone.0310194.ref042","doi-asserted-by":"crossref","first-page":"1604","DOI":"10.1109\/TIT.2011.2104993","article-title":"Minimax robust quickest change detection","volume":"57","author":"J Unnikrishnan","year":"2011","journal-title":"IEEE Transactions on Information Theory"},{"issue":"1\/2","key":"pone.0310194.ref043","doi-asserted-by":"crossref","first-page":"100","DOI":"10.2307\/2333009","article-title":"Continuous inspection schemes","volume":"41","author":"ES Page","year":"1954","journal-title":"Biometrika"},{"issue":"3","key":"pone.0310194.ref044","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s10994-011-5266-3","article-title":"Statistical analysis of kernel-based least-squares density-ratio estimation","volume":"86","author":"T Kanamori","year":"2012","journal-title":"Machine Learning"},{"key":"pone.0310194.ref045","unstructured":"Makiyama K. densratio: Density Ratio Estimation; 2019. Available from: https:\/\/CRAN.R-project.org\/package=densratio."},{"issue":"4","key":"pone.0310194.ref046","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1198\/TECH.2011.10069","article-title":"Nonparametric monitoring of data streams for changes in location and scale","volume":"53","author":"GJ Ross","year":"2011","journal-title":"Technometrics"},{"issue":"3","key":"pone.0310194.ref047","first-page":"1","article-title":"Parametric and Nonparametric Sequential Change Detection in R: The cpm Package","volume":"66","author":"GJ Ross","year":"2015","journal-title":"Journal of Statistical Software"},{"key":"pone.0310194.ref048","unstructured":"Chen H, Chu L. gStream: Graph-Based Sequential Change-Point Detection for Streaming Data; 2019. Available from: https:\/\/CRAN.R-project.org\/package=gStream."},{"key":"pone.0310194.ref049","article-title":"Assessing generalizability of a dengue classifier across multiple datasets","author":"B Lu","year":"2024","journal-title":"bioRxiv"},{"key":"pone.0310194.ref050","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12879-016-1849-8","article-title":"Early diagnosis of dengue disease severity in a resource-limited Asian country","volume":"16","author":"P Cavailler","year":"2016","journal-title":"BMC Infectious Diseases"},{"issue":"11","key":"pone.0310194.ref051","doi-asserted-by":"crossref","first-page":"e0008843","DOI":"10.1371\/journal.pntd.0008843","article-title":"Comparing machine learning with case-control models to identify confirmed dengue cases","volume":"14","author":"TS Ho","year":"2020","journal-title":"PLOS Neglected Tropical Diseases"},{"issue":"9","key":"pone.0310194.ref052","doi-asserted-by":"crossref","first-page":"e1322","DOI":"10.1371\/journal.pntd.0001322","article-title":"Quantifying the emergence of dengue in Hanoi, Vietnam: 1998\u20132009","volume":"5","author":"HQ Cuong","year":"2011","journal-title":"PLOS Neglected Tropical Diseases"},{"issue":"1","key":"pone.0310194.ref053","doi-asserted-by":"crossref","first-page":"e0007997","DOI":"10.1371\/journal.pntd.0007997","article-title":"Heatwaves and dengue outbreaks in Hanoi, Vietnam: New evidence on early warning","volume":"14","author":"J Cheng","year":"2020","journal-title":"PLOS Neglected Tropical Diseases"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0310194","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T13:38:01Z","timestamp":1726493881000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0310194"}},"subtitle":[],"editor":[{"given":"Jos\u00e9","family":"Ramos-Casta\u00f1eda","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,9,16]]},"references-count":53,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,9,16]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0310194","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,16]]}}}