{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,4]],"date-time":"2026-11-04T17:22:55Z","timestamp":1793812975364,"version":"4.6.1"},"reference-count":35,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,5,17]],"date-time":"2025-05-17T00:00:00Z","timestamp":1747440000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Cancers"],"abstract":"<jats:p>Background: Flow cytometric detection of minimal residual disease (MRD) in chronic lymphocytic leukemia (CLL) is complex, time-consuming, and subject to inter-operator variability. Deep neural networks (DNNs) offer potential for standardization and efficiency improvement, but require rigorous validation and monitoring for safe clinical implementation. Methods: We evaluated a DNN-assisted human-in-the-loop approach for CLL MRD detection. Initial validation included method comparison against manual analysis (n = 240), precision studies, and analytical sensitivity verification. Post-implementation monitoring comprised four components: daily electronic quality control, input data drift detection, error analysis, and attribute acceptance sampling. Laboratory efficiency was assessed through a timing study of 161 cases analyzed by five technologists. Results: Method comparison demonstrated 97.5% concordance with manual analysis for qualitative classification (sensitivity 100%, specificity 95%) and excellent correlation for quantitative assessment (r = 0.99, Deming slope = 0.99). Precision studies confirmed high repeatability and within-laboratory precision across multiple operators. Analytical sensitivity was verified at 0.002% MRD. Post-implementation monitoring identified 2.97% of cases (26\/874) with input data drift, primarily high-burden CLL and non-CLL neoplasms. Error analysis showed the DNN alone achieved 97% sensitivity compared to human-in-the-loop-reviewed results, with 13 missed cases (1.5%) showing atypical immunophenotypes. Attribute acceptance sampling confirmed 98.8% of reported negative cases were true negatives. The DNN-assisted workflow reduced average analysis time by 60.3% compared to manual analysis (4.2 \u00b1 2.3 vs. 10.5 \u00b1 5.8 min). Conclusions: The implementation of a DNN-assisted approach for CLL MRD detection in a clinical laboratory provides diagnostic performance equivalent to expert manual analysis while substantially reducing analysis time. Comprehensive performance monitoring ensures ongoing safety and effectiveness in routine clinical practice. This approach provides a model for responsible AI integration in clinical laboratories, balancing automation benefits with expert oversight.<\/jats:p>","DOI":"10.3390\/cancers17101688","type":"journal-article","created":{"date-parts":[[2025,5,19]],"date-time":"2025-05-19T03:51:03Z","timestamp":1747626663000},"page":"1688","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Clinical Validation and Post-Implementation Performance Monitoring of a Neural Network-Assisted Approach for Detecting Chronic Lymphocytic Leukemia Minimal Residual Disease by Flow Cytometry"],"prefix":"10.3390","volume":"17","author":[{"given":"Jansen N.","family":"Seheult","sequence":"first","affiliation":[{"name":"Cell Kinetics Laboratory, DLMP, Mayo Clinic, Rochester, MN 55905, USA"},{"name":"Division of Computational Pathology and Informatics, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregory E.","family":"Otteson","sequence":"additional","affiliation":[{"name":"Cell Kinetics Laboratory, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5658-936X","authenticated-orcid":false,"given":"Matthew J.","family":"Weybright","sequence":"additional","affiliation":[{"name":"Cell Kinetics Laboratory, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael M.","family":"Timm","sequence":"additional","affiliation":[{"name":"Cell Kinetics Laboratory, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenchao","family":"Han","sequence":"additional","affiliation":[{"name":"Division of Computational Pathology and Informatics, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dragan","family":"Jevremovic","sequence":"additional","affiliation":[{"name":"Cell Kinetics Laboratory, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4909-6157","authenticated-orcid":false,"given":"Pedro","family":"Horna","sequence":"additional","affiliation":[{"name":"Cell Kinetics Laboratory, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5783-4485","authenticated-orcid":false,"given":"Horatiu","family":"Olteanu","sequence":"additional","affiliation":[{"name":"Cell Kinetics Laboratory, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5741-4293","authenticated-orcid":false,"given":"Min","family":"Shi","sequence":"additional","affiliation":[{"name":"Cell Kinetics Laboratory, DLMP, Mayo Clinic, Rochester, MN 55905, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,5,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"4049","DOI":"10.1200\/JCO.21.01181","article-title":"Minimal Residual Disease Dynamics after Venetoclax-Obinutuzumab Treatment: Extended Off-Treatment Follow-up from the Randomized CLL14 Study","volume":"39","author":"Zhang","year":"2021","journal-title":"J. Clin. Oncol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"955","DOI":"10.1182\/blood-2017-06-792333","article-title":"A model for predicting effect of treatment on progression-free survival using MRD as a surrogate end point in CLL","volume":"131","author":"Dimier","year":"2018","journal-title":"Blood"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1038\/s41408-022-00756-9","article-title":"Chronic lymphocytic leukemia treatment algorithm 2022","volume":"12","author":"Hampel","year":"2022","journal-title":"Blood Cancer J."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1056\/NEJMoa2310063","article-title":"Chronic Lymphocytic Leukemia Therapy Guided by Measurable Residual Disease","volume":"390","author":"Munir","year":"2024","journal-title":"N. Engl. J. Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1182\/hematology.2023000441","article-title":"MRD-directed therapy in CLL: Ready for prime time?","volume":"2023","author":"Rhodes","year":"2023","journal-title":"Hematol. Am. Soc. Hematol. Educ. Program"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"3364","DOI":"10.1038\/s41375-021-01419-7","article-title":"Measurable residual disease testing in chronic lymphocytic leukaemia: Hype, hope neither or both?","volume":"35","author":"Yang","year":"2021","journal-title":"Leukemia"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"929","DOI":"10.1038\/leu.2015.313","article-title":"A complementary role of multiparameter flow cytometry and high-throughput sequencing for minimal residual disease detection in chronic lymphocytic leukemia: An European Research Initiative on CLL study","volume":"30","author":"Rawstron","year":"2016","journal-title":"Leukemia"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Salama, M.E., Otteson, G.E., Camp, J.J., Seheult, J.N., Jevremovic, D., Holmes, D.R., Olteanu, H., and Shi, M. (2022). Artificial Intelligence Enhances Diagnostic Flow Cytometry Workflow in the Detection of Minimal Residual Disease of Chronic Lymphocytic Leukemia. Cancers, 14.","DOI":"10.3390\/cancers14102537"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1002\/cyto.b.21595","article-title":"Reproducible diagnosis of chronic lymphocytic leukemia by flow cytometry: An European Research Initiative on CLL (ERIC) & European Society for Clinical Cell Analysis (ESCCA) Harmonisation project","volume":"94","author":"Rawstron","year":"2018","journal-title":"Cytom. Part B Clin. Cytom."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1007\/978-1-4939-9650-6_17","article-title":"Clinical Flow-Cytometric Testing in Chronic Lymphocytic Leukemia","volume":"Volume 2032","author":"Salem","year":"2019","journal-title":"Methods in Molecular Biology"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1002\/cyto.b.22170","article-title":"Performance of a novel eight-color flow cytometry panel for measurable residual disease assessment of chronic lymphocytic leukemia","volume":"106","author":"Chen","year":"2024","journal-title":"Cytom. Part B Clin. Cytom."},{"key":"ref_12","unstructured":"Code of Federal Regulations (1990). Laboratory Requirements. Clinical Laboratory Improvement Amendments. Title 42, Chapter IV, Subchapter G, Part 493, National Archives and Records Administration."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e235514","DOI":"10.1001\/jamahealthforum.2023.5514","article-title":"Patient Safety and Artificial Intelligence in Clinical Care","volume":"5","author":"Ratwani","year":"2024","journal-title":"JAMA Health Forum"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1148\/radiol.2017171920","article-title":"Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction","volume":"286","author":"Park","year":"2018","journal-title":"Radiology"},{"key":"ref_15","first-page":"2346","article-title":"Learning under Concept Drift: A Review","volume":"31","author":"Lu","year":"2019","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1145\/2523813","article-title":"A Survey on Concept Drift Adaptation","volume":"46","author":"Gama","year":"2014","journal-title":"ACM Comput. Surv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"e335","DOI":"10.5858\/arpa.2023-0042-CP","article-title":"Recommendations for Performance Evaluation of Machine Learning in Pathology: A Concept Paper from the College of American Pathologists","volume":"148","author":"Hanna","year":"2024","journal-title":"Arch. Pathol. Lab. Med."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1016\/S0168-1605(02)00107-1","article-title":"Analysing collaborative trials for qualitative microbiological methods: Accordance and concordance","volume":"79","author":"Langton","year":"2002","journal-title":"Int. J. Food Microbiol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1007\/s00769-010-0661-1","article-title":"The determination of precision of qualitative measurement methods by interlaboratory experiments","volume":"15","author":"Wilrich","year":"2010","journal-title":"Accredit. Qual. Assur."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"622","DOI":"10.6004\/jnccn.2022.0031","article-title":"NCCN Guidelines(R) Insights: Chronic Lymphocytic Leukemia\/Small Lymphocytic Lymphoma, Version 3.2022","volume":"20","author":"Wierda","year":"2022","journal-title":"J. Natl. Compr. Cancer Netw."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2745","DOI":"10.1182\/blood-2017-09-806398","article-title":"iwCLL guidelines for diagnosis, indications for treatment, response assessment, and supportive management of CLL","volume":"131","author":"Hallek","year":"2018","journal-title":"Blood"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Alsulimani, A., Akhter, N., Jameela, F., Ashgar, R.I., Jawed, A., Hassani, M.A., and Dar, S.A. (2024). The Impact of Artificial Intelligence on Microbial Diagnosis. Microorganisms, 12.","DOI":"10.3390\/microorganisms12061051"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1226","DOI":"10.1016\/j.echo.2022.07.004","article-title":"Human versus Artificial Intelligence-Based Echocardiographic Analysis as a Predictor of Outcomes: An Analysis from the World Alliance Societies of Echocardiography COVID Study","volume":"35","author":"Asch","year":"2022","journal-title":"J. Am. Soc. Echocardiogr."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1038\/s41591-024-03301-2","article-title":"Clinical validation of an AI-based pathology tool for scoring of metabolic dysfunction-associated steatohepatitis","volume":"31","author":"Pulaski","year":"2025","journal-title":"Nat. Med."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"471","DOI":"10.1182\/blood-2015-03-585075","article-title":"Targeted therapies in CLL: Mechanisms of resistance and strategies for management","volume":"126","author":"Woyach","year":"2015","journal-title":"Blood"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"106031","DOI":"10.1016\/j.cie.2019.106031","article-title":"Machine learning based concept drift detection for predictive maintenance","volume":"137","author":"Zenisek","year":"2019","journal-title":"Comput. Ind. Eng."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhou, X., Faro, W., Zhang, X., and Arvapally, R. (2019). A Framework to Monitor Machine Learning Systems Using Concept Drift Detection. Business Information Systems, Springer.","DOI":"10.1007\/978-3-030-20485-3_17"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1038\/s41746-024-01093-w","article-title":"Understanding the errors made by artificial intelligence algorithms in histopathology in terms of patient impact","volume":"7","author":"Evans","year":"2024","journal-title":"NPJ Digit. Med."},{"key":"ref_29","unstructured":"US Food and Drug Administration (2025, May 14). Guidance for Industry: Pre-Storage Leukocyte Reduction of Whole Blood and Blood Components Intended for Transfusion, Available online: https:\/\/www.fda.gov\/media\/84460\/download."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1002\/cyto.b.22116","article-title":"Automated quantification of measurable residual disease in chronic lymphocytic leukemia using an artificial intelligence-assisted workflow","volume":"106","author":"Bazinet","year":"2024","journal-title":"Cytom. Part B Clin. Cytom."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1613\/jair.1.16709","article-title":"Confidence-based Estimators for Predictive Performance in Model Monitoring","volume":"82","author":"Nurminen","year":"2025","journal-title":"J. Artif. Intell. Res."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"111902","DOI":"10.1016\/j.jcp.2022.111902","article-title":"Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons","volume":"477","author":"Psaros","year":"2023","journal-title":"J. Comput. Phys."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"123149","DOI":"10.1109\/ACCESS.2023.3329685","article-title":"Improved Uncertainty Quantification for Neural Networks with Bayesian Last Layer","volume":"11","author":"Fiedler","year":"2023","journal-title":"IEEE Access"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"106816","DOI":"10.1016\/j.csda.2019.106816","article-title":"Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation","volume":"142","author":"Kwon","year":"2020","journal-title":"Comput. Stat. Data Anal."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Bethell, D., Gerasimou, S., and Calinescu, R. (March, January 25). Robust uncertainty quantification using conformalised Monte Carlo prediction. Proceedings of the AAAI Conference on Artificial Intelligence 2024, Philadelphia, PA, USA.","DOI":"10.1609\/aaai.v38i19.30084"}],"container-title":["Cancers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-6694\/17\/10\/1688\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:34:20Z","timestamp":1760031260000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-6694\/17\/10\/1688"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,17]]},"references-count":35,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,5]]}},"alternative-id":["cancers17101688"],"URL":"https:\/\/doi.org\/10.3390\/cancers17101688","relation":{},"ISSN":["2072-6694"],"issn-type":[{"value":"2072-6694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,17]]}}}