{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,25]],"date-time":"2026-10-25T17:35:48Z","timestamp":1792949748420,"version":"4.5.0"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1012803","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T00:00:00Z","timestamp":1740700800000}}],"reference-count":46,"publisher":"Public Library of Science (PLoS)","issue":"2","license":[{"start":{"date-parts":[[2025,2,13]],"date-time":"2025-02-13T00:00:00Z","timestamp":1739404800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Accurately labeling large datasets is important for biomedical machine learning yet challenging while modern data augmentation methods may generate noise in the training data, which may deteriorate machine learning model performance. Existing approaches addressing noisy training data typically rely on strict modeling assumptions, classification models and well-curated dataset. To address these, we propose a novel reliability-based training-data-cleaning method employing inductive conformal prediction (ICP). This method uses a small set of well-curated training data and leverages ICP-calculated reliability metrics to selectively correct mislabeled data and outliers within vast quantities of noisy training data. The efficacy is validated across three classification tasks with distinct modalities: filtering drug-induced-liver-injury (DILI) literature with free-text title and abstract, predicting ICU admission of COVID-19 patients through CT radiomics and electronic health records, and subtyping breast cancer using RNA-sequencing data. Varying levels of noise to the training labels were introduced via label permutation. Our training-data-cleaning method significantly enhanced the downstream classification performance (paired t-tests, <jats:italic>p<\/jats:italic>\u00a0\u2264\u00a00\u00a0.\u00a005 among 30 random train\/test partitions): significant accuracy enhancement in 86 out of 96 DILI experiments (up to 11.4% increase from 0.812 to 0.905), significant AUROC and AUPRC enhancements in all 48 COVID-19 experiments (up to 23.8% increase from 0.597 to 0.739 for AUROC, and 69.8% increase from 0.183 to 0.311 for AUPRC), and significant accuracy and macro-average F1-score improvements in 47 out of 48 RNA-sequencing experiments (up to 74.6% increase from 0.351 to 0.613 for accuracy, and 89.0% increase from 0.267 to 0.505 for F1-score). The improvement can be both statistically and clinically significant for information retrieval, disease diagnosis and prognosis. The method offers the potential to substantially boost classification performance in biomedical machine learning tasks without necessitating an excessive volume of well-curated training data or strong data distribution and modeling assumptions in existing semi-supervised learning methods.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1012803","type":"journal-article","created":{"date-parts":[[2025,2,13]],"date-time":"2025-02-13T18:51:16Z","timestamp":1739472676000},"page":"e1012803","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":9,"title":["Reliability-enhanced data cleaning in biomedical machine learning using inductive\nconformal prediction"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7058-2603","authenticated-orcid":true,"given":"Xianghao","family":"Zhan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qinmei","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuanning","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guangming","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9965-5466","authenticated-orcid":true,"given":"Olivier","family":"Gevaert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,2,13]]},"reference":[{"issue":"3","key":"pcbi.1012803.ref001","doi-asserted-by":"crossref","first-page":"e28","DOI":"10.2196\/medinform.5909","article-title":"Prediction of sepsis in the intensive care unit with minimal electronic health record data: a machine learning approach","volume":"4","author":"T Desautels","year":"2016","journal-title":"JMIR Med Inform"},{"issue":"1","key":"pcbi.1012803.ref002","first-page":"1","article-title":"AI-based analysis of CT images for rapid triage of COVID-19 patients","volume":"4","author":"Q Xu","year":"2021","journal-title":"NPJ Digitl Med."},{"issue":"7","key":"pcbi.1012803.ref003","doi-asserted-by":"crossref","first-page":"100289","DOI":"10.1016\/j.patter.2021.100289","article-title":"Structuring clinical text with AI: old versus new natural language processing techniques evaluated on eight common cardiovascular diseases","volume":"2","author":"X Zhan","year":"2021","journal-title":"Patterns (N Y)"},{"issue":"4","key":"pcbi.1012803.ref004","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1038\/s42256-023-00633-5","article-title":"Multimodal data fusion for cancer biomarker discovery with deep learning","volume":"5","author":"S Steyaert","year":"2023","journal-title":"Nat Mach Intell"},{"issue":"1","key":"pcbi.1012803.ref005","doi-asserted-by":"crossref","first-page":"4122","DOI":"10.1038\/s41467-023-39933-0","article-title":"Spatial cellular architecture predicts prognosis in glioblastoma","volume":"14","author":"Y Zheng","year":"2023","journal-title":"Nat Commun"},{"issue":"11","key":"pcbi.1012803.ref006","doi-asserted-by":"crossref","first-page":"8765","DOI":"10.1007\/s00330-021-07962-2","article-title":"Advanced gastric cancer: CT radiomics prediction and early detection of downstaging with neoadjuvant chemotherapy","volume":"31","author":"Q Xu","year":"2021","journal-title":"Eur Radiol"},{"issue":"10","key":"pcbi.1012803.ref007","doi-asserted-by":"crossref","first-page":"5033","DOI":"10.1109\/JBHI.2022.3193365","article-title":"Reliably filter drug-induced liver injury literature with natural language processing and conformal prediction","volume":"26","author":"X Zhan","year":"2022","journal-title":"IEEE J Biomed Health Inform"},{"key":"pcbi.1012803.ref008"},{"issue":"1","key":"pcbi.1012803.ref009","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1186\/s12880-020-00435-w","article-title":"Weakly-supervised convolutional neural networks of renal tumor segmentation in abdominal CTA images","volume":"20","author":"G Yang","year":"2020","journal-title":"BMC Med Imaging"},{"key":"pcbi.1012803.ref010"},{"key":"pcbi.1012803.ref011"},{"key":"pcbi.1012803.ref012","doi-asserted-by":"crossref","first-page":"101759","DOI":"10.1016\/j.media.2020.101759","article-title":"Deep learning with noisy labels: exploring techniques and remedies in medical image analysis","volume":"65","author":"D Karimi","year":"2020","journal-title":"Med Image Anal"},{"key":"pcbi.1012803.ref013"},{"key":"pcbi.1012803.ref014"},{"key":"pcbi.1012803.ref015","article-title":"Making deep neural networks robust to label noise: a loss correction approach. In: Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition","author":"G Patrini","year":"2017"},{"key":"pcbi.1012803.ref016"},{"key":"pcbi.1012803.ref017"},{"key":"pcbi.1012803.ref018","article-title":"Joint optimization framework for learning with noisy labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"D Tanaka","year":"2018"},{"key":"pcbi.1012803.ref019","article-title":"Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In: International Conference on Machine Learning. PMLR","author":"L Jiang","year":"2018"},{"key":"pcbi.1012803.ref020"},{"key":"pcbi.1012803.ref021"},{"key":"pcbi.1012803.ref022"},{"key":"pcbi.1012803.ref023"},{"key":"pcbi.1012803.ref024","article-title":"Learning to reweight examples for robust deep learning. In: International Conference on Machine Learning. PMLR","author":"M Ren","year":"2018"},{"key":"pcbi.1012803.ref025","article-title":"Learning from noisy large-scale datasets with minimal supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"A Veit","year":"2017"},{"key":"pcbi.1012803.ref026"},{"key":"pcbi.1012803.ref027"},{"key":"pcbi.1012803.ref028"},{"key":"pcbi.1012803.ref029","doi-asserted-by":"crossref","first-page":"107588","DOI":"10.1016\/j.measurement.2020.107588","article-title":"An electronic nose-based assistive diagnostic prototype for lung cancer detection with conformal prediction","volume":"158","author":"X Zhan","year":"2020","journal-title":"Measurement"},{"issue":"7418","key":"pcbi.1012803.ref030","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1038\/nature11412","article-title":"Comprehensive molecular portraits of human breast tumours","volume":"490","author":"Network Cancer Genome Atlas","year":"2012","journal-title":"Nature"},{"key":"pcbi.1012803.ref031","article-title":"Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. Valletta, Malta: ELRA","author":"R Rehurek","year":"2010"},{"key":"pcbi.1012803.ref032"},{"issue":"1","key":"pcbi.1012803.ref033","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1038\/s41597-019-0055-0","article-title":"BioWordVec, improving biomedical word embeddings with subword information and MeSH","volume":"6","author":"Y Zhang","year":"2019","journal-title":"Sci Data"},{"issue":"6","key":"pcbi.1012803.ref034","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1038\/s41587-020-0546-8","article-title":"Visualizing and interpreting cancer genomics data via the Xena platform","volume":"38","author":"MJ Goldman","year":"2020","journal-title":"Nat Biotechnol"},{"issue":"3","key":"pcbi.1012803.ref035","doi-asserted-by":"crossref","first-page":"100067","DOI":"10.1016\/j.xgen.2021.100067","article-title":"Molecular analysis of TCGA breast cancer histologic types","volume":"1","author":"A Thennavan","year":"2021","journal-title":"Cell Genom"},{"key":"pcbi.1012803.ref036"},{"key":"pcbi.1012803.ref037","first-page":"1","article-title":"CPSC: conformal prediction with shrunken centroids for efficient prediction reliability quantification and data augmentation, a case in alternative herbal medicine classification with electronic nose","volume":"71","author":"L Liu","year":"2022","journal-title":"IEEE Trans Instrum Meas"},{"key":"pcbi.1012803.ref038","doi-asserted-by":"crossref","first-page":"100603","DOI":"10.1016\/j.ejro.2024.100603","article-title":"A tri-light warning system for hospitalized COVID-19 patients: credibility-based risk stratification for future pandemic preparedness","volume":"13","author":"C Xu","year":"2024","journal-title":"Eur J Radiol Open"},{"issue":"10","key":"pcbi.1012803.ref039","doi-asserted-by":"crossref","first-page":"6567","DOI":"10.1073\/pnas.082099299","article-title":"Diagnosis of multiple cancer types by shrunken centroids of gene expression","volume":"99","author":"R Tibshirani","year":"2002","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pcbi.1012803.ref040"},{"key":"pcbi.1012803.ref041"},{"key":"pcbi.1012803.ref042","first-page":"1","article-title":"Unsupervised cross-user adaptation in taste sensation recognition based on surface electromyography","volume":"71","author":"H Wang","year":"2022","journal-title":"IEEE Trans Instrum Meas"},{"key":"pcbi.1012803.ref043"},{"issue":"20","key":"pcbi.1012803.ref044","doi-asserted-by":"crossref","first-page":"22995","DOI":"10.1109\/JSEN.2021.3102488","article-title":"Boost AI power: data augmentation strategies with unlabeled data and conformal prediction, a case in alternative herbal medicine discrimination with electronic nose","volume":"21","author":"L Liu","year":"2021","journal-title":"IEEE Sensors J"},{"key":"pcbi.1012803.ref045"},{"key":"pcbi.1012803.ref046","article-title":"Combining active learning and semi-supervised learning by using selective label spreading. In: 2017 IEEE International Conference on Data Mining Workshops (ICDMW). IEEE","author":"X Chen","year":"2017"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1012803","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T00:00:00Z","timestamp":1740700800000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012803","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T18:37:58Z","timestamp":1740767878000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012803"}},"subtitle":[],"editor":[{"given":"Anders","family":"Wallqvist","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,2,13]]},"references-count":46,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,2,13]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1012803","relation":{"new_version":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1012803","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,13]]}}}