{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T05:49:56Z","timestamp":1792043396160,"version":"4.4.0"},"reference-count":18,"publisher":"Public Library of Science (PLoS)","issue":"4","license":[{"start":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T00:00:00Z","timestamp":1651190400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>A standardized objective evaluation method is needed to compare machine learning (ML) algorithms as these tools become available for clinical use. Therefore, we designed, built, and tested an evaluation pipeline with the goal of normalizing performance measurement of independently developed algorithms, using a common test dataset of our clinical imaging. Three vendor applications for detecting solid, part-solid, and groundglass lung nodules in chest CT examinations were assessed in this retrospective study using our data-preprocessing and algorithm assessment chain. The pipeline included tools for image cohort creation and de-identification; report and image annotation for ground-truth labeling; server partitioning to receive vendor \u201cblack box\u201d algorithms and to enable model testing on our internal clinical data (100 chest CTs with 243 nodules) from within our security firewall; model validation and result visualization; and performance assessment calculating algorithm recall, precision, and receiver operating characteristic curves (ROC). Algorithm true positives, false positives, false negatives, recall, and precision for detecting lung nodules were as follows: Vendor-1 (194, 23, 49, 0.80, 0.89); Vendor-2 (182, 270, 61, 0.75, 0.40); Vendor-3 (75, 120, 168, 0.32, 0.39). The AUCs for detection of solid (0.61\u20130.74), groundglass (0.66\u20130.86) and part-solid (0.52\u20130.86) nodules varied between the three vendors. Our ML model validation pipeline enabled testing of multi-vendor algorithms within the institutional firewall. Wide variations in algorithm performance for detection as well as classification of lung nodules justifies the premise for a standardized objective ML algorithm evaluation process.<\/jats:p>","DOI":"10.1371\/journal.pone.0267213","type":"journal-article","created":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T17:31:50Z","timestamp":1651253510000},"page":"e0267213","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":5,"title":["Validation pipeline for machine learning algorithm assessment for multiple vendors"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9686-6751","authenticated-orcid":true,"given":"Bernardo C.","family":"Bizzo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shadi","family":"Ebrahimian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark E.","family":"Walters","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark H.","family":"Michalski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katherine P.","family":"Andriole","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keith J.","family":"Dreyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mannudeep K.","family":"Kalra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tarik","family":"Alkasab","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Subba R.","family":"Digumarthy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2022,4,29]]},"reference":[{"key":"pone.0267213.ref001","unstructured":"RSNA Pediatric Bone Age Challenge. Radiological Society of North America 2017. http:\/\/rsnachallenges.cloudapp.net\/competitions\/4. Accessed August 23, 2018."},{"key":"pone.0267213.ref002","unstructured":"Data Science Bowl 2017. Kaggle. https:\/\/www.kaggle.com\/c\/data-science-bowl-2017. Accessed August 23, 2018."},{"key":"pone.0267213.ref003","unstructured":"ISLES: Ischemic Stroke Lesion Segmentation Challenge. http:\/\/www.isles-challenge.org. Accessed August 23, 2018."},{"issue":"10","key":"pone.0267213.ref004","doi-asserted-by":"crossref","first-page":"1993","DOI":"10.1109\/TMI.2014.2377694","article-title":"The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)","volume":"34","author":"BH Menze","year":"2015","journal-title":"IEEE Trans Med Imaging"},{"key":"pone.0267213.ref005","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.media.2017.06.015","article-title":"Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge","volume":"42","author":"AAA Setio","year":"2017","journal-title":"Med Image Anal"},{"issue":"20","key":"pone.0267213.ref006","first-page":"2525","article-title":"Cancer Facts and Figures 2017","volume":"21","author":"American Cancer Society","year":"2017","journal-title":"Genes Dev"},{"issue":"24","key":"pone.0267213.ref007","doi-asserted-by":"crossref","first-page":"1763","DOI":"10.1093\/jnci\/djn384","article-title":"Productivity costs of cancer mortality in the United States: 2000\u20132020","volume":"100","author":"CJ Bradley","year":"2008","journal-title":"J Natl Cancer Inst"},{"key":"pone.0267213.ref008","unstructured":"Centers for Medicare & Medicaid Services. Decision Memo for Screening for Lung Cancer with Low Dose Computed Tomography (CAG-00439N). http:\/\/www.cms.gov\/medicare-coverage-database\/details\/nca-decision-memo.aspx?NCAId=274. Published February 5, 2015. Accessed August 23, 2018."},{"issue":"1","key":"pone.0267213.ref009","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1016\/j.acra.2008.05.022","article-title":"Assessment of Radiologist Performance in the Detection of Lung Nodules","volume":"16","author":"SG Armato","year":"2009","journal-title":"Acad Radiol"},{"issue":"9","key":"pone.0267213.ref010","doi-asserted-by":"crossref","first-page":"1184","DOI":"10.1016\/j.jacr.2017.04.031","article-title":"Creation of an Open Framework for Point-of-Care Computer-Assisted Reporting and Decision Support Tools for Radiologists","volume":"14","author":"TK Alkasab","year":"2017","journal-title":"J Am Coll Radiol"},{"key":"pone.0267213.ref011","first-page":"1","article-title":"A fully automatic artificial intelligence-based CT image analysis system for accurate detection, diagnosis, and quantitative severity evaluation of pulmonary tuberculosis","author":"C Yan","year":"2021","journal-title":"Eur Radiol"},{"key":"pone.0267213.ref012","doi-asserted-by":"crossref","first-page":"106239","DOI":"10.1016\/j.cmpb.2021.106239","article-title":"Automated Data Quality Control in FDOPA brain PET Imaging using Deep Learning","volume":"208","author":"AD Pontoriero","year":"2021","journal-title":"Comput Methods Programs Biomed"},{"issue":"1","key":"pone.0267213.ref013","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/s12350-019-01906-6","article-title":"Quantitative Clinical Nuclear Cardiology, Part 1: Established Applications","volume":"27","author":"EV Garcia","year":"2020","journal-title":"J Nucl Cardiol"},{"issue":"4","key":"pone.0267213.ref014","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1016\/j.acra.2021.09.002","article-title":"FDA-regulated AI Algorithms: Trends, Strengths, and Gaps of Validation Studies","volume":"29","author":"S Ebrahimian","year":"2022","journal-title":"Acad Radiol"},{"issue":"4","key":"pone.0267213.ref015","doi-asserted-by":"crossref","first-page":"582","DOI":"10.1038\/s41591-021-01312-x","article-title":"How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals","volume":"27","author":"E Wu","year":"2021","journal-title":"Nat Med"},{"key":"pone.0267213.ref016","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","article-title":"Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans","volume":"3","author":"M Roberts","year":"2021","journal-title":"Nat Mach Intell"},{"key":"pone.0267213.ref017","unstructured":"https:\/\/www.accessdata.fda.gov\/scripts\/cdrh\/cfdocs\/cfPCD\/classification.cfm?id=QFM, Accessed February 22, 2022."},{"issue":"2","key":"pone.0267213.ref018","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1148\/radiographics.21.2.g01mr17403","article-title":"Atypical Pulmonary Metastases: Spectrum of Radiologic Findings","volume":"21","author":"JB Seo","year":"2001","journal-title":"Radiographics"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0267213","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T17:32:15Z","timestamp":1651253535000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0267213"}},"subtitle":[],"editor":[{"given":"Peter M.A.","family":"van Ooijen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,4,29]]},"references-count":18,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,4,29]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0267213","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,29]]}}}