{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:40:00Z","timestamp":1760244000102,"version":"build-2065373602"},"reference-count":42,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2009,11,30]],"date-time":"2009-11-30T00:00:00Z","timestamp":1259539200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>This paper uses an ensemble of classifiers and active learning strategies to predict radiologists\u2019 assessment of the nodules of the Lung Image Database Consortium (LIDC). In particular, the paper presents machine learning classifiers that model agreement among ratings in seven semantic characteristics: spiculation, lobulation, texture, sphericity, margin, subtlety, and malignancy. The ensemble of classifiers (which can be considered as a computer panel of experts) uses 64 image features of the nodules across four categories (shape, intensity, texture, and size) to predict semantic characteristics. The active learning begins the training phase with nodules on which radiologists\u2019 semantic ratings agree, and incrementally learns how to classify nodules on which the radiologists do not agree. Using our proposed approach, the classification accuracy of the ensemble of classifiers is higher than the accuracy of a single classifier. In the long run, our proposed approach can be used to increase consistency among radiological interpretations by providing physicians a \u201csecond read\u201d.<\/jats:p>","DOI":"10.3390\/a2041473","type":"journal-article","created":{"date-parts":[[2009,11,30]],"date-time":"2009-11-30T11:14:33Z","timestamp":1259579673000},"page":"1473-1502","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":44,"title":["Predicting Radiological Panel Opinions Using a Panel of Machine Learning Classifiers"],"prefix":"10.3390","volume":"2","author":[{"given":"Dmitriy","family":"Zinovev","sequence":"first","affiliation":[{"name":"Intelligent Multimedia Processing Laboratory, College of Computing and Digital Media, DePaul University, Chicago, IL 60604, USA"}]},{"given":"Daniela","family":"Raicu","sequence":"additional","affiliation":[{"name":"Intelligent Multimedia Processing Laboratory, College of Computing and Digital Media, DePaul University, Chicago, IL 60604, USA"}]},{"given":"Jacob","family":"Furst","sequence":"additional","affiliation":[{"name":"Intelligent Multimedia Processing Laboratory, College of Computing and Digital Media, DePaul University, Chicago, IL 60604, USA"}]},{"given":"Samuel  G.","family":"Armato III","sequence":"additional","affiliation":[{"name":"Department of Radiology, The University of Chicago, Chicago, IL 60637, USA"}]}],"member":"1968","published-online":{"date-parts":[[2009,11,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"739","DOI":"10.1148\/radiol.2323032035","article-title":"Lung Image Database Consortium Research Group. Lung image database consortium: Developing a resource for the medical imaging research community","volume":"232","author":"Armato","year":"2004","journal-title":"Radiology"},{"key":"ref_2","first-page":"No. 2","article-title":"Semi-supervised learning approaches for predicting lung nodules semantic characteristics","volume":"3","author":"Raicu","year":"2009","journal-title":"Intell. Decis. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Chapelle, O., Sch\u00f6lkopf, B., and Zien, A. (2006). Semi-Supervised Learning, MIT.","DOI":"10.7551\/mitpress\/9780262033589.001.0001"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted kappa; nominal scale agreement with provision for scaled disagreement or partial credit","volume":"70","author":"Cohen","year":"1968","journal-title":"Psychol. Bull."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"880","DOI":"10.1118\/1.598603","article-title":"A pattern classification approach to characterizing solitary pulmonary nodules imaged on high resolution CT: Preliminary results","volume":"26","author":"Hart","year":"1999","journal-title":"Med. Phys."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1016\/S0895-6111(99)00033-6","article-title":"The effects of co-occurrence matrix based texture parameters on the classification of solitary pulmonary nodules imaged on computed tomography","volume":"23","author":"Wyckoff","year":"1999","journal-title":"Comput. Med. Imaging Graph."},{"key":"ref_7","unstructured":"Lo, S.C.B., Hsu, L.Y., Freedman, M.T., Lure, Y.M.F., and Zhao, H. (, January February,). Classification of lung nodules in diagnostic CT: An approach based on 3-D vascular features,nodule density distributions, and shape features. Proceedings of SPIE Medical Imaging Conference, San Diego, CA, USA,."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1188","DOI":"10.1118\/1.1573210","article-title":"Automated lung nodule classification following automated nodule detection on CT: A serial approach","volume":"30","author":"Armato","year":"2003","journal-title":"Med. Phys."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1255","DOI":"10.2214\/ajr.180.5.1801255","article-title":"Indeterminate solitary pulmonary nodules revealed at population-based CT screening of the lung: using first follow-up diagnostic CT to differentiate benign and malignant lesions","volume":"180","author":"Takashima","year":"2003","journal-title":"Am. J. Roentgenol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"955","DOI":"10.2214\/ajr.180.4.1800955","article-title":"Small solitary pulmonary nodules (<1 cm) detected at population-based CT screening for lung cancer: reliable high-resolution CT features of benign lesions","volume":"180","author":"Takashima","year":"2003","journal-title":"Am. J. Roentgenol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1016\/j.ics.2004.03.292","article-title":"Computer-aided lung nodule diagnosis using a simple classifier","volume":"6","author":"Shah","year":"2004","journal-title":"Int. Congr. Ser."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1023\/A:1022631118932","article-title":"Very simple classification rules perform well on most commonly used datasets","volume":"11","author":"Holte","year":"1993","journal-title":"Mach. Learning"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Samuel, C.C., Saravanan, V., and Vimala, D.M.R. (2007, January December). Lung nodule diagnosis from CT images using fuzzy logic. Proceedings of International Conference on Computational Intelligence and Multimedia Applications, Sivakasi, Tamilnadu, India.","DOI":"10.1109\/ICCIMA.2007.236"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1109\/TMI.2005.862753","article-title":"Computer analysis of computed tomography scans of the Lung: A survey","volume":"4","author":"Sluimer","year":"2006","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1097\/RTI.0b013e318173dd1f","article-title":"Computer-aided diagnosis in lung nodule assessment","volume":"23","author":"Goldin","year":"2008","journal-title":"J. Thoracic Imaging"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1148\/radiology.186.2.8421743","article-title":"Determining the likelihood of malignancy in solitary pulmonary nodules with Bayesian analysis. Part I. Theory","volume":"186","author":"Gurney","year":"1993","journal-title":"Radiology"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1148\/radiology.186.2.8421744","article-title":"Determining the likelihood of malignancy in solitary pulmonary nodules with Bayesian analysis. Part II. Application","volume":"186","author":"Gurney","year":"1993","journal-title":"Radiology"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"657","DOI":"10.2214\/ajr.178.3.1780657","article-title":"Usefulness of an artificial neural network for differentiating benign from malignant pulmonary nodules on high-resolution CT: Evaluation with receiver operating characteristic analysis","volume":"178","author":"Matsuki","year":"2002","journal-title":"Am. J. Roentgenol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1118\/1.1543575","article-title":"Computerized scheme for determination of the likelihood measure of malignancy for pulmonary nodules on low-dose CT images","volume":"30","author":"Aoyama","year":"2003","journal-title":"Med. Phys."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1007\/s10278-006-0627-3","article-title":"An ontology for PACS integration","volume":"12","author":"Kahn","year":"2006","journal-title":"J. Digital Imaging"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"538","DOI":"10.1109\/TITB.2005.855563","article-title":"Knowledge representation and sharing using visual semantic modeling for diagnostic medical image databases","volume":"9","author":"Barb","year":"2005","journal-title":"IEEE Trans. Inf. Technol. Biomed."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ebadollahi, S., Coden, A., Tanenblatt, M.A., Chang, S.F., Syeda-Mahmood, T.F., and Amir, A. (2006). Concept-based electronic health records: Opportunities and challenges. ACM Multimed., 997\u20131006.","DOI":"10.1145\/1180639.1180859"},{"key":"ref_23","unstructured":"Ebadollahi, S., Johnson, D.E., and Diao, M. (2008). Retrieving clinical cases through a concept space representation of text and images. SPIE Med. Imaging Symp., (submitted)."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1513","DOI":"10.1016\/j.acra.2008.06.005","article-title":"Quantitative analysis of lesion morphology and texture features for diagnostic prediction in breast MRI","volume":"15","author":"Nie","year":"2008","journal-title":"Acad. Radiol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1002\/jmri.20541","article-title":"Breast lesion analysis of shape technique: Semi-automated vs. Manual morphological description","volume":"23","author":"Liney","year":"2006","journal-title":"J. Magn. Reson. Imaging 2006,"},{"key":"ref_26","unstructured":"Raicu, D.S., Varutbangkul, E., Cisneros, J.G., Furst, J.D., Channin, D.S., and Armato, S.G. (, January February,). Semantics and image content integration for pulmonary nodule interpretation in thoracic computed tomography. Proceedings of SPIE Medical Imaging Conference, San Diego, CA, USA."},{"key":"ref_27","unstructured":"Raicu, D.S., Varutbangkul, E., Furst, J.D., and Armato, S.G. (2008). Modeling semantics from image data: opportunities from LIDC. Int. J. Biomed. Eng. Technol., 1\u201322."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Opulencia, P., Channin, D.S., Raicu, D.S., and Furst, J.D. (2009). Mapping LIDC, RadLex, and Lung nodule image features. J. Digital Imaging, (in press).","DOI":"10.1007\/s10278-010-9285-6"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scale","volume":"20","author":"Cohen","year":"1960","journal-title":"Educat. Psychol. Measure."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychol. Bull."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"363","DOI":"10.2307\/2529786","article-title":"An application of hierarchical Kappa-type statistics in the assessment of majority agreement among multiple observers","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"207","DOI":"10.2307\/2529972","article-title":"Extension of the kappa coefficient","volume":"36","author":"Kraemer","year":"1980","journal-title":"Biometrics"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychol. Bull."},{"key":"ref_35","first-page":"360","article-title":"Understanding interobserver agreement: The Kappa statistic","volume":"5","author":"Viera","year":"2005","journal-title":"Fam Med."},{"key":"ref_36","first-page":"1464","article-title":"The Lung image database consortium (LIDC) data collection process for nodule detection and annotation","volume":"12","author":"Armato","year":"2007","journal-title":"Acad. Radiol."},{"key":"ref_37","unstructured":"Philips, C., Li, D., Furst, J., and Raicu, D. An analysis of Co-occurrence and gabor texture classification in 2D and 3D. Proceedings of CARS, Barcelona, Spain."},{"key":"ref_38","unstructured":"Melville, P., and Mooney, R. Constructing diverse classifier ensembles using artificial training examples. Proceedings of 18th International Joint Conference on Artificial Intelligence, Acapulco, Mexico."},{"key":"ref_39","unstructured":"Melville, P., and Mooney, R. (, January July,). Diverse ensembles for active learning. Proceedings of International Conference on Machine Learning, Banff, Alberta, Canada."},{"key":"ref_40","unstructured":"Available online: http:\/\/www.cs.waikato.ac.nz\/ml\/weka\/."},{"key":"ref_41","unstructured":"Mitchell, T.M. (1997). Machine Learning, McGraw-Hill."},{"key":"ref_42","unstructured":"Siena, S., Zinoveva, O., Raicu, D., and Furst, J. (, January February,). Area and shape-dependent variability metric for evaluating panel segmentations of lung nodules in LIDC data. Proceedings of SPIE Medical Imaging Conference, San Francisco, CA, USA. (accepted)."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/2\/4\/1473\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T22:11:49Z","timestamp":1760220709000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/2\/4\/1473"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,11,30]]},"references-count":42,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2009,12]]}},"alternative-id":["a2041473"],"URL":"https:\/\/doi.org\/10.3390\/a2041473","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2009,11,30]]}}}