{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T19:34:50Z","timestamp":1778700890556,"version":"3.51.4"},"reference-count":38,"publisher":"Georg Thieme Verlag KG","issue":"01","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Methods Inf Med"],"published-print":{"date-parts":[[2015]]},"abstract":"<jats:title>Summary<\/jats:title><jats:p>Objective: Inter-expert variability in image-based clinical diagnosis has been demonstrated in many diseases including retinopathy of prematurity (ROP), which is a disease affecting low birth weight infants and is a major cause of childhood blindness. In order to better understand the underlying causes of variability among experts, we propose a method to quantify the variability of expert decisions and analyze the relationship between expert diagnoses and features computed from the images. Identification of these features is relevant for development of computer-based decision support systems and educational systems in ROP, and these methods may be applicable to other diseases where inter-expert variability is observed.<\/jats:p><jats:p>Methods: The experiments were carried out on a dataset of 34 retinal images, each with diagnoses provided independently by 22 experts. Analysis was performed using concepts of Mutual Information (MI) and Kernel Density Estimation. A large set of structural features (a total of 66) were extracted from retinal images. Feature selection was utilized to identify the most important features that correlated to actual clinical decisions by the 22 study experts. The best three features for each observer were selected by an exhaustive search on all possible feature subsets and considering joint MI as a relevance criterion. We also compared our results with the results of Cohen\u2019s Kappa [36] as an inter-rater reliability measure.<\/jats:p><jats:p>Results: The results demonstrate that a group of observers (17 among 22) decide consistently with each other. Mean and second central moment of arteriolar tortuosity is among the reasons of disagreement between this group and the rest of the observers, meaning that the group of experts consider amount of tortuosity as well as the variation of tortuosity in the image.<\/jats:p><jats:p>Conclusion: Given a set of image-based features, the proposed analysis method can identify critical image-based features that lead to expert agreement and disagreement in diagnosis of ROP. Although tree-based features and various statistics such as central moment are not popular in the literature, our results suggest that they are important for diagnosis.<\/jats:p>","DOI":"10.3414\/me13-01-0081","type":"journal-article","created":{"date-parts":[[2014,12,1]],"date-time":"2014-12-01T14:24:53Z","timestamp":1417443893000},"page":"93-102","source":"Crossref","is-referenced-by-count":20,"title":["Analysis of Underlying Causes of Inter-expert Disagreement in Retinopathy of Prematurity Diagnosis"],"prefix":"10.3414","volume":"54","author":[{"given":"J.","family":"Kalpathy-Cramer","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"S.","family":"You","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"K.","family":"Keck","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"D.","family":"Erdogmus","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M. F.","family":"Chiang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"E.","family":"Ataer-Cansizoglu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"194","published-online":{"date-parts":[[2018,1,22]]},"reference":[{"key":"10.3414\/ME13-01-0081-1","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(92)90128-A"},{"key":"10.3414\/ME13-01-0081-2","doi-asserted-by":"publisher","DOI":"10.3414\/ME9230"},{"key":"10.3414\/ME13-01-0081-3","doi-asserted-by":"crossref","unstructured":"Elmore JG, Brenner RJ. The more eyes, the better to see? from double to quadruple reading of screening mammograms. Journal of the National Cancer Institute 2007: 1141-1143","DOI":"10.1093\/jnci\/djm079"},{"key":"10.3414\/ME13-01-0081-4","doi-asserted-by":"publisher","DOI":"10.1001\/archopht.125.7.875"},{"issue":"4","key":"10.3414\/ME13-01-0081-5","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1055\/s-0038-1633886","volume":"43","author":"Van","year":"2004","journal-title":"Methods Inf Med"},{"key":"10.3414\/ME13-01-0081-6","doi-asserted-by":"crossref","unstructured":"Early Treatment For ROP Cooperative Group. Revised indications for the treatment of retinopathy of prematurity; results of the early treatment for retinopathy of prematurity randomized trial. Arch Ophthalmol 2003; (121): 1684\u2006-1694","DOI":"10.1001\/archopht.121.12.1684"},{"issue":"3","key":"10.3414\/ME13-01-0081-7","first-page":"227","volume":"79","author":"Gilbert","year":"2001","journal-title":"Bull World Health Organ"},{"key":"10.3414\/ME13-01-0081-8","doi-asserted-by":"publisher","DOI":"10.1542\/peds.2004-1180"},{"key":"10.3414\/ME13-01-0081-9","doi-asserted-by":"publisher","DOI":"10.1542\/peds.2007-0378"},{"issue":"5","key":"10.3414\/ME13-01-0081-10","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1542\/peds.91.5.859","volume":"91","author":"Javitt,","year":"1993","journal-title":"Pediatrics"},{"key":"10.3414\/ME13-01-0081-11","doi-asserted-by":"crossref","unstructured":"The Committee for the Classi\u04ffcation of Retinopathy of Prematurity. An international classification of retinopathy of prematurity. Arch Ophthalmol 1984: 1130\u2006-1134","DOI":"10.1001\/archopht.1984.01040030908011"},{"key":"10.3414\/ME13-01-0081-12","doi-asserted-by":"crossref","unstructured":"The Committee for the Classi\u04ffcation of Retinopathy of Prematurity. The international classi\u04ffcation of retinopathy of prematurity revisited. Arch Ophthalmol 2005: 991-\u2006999","DOI":"10.1001\/archopht.123.7.991"},{"key":"10.3414\/ME13-01-0081-13","doi-asserted-by":"publisher","DOI":"10.1001\/archopht.125.7.875"},{"key":"10.3414\/ME13-01-0081-14","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaapos.2007.11.022"},{"key":"10.3414\/ME13-01-0081-15","doi-asserted-by":"publisher","DOI":"10.1097\/IAE.0b013e31823ac3c3"},{"key":"10.3414\/ME13-01-0081-16","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajo.2010.04.027"},{"key":"10.3414\/ME13-01-0081-17","doi-asserted-by":"publisher","DOI":"10.3928\/01913913-20110222-01"},{"key":"10.3414\/ME13-01-0081-18","unstructured":"Guyon I, Andr\u00e9 Elisseeff. An introduction to variable and feature selection. The Journal of Machine Learning Research 2003: 1157-1182"},{"key":"10.3414\/ME13-01-0081-19","unstructured":"Molina LC, Llu\u00eds B, Nebot A. Feature selection algorithms: A survey and experimental evaluation. IEEE International Conference on Data Mining (ICDM) 2002: 306\u2006-313"},{"key":"10.3414\/ME13-01-0081-20","unstructured":"Rong J, Ghahramani Z. Learning with multiple labels. Advances in Neural Information Processing Systems 2002: 897-\u2006904"},{"key":"10.3414\/ME13-01-0081-21","first-page":"1297","volume":"99","author":"Raykar","year":"2010","journal-title":"The Journal of Machine Learning Research"},{"key":"10.3414\/ME13-01-0081-22","doi-asserted-by":"publisher","DOI":"10.1016\/S1091-8531(02)00015-0"},{"key":"10.3414\/ME13-01-0081-23","doi-asserted-by":"publisher","DOI":"10.1167\/iovs.05-0646"},{"key":"10.3414\/ME13-01-0081-24","doi-asserted-by":"publisher","DOI":"10.1167\/iovs.07-1353"},{"key":"10.3414\/ME13-01-0081-25","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaapos.2009.06.007"},{"key":"10.3414\/ME13-01-0081-26","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaapos.2007.09.005"},{"key":"10.3414\/ME13-01-0081-27","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2007.10.006"},{"key":"10.3414\/ME13-01-0081-28","doi-asserted-by":"publisher","DOI":"10.1001\/archopht.125.11.1523"},{"key":"10.3414\/ME13-01-0081-29","doi-asserted-by":"publisher","DOI":"10.1001\/archopht.1988.01060130517027"},{"key":"10.3414\/ME13-01-0081-30","doi-asserted-by":"publisher","DOI":"10.1118\/1.2409238"},{"key":"10.3414\/ME13-01-0081-31","doi-asserted-by":"publisher","DOI":"10.3138\/FM57-6770-U75U-7727"},{"key":"10.3414\/ME13-01-0081-32","doi-asserted-by":"crossref","unstructured":"De Boor, C. A Practical Guide to Splines. New York: Springer Verlag; 1978","DOI":"10.1007\/978-1-4612-6333-3"},{"key":"10.3414\/ME13-01-0081-33","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/16.1.30"},{"key":"10.3414\/ME13-01-0081-34","doi-asserted-by":"crossref","unstructured":"Ataer-Cansizoglu E, You S, Kalpathy-Cramer J, Keck K, Chiang MF, Erdogmus D. Observer and feature analysis on diagnosis of retinopathy of prematurity. IEEE International Workshop on Machine Learning for Signal Processing (MLSP) 2012: 1-\u20066","DOI":"10.1109\/MLSP.2012.6349809"},{"issue":"4","key":"10.3414\/ME13-01-0081-35","first-page":"349","volume":"7","author":"Finn","year":"1993","journal-title":"International Journal of Information Systems"},{"key":"10.3414\/ME13-01-0081-36","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"issue":"3","key":"10.3414\/ME13-01-0081-37","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1093\/ptj\/85.3.257","volume":"85","author":"Sim","year":"2005","journal-title":"Physical therapy"},{"issue":"3","key":"10.3414\/ME13-01-0081-38","first-page":"395","volume":"28","author":"Klemens","year":"2012","journal-title":"Journal of Official Statistics"}],"container-title":["Methods of Information in Medicine"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.3414\/ME13-01-0081.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,23]],"date-time":"2022-04-23T15:55:36Z","timestamp":1650729336000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.3414\/ME13-01-0081"}},"subtitle":["Application of Machine Learning Principles"],"short-title":[],"issued":{"date-parts":[[2015]]},"references-count":38,"journal-issue":{"issue":"01","published-online":{"date-parts":[[2018,1,22]]},"published-print":{"date-parts":[[2015]]}},"URL":"https:\/\/doi.org\/10.3414\/me13-01-0081","relation":{},"ISSN":["0026-1270","2511-705X"],"issn-type":[{"value":"0026-1270","type":"print"},{"value":"2511-705X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015]]}}}