{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T04:13:12Z","timestamp":1747195992447,"version":"3.40.5"},"reference-count":27,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T00:00:00Z","timestamp":1621900800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Calgary Eye Foundation"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:p>Glaucoma is a leading cause of blindness worldwide whose detection is based on multiple factors, including measuring the cup to disc ratio, retinal nerve fiber layer and visual field defects. Advances in image processing and machine learning have allowed the development of automated approached for segmenting objects from fundus images. However, to build a robust system, a reliable ground truth dataset is required for proper training and validation of the model. In this study, we investigate the level of agreement in properly detecting the retinal disc in fundus images using an online portal built for such purposes. Two Doctors of Optometry independently traced the discs for 159 fundus images obtained from publicly available datasets using a purpose-built online portal. Additionally, we studied the effectiveness of ellipse fitting in handling misalignments in tracing. We measured tracing precision, interobserver variability, and average boundary distance between the results provided by ophthalmologists, and optometrist tracing. We also studied whether ellipse fitting has a positive or negative impact on properly detecting disc boundaries. The overall agreement between the optometrists in terms of locating the disc region in these images was 0.87. However, we found that there was a fair agreement on the disc border with kappa = 0.21. Disagreements were mainly in fundus images obtained from glaucomatous patients. The resulting dataset was deemed to be an acceptable ground truth dataset for training a validation of models for automatic detection of objects in fundus images.<\/jats:p>","DOI":"10.1371\/journal.pone.0251703","type":"journal-article","created":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T17:23:32Z","timestamp":1621963412000},"page":"e0251703","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":2,"title":["Utilizing a responsive web portal for studying disc tracing agreement in retinal images"],"prefix":"10.1371","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6349-2821","authenticated-orcid":true,"given":"Abdullah","family":"Sarhan","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrew","family":"Swift","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adam","family":"Gorner","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jon","family":"Rokne","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Reda","family":"Alhajj","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gavin","family":"Docherty","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrew","family":"Crichton","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2021,5,25]]},"reference":[{"issue":"9","key":"pone.0251703.ref001","doi-asserted-by":"crossref","first-page":"e888","DOI":"10.1016\/S2214-109X(17)30293-0","article-title":"Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis","volume":"5","author":"RR Bourne","year":"2017","journal-title":"The Lancet Global Health"},{"issue":"9","key":"pone.0251703.ref002","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1109\/TMI.2017.2703147","article-title":"Segmentation and quantification for angle-closure glaucoma assessment in anterior segment OCT","volume":"36","author":"H Fu","year":"2017","journal-title":"IEEE transactions on medical imaging"},{"issue":"11","key":"pone.0251703.ref003","doi-asserted-by":"crossref","first-page":"2081","DOI":"10.1016\/j.ophtha.2014.05.013","article-title":"Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis","volume":"121","author":"YC Tham","year":"2014","journal-title":"Ophthalmology"},{"key":"pone.0251703.ref004","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/j.knosys.2012.02.010","article-title":"Data mining technique for automated diagnosis of glaucoma using higher order spectra and wavelet energy features","volume":"33","author":"MRK Mookiah","year":"2012","journal-title":"Knowledge-Based Systems"},{"key":"pone.0251703.ref005","doi-asserted-by":"crossref","unstructured":"Kumar BN, Chauhan R, Dahiya N. Detection of Glaucoma using image processing techniques: A review. In: 2016 International Conference on Microelectronics, Computing and Communications (MicroCom). IEEE; 2016. p. 1\u20136.","DOI":"10.1109\/MicroCom.2016.7522515"},{"key":"pone.0251703.ref006","doi-asserted-by":"crossref","first-page":"101657","DOI":"10.1016\/j.compmedimag.2019.101657","article-title":"Glaucoma detection using image processing techniques: A literature review","author":"A Sarhan","year":"2019","journal-title":"Computerized Medical Imaging and Graphics"},{"journal-title":"Canadian association of optometrists pre-budget submission","year":"2016","author":"B Thienes","key":"pone.0251703.ref007"},{"issue":"22","key":"pone.0251703.ref008","article-title":"Connecting technological innovation in artificial intelligence to real-world medical practice through rigorous clinical validation: what peer-reviewed medical journals could do","volume":"33","author":"SH Park","year":"2018","journal-title":"Journal of Korean Medical Science"},{"issue":"4","key":"pone.0251703.ref009","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1007\/s13246-015-0377-y","article-title":"Glaucoma detection using novel optic disc localization, hybrid feature set and classification techniques","volume":"38","author":"MU Akram","year":"2015","journal-title":"Australasian physical & engineering sciences in medicine"},{"key":"pone.0251703.ref010","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.compbiomed.2014.01.005","article-title":"Application of vascular bundle displacement in the optic disc for glaucoma detection using fundus images","volume":"47","author":"JA De La Fuente-Arriaga","year":"2014","journal-title":"Computers in biology and medicine"},{"issue":"3","key":"pone.0251703.ref011","doi-asserted-by":"crossref","first-page":"466","DOI":"10.1016\/j.bbe.2017.05.008","article-title":"Robust and accurate optic disk localization using vessel symmetry line measure in fundus images","volume":"37","author":"R Panda","year":"2017","journal-title":"Biocybernetics and Biomedical Engineering"},{"key":"pone.0251703.ref012","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1016\/j.media.2019.03.009","article-title":"Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis","volume":"54","author":"V Cheplygina","year":"2019","journal-title":"Medical image analysis"},{"key":"pone.0251703.ref013","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"G Litjens","year":"2017","journal-title":"Medical image analysis"},{"issue":"6","key":"pone.0251703.ref014","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"J Howe","year":"2006","journal-title":"Wired magazine"},{"issue":"2","key":"pone.0251703.ref015","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0117401","article-title":"Crowdsourcing as a screening tool to detect clinical features of glaucomatous optic neuropathy from digital photography","volume":"10","author":"D Mitry","year":"2015","journal-title":"PloS one"},{"key":"pone.0251703.ref016","unstructured":"Amazon Mechanical Turk;. https:\/\/www.mturk.com\/."},{"issue":"5","key":"pone.0251703.ref017","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1167\/tvst.5.5.6","article-title":"The accuracy and reliability of crowdsource annotations of digital retinal images","volume":"5","author":"D Mitry","year":"2016","journal-title":"Translational vision science & technology"},{"key":"pone.0251703.ref018","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/978-3-030-01364-6_11","volume-title":"Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis","author":"J Son","year":"2018"},{"issue":"7","key":"pone.0251703.ref019","doi-asserted-by":"crossref","first-page":"3103","DOI":"10.1167\/iovs.16-19646","article-title":"Beta and gamma peripapillary atrophy in myopic eyes with and without glaucoma","volume":"57","author":"JR Vianna","year":"2016","journal-title":"Investigative ophthalmology & visual science"},{"issue":"43","key":"pone.0251703.ref020","article-title":"A novel fundus image reading tool for efficient generation of a multi-dimensional categorical image database for machine learning algorithm training","volume":"33","author":"SJ Park","year":"2018","journal-title":"Journal of Korean medical science"},{"key":"pone.0251703.ref021","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/ISBI.2014.6867807","volume-title":"2014 IEEE 11th international symposium on biomedical imaging (ISBI)","author":"J Sivaswamy","year":"2014"},{"issue":"3","key":"pone.0251703.ref022","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0032435","article-title":"Fast retinal vessel detection and measurement using wavelets and edge location refinement","volume":"7","author":"P Bankhead","year":"2012","journal-title":"PloS one"},{"key":"pone.0251703.ref023","unstructured":"Tracing Done By Optometrists;. https:\/\/data.mendeley.com\/datasets\/7xv5rzxgrh\/1."},{"issue":"5","key":"pone.0251703.ref024","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1037\/h0028106","article-title":"Large sample standard errors of kappa and weighted kappa","volume":"72","author":"JL Fleiss","year":"1969","journal-title":"Psychological bulletin"},{"issue":"5","key":"pone.0251703.ref025","first-page":"360","article-title":"Understanding interobserver agreement: the kappa statistic","volume":"37","author":"AJ Viera","year":"2005","journal-title":"Fam med"},{"issue":"6","key":"pone.0251703.ref026","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/TMI.2011.2106509","article-title":"Optic disk and cup segmentation from monocular color retinal images for glaucoma assessment","volume":"30","author":"GD Joshi","year":"2011","journal-title":"IEEE transactions on medical imaging"},{"key":"pone.0251703.ref027","unstructured":"Hal\u0131r R, Flusser J. Numerically stable direct least squares fitting of ellipses. In: Proc. 6th International Conference in Central Europe on Computer Graphics and Visualization. WSCG. vol. 98. Citeseer; 1998. p. 125\u2013132."}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0251703","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T17:23:56Z","timestamp":1621963436000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0251703"}},"subtitle":[],"editor":[{"given":"Yuchen","family":"Qiu","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2021,5,25]]},"references-count":27,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,5,25]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0251703","relation":{},"ISSN":["1932-6203"],"issn-type":[{"type":"electronic","value":"1932-6203"}],"subject":[],"published":{"date-parts":[[2021,5,25]]}}}