{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T16:42:00Z","timestamp":1780418520922,"version":"3.54.1"},"reference-count":38,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2023,8,24]],"date-time":"2023-08-24T00:00:00Z","timestamp":1692835200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background: Optic nerve head (ONH) interpretation is a glaucoma screening method which may be influenced by criteria variability. Laguna ONhE software is a low-cost and non-invasive method of ONH analysis. Methods: We analysed the results of the Laguna ONhE application, interpreting 552 ONH images from the ACRIMA database, publicly available on the Internet, and compared them with the opinion of five experts. Diagnostic agreement was investigated using Cohen\u2019s kappa (\u03ba) with 95% confidence. Results: The kappa concordance index obtained with Laguna ONhE and the majority of the experts\u2019 criterion (0.77) was significantly higher compared to that obtained with ACRIMA and the majority of the experts\u2019 criterion (0.61). In 44.7% of the cases there was absolute agreement among the 5 experts and the Laguna ONhE program. Removing borderline cases from the analysis yielded increased diagnostic agreement (0.81). The area under the receiver operating characteristic (AUROC) of the Laguna ONhE program (0.953, p &lt; 0.001) was not significantly different than AUROC of the majority of the experts\u2019 criterion (0.925, p &lt; 0.001), p = 0.052. Individually obtained expert\u2019s AUROCs were significantly lower (0.636 to 0.913; p &lt; 0.01). Conclusions: Laguna ONhE\u2019s agreement with the experts is high, particularly where the diagnosis may be more obvious by the appearance of the ONH.<\/jats:p>","DOI":"10.3390\/jcm12175485","type":"journal-article","created":{"date-parts":[[2023,8,24]],"date-time":"2023-08-24T10:23:40Z","timestamp":1692872620000},"page":"5485","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Agreement between Five Experts and the Laguna ONhE Automatic Colourimetric Application Interpreting the Glaucomatous Aspect of the Optic Nerve"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3413-5248","authenticated-orcid":false,"given":"Carmen","family":"Mendez-Hernandez","sequence":"first","affiliation":[{"name":"Ophthalmology Department, Hospital Clinico San Carlos, Institute of Health Research (IdISSC), 28232 Madrid, Spain"},{"name":"Department of Immunology, Ophthalmology and ORL IIORC, Complutense University of Madrid, 28232 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1206-4856","authenticated-orcid":false,"given":"Esperanza","family":"Gutierrez-Diaz","sequence":"additional","affiliation":[{"name":"Ophthalmology Department, Hospital 12 de Octubre, 28041 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4094-6346","authenticated-orcid":false,"given":"Marta","family":"Pazos","sequence":"additional","affiliation":[{"name":"Institut of Ophthalmology, Hospital Clinic Barcelona, Universitat de Barcelona, 08036 Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4675-9373","authenticated-orcid":false,"given":"Rafael","family":"Gimenez-Gomez","sequence":"additional","affiliation":[{"name":"Ophthalmology Department, Hospital Universitario Reina Sofia, Instituto Maimonides de Investigaci\u00f3n M\u00e9dica, 14014 Cordoba, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0118-7837","authenticated-orcid":false,"given":"Maria Dolores","family":"Pinazo-Duran","sequence":"additional","affiliation":[{"name":"Ophthalmic Research Unit \u201cSantiago Grisolia\u201d, FISABIO and Department of Surgery, Medical School, University of Valencia, 46010 Valencia, Spain"},{"name":"Spanish Network of Inflammatory Diseases REI-RICORS RD21\/0002\/0032, Institute of Health Carlos III, 28029 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2081","DOI":"10.1016\/j.ophtha.2014.05.013","article-title":"Global Prevalence of Glaucoma and Projections of Glaucoma Burden through 2040: A Systematic Review and Meta-Analysis","volume":"121","author":"Tham","year":"2014","journal-title":"Ophthalmology"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zedan, M.J.M., Zulkifley, M.A., Ibrahim, A.A., Moubark, A.M., Kamari, N.A.M., and Abdani, S.R. (2023). Automated Glaucoma Screening and Diagnosis Based on Retinal Fundus Images Using Deep Learning Approaches: A Comprehensive Review. Diagnostics, 13.","DOI":"10.3390\/diagnostics13132180"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1186\/s12938-019-0649-y","article-title":"NACNNs for Automatic Glaucoma Assessment Using Fundus Images: An Extensive Validation","volume":"18","author":"Morales","year":"2019","journal-title":"Biomed. Eng. Online"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1167\/iovs.12-10761","article-title":"Measuring Hemoglobin Levels in the Optic Nerve Head: Comparisons with Other Structural and Functional Parameters of Glaucoma","volume":"54","author":"Sigut","year":"2013","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1562","DOI":"10.1167\/iovs.14-15592","article-title":"Estimation of the Relative Amount of Hemoglobin in the Cup and Neuroretinal Rim Using Stereoscopic Color Fundus Images","volume":"56","author":"Sigut","year":"2015","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"798","DOI":"10.3109\/02713683.2015.1062112","article-title":"Estimating the Amount of Hemoglobin in the Neuroretinal Rim Using Color Images and OCT","volume":"41","author":"Sigut","year":"2016","journal-title":"Curr. Eye Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1111\/aos.13050","article-title":"Glaucoma Diagnostic Capacity of Optic Nerve Head Haemoglobin Measures Compared with Spectral Domain OCT and HRT III Confocal Tomography","volume":"94","year":"2016","journal-title":"Acta Ophthalmol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Gonzalez-Hernandez, M., Gonzalez-Hernandez, D., Perez-Barbudo, D., Rodriguez-Esteve, P., Betancor-Caro, N., and de la Rosa, M.G. (2021). Fully Automated Colorimetric Analysis of the Optic Nerve Aided by Deep Learning and Its Association with Perimetry and Oct for the Study of Glaucoma. J. Clin. Med., 10.","DOI":"10.3390\/jcm10153231"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"9916102","DOI":"10.1155\/2021\/9916102","article-title":"Optic Nerve Head Hemoglobin Levels in Glaucoma: A Structural and Functional Correlation Study","volume":"2021","author":"Rocha","year":"2021","journal-title":"J. Ophthalmol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e000972","DOI":"10.1136\/bmjophth-2022-000972","article-title":"Optic Disc Area Frequency Distribution in a Large Sample of Retinographic Images","volume":"7","year":"2022","journal-title":"BMJ Open Ophthalmol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"797","DOI":"10.1016\/j.jclinepi.2009.02.005","article-title":"A Review of Solutions for Diagnostic Accuracy Studies with an Imperfect or Missing Reference Standard","volume":"62","author":"Reitsma","year":"2009","journal-title":"J. Clin. Epidemiol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1353","DOI":"10.1001\/jamaophthalmol.2019.3501","article-title":"Development and Validation of a Deep Learning System to Detect Glaucomatous Optic Neuropathy Using Fundus Photographs","volume":"137","author":"Liu","year":"2019","journal-title":"JAMA Ophthalmol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"6694784","DOI":"10.1155\/2021\/6694784","article-title":"An Artificial-Intelligence- And Telemedicine-Based Screening Tool to Identify Glaucoma Suspects from Color Fundus Imaging","volume":"2021","author":"Bhuiyan","year":"2021","journal-title":"J. Ophthalmol."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Gonzalez-Hernandez, M., Gonzalez-Hernandez, D., Betancor-Caro, N., Guedes-Guedes, I., Guldager, M.K., and Gonzalez de la Rosa, M. (2022). Glaucoma Incidence and Progression in Diabetics: The Canary Islands Study Using the Laguna ONhE Application. J. Clin. Med., 11.","DOI":"10.3390\/jcm11247294"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"105717","DOI":"10.1016\/j.cmpb.2020.105717","article-title":"Optic Disc and Optic Cup Segmentation Based on Anatomy Guided Cascade Network","volume":"197","author":"Bian","year":"2020","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"2493","DOI":"10.1109\/TMI.2018.2837012","article-title":"Disc-Aware Ensemble Network for Glaucoma Screening From Fundus Image","volume":"37","author":"Fu","year":"2018","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Jin, B., Liu, P., Wang, P., Shi, L., and Zhao, J. (2020). Optic Disc Segmentation Using Attention-Based U-Net and the Improved Cross-Entropy Convolutional Neural Network. Entropy, 22.","DOI":"10.3390\/e22080844"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"6874765","DOI":"10.1155\/2018\/6874765","article-title":"Comparison of Machine-Learning Classification Models for Glaucoma Management","volume":"2018","author":"An","year":"2018","journal-title":"J. Healthc. Eng."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1109\/TBME.2019.2913211","article-title":"JointRCNN: A Region-Based Convolutional Neural Network for Optic Disc and Cup Segmentation","volume":"67","author":"Jiang","year":"2020","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ahn, J.M., Kim, S., Ahn, K.-S., Cho, S.-H., Lee, K.B., and Kim, U.S. (2018). A Deep Learning Model for the Detection of Both Advanced and Early Glaucoma Using Fundus Photography. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0207982"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1136\/bjophthalmol-2020-316455","article-title":"Diagnostic Validity of Optic Nerve Head Colorimetric Assessment and Optical Coherence Tomography Angiography in Patients with Glaucoma","volume":"105","author":"Wang","year":"2021","journal-title":"Br. J. Ophthalmol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1367","DOI":"10.1016\/S0140-6736(10)61423-7","article-title":"Glaucoma","volume":"377","author":"Quigley","year":"2011","journal-title":"Lancet"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1038\/s41433-019-0667-9","article-title":"Care Pathways for Glaucoma Detection and Monitoring in the UK","volume":"34","author":"Harper","year":"2020","journal-title":"Eye"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1016\/S0161-6420(92)31990-6","article-title":"Expert Agreement in Evaluating the Optic Disc for Glaucoma","volume":"99","author":"Varma","year":"1992","journal-title":"Ophthalmology"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"446","DOI":"10.1097\/00006324-200009000-00005","article-title":"Optic Nerve Evaluation among Optometrists","volume":"77","author":"Spalding","year":"2000","journal-title":"Optom. Vis. Sci."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1007\/s004170100272","article-title":"Agreement between Ophthalmologists and Optometrists in Optic Disc Assessment: Training Implications for Glaucoma Co-Management","volume":"239","author":"Harper","year":"2001","journal-title":"Graefes Arch. Clin. Exp. Ophthalmol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"836","DOI":"10.1016\/S0002-9394(01)01254-5","article-title":"Agreement among Clinicians in the Recognition of Patterns of Optic Disk Damage in Glaucoma","volume":"132","author":"Nicolela","year":"2001","journal-title":"Am. J. Ophthalmol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"742","DOI":"10.1016\/j.ophtha.2010.08.019","article-title":"Agreement and Accuracy of Non-Expert Ophthalmologists in Assessing Glaucomatous Changes in Serial Stereo Optic Disc Photographs","volume":"118","author":"Breusegem","year":"2011","journal-title":"Ophthalmology"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1111\/j.1442-9071.2010.02462.x","article-title":"Glaucomatous Optic Neuropathy Evaluation Project: A Standardized Internet System for Assessing Skills in Optic Disc Examination","volume":"39","author":"Kong","year":"2011","journal-title":"Clin. Exp. Ophthalmol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"931738","DOI":"10.1155\/2014\/931738","article-title":"Correlation between Optic Nerve Parameters Obtained Using 3D Nonmydriatic Retinal Camera and Optical Coherence Tomography: Interobserver Agreement on the Disc Damage Likelihood Scale","volume":"2014","author":"Han","year":"2014","journal-title":"J. Ophthalmol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"OCT80","DOI":"10.1167\/iovs.15-18931","article-title":"Technology and the Glaucoma Suspect","volume":"57","author":"Blumberg","year":"2016","journal-title":"Investig. Ophthalmol. Vis. Sci."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Anton, A., Nolivos, K., Pazos, M., Fatti, G., Herranz, A., Ayala-Fuentes, M.E., Mart\u00ednez-Prats, E., Peral, O., Vega-Lopez, Z., and Monleon-Getino, A. (2021). Interobserver and Intertest Agreement in Telemglaucoma Screening with Optic Disk Photos Ancoherence Tomography. J. Clin. Med., 10.","DOI":"10.3390\/jcm10153337"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1016\/S0161-6420(88)33177-5","article-title":"Intraobserver and Interobserver Agreement in Measurement of Optic Disc Characteristics","volume":"95","author":"Tielsch","year":"1988","journal-title":"Ophthalmology"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1662","DOI":"10.1016\/S0161-6420(94)31118-3","article-title":"Agreement among Optometrists, Ophthalmologists, and Residents in Evaluating the Optic Disc for Glaucoma","volume":"101","author":"Abrams","year":"1994","journal-title":"Ophthalmology"},{"key":"ref_36","first-page":"2340236","article-title":"Relationship between Retinal Nerve Fiber Layer Thickness and Hemoglobin Present in the Optic Nerve Head in Glaucoma","volume":"2017","author":"Saavedra","year":"2017","journal-title":"J. Ophthalmol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e001100","DOI":"10.1136\/bmjophth-2022-001100","article-title":"Comparison of Age-Related Vascular Changes in the Optic Disc of Patients with Diabetes, with Glaucomatous and Non-Glaucomatous Features","volume":"7","year":"2022","journal-title":"BMJ Open Ophthalmol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1097\/IJG.0000000000002026","article-title":"Discrimination Between Healthy Eyes and Those With Mild Glaucoma Damage Using Hemoglobin Measurements of the Optic Nerve Head","volume":"31","author":"Meneses","year":"2022","journal-title":"J. Glaucoma"}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/12\/17\/5485\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:38:01Z","timestamp":1760128681000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/12\/17\/5485"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,24]]},"references-count":38,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["jcm12175485"],"URL":"https:\/\/doi.org\/10.3390\/jcm12175485","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,24]]}}}