{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:00:36Z","timestamp":1782842436902,"version":"3.54.5"},"reference-count":40,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"\n Background<\/jats:title>\n As trachoma is eliminated, skilled field graders become less adept at correctly identifying active disease (trachomatous inflammation\u2014follicular [TF]). Deciding if trachoma has been eliminated from a district or if treatment strategies need to be continued or reinstated is of critical public health importance. Telemedicine solutions require both connectivity, which can be poor in the resource-limited regions of the world in which trachoma occurs, and accurate grading of the images.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n Our purpose was to develop and validate a cloud-based \u201cvirtual reading center\u201d (VRC) model using crowdsourcing for image interpretation.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n The Amazon Mechanical Turk (AMT) platform was used to recruit lay graders to interpret 2299 gradable images from a prior field trial of a smartphone-based camera system. Each image received 7 grades for US $0.05 per grade in this VRC. The resultant data set was divided into training and test sets to internally validate the VRC. In the training set, crowdsourcing scores were summed, and the optimal raw score cutoff was chosen to optimize kappa agreement and the resulting prevalence of TF. The best method was then applied to the test set, and the sensitivity, specificity, kappa, and TF prevalence were calculated.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n In this trial, over 16,000 grades were rendered in just over 60 minutes for US $1098 including AMT fees. After choosing an AMT raw score cut point to optimize kappa near the World Health Organization (WHO)\u2013endorsed level of 0.7 (with a simulated 40% prevalence TF), crowdsourcing was 95% sensitive and 87% specific for TF in the training set with a kappa of 0.797. All 196 crowdsourced-positive images received a skilled overread to mimic a tiered reading center and specificity improved to 99%, while sensitivity remained above 78%. Kappa for the entire sample improved from 0.162 to 0.685 with overreads, and the skilled grader burden was reduced by over 80%. This tiered VRC model was then applied to the test set and produced a sensitivity of 99% and a specificity of 76% with a kappa of 0.775 in the entire set. The prevalence estimated by the VRC was 2.70% (95% CI 1.84%-3.80%) compared to the ground truth prevalence of 2.87% (95% CI 1.98%-4.01%).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n A VRC model using crowdsourcing as a first pass with skilled grading of positive images was able to identify TF rapidly and accurately in a low prevalence setting. The findings from this study support further validation of a VRC and crowdsourcing for image grading and estimation of trachoma prevalence from field-acquired images, although further prospective field testing is required to determine if diagnostic characteristics are acceptable in real-world surveys with a low prevalence of the disease.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/41233","type":"journal-article","created":{"date-parts":[[2023,2,19]],"date-time":"2023-02-19T04:39:05Z","timestamp":1676781545000},"page":"e41233","source":"Crossref","is-referenced-by-count":0,"title":["A Virtual Reading Center Model Using Crowdsourcing to Grade Photographs for Trachoma: Validation Study"],"prefix":"10.2196","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7847-3914","authenticated-orcid":false,"given":"Christopher J","family":"Brady","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3224-7617","authenticated-orcid":false,"given":"R Chase","family":"Cockrell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1371-3515","authenticated-orcid":false,"given":"Lindsay R","family":"Aldrich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5736-5850","authenticated-orcid":false,"given":"Meraf A","family":"Wolle","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0818-8100","authenticated-orcid":false,"given":"Sheila K","family":"West","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2023,4,6]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1016\/s2214-109x(19)30450-4"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1186\/s40249-017-0345-8"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1016\/S2214-109X(17)30393-5"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2019.11.001"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1007\/s40135-020-00245-x"},{"issue":"31","key":"ref6","first-page":"353","volume":"96","year":"2021","journal-title":"Wkly Epidemiol Rec"},{"key":"ref7","unstructured":"WHO Alliance for the Global Elimination of TrachomaReport of the third meeting of the WHO Alliance for the Global Elimination of TrachomaWorld Health Organization19982022-06-21https:\/\/apps.who.int\/iris\/handle\/10665\/65933"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1111\/opo.12666"},{"key":"ref9","unstructured":"Reaching the Last Mile Forum: Keynote AddressWorld Health Organization20192022-06-21https:\/\/www.who.int\/director-general\/speeches\/detail\/reaching-the-last-mile-forum"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1093\/inthealth\/ihv049"},{"key":"ref11","unstructured":"Network of WHO collaborating centres for trachoma: second meeting reportWorld Health Organization20172022-06-21https:\/\/apps.who.int\/iris\/handle\/10665\/258687"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1093\/trstmh\/traa171"},{"key":"ref13","unstructured":"Report of the 23rd meeting of the WHO alliance for the global elimination of trachoma by 2020World Health Organization20212022-06-21https:\/\/apps.who.int\/iris\/handle\/10665\/341049"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.4269\/ajtmh.20-0386"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1093\/inthealth\/ihz035"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.1136\/bjo.74.1.12"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1016\/j.survophthal.2021.08.005"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1097\/IJG.0000000000000660"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.7984"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.3807"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pntd.0009928"},{"issue":"7","key":"ref22","first-page":"492","volume":"61","author":"Brady, C","year":"2020","journal-title":"Investig Ophthalmol Visual Sci"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.2471\/BLT.19.248708"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.3109\/09286586.2015.1037401"},{"key":"ref25","unstructured":"Report of the third global scientific meeting on trachomaWorld Health Organization2022-06-21https:\/\/apps.who.int\/iris\/handle\/10665\/329074"},{"key":"ref26","author":"Courtright, P","year":"2019","journal-title":"Tropical Data: Training System for Trachoma Prevalence Surveys"},{"issue":"3","key":"ref27","doi-asserted-by":"crossref","first-page":"505","DOI":"10.4269\/ajtmh.2006.74.505","volume":"74","author":"Solomon, AW","year":"2006","journal-title":"Am J Trop Med Hyg"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1089\/tmj.2020.0006"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0071154"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0117401"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1097\/IOP.0000000000001515"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsurg.2021.02.004"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1007\/s11548-019-01956-8"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1167\/tvst.8.6.40"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2016.17216"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0210463"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1167\/tvst.10.3.19"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.1089\/tmj.2020.0008"},{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pntd.0010943"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-018-0040-6"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2023\/1\/e41233","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:40:45Z","timestamp":1782841245000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2023\/1\/e41233"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,6]]},"references-count":40,"URL":"https:\/\/doi.org\/10.2196\/41233","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.41233","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,6]]}}}