{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T18:11:46Z","timestamp":1790273506788,"version":"4.1.0"},"reference-count":45,"publisher":"SPIE-Intl Soc Optical Eng","issue":"05","license":[{"start":{"date-parts":[[2020,10,16]],"date-time":"2020-10-16T00:00:00Z","timestamp":1602806400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Med. Imag."],"published-print":{"date-parts":[[2020,10,16]]},"abstract":"<jats:p>Purpose: Deep learning (DL) algorithms have shown promising results for brain tumor segmentation in MRI. However, validation is required prior to routine clinical use. We report the first randomized and blinded comparison of DL and trained technician segmentations.\nApproach: We compiled a multi-institutional database of 741 pretreatment MRI exams. Each contained a postcontrast T1-weighted exam, a T2-weighted fluid-attenuated inversion recovery exam, and at least one technician-derived tumor segmentation. The database included 729 unique patients (470 males and 259 females). Of these exams, 641 were used for training the DL system, and 100 were reserved for testing. We developed a platform to enable qualitative, blinded, controlled assessment of lesion segmentations made by technicians and the DL method. On this platform, 20 neuroradiologists performed 400 side-by-side comparisons of segmentations on 100 test cases. They scored each segmentation between 0 (poor) and 10 (perfect). Agreement between segmentations from technicians and the DL method was also evaluated quantitatively using the Dice coefficient, which produces values between 0 (no overlap) and 1 (perfect overlap).\nResults: The neuroradiologists gave technician and DL segmentations mean scores of 6.97 and 7.31, respectively (p\u2009\u2009&lt;\u2009\u20090.00007). The DL method achieved a mean Dice coefficient of 0.87 on the test cases.\nConclusions: This was the first objective comparison of automated and human segmentation using a blinded controlled assessment study. Our DL system learned to outperform its \u201chuman teachers\u201d and produced output that was better, on average, than its training data.<\/jats:p>","DOI":"10.1117\/1.jmi.7.5.055501","type":"journal-article","created":{"date-parts":[[2020,10,16]],"date-time":"2020-10-16T13:19:55Z","timestamp":1602854395000},"page":"1-19","source":"Crossref","is-referenced-by-count":13,"title":["Deep neural network to locate and segment brain tumors outperformed the expert technicians who created the training data"],"prefix":"10.1117","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2340-4708","authenticated-orcid":false,"given":"Joseph Ross","family":"Mitchell","sequence":"first","affiliation":[{"name":"Moffitt Cancer Ctr."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Konstantinos","family":"Kamnitsas","sequence":"additional","affiliation":[{"name":"Imperial College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyle W.","family":"Singleton","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Scott A.","family":"Whitmire","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kamala R.","family":"Clark-Swanson","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4344-1282","authenticated-orcid":false,"given":"Sara","family":"Ranjbar","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cassandra R.","family":"Rickertsen","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sandra K.","family":"Johnston","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathleen M.","family":"Egan","sequence":"additional","affiliation":[{"name":"Moffitt Cancer Ctr."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dana E.","family":"Rollison","sequence":"additional","affiliation":[{"name":"Moffitt Cancer Ctr."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8478-7792","authenticated-orcid":false,"given":"John","family":"Arrington","sequence":"additional","affiliation":[{"name":"Moffitt Cancer Ctr."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karl N.","family":"Krecke","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Theodore J.","family":"Passe","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1592-1182","authenticated-orcid":false,"given":"Jared T.","family":"Verdoorn","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex A.","family":"Nagelschneider","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carrie  M.","family":"Carr","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4237-7776","authenticated-orcid":false,"given":"John D.","family":"Port","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alice","family":"Patton","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Norbert G.","family":"Campeau","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Greta B.","family":"Liebo","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laurence J.","family":"Eckel","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher P.","family":"Wood","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0301-0493","authenticated-orcid":false,"given":"Christopher H.","family":"Hunt","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4767-1867","authenticated-orcid":false,"given":"Prasanna","family":"Vibhute","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kent D.","family":"Nelson","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7179-5846","authenticated-orcid":false,"given":"Joseph M.","family":"Hoxworth","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ameet C.","family":"Patel","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6946-4493","authenticated-orcid":false,"given":"Brian W.","family":"Chong","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5865-6828","authenticated-orcid":false,"given":"Jeffrey S.","family":"Ross","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0594-1542","authenticated-orcid":false,"given":"Jerrold L.","family":"Boxerman","sequence":"additional","affiliation":[{"name":"Rhode Island Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael A.","family":"Vogelbaum","sequence":"additional","affiliation":[{"name":"Moffitt Cancer Ctr."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leland S.","family":"Hu","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4897-9356","authenticated-orcid":false,"given":"Ben","family":"Glocker","sequence":"additional","affiliation":[{"name":"Imperial College London"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kristin R.","family":"Swanson","sequence":"additional","affiliation":[{"name":"Mayo Clinic"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"189","reference":[{"key":"r1","doi-asserted-by":"publisher","DOI":"10.1016\/0730-725X(94)00124-L"},{"key":"r2","doi-asserted-by":"publisher","DOI":"10.1016\/S0730-725X(96)00386-4"},{"key":"r3","doi-asserted-by":"publisher","DOI":"10.1148\/radiology.211.3.r99ma07781"},{"key":"r4","first-page":"123","article-title":"A work-efficient GPU algorithm for level set segmentation","author":"Roberts","year":"2010"},{"key":"r5","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-45786-0_46"},{"key":"r6","doi-asserted-by":"publisher","DOI":"10.1148\/radiology.218.2.r01fe44586"},{"key":"r7","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2007.912817"},{"key":"r8","doi-asserted-by":"publisher","DOI":"10.1016\/j.cmpb.2013.04.011"},{"key":"r9","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-55524-9_14"},{"key":"r10","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2014.2377694"},{"key":"r11","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-75238-9_16"},{"key":"r12","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-75238-9_38"},{"key":"r13","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-75238-9_25"},{"key":"r14","doi-asserted-by":"publisher","DOI":"10.1016\/S1470-2045(19)30098-1"},{"key":"r15","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-00536-8_1"},{"key":"r16","doi-asserted-by":"publisher","DOI":"10.1093\/neuonc\/noz106"},{"key":"r17","author":"Swanson","year":"1999"},{"key":"r18","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2016.78"},{"key":"r19","article-title":"Level set methods and fast marching methods: evolving interfaces in computational geometry, fluid mechanics, computer vision, and materials science","author":"Sethian","year":"1999"},{"key":"r20","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2010.2046908"},{"key":"r21","article-title":"Robust automatic whole brain extraction on magnetic resonance imaging of brain tumor patients using dense-Vnet","author":"Ranjbar","year":"2020"},{"key":"r22","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2006.01.021"},{"key":"r23","doi-asserted-by":"publisher","DOI":"10.1109\/ISBI.2009.5193023"},{"key":"r24","doi-asserted-by":"publisher","DOI":"10.1006\/nimg.1995.1012"},{"key":"r25","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2006.06.005"},{"key":"r26","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2016.10.004"},{"key":"r27","doi-asserted-by":"publisher","DOI":"10.2307\/1932409"},{"key":"r28","article-title":"SGDR: stochastic gradient descent with warm restarts","author":"Loshchilov","year":"2016"},{"key":"r29","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2006.01.015"},{"key":"r30","article-title":"AutoHotkey","author":"Mallett","year":"2020"},{"key":"r31","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.c3666"},{"key":"r32","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-11726-9_28"},{"key":"r33","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-11726-9_21"},{"key":"r34","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-11726-9_40"},{"key":"r35","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-11726-9_44"},{"key":"r36","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002683"},{"key":"r37","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-013-9622-7"},{"key":"r38","doi-asserted-by":"publisher","DOI":"10.1016\/0024-3205(78)90098-X"},{"key":"r39","doi-asserted-by":"publisher","DOI":"10.1191\/1352458504ms1061oa"},{"key":"r40","doi-asserted-by":"publisher","DOI":"10.1118\/1.597689"},{"key":"r41","doi-asserted-by":"publisher","DOI":"10.1118\/1.597685"},{"key":"r42","doi-asserted-by":"crossref","DOI":"10.1145\/1401890.1401965","article-title":"Get another label? Improving data quality and data mining using multiple, noisy labelers","author":"Sheng","year":"2008"},{"key":"r43","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"r44","doi-asserted-by":"publisher","DOI":"10.1109\/icdm.2010.147"},{"key":"r45","doi-asserted-by":"publisher","DOI":"10.1109\/icdm.2011.133"}],"container-title":["Journal of Medical Imaging"],"original-title":[],"link":[{"URL":"https:\/\/www.spiedigitallibrary.org\/journals\/journal-of-medical-imaging\/volume-7\/issue-05\/055501\/Deep-neural-network-to-locate-and-segment-brain-tumors-outperformed\/10.1117\/1.JMI.7.5.055501.full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T17:54:59Z","timestamp":1790272499000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.spiedigitallibrary.org\/journals\/journal-of-medical-imaging\/volume-7\/issue-05\/055501\/Deep-neural-network-to-locate-and-segment-brain-tumors-outperformed\/10.1117\/1.JMI.7.5.055501.full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,16]]},"references-count":45,"journal-issue":{"issue":"05"},"URL":"https:\/\/doi.org\/10.1117\/1.jmi.7.5.055501","relation":{},"ISSN":["2329-4302","2329-4310"],"issn-type":[{"value":"2329-4302","type":"print"},{"value":"2329-4310","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,16]]}}}