{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T15:35:04Z","timestamp":1789140904862,"version":"build-2803163510"},"reference-count":38,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Invest Radiol"],"published-print":{"date-parts":[[2019,1]]},"abstract":"<jats:sec>\n            <jats:title>Objectives<\/jats:title>\n            <jats:p>The aim of this study was to compare the diagnostic performance of a deep learning algorithm with that of radiologists in diagnosing maxillary sinusitis on Waters\u2019 view radiographs.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Materials and Methods<\/jats:title>\n            <jats:p>Among 80,475 Waters\u2019 view radiographs, examined between May 2003 and February 2017, 9000 randomly selected cases were classified as normal or maxillary sinusitis based on radiographic findings and divided into training (n = 8000) and validation (n = 1000) sets to develop a deep learning algorithm. Two test sets composed of Waters\u2019 view radiographs with concurrent paranasal sinus computed tomography were labeled based on computed tomography findings: one with temporal separation (n = 140) and the other with geographic separation (n = 200) from the training set. Area under the receiver operating characteristics curve (AUC), sensitivity, and specificity of the algorithm and 5 radiologists were assessed. Interobserver agreement between the algorithm and majority decision of the radiologists was measured. The correlation coefficient between the predicted probability of the algorithm and average confidence level of the radiologists was determined.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>The AUCs of the deep learning algorithm were 0.93 and 0.88 for the temporal and geographic external test sets, respectively. The AUCs of the radiologists were 0.83 to 0.89 for the temporal and 0.75 to 0.84 for the geographic external test sets. The deep learning algorithm showed statistically significantly higher AUC than radiologist in both test sets. In terms of sensitivity and specificity, the deep learning algorithm was comparable to the radiologists. A strong interobserver agreement was noted between the algorithm and radiologists (Cohen \u03ba coefficient, 0.82). The correlation coefficient between the predicted probability of the algorithm and confidence level of radiologists was 0.89 and 0.84 for the 2 test sets, respectively.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusions<\/jats:title>\n            <jats:p>The deep learning algorithm could diagnose maxillary sinusitis on Waters\u2019 view radiograph with superior AUC and comparable sensitivity and specificity to those of radiologists.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1097\/rli.0000000000000503","type":"journal-article","created":{"date-parts":[[2018,8,1]],"date-time":"2018-08-01T20:21:43Z","timestamp":1533154903000},"page":"7-15","source":"Crossref","is-referenced-by-count":103,"title":["Deep Learning in Diagnosis of Maxillary Sinusitis Using Conventional Radiography"],"prefix":"10.1097","volume":"54","author":[{"given":"Youngjune","family":"Kim","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyong Joon","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leonard","family":"Sunwoo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dongjun","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Bundang Hospital, Seongnam;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang-Mo","family":"Nam","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Bundang Hospital, Seongnam;"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jungheum","family":"Cho","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jihyun","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Radiology, Hallym University Sacred Heart Hospital, Anyang; and"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yun Jung","family":"Bae","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roh-Eul","family":"Yoo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Byung Se","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cheolkyu","family":"Jung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jae Hyoung","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","reference":[{"key":"bib1-20230926","first-page":"770","article-title":"Deep residual learning for image recognition","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"bib2-20230926","doi-asserted-by":"crossref","first-page":"754","DOI":"10.2214\/AJR.16.17224","article-title":"Implementing machine learning in radiology practice and research","volume":"208","year":"2017","journal-title":"Am J Roentgenol"},{"key":"bib3-20230926","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1148\/rg.2017160130","article-title":"Machine learning for medical imaging","volume":"37","year":"2017","journal-title":"Radiographics"},{"key":"bib4-20230926","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1148\/radiol.2017170236","article-title":"Performance of a deep-learning neural network model in assessing skeletal maturity on pediatric hand radiographs","volume":"287","year":"2018","journal-title":"Radiology"},{"key":"bib5-20230926","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1097\/RLI.0000000000000341","article-title":"Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs","volume":"52","year":"2017","journal-title":"Invest Radiol"},{"key":"bib6-20230926","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1097\/RLI.0000000000000448","article-title":"Texture analysis and machine learning for detecting myocardial infarction in noncontrast low-dose computed tomography: unveiling the invisible","volume":"53","year":"2018","journal-title":"Invest Radiol"},{"key":"bib7-20230926","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1097\/RLI.0000000000000400","article-title":"Intravoxel incoherent motion: model-free determination of tissue type in abdominal organs using machine learning","volume":"52","year":"2017","journal-title":"Invest Radiol"},{"key":"bib8-20230926","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1097\/RLI.0000000000000358","article-title":"Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer","volume":"52","year":"2017","journal-title":"Invest Radiol"},{"key":"bib9-20230926","first-page":"1795","article-title":"Acute sinusitis: a cost-effective approach to diagnosis and treatment","volume":"58","year":"1998","journal-title":"Am Fam Physician"},{"key":"bib10-20230926","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.pop.2007.09.002","article-title":"The diagnosis and management of acute and chronic sinusitis","volume":"35","year":"2008","journal-title":"Prim Care"},{"key":"bib11-20230926","doi-asserted-by":"crossref","first-page":"427","DOI":"10.4065\/mcp.2010.0392","article-title":"Rhinosinusitis diagnosis and management for the clinician: a synopsis of recent consensus guidelines","volume":"86","year":"2011","journal-title":"Mayo Clin Proc"},{"key":"bib12-20230926","first-page":"1","article-title":"European position paper on rhinosinusitis and nasal polyps 2007","volume":"20","year":"2007","journal-title":"Rhinol Suppl"},{"key":"bib13-20230926","first-page":"S1","article-title":"Clinical practice guideline (update): adult sinusitis","volume":"152","year":"2015","journal-title":"Otolaryngol Head Neck Surg"},{"key":"bib14-20230926","doi-asserted-by":"crossref","first-page":"S13","DOI":"10.1016\/j.jaci.2005.09.048","article-title":"The diagnosis and management of sinusitis: a practice parameter update","volume":"116","year":"2005","journal-title":"J Allergy Clin Immunol"},{"key":"bib15-20230926","doi-asserted-by":"crossref","first-page":"381","DOI":"10.2214\/ajr.156.2.1898819","article-title":"Paranasal sinuses on MR images of the brain: significance of mucosal thickening","volume":"156","year":"1991","journal-title":"Am J Roentgenol"},{"key":"bib16-20230926","doi-asserted-by":"crossref","first-page":"1014","DOI":"10.1097\/00004728-198811000-00019","article-title":"MR imaging of normal nasal cycle: comparison with sinus pathology","volume":"12","year":"1988","journal-title":"J Comput Assist Tomogr"},{"key":"bib17-20230926","doi-asserted-by":"crossref","first-page":"271","DOI":"10.4193\/Rhin08.147","article-title":"Reappraising the role of radiography in the diagnosis of chronic rhinosinusitis","volume":"47","year":"2009","journal-title":"Rhinology"},{"key":"bib18-20230926","first-page":"55","article-title":"Study on the reproducibility of the Waters' views of the maxillary sinuses","volume":"43","year":"2005","journal-title":"Rhinology"},{"key":"bib19-20230926","doi-asserted-by":"crossref","first-page":"856","DOI":"10.1053\/crad.2000.0550","article-title":"The value of the occipitomental (Waters') view in diagnosis of sinusitis: a comparative study with computed tomography","volume":"55","year":"2000","journal-title":"Clin Radiol"},{"key":"bib20-20230926","first-page":"7","article-title":"Antibiotic treatment of patients with mucosal thickening in the paranasal sinuses, and validation of cut-off points in sinus CT","volume":"36","year":"1998","journal-title":"Rhinology"},{"key":"bib21-20230926","doi-asserted-by":"crossref","first-page":"802","DOI":"10.1177\/019459989110500606","article-title":"The impact of screening sinus CT on the planning of functional endoscopic sinus surgery","volume":"105","year":"1991","journal-title":"Otolaryngol Head Neck Surg"},{"key":"bib22-20230926","first-page":"183","article-title":"Use of symptoms, signs, and blood tests to diagnose acute sinus infections in primary care: comparison with computed tomography","volume":"28","year":"1996","journal-title":"Fam Med"},{"key":"bib23-20230926","doi-asserted-by":"crossref","first-page":"S550","DOI":"10.1016\/j.jacr.2017.08.041","article-title":"ACR Appropriateness Criteria\u00ae Sinonasal Disease","volume":"14","year":"2017","journal-title":"J Am Coll Radiol"},{"key":"bib24-20230926","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1259\/dmfr\/65139094","article-title":"Conventional sinus radiography compared with CT in the diagnosis of acute sinusitis","volume":"32","year":"2003","journal-title":"Dentomaxillofac Radiol"},{"key":"bib25-20230926","first-page":"1","article-title":"Diagnosis and treatment of acute bacterial rhinosinusitis","year":"1999","journal-title":"Evid Rep Technol Assess (Summ)"},{"key":"bib26-20230926","doi-asserted-by":"crossref","first-page":"1393","DOI":"10.1016\/j.ijporl.2015.05.032","article-title":"Development of the maxillary sinus from birth to age 18. Postnatal growth pattern","volume":"79","year":"2015","journal-title":"Int J Pediatr Otorhinolaryngol"},{"key":"bib27-20230926","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1093\/bioinformatics\/btt662","article-title":"Effect of separate sampling on classification accuracy","volume":"30","year":"2014","journal-title":"Bioinformatics"},{"key":"bib29-20230926","first-page":"249","article-title":"Understanding the difficulty of training deep feedforward neural networks","year":"2010","journal-title":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics"},{"key":"bib30-20230926","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach","volume":"44","year":"1988","journal-title":"Biometrics"},{"key":"bib31-20230926","first-page":"276","article-title":"Interrater reliability: the kappa statistic","volume":"22","year":"2012","journal-title":"Biochem Med (Zagreb)"},{"key":"bib32-20230926","first-page":"2921","article-title":"Learning deep features for discriminative localization","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"bib34-20230926","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1148\/radiol.2017171920","article-title":"Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction","volume":"286","year":"2018","journal-title":"Radiology"},{"key":"bib35-20230926","doi-asserted-by":"crossref","first-page":"2402","DOI":"10.1001\/jama.2016.17216","article-title":"Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs","volume":"316","year":"2016","journal-title":"JAMA"},{"key":"bib36-20230926","first-page":"489","article-title":"Radiologic correlates of symptom-based diagnostic criteria for chronic rhinosinusitis","volume":"128","year":"2003","journal-title":"Otolaryngol Head Neck Surg"},{"key":"bib37-20230926","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1148\/radiol.2481071451","article-title":"Effective doses in radiology and diagnostic nuclear medicine: a catalog","volume":"248","year":"2008","journal-title":"Radiology"},{"key":"bib38-20230926","doi-asserted-by":"crossref","first-page":"306","DOI":"10.1016\/j.otohns.2008.12.007","article-title":"Revisiting the interpretation of positive sinus CT findings: a radiological and symptom-based review","volume":"140","year":"2009","journal-title":"Otolaryngol Head Neck Surg"},{"key":"bib39-20230926","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1177\/194589240501900411","article-title":"Lack of significant correlation between rhinosinusitis symptoms and specific regions of sinus computer tomography scans","volume":"19","year":"2005","journal-title":"Am J Rhinol"},{"key":"bib40-20230926","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1056\/NEJM199401063300105","article-title":"Computed tomographic study of the common cold","volume":"330","year":"1994","journal-title":"N Engl J Med"}],"container-title":["Investigative Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.lww.com\/00004424-201901000-00002","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,26]],"date-time":"2023-09-26T17:44:44Z","timestamp":1695750284000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.lww.com\/00004424-201901000-00002"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1]]},"references-count":38,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019]]}},"URL":"https:\/\/doi.org\/10.1097\/rli.0000000000000503","relation":{},"ISSN":["1536-0210","0020-9996"],"issn-type":[{"value":"1536-0210","type":"electronic"},{"value":"0020-9996","type":"print"}],"subject":[],"published":{"date-parts":[[2019,1]]}}}