{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T12:48:48Z","timestamp":1792759728690,"version":"4.5.0"},"reference-count":25,"publisher":"Oxford University Press (OUP)","issue":"1123","license":[{"start":{"date-parts":[[2021,6,18]],"date-time":"2021-06-18T00:00:00Z","timestamp":1623974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,7,1]]},"abstract":"<jats:sec>\n                    <jats:title>Objective:<\/jats:title>\n                    <jats:p>Demonstrate the importance of combining multiple readers' opinions, in a context-aware manner, when establishing the reference standard for validation of artificial intelligence (AI) applications for, e.g. chest radiographs. By comparing individual readers, majority vote of a panel, and panel-based discussion, we identify methods which maximize interobserver agreement and label reproducibility.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods:<\/jats:title>\n                    <jats:p>1100 frontal chest radiographs were evaluated for 6 findings: airspace opacity, cardiomegaly, pulmonary edema, fracture, nodules, and pneumothorax. Each image was reviewed by six radiologists, first individually and then via asynchronous adjudication (web-based discussion) in two panels of three readers to resolve disagreements within each panel. We quantified the reproducibility of each method by measuring interreader agreement.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>Panel-based majority vote improved agreement relative to individual readers for all findings. Most disagreements were resolved with two rounds of adjudication, which further improved reproducibility for some findings, particularly reducing misses. Improvements varied across finding categories, with adjudication improving agreement for cardiomegaly, fractures, and pneumothorax.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion:<\/jats:title>\n                    <jats:p>The likelihood of interreader agreement, even within panels of US board-certified radiologists, must be considered before reads can be used as a reference standard for validation of proposed AI tools. Agreement and, by extension, reproducibility can be improved by applying majority vote, maximum sensitivity, or asynchronous adjudication for different findings, which supports the development of higher quality clinical research.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Advances in knowledge:<\/jats:title>\n                    <jats:p>A panel of three experts is a common technique for establishing reference standards when ground truth is not available for use in AI validation. The manner in which differing opinions are resolved is shown to be important, and has not been previously explored.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1259\/bjr.20210435","type":"journal-article","created":{"date-parts":[[2021,6,18]],"date-time":"2021-06-18T10:38:52Z","timestamp":1624012732000},"source":"Crossref","is-referenced-by-count":24,"title":["Improving reference standards for validation of AI-based radiography"],"prefix":"10.1093","volume":"94","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9869-4447","authenticated-orcid":false,"given":"Gavin E","family":"Duggan","sequence":"first","affiliation":[{"name":"Google Health (G.E.D., Y.L., D.T., S.S.), Stanford Health Care and Palo Alto Veterans Affairs (J.J.R.) , California, California , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua J","family":"Reicher","sequence":"additional","affiliation":[{"name":"Google Health (G.E.D., Y.L., D.T., S.S.), Stanford Health Care and Palo Alto Veterans Affairs (J.J.R.) , California, California , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yun","family":"Liu","sequence":"additional","affiliation":[{"name":"Google Health (G.E.D., Y.L., D.T., S.S.), Stanford Health Care and Palo Alto Veterans Affairs (J.J.R.) , California, California , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Tse","sequence":"additional","affiliation":[{"name":"Google Health (G.E.D., Y.L., D.T., S.S.), Stanford Health Care and Palo Alto Veterans Affairs (J.J.R.) , California, California , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shravya","family":"Shetty","sequence":"additional","affiliation":[{"name":"Google Health (G.E.D., Y.L., D.T., S.S.), Stanford Health Care and Palo Alto Veterans Affairs (J.J.R.) , California, California , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,6,18]]},"reference":[{"key":"2024050310314660800_b1","doi-asserted-by":"crossref","first-page":"1205","DOI":"10.3758\/APP.72.5.1205","article-title":"Current perspectives in medical image perception","volume":"72","author":"Krupinski","year":"2010","journal-title":"Atten Percept Psychophys"},{"key":"2024050310314660800_b2","first-page":"889","volume-title":"In: Proceedings of the 26th Annual International Conference on Machine Learning. New York, NY, USA: ACM","author":"Raykar","year":"2009"},{"key":"2024050310314660800_b3","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1148\/radiol.2462070712","article-title":"Fleischner Society: glossary of terms for thoracic imaging","volume":"246","author":"Hansell","year":"2008","journal-title":"Radiology"},{"key":"2024050310314660800_b4","doi-asserted-by":"crossref","first-page":"2729","DOI":"10.1001\/archinte.1994.00420230122014","article-title":"Interobserver variability in the interpretation of chest roentgenograms of patients with possible pneumonia","volume":"154","author":"Young","year":"1994","journal-title":"Arch Intern Med"},{"key":"2024050310314660800_b5","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1148\/radiology.217.2.r00nv14456","article-title":"Measuring performance in chest radiography","volume":"217","author":"Potchen","year":"2000","journal-title":"Radiology"},{"key":"2024050310314660800_b6","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1111\/j.1754-9485.2012.02348.x","article-title":"Common patterns in 558 diagnostic radiology errors","volume":"56","author":"Donald","year":"2012","journal-title":"J Med Imaging Radiat Oncol"},{"key":"2024050310314660800_b7","doi-asserted-by":"crossref","first-page":"1472","DOI":"10.1016\/j.acra.2018.02.018","article-title":"Deep learning in radiology","volume":"25","author":"McBee","year":"2018","journal-title":"Acad Radiol"},{"key":"2024050310314660800_b8","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1148\/radiol.2017162326","article-title":"Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using Convolutional neural networks","volume":"284","author":"Lakhani","year":"2017","journal-title":"Radiology"},{"key":"2024050310314660800_b9","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0204155","article-title":"Deep learning in chest radiography: detection of findings and presence of change","volume":"13","author":"Singh","year":"2018","journal-title":"PLoS One"},{"key":"2024050310314660800_b10","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pmed.1002686","article-title":"Deep learning for chest radiograph diagnosis: a retrospective comparison of the CheXNeXt algorithm to practicing radiologists","volume":"15","author":"Rajpurkar","year":"2018","journal-title":"PLoS Med"},{"key":"2024050310314660800_b11","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pmed.1002697","article-title":"Automated detection of moderate and large pneumothorax on frontal chest x-rays using deep convolutional neural networks: a retrospective study","volume":"15","author":"Taylor","year":"2018","journal-title":"PLoS Med"},{"key":"2024050310314660800_b12","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pmed.1002683","article-title":"Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study","volume":"15","author":"Zech","year":"2018","journal-title":"PLoS Med"},{"key":"2024050310314660800_b13","doi-asserted-by":"crossref","first-page":"1085","DOI":"10.1259\/bjr.70.839.9536897","article-title":"Radiology\u2019s Achilles\u2019 heel: error and variation in the interpretation of the R\u00f6ntgen image","volume":"70","author":"Robinson","year":"1997","journal-title":"Br J Radiol"},{"key":"2024050310314660800_b14","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1148\/radiol.2019192515","article-title":"Assessing Radiology research on artificial intelligence: a brief guide for Authors, Reviewers, and Readers\u2014from the Radiology editorial board","volume":"294","author":"Bluemke","year":"2020","journal-title":"Radiology"},{"key":"2024050310314660800_b15","doi-asserted-by":"crossref","DOI":"10.1186\/1472-6947-13-90","article-title":"Automated identification of pneumonia in chest radiograph reports in critically ill patients","volume":"13","author":"Liu","year":"2013","journal-title":"BMC Med Inform Decis Mak"},{"key":"2024050310314660800_b16","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1148\/radiol.10100252","article-title":"Consensus interpretation in imaging research: is there a better way?","volume":"257","author":"Bankier","year":"2010","journal-title":"Radiology"},{"key":"2024050310314660800_b17","doi-asserted-by":"crossref","first-page":"1264","DOI":"10.1016\/j.ophtha.2018.01.034","article-title":"Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy","volume":"125","author":"Krause","year":"2018","journal-title":"Ophthalmology"},{"key":"2024050310314660800_b18","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1148\/radiol.2019191293","article-title":"Chest radiograph interpretation with deep learning models: assessment with Radiologist-adjudicated reference standards and Population-adjusted evaluation","volume":"294","author":"Majkowska","year":"2020","journal-title":"Radiology"},{"key":"2024050310314660800_b19","unstructured":"Krippendorff \u00a0K\n          \n.\nComputing Krippendorff\u2019s alpha-reliability\n.\n2011\n. Available from:\nhttps:\/\/repository.upenn.edu\/asc_papers\/43\n."},{"key":"2024050310314660800_b20","doi-asserted-by":"crossref","first-page":"720","DOI":"10.1378\/chest.115.3.720","article-title":"Miss rate of lung cancer on the chest radiograph in clinical practice","volume":"115","author":"Quekel","year":"1999","journal-title":"Chest"},{"key":"2024050310314660800_b21","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1053\/j.sult.2012.01.015","article-title":"Learning from errors in radiology: a comprehensive review","volume":"33","author":"Pinto","year":"2012","journal-title":"Seminars in Ultrasound, CT and MRI"},{"key":"2024050310314660800_b22","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1007\/s002470050617","article-title":"Inter- and intra-observer variability in the assessment of atelectasis and consolidation in neonatal chest radiographs","volume":"29","author":"Bloomfield","year":"1999","journal-title":"Pediatr Radiol"},{"key":"2024050310314660800_b23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.15172\/pneu.2014.5\/464","article-title":"The diagnosis of pneumonia requires a chest radiograph (x-ray)\u2013yes, no or sometimes?","volume":"5","author":"Wootton","year":"2014","journal-title":"Pneumonia"},{"key":"2024050310314660800_b24","doi-asserted-by":"crossref","first-page":"1055","DOI":"10.2214\/ajr.137.5.1055","article-title":"Interpretation of radiographs: effect of clinical history","volume":"137","author":"Doubilet","year":"1981","journal-title":"AJR Am of Roentgenol"},{"key":"2024050310314660800_b25","doi-asserted-by":"crossref","first-page":"749","DOI":"10.1148\/radiol.2018171010","article-title":"Double reading in breast cancer screening: cohort evaluation in the CO-OPS trial","volume":"287","author":"Taylor-Phillips","year":"2018","journal-title":"Radiology"}],"container-title":["The British Journal of Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bjr\/article-pdf\/94\/1123\/20210435\/57393127\/bjr.20210435.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bjr\/article-pdf\/94\/1123\/20210435\/57393127\/bjr.20210435.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,3]],"date-time":"2024-05-03T06:32:53Z","timestamp":1714717973000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bjr\/article\/doi\/10.1259\/bjr.20210435\/7663681"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,18]]},"references-count":25,"journal-issue":{"issue":"1123","published-online":{"date-parts":[[2021,6,18]]},"published-print":{"date-parts":[[2021,7,1]]}},"URL":"https:\/\/doi.org\/10.1259\/bjr.20210435","relation":{},"ISSN":["0007-1285","1748-880X"],"issn-type":[{"value":"0007-1285","type":"print"},{"value":"1748-880X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,7,1]]},"published":{"date-parts":[[2021,6,18]]},"article-number":"20210435"}}