{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T03:40:10Z","timestamp":1791517210864,"version":"4.3.4"},"reference-count":28,"publisher":"BMJ","issue":"12","license":[{"start":{"date-parts":[[2021,12,6]],"date-time":"2021-12-06T00:00:00Z","timestamp":1638748800000},"content-version":"unspecified","delay-in-days":5,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Annalise.ai","award":["N\/A"],"award-info":[{"award-number":["N\/A"]}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open"],"accepted":{"date-parts":[[2021,11,11]]},"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>To evaluate the ability of a commercially available comprehensive chest radiography deep convolutional neural network (DCNN) to detect simple and tension pneumothorax, as stratified by the following subgroups: the presence of an intercostal drain; rib, clavicular, scapular or humeral fractures or rib resections; subcutaneous emphysema and erect versus non-erect positioning. The hypothesis was that performance would not differ significantly in each of these subgroups when compared with the overall test dataset.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Design<\/jats:title>\n                  <jats:p>A retrospective case\u2013control study was undertaken.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Setting<\/jats:title>\n                  <jats:p>Community radiology clinics and hospitals in Australia and the USA.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Participants<\/jats:title>\n                  <jats:p>A test dataset of 2557 chest radiography studies was ground-truthed by three subspecialty thoracic radiologists for the presence of simple or tension pneumothorax as well as each subgroup other than positioning. Radiograph positioning was derived from radiographer annotations on the images.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Outcome measures<\/jats:title>\n                  <jats:p>DCNN performance for detecting simple and tension pneumothorax was evaluated over the entire test set, as well as within each subgroup, using the area under the receiver operating characteristic curve (AUC). A difference in AUC of more than 0.05 was considered clinically significant.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>When compared with the overall test set, performance of the DCNN for detecting simple and tension pneumothorax was statistically non-inferior in all subgroups. The DCNN had an AUC of 0.981 (0.976\u20130.986) for detecting simple pneumothorax and 0.997 (0.995\u20130.999) for detecting tension pneumothorax.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Hidden stratification has significant implications for potential failures of deep learning when applied in clinical practice. This study demonstrated that a comprehensively trained DCNN can be resilient to hidden stratification in several clinically meaningful subgroups in detecting pneumothorax.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1136\/bmjopen-2021-053024","type":"journal-article","created":{"date-parts":[[2021,12,7]],"date-time":"2021-12-07T11:27:02Z","timestamp":1638876422000},"page":"e053024","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":18,"title":["Do comprehensive deep learning algorithms suffer from hidden stratification? A retrospective study on pneumothorax detection in chest radiography"],"prefix":"10.1136","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2305-7873","authenticated-orcid":false,"given":"Jarrel","family":"Seah","sequence":"first","affiliation":[{"name":"Radiology, Alfred Health, Melbourne, Victoria, Australia"},{"name":"annalise.ai, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cyril","family":"Tang","sequence":"additional","affiliation":[{"name":"annalise.ai, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Quinlan D","family":"Buchlak","sequence":"additional","affiliation":[{"name":"annalise.ai, Sydney, New South Wales, Australia"},{"name":"University of Notre Dame Australia, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2082-5723","authenticated-orcid":false,"given":"Michael Robert","family":"Milne","sequence":"additional","affiliation":[{"name":"annalise.ai, Sydney, New South Wales, Australia"},{"name":"annalise.ai, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xavier","family":"Holt","sequence":"additional","affiliation":[{"name":"annalise.ai, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hassan","family":"Ahmad","sequence":"additional","affiliation":[{"name":"annalise.ai, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Lambert","sequence":"additional","affiliation":[{"name":"annalise.ai, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nazanin","family":"Esmaili","sequence":"additional","affiliation":[{"name":"University of Notre Dame Australia, Sydney, New South Wales, Australia"},{"name":"University of Technology Sydney, Sydney, New South Wales, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luke","family":"Oakden-Rayner","sequence":"additional","affiliation":[{"name":"Australian Institute for Machine Learning, The University of Adelaide, Adelaide, South Australia, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Brotchie","sequence":"additional","affiliation":[{"name":"annalise.ai, Sydney, New South Wales, Australia"},{"name":"Radiology, St Vincent's Hospital Melbourne Pty Ltd, Fitzroy, Victoria, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Catherine M","family":"Jones","sequence":"additional","affiliation":[{"name":"annalise.ai, Sydney, New South Wales, Australia"},{"name":"I-MED Radiology, Brisbane, Queensland, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2021,12,7]]},"reference":[{"key":"2025101814581324000_11.12.e053024.1","doi-asserted-by":"crossref","first-page":"5455","DOI":"10.1007\/s10462-020-09825-6","article-title":"A survey of the recent architectures of deep convolutional neural networks","volume":"53","author":"Khan","year":"2020","journal-title":"Artif Intell Rev"},{"key":"2025101814581324000_11.12.e053024.2","doi-asserted-by":"publisher","DOI":"10.1162\/NECO_a_00990"},{"key":"2025101814581324000_11.12.e053024.3","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1002686"},{"key":"2025101814581324000_11.12.e053024.4","article-title":"Convolutional neural networks in medical image understanding: a survey","volume":"1","author":"Sarvamangala","year":"2021","journal-title":"Evol Intell"},{"key":"2025101814581324000_11.12.e053024.5","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00513-5"},{"key":"2025101814581324000_11.12.e053024.6","doi-asserted-by":"crossref","unstructured":"Oakden-Rayner L , Dunnmon J , Carneiro G . Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In: ACM CHIL 2020 - Proceedings of the 2020 ACM conference on health, inference, and learning. Association for Computing Machinery, Inc, 2020: 151\u20139.","DOI":"10.1145\/3368555.3384468"},{"key":"2025101814581324000_11.12.e053024.7","doi-asserted-by":"publisher","DOI":"10.4103\/2229-5151.134182"},{"key":"2025101814581324000_11.12.e053024.8","unstructured":"Wang X , Peng Y , Lu L . ChestX-ray8: hospital-scale chest X-ray database and benchmarks on Weakly-Supervised classification and localization of common thorax diseases. Available: https:\/\/uts.nlm.nih.gov\/metathesaurus.html [Accessed 18 Apr 2021]."},{"key":"2025101814581324000_11.12.e053024.9","doi-asserted-by":"crossref","first-page":"665","DOI":"10.1038\/s42256-020-00257-z","article-title":"Shortcut learning in deep neural networks","volume":"2","author":"Geirhos","year":"2020","journal-title":"Nat Mach Intell"},{"key":"2025101814581324000_11.12.e053024.10","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1007\/s12663-010-0158-9","article-title":"Subcutaneous emphysema","volume":"14","author":"Balaji","year":"2015","journal-title":"J Maxillofac Oral Surg"},{"key":"2025101814581324000_11.12.e053024.11","doi-asserted-by":"publisher","DOI":"10.1016\/S1010-7940(03)00256-2"},{"key":"2025101814581324000_11.12.e053024.12","doi-asserted-by":"publisher","DOI":"10.1186\/1752-2897-4-12"},{"key":"2025101814581324000_11.12.e053024.13","first-page":"495","article-title":"Video-assisted thoracic surgery complications","volume":"9","author":"\u0141ochowski","year":"2014","journal-title":"Wideochir Inne Tech Maloinwazyjne"},{"key":"2025101814581324000_11.12.e053024.14","unstructured":"Annalise.ai - Annalise CXR comprehensive medical imaging AI. Available: https:\/\/annalise.ai\/products\/annalise-cxr\/ [Accessed 18 Apr 2021]."},{"key":"2025101814581324000_11.12.e053024.15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41597-019-0322-0","article-title":"MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports","volume":"6","author":"Johnson","year":"2019","journal-title":"Sci Data"},{"key":"2025101814581324000_11.12.e053024.16","doi-asserted-by":"crossref","DOI":"10.2307\/2346806","article-title":"Maximum likelihood estimation of observer error-rates using the em algorithm","volume":"28","author":"Dawid","year":"1979","journal-title":"Appl Stat"},{"key":"2025101814581324000_11.12.e053024.17","doi-asserted-by":"publisher","DOI":"10.1097\/JTO.0b013e3181ec173d"},{"key":"2025101814581324000_11.12.e053024.18","doi-asserted-by":"crossref","DOI":"10.1016\/j.conctc.2019.100434","article-title":"Statistical considerations for testing an AI algorithm used for prescreening lung CT images","volume":"16","author":"Obuchowski","year":"2019","journal-title":"Contemp Clin Trials Commun"},{"key":"2025101814581324000_11.12.e053024.19","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-019-0686-2"},{"key":"2025101814581324000_11.12.e053024.20","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpc.2010.04.018"},{"key":"2025101814581324000_11.12.e053024.21","unstructured":"Bird S , Klein E , Loper E . Natural language processing with python: analyzing text with the natural language toolkit. O\u2019Reilly Media, Inc, 2009."},{"key":"2025101814581324000_11.12.e053024.22","unstructured":"Rehurek R . Scalability of semantic analysis in natural language processing 2011."},{"key":"2025101814581324000_11.12.e053024.23","unstructured":"Chollet F . Keras, 2015. Available: https:\/\/github.com\/fchollet\/keras"},{"key":"2025101814581324000_11.12.e053024.24","unstructured":"Goo\u00dfen A , Deshpande H , Harder T . Deep learning for pneumothorax detection and localization in chest radiographs. Available: http:\/\/arxiv.org\/abs\/1907.07324 [Accessed 18 Apr 2021]."},{"key":"2025101814581324000_11.12.e053024.25","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.10026977"},{"key":"2025101814581324000_11.12.e053024.26","doi-asserted-by":"crossref","first-page":"1235","DOI":"10.1007\/s10143-019-01163-8","article-title":"Machine learning applications to clinical decision support in neurosurgery: an artificial intelligence augmented systematic review","volume":"43","author":"Buchlak","year":"2020","journal-title":"Neurosurg Rev"},{"key":"2025101814581324000_11.12.e053024.27","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1007\/s10994-005-4257-7","article-title":"On the application of ROC analysis to predict classification performance under varying class distributions","volume":"58","author":"Webb","year":"2005","journal-title":"Mach Learn"},{"key":"2025101814581324000_11.12.e053024.28","doi-asserted-by":"crossref","first-page":"1372","DOI":"10.1016\/j.amjsurg.2020.06.073","article-title":"Ethical thinking machines in surgery and the requirement for clinical leadership","volume":"220","author":"Buchlak","year":"2020","journal-title":"Am J Surg"}],"container-title":["BMJ Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjopen-2021-053024","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T21:58:20Z","timestamp":1760824700000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjopen.bmj.com\/lookup\/doi\/10.1136\/bmjopen-2021-053024"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":28,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12,7]]},"published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1136\/bmjopen-2021-053024"],"URL":"https:\/\/doi.org\/10.1136\/bmjopen-2021-053024","relation":{},"ISSN":["2044-6055","2044-6055"],"issn-type":[{"value":"2044-6055","type":"print"},{"value":"2044-6055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12]]}}}