{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T19:37:45Z","timestamp":1772048265278,"version":"3.50.1"},"reference-count":20,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2020,8,10]],"date-time":"2020-08-10T00:00:00Z","timestamp":1597017600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,8,10]],"date-time":"2020-08-10T00:00:00Z","timestamp":1597017600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100009228","name":"Region \u00d6rebro l\u00e4n","doi-asserted-by":"publisher","award":["OLL-878081"],"award-info":[{"award-number":["OLL-878081"]}],"id":[{"id":"10.13039\/501100009228","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Nyckelfonden, Sweden","award":["OLL-881491"],"award-info":[{"award-number":["OLL-881491"]}]},{"name":"Analytic Imaging Diagnostics Arena","award":["-"],"award-info":[{"award-number":["-"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging"],"published-print":{"date-parts":[[2020,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Emphysema is visible on computed tomography (CT) as low-density lesions representing the destruction of the pulmonary alveoli. To train a machine learning model on the emphysema extent in CT images, labeled image data is needed. The provision of these labels requires trained readers, who are a limited resource. The purpose of the study was to test the reading time, inter-observer reliability and validity of the multi-reader\u2013multi-split method for acquiring CT image labels from radiologists. The approximately 500 slices of each stack of lung CT images were split into 1-cm chunks, with 17 thin axial slices per chunk. The chunks were randomly distributed to 26 readers, radiologists and radiology residents. Each chunk was given a quick score concerning emphysema type and severity in the left and right lung separately. A cohort of 102 subjects, with varying degrees of visible emphysema in the lung CT images, was selected from the SCAPIS pilot, performed in 2012 in Gothenburg, Sweden. In total, the readers created 9050 labels for 2881 chunks. Image labels were compared with regional annotations already provided at the SCAPIS pilot inclusion. The median reading time per chunk was 15\u00a0s. The inter-observer Krippendorff\u2019s alpha was 0.40 and 0.53 for emphysema type and score, respectively, and higher in the apical part than in the basal part of the lungs. The multi-split emphysema scores were generally consistent with regional annotations. In conclusion, the multi-reader\u2013multi-split method provided reasonably valid image labels, with an estimation of the inter-observer reliability.<\/jats:p>","DOI":"10.1007\/s10278-020-00378-2","type":"journal-article","created":{"date-parts":[[2020,8,10]],"date-time":"2020-08-10T19:04:05Z","timestamp":1597086245000},"page":"1185-1193","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Multi-Reader\u2013Multi-Split Annotation of Emphysema in Computed Tomography"],"prefix":"10.1007","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1346-1450","authenticated-orcid":false,"given":"Mats","family":"Lid\u00e9n","sequence":"first","affiliation":[]},{"given":"Ola","family":"Hjelmgren","sequence":"additional","affiliation":[]},{"given":"Jenny","family":"Vikgren","sequence":"additional","affiliation":[]},{"given":"Per","family":"Thunberg","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,8,10]]},"reference":[{"key":"378_CR1","doi-asserted-by":"publisher","first-page":"e442","DOI":"10.1371\/journal.pmed.0030442","volume":"3","author":"CD Mathers","year":"2006","unstructured":"Mathers CD, Loncar D: Projections of global mortality and burden of disease from 2002 to 2030. PLoS Med. 3:e442, 2006","journal-title":"PLoS Med."},{"key":"378_CR2","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1164\/rccm.201204-0596PP","volume":"187","author":"J Vestbo","year":"2013","unstructured":"Vestbo J, Hurd SS, Agust\u00ed AG, Jones PW, Vogelmeier C, Anzueto A, Barnes PJ, Fabbri LM, Martinez FJ, Nishimura M, Stockley RA, Sin DD, Rodriguez-Roisin R: Global strategy for the diagnosis, management, and prevention of chronic obstructive pulmonary disease: GOLD executive summary. Am J Respir Crit Care Med 187:347\u201365, 2013","journal-title":"Am J Respir Crit Care Med"},{"key":"378_CR3","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1080\/15412550802093025","volume":"5","author":"GR Washko","year":"2008","unstructured":"Washko GR, Criner GJ, Mohsenifar Z, Sciurba FC, Sharafkhaneh A, Make BJ, Hoffman EA, Reilly JJ: Computed tomographic-based quantification of emphysema and correlation to pulmonary function and mechanics. COPD 5:177\u201386, 2008","journal-title":"COPD"},{"key":"378_CR4","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.rmed.2016.08.023","volume":"119","author":"V Andrianopoulos","year":"2016","unstructured":"Andrianopoulos V, Celli BR, Franssen FME, Pinto-Plata VM, Calverley PMA, Vanfleteren LEGW, Vogiatzis I, Vestbo J, Agusti A, Bakke PS, Rennard SI, MacNee W, Tal-Singer R, Yates JC, Wouters EFM, Spruit MA: Determinants of exercise-induced oxygen desaturation including pulmonary emphysema in COPD: Results from the ECLIPSE study. Respir Med 119:87\u201395, 2016","journal-title":"Respir Med"},{"key":"378_CR5","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1136\/thoraxjnl-2011-201185","volume":"67","author":"CH Martinez","year":"2012","unstructured":"Martinez CH, Chen Y-H, Westgate PM, Liu LX, Murray S, Curtis JL, Make BJ, Kazerooni EA, Lynch DA, Marchetti N, Washko GR, Martinez FJ, Han MK, COPDGene Investigators: Relationship between quantitative CT metrics and health status and BODE in chronic obstructive pulmonary disease. Thorax 67:399\u2013406, 2012","journal-title":"Thorax"},{"key":"378_CR6","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1148\/radiol.2018172294","volume":"288","author":"DA Lynch","year":"2018","unstructured":"Lynch DA, Moore CM, Wilson C, et al: CT-based Visual Classification of Emphysema: Association with Mortality in the COPDGene Study. Radiology 288:859\u2013866, 2018","journal-title":"Radiology"},{"key":"378_CR7","doi-asserted-by":"publisher","unstructured":"Vikgren J, Khalil M, Cederlund K, S\u00f6rensen K, Boijsen M, Brandberg J, Lampa E, Sk\u00f6ld MC, Wollmer P, Lindberg E, Engvall JE, Bergstr\u00f6m G, Tor\u00e9n K, Johnsson \u00c5A: Visual and Quantitative Evaluation of Emphysema: A Case-Control Study of 1111 Participants in the Pilot Swedish CArdioPulmonary BioImage Study (SCAPIS). Acad Radiol. https:\/\/doi.org\/10.1016\/j.acra.2019.06.019. July 17, 2019","DOI":"10.1016\/j.acra.2019.06.019"},{"key":"378_CR8","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1164\/rccm.201705-0860OC","volume":"197","author":"G Gonz\u00e1lez","year":"2018","unstructured":"Gonz\u00e1lez G, Ash SY, Vegas-S\u00e1nchez-Ferrero G, Onieva Onieva J, Rahaghi FN, Ross JC, D\u00edaz A, San Jos\u00e9 Est\u00e9par R, Washko GR, COPDGene and ECLIPSE Investigators: Disease Staging and Prognosis in Smokers Using Deep Learning in Chest Computed Tomography. Am J Respir Crit Care Med 197:193\u2013203, 2018","journal-title":"Am J Respir Crit Care Med"},{"key":"378_CR9","doi-asserted-by":"publisher","first-page":"590","DOI":"10.1148\/radiol.2018180547","volume":"290","author":"S Soffer","year":"2019","unstructured":"Soffer S, Ben-Cohen A, Shimon O, Amitai MM, Greenspan H, Klang E: Convolutional Neural Networks for Radiologic Images: A Radiologist\u2019s Guide. Radiology 290:590\u2013606, 2019","journal-title":"Radiology"},{"key":"378_CR10","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1148\/radiol.2018171820","volume":"288","author":"G Choy","year":"2018","unstructured":"Choy G, Khalilzadeh O, Michalski M, Do S, Samir AE, Pianykh OS, Geis JR, Pandharipande P V, Brink JA, Dreyer KJ: Current Applications and Future Impact of Machine Learning in Radiology. Radiology 288:318\u2013328, 2018","journal-title":"Radiology"},{"key":"378_CR11","doi-asserted-by":"publisher","first-page":"570","DOI":"10.3348\/kjr.2017.18.4.570","volume":"18","author":"J-G Lee","year":"2017","unstructured":"Lee J-G, Jun S, Cho Y-W, Lee H, Kim GB, Seo JB, Kim N: Deep Learning in Medical Imaging: General Overview. Korean J Radiol 18:570\u2013584, 2017","journal-title":"Korean J Radiol"},{"key":"378_CR12","unstructured":"L\u00e4ngkvist M, Widell J, Thunberg P, Loutfi A, Lid\u00e9n M: Interactive user interface based on Convolutional Auto-encoders for annotating CTscans. arXiv:1904.11701, 2019"},{"key":"378_CR13","doi-asserted-by":"publisher","unstructured":"Parakh A, Lee H, Lee JH, Eisner BH, Sahani D V, Do S: Urinary Stone Detection on CT Images Using Deep Convolutional Neural Networks: Evaluation of Model Performance and Generalization. Radiol Artif Intell. https:\/\/doi.org\/10.1148\/ryai.2019180066. July 14, 2019","DOI":"10.1148\/ryai.2019180066"},{"key":"378_CR14","doi-asserted-by":"crossref","unstructured":"L\u00e4ngkvist M, Jendeberg J, Thunberg P, Loutfi A, Lid\u00e9n M: Computer aided detection of ureteral stones in thin slice computed tomography volumes using Convolutional Neural Networks. Comput Biol Med 97:153\u201360, 2018","DOI":"10.1016\/j.compbiomed.2018.04.021"},{"key":"378_CR15","doi-asserted-by":"publisher","first-page":"487","DOI":"10.1007\/s10278-017-9988-z","volume":"30","author":"H Lee","year":"2017","unstructured":"Lee H, Troschel FM, Tajmir S, Fuchs G, Mario J, Fintelmann FJ, Do S: Pixel-Level Deep Segmentation: Artificial Intelligence Quantifies Muscle on Computed Tomography for Body Morphometric Analysis. J Digit Imaging 30:487\u2013498, 2017","journal-title":"J Digit Imaging"},{"key":"378_CR16","doi-asserted-by":"publisher","first-page":"23","DOI":"10.20982\/tqmp.08.1.p023","volume":"8","author":"KA Hallgren","year":"2012","unstructured":"Hallgren KA: Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 8:23\u201334, 2012","journal-title":"Tutor Quant Methods Psychol."},{"key":"378_CR17","doi-asserted-by":"publisher","first-page":"645","DOI":"10.1111\/joim.12384","volume":"278","author":"G Bergstr\u00f6m","year":"2015","unstructured":"Bergstr\u00f6m G, Berglund G, Blomberg A, et al: The Swedish CArdioPulmonary BioImage Study: objectives and design. J Intern Med 278:645\u201359, 2015","journal-title":"J Intern Med"},{"key":"378_CR18","volume-title":"Content Analysis \u2013 An introduction to Its Methodology","author":"K Krippendorff","year":"2004","unstructured":"Krippendorff K: Content Analysis \u2013 An introduction to Its Methodology, 2nd edition, Thousand Oaks, CA, 2004.","edition":"2"},{"key":"378_CR19","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1186\/s12874-016-0200-9","volume":"16","author":"A Zapf","year":"2016","unstructured":"Zapf A, Castell S, Morawietz L, Karch A: Measuring inter-rater reliability for nominal data \u2013 which coefficients and confidence intervals are appropriate? BMC Med Res Methodol 16:93, 2016","journal-title":"BMC Med Res Methodol"},{"key":"378_CR20","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1080\/19312450709336664","volume":"1","author":"AF Hayes","year":"2007","unstructured":"Hayes AF, Krippendorff K: Answering the Call for a Standard Reliability Measure for Coding Data. Commun Methods Meas. 1:77\u201389, 2007","journal-title":"Commun Methods Meas."}],"container-title":["Journal of Digital Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-020-00378-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-020-00378-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-020-00378-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,8,10]],"date-time":"2021-08-10T00:21:08Z","timestamp":1628554868000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-020-00378-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,10]]},"references-count":20,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,10]]}},"alternative-id":["378"],"URL":"https:\/\/doi.org\/10.1007\/s10278-020-00378-2","relation":{},"ISSN":["0897-1889","1618-727X"],"issn-type":[{"value":"0897-1889","type":"print"},{"value":"1618-727X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,8,10]]},"assertion":[{"value":"11 December 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 June 2020","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 July 2020","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 August 2020","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with Ethical Standards"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}},{"value":"All procedures performed were in accordance with the ethical standards of the regional research ethics board and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}}]}}