{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T20:53:36Z","timestamp":1774644816209,"version":"3.50.1"},"publisher-location":"Cham","reference-count":49,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031732287","type":"print"},{"value":"9783031732294","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,10,25]],"date-time":"2024-10-25T00:00:00Z","timestamp":1729814400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,10,25]],"date-time":"2024-10-25T00:00:00Z","timestamp":1729814400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper does not describe a novel method. Instead, it studies an essential foundation for reliable benchmarking and ultimately real-world application of AI-based image analysis: generating high-quality reference annotations. Previous research has focused on crowdsourcing as a means of outsourcing annotations. However, little attention has so far been given to annotation companies, specifically regarding their internal quality assurance (QA) processes. Therefore, our aim is to evaluate the influence of QA employed by annotation companies on annotation quality and devise methodologies for maximizing data annotation efficacy. Based on a total of 57,648 instance segmented images obtained from a total of 924 annotators and 34 QA workers from four annotation companies and Amazon Mechanical Turk (MTurk), we derived the following insights: (1) Annotation companies perform better both in terms of quantity and quality compared to the widely used platform MTurk. (2) Annotation companies\u2019 internal QA only provides marginal improvements, if any. However, improving labeling instructions instead of investing in QA can substantially boost annotation performance. (3) The benefit of internal QA depends on specific image characteristics. Our work could enable researchers to derive substantially more value from a fixed annotation budget and change the way annotation companies conduct internal QA.<\/jats:p>","DOI":"10.1007\/978-3-031-73229-4_4","type":"book-chapter","created":{"date-parts":[[2024,10,24]],"date-time":"2024-10-24T15:03:09Z","timestamp":1729782189000},"page":"52-69","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Quality Assured: Rethinking Annotation Strategies in\u00a0Imaging AI"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3518-0315","authenticated-orcid":false,"given":"Tim","family":"R\u00e4dsch","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4363-1876","authenticated-orcid":false,"given":"Annika","family":"Reinke","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7509-3307","authenticated-orcid":false,"given":"Vivienn","family":"Weru","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3687-6381","authenticated-orcid":false,"given":"Minu D.","family":"Tizabi","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8516-8707","authenticated-orcid":false,"given":"Nicholas","family":"Heller","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3519-5886","authenticated-orcid":false,"given":"Fabian","family":"Isensee","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1810-0267","authenticated-orcid":false,"given":"Annette","family":"Kopp-Schneider","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4910-9368","authenticated-orcid":false,"given":"Lena","family":"Maier-Hein","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,10,25]]},"reference":[{"key":"4_CR1","doi-asserted-by":"publisher","unstructured":"Ali, S., et al.: An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy. Sci. Rep. 10(1), 2748 (2020). https:\/\/doi.org\/10.1038\/s41598-020-59413-5, https:\/\/www.nature.com\/articles\/s41598-020-59413-5","DOI":"10.1038\/s41598-020-59413-5"},{"key":"4_CR2","doi-asserted-by":"publisher","unstructured":"Bommasani, R., et al.: On the opportunities and risks of foundation models (2022). https:\/\/doi.org\/10.48550\/arXiv.2108.07258, http:\/\/arxiv.org\/abs\/2108.07258","DOI":"10.48550\/arXiv.2108.07258"},{"key":"4_CR3","doi-asserted-by":"publisher","unstructured":"Bragg, J., Mausam, Weld, D.S.: Sprout: crowd-powered task design for crowdsourcing. In: Proceedings of the ACM on User Interface Software and Technology (2018). https:\/\/doi.org\/10.1145\/3242587.3242598","DOI":"10.1145\/3242587.3242598"},{"key":"4_CR4","doi-asserted-by":"publisher","unstructured":"Chang, J.C., Amershi, S., Kamar, E.: Revolt: collaborative crowdsourcing for labeling machine learning datasets. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI 2017, pp. 2334\u20132346. Association for Computing Machinery, New York (2017). https:\/\/doi.org\/10.1145\/3025453.3026044","DOI":"10.1145\/3025453.3026044"},{"key":"4_CR5","doi-asserted-by":"publisher","unstructured":"Chen, L., et al.: AlpaGasus: training a better alpaca with fewer data (2023). https:\/\/doi.org\/10.48550\/arXiv.2307.08701, http:\/\/arxiv.org\/abs\/2307.08701","DOI":"10.48550\/arXiv.2307.08701"},{"key":"4_CR6","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1007\/978-3-319-46976-8_22","volume-title":"Deep Learning and Data Labeling for Medical Applications","author":"V Cheplygina","year":"2016","unstructured":"Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H.A.W.M., de Bruijne, M.: Early experiences with crowdsourcing airway annotations in chest CT. In: Carneiro, G., et al. (eds.) LABELS\/DLMIA -2016. LNCS, vol. 10008, pp. 209\u2013218. Springer, Cham (2016). https:\/\/doi.org\/10.1007\/978-3-319-46976-8_22"},{"issue":"4","key":"4_CR7","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1177\/1948550619875149","volume":"11","author":"M Chmielewski","year":"2020","unstructured":"Chmielewski, M., Kucker, S.C.: An MTurk crisis? Shifts in data quality and the impact on study results. Soc. Psychol. Pers. Sci. 11(4), 464\u2013473 (2020). https:\/\/doi.org\/10.1177\/1948550619875149","journal-title":"Soc. Psychol. Pers. Sci."},{"key":"4_CR8","unstructured":"Cognilytica: Data engineering, preparation, and labeling for AI 2020 (2020). https:\/\/www.cognilytica.com\/2020\/01\/31\/data-preparation-labeling-for-ai-2020\/"},{"key":"4_CR9","doi-asserted-by":"publisher","unstructured":"Cr\u00e9quit, P., Mansouri, G., Benchoufi, M., Vivot, A., Ravaud, P.: Mapping of crowdsourcing in health: systematic review. J. Med. Internet Res. 20(5), 187 (2018). https:\/\/doi.org\/10.2196\/jmir.9330, https:\/\/www.jmir.org\/2018\/5\/e187\/","DOI":"10.2196\/jmir.9330"},{"key":"4_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3148148","volume":"51","author":"F Daniel","year":"2019","unstructured":"Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., Allahbakhsh, M.: Quality control in crowdsourcing: a survey of quality attributes, assessment techniques and assurance actions. ACM Comput. Surv. 51, 1\u201340 (2019). https:\/\/doi.org\/10.1145\/3148148","journal-title":"ACM Comput. Surv."},{"issue":"3","key":"4_CR11","doi-asserted-by":"publisher","first-page":"297","DOI":"10.2307\/1932409","volume":"26","author":"LR Dice","year":"1945","unstructured":"Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297\u2013302 (1945)","journal-title":"Ecology"},{"key":"4_CR12","doi-asserted-by":"publisher","unstructured":"Dow, S., Kulkarni, A., Klemmer, S., Hartmann, B.: Shepherding the crowd yields better work. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW 2012, pp. 1013\u20131022. Association for Computing Machinery, New York (2012). https:\/\/doi.org\/10.1145\/2145204.2145355","DOI":"10.1145\/2145204.2145355"},{"key":"4_CR13","doi-asserted-by":"publisher","unstructured":"Duggan, N.M., et al.: Gamified crowdsourcing as a novel approach to lung ultrasound dataset labeling (2023). https:\/\/doi.org\/10.48550\/arXiv.2306.06773, http:\/\/arxiv.org\/abs\/2306.06773","DOI":"10.48550\/arXiv.2306.06773"},{"key":"4_CR14","doi-asserted-by":"publisher","unstructured":"Gaikwad, S., et\u00a0al.: Prototype tasks: improving crowdsourcing results through rapid, iterative task design (2017). https:\/\/doi.org\/10.48550\/arXiv.1707.05645, https:\/\/arxiv.org\/abs\/1707.05645","DOI":"10.48550\/arXiv.1707.05645"},{"key":"4_CR15","doi-asserted-by":"publisher","unstructured":"Halevy, A., Norvig, P., Pereira, F.: The unreasonable effectiveness of data. IEEE Intell. Syst. 24(2), 8\u201312 (2009). https:\/\/doi.org\/10.1109\/MIS.2009.36. iSBN 1541-1672","DOI":"10.1109\/MIS.2009.36"},{"key":"4_CR16","doi-asserted-by":"publisher","unstructured":"Han, D., et al.: Neglected free lunch\u2013learning image classifiers using annotation byproducts (2023). https:\/\/doi.org\/10.48550\/arXiv.2303.17595, https:\/\/arxiv.org\/abs\/2303.17595","DOI":"10.48550\/arXiv.2303.17595"},{"issue":"3","key":"4_CR17","doi-asserted-by":"publisher","first-page":"034002","DOI":"10.1117\/1.JMI.5.3.034002","volume":"5","author":"E Heim","year":"2018","unstructured":"Heim, E., et al.: Large-scale medical image annotation with crowd-powered algorithms. J. Med. Imaging 5(3), 034002 (2018). https:\/\/doi.org\/10.1117\/1.JMI.5.3.034002","journal-title":"J. Med. Imaging"},{"key":"4_CR18","doi-asserted-by":"publisher","unstructured":"Hettiachchi, D., Schaekermann, M., McKinney, T.J., Lease, M.: The challenge of variable effort crowdsourcing and how visible gold can help. In: Proceedings of the ACM on Human-Computer Interaction, vol. 5, pp. 332:1\u2013332:26 (2021). https:\/\/doi.org\/10.1145\/3476073","DOI":"10.1145\/3476073"},{"key":"4_CR19","unstructured":"Hutson, M., Kanzheleva, O., Taggart, C., Campana, B.J.L., Duong, Q.: Quality control challenges in crowdsourcing medical labeling (2019)"},{"key":"4_CR20","doi-asserted-by":"publisher","unstructured":"Idrissi, B.Y., et al.: ImageNet-X: understanding model mistakes with factor of variation annotations (2022). https:\/\/doi.org\/10.48550\/arXiv.2211.01866, http:\/\/arxiv.org\/abs\/2211.01866","DOI":"10.48550\/arXiv.2211.01866"},{"key":"4_CR21","doi-asserted-by":"publisher","unstructured":"Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on Amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 64\u201367. ACM, Washington (2010). https:\/\/doi.org\/10.1145\/1837885.1837906, https:\/\/dl.acm.org\/doi\/10.1145\/1837885.1837906","DOI":"10.1145\/1837885.1837906"},{"issue":"5","key":"4_CR22","doi-asserted-by":"publisher","first-page":"720","DOI":"10.1177\/1461444813511926","volume":"17","author":"L Irani","year":"2015","unstructured":"Irani, L.: The cultural work of microwork. New Media Soc. 17(5), 720\u2013739 (2015). https:\/\/doi.org\/10.1177\/1461444813511926","journal-title":"New Media Soc."},{"key":"4_CR23","doi-asserted-by":"publisher","unstructured":"Irshad, H., et al.: Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: evaluating experts, automated methods, and the crowd. In: Biocomputing 2015, pp. 294\u2013305. WORLD SCIENTIFIC, Kohala Coast (2014). https:\/\/doi.org\/10.1142\/9789814644730_0029, http:\/\/www.worldscientific.com\/doi\/abs\/10.1142\/9789814644730_0029","DOI":"10.1142\/9789814644730_0029"},{"key":"4_CR24","doi-asserted-by":"publisher","unstructured":"Joskowicz, L., Cohen, D., Caplan, N., Sosna, J.: Inter-observer variability of manual contour delineation of structures in CT. Eur. Radiol. 29(3), 1391\u20131399 (2019). https:\/\/doi.org\/10.1007\/s00330-018-5695-5","DOI":"10.1007\/s00330-018-5695-5"},{"key":"4_CR25","doi-asserted-by":"publisher","unstructured":"Kentley, J., et al.: Agreement between experts and an untrained crowd for identifying dermoscopic features using a gamified app: reader feasibility study. JMIR Med. Inform. 11, e38412 (2023). https:\/\/doi.org\/10.2196\/38412, https:\/\/medinform.jmir.org\/2023\/1\/e38412","DOI":"10.2196\/38412"},{"issue":"4","key":"4_CR26","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1007\/s10278-017-9976-3","volume":"30","author":"MD Kohli","year":"2017","unstructured":"Kohli, M.D., Summers, R.M., Geis, J.R.: Medical image data and datasets in the era of machine learning-whitepaper from the 2016 C-MIMI meeting dataset session. J. Digit. Imaging 30(4), 392\u2013399 (2017). https:\/\/doi.org\/10.1007\/s10278-017-9976-3","journal-title":"J. Digit. Imaging"},{"key":"4_CR27","unstructured":"Le, J., Edmonds, A., Hester, V., Biewald, L.: Ensuring quality in crowdsourced search relevance evaluation: the effects of training question distribution. In: SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, vol.\u00a02126, pp. 22\u201332 (2010)"},{"key":"4_CR28","doi-asserted-by":"publisher","unstructured":"Le\u00a0Ludec, C., Cornet, M., Casilli, A.A.: The problem with annotation. Human labour and outsourcing between France and Madagascar. Big Data Soc. 10(2), 20539517231188723 (2023). https:\/\/doi.org\/10.1177\/20539517231188723, http:\/\/journals.sagepub.com\/doi\/10.1177\/20539517231188723","DOI":"10.1177\/20539517231188723"},{"key":"4_CR29","doi-asserted-by":"publisher","unstructured":"Lee, A.N., Hunter, C.J., Ruiz, N.: Platypus: quick, cheap, and powerful refinement of LLMs (2023). https:\/\/doi.org\/10.48550\/arXiv.2308.07317, http:\/\/arxiv.org\/abs\/2308.07317","DOI":"10.48550\/arXiv.2308.07317"},{"key":"4_CR30","doi-asserted-by":"publisher","unstructured":"Litman, L., Robinson, J., Rosenzweig, C.: The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk. Behav. Res. Methods 47(2), 519\u2013528 (2015). https:\/\/doi.org\/10.3758\/s13428-014-0483-x","DOI":"10.3758\/s13428-014-0483-x"},{"key":"4_CR31","doi-asserted-by":"publisher","unstructured":"Lu, X., et al.: Rethinking quality assurance for crowdsourced multi-ROI image segmentation. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 11, no. 1, pp. 103\u2013114 (2023). https:\/\/doi.org\/10.1609\/hcomp.v11i1.27552, https:\/\/ojs.aaai.org\/index.php\/HCOMP\/article\/view\/27552","DOI":"10.1609\/hcomp.v11i1.27552"},{"key":"4_CR32","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"438","DOI":"10.1007\/978-3-319-10470-6_55","volume-title":"Medical Image Computing and Computer-Assisted Intervention \u2013 MICCAI 2014","author":"L Maier-Hein","year":"2014","unstructured":"Maier-Hein, L., et al.: Can masses of non-experts train highly accurate image classifiers? In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8674, pp. 438\u2013445. Springer, Cham (2014). https:\/\/doi.org\/10.1007\/978-3-319-10470-6_55"},{"key":"4_CR33","doi-asserted-by":"publisher","unstructured":"Maier-Hein, L., et al.: Metrics reloaded: recommendations for image analysis validation. Nat. Methods 21, 195\u2013212 (2024). https:\/\/doi.org\/10.1038\/s41592-023-02151-z, https:\/\/www.nature.com\/articles\/s41592-023-02151-z","DOI":"10.1038\/s41592-023-02151-z"},{"key":"4_CR34","doi-asserted-by":"publisher","unstructured":"Maier-Hein, L., et al.: Heidelberg colorectal data set for surgical data science in the sensor operating room. Sci. Data 8(1), 101 (2021). https:\/\/doi.org\/10.1038\/s41597-021-00882-2 , https:\/\/www.nature.com\/articles\/s41597-021-00882-2","DOI":"10.1038\/s41597-021-00882-2"},{"key":"4_CR35","doi-asserted-by":"publisher","unstructured":"McCulloch, C.E., Searle, S.R.: Generalized, Linear, and Mixed Models. Wiley (2004). https:\/\/doi.org\/10.1002\/0471722073, https:\/\/books.google.com\/books?hl=en&lr= &id=bWDPukohugQC &oi=fnd &pg=PR6 &dq=Generalized,+Linear,+and+MixedModels. &ots=sSy3HM7CGo &sig=Ss04EHKG_xkcZZCYWSEJl6paHos","DOI":"10.1002\/0471722073"},{"key":"4_CR36","doi-asserted-by":"publisher","unstructured":"Miceli, M., Posada, J.: The data-production dispositif. Proc. ACM Hum.-Comput. Interact. 6(CSCW2), 1\u201337 (2022). https:\/\/doi.org\/10.1145\/3555561, https:\/\/dl.acm.org\/doi\/10.1145\/3555561","DOI":"10.1145\/3555561"},{"key":"4_CR37","doi-asserted-by":"publisher","unstructured":"Michaelis, C., et al.: Benchmarking robustness in object detection: autonomous driving when winter is coming (2020). https:\/\/doi.org\/10.48550\/arXiv.1907.07484, http:\/\/arxiv.org\/abs\/1907.07484","DOI":"10.48550\/arXiv.1907.07484"},{"key":"4_CR38","doi-asserted-by":"publisher","unstructured":"Petrovi\u0107, N., Moy\u00e0-Alcover, G., Varona, J., Jaume-i Cap\u00f3, A.: Crowdsourcing human-based computation for medical image analysis: a systematic literature review. Health Inform. J. 26(4), 2446\u20132469 (2020). https:\/\/doi.org\/10.1177\/1460458220907435, http:\/\/journals.sagepub.com\/doi\/10.1177\/1460458220907435","DOI":"10.1177\/1460458220907435"},{"key":"4_CR39","doi-asserted-by":"publisher","unstructured":"Reinke, A., et al.: Understanding metric-related pitfalls in image analysis validation. Nat. Methods 1\u201313 (2024). https:\/\/doi.org\/10.1038\/s41592-023-02150-0, https:\/\/www.nature.com\/articles\/s41592-023-02150-0, publisher: Nature Publishing Group","DOI":"10.1038\/s41592-023-02150-0"},{"key":"4_CR40","unstructured":"Research, G.V.: Data collection and labeling market size & share report 2030 (2022). https:\/\/www.grandviewresearch.com\/industry-analysis\/data-collection-labeling-market"},{"key":"4_CR41","doi-asserted-by":"publisher","unstructured":"Ro\u00df, T., et al.: How can we learn (more) from challenges? A statistical approach to driving future algorithm development. arXiv preprint arXiv:2106.09302 (2021). https:\/\/doi.org\/10.48550\/arXiv.2106.09302","DOI":"10.48550\/arXiv.2106.09302"},{"key":"4_CR42","doi-asserted-by":"publisher","unstructured":"R\u00e4dsch, T., et al.: Labelling instructions matter in biomedical image analysis. Nat. Mach. Intell. 5(3), 273\u2013283 (2023). https:\/\/doi.org\/10.1038\/s42256-023-00625-5, https:\/\/www.nature.com\/articles\/s42256-023-00625-5","DOI":"10.1038\/s42256-023-00625-5"},{"key":"4_CR43","doi-asserted-by":"publisher","unstructured":"Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., Aroyo, L.M.: \u201cEveryone wants to do the model work, not the data work\u201d: data cascades in high-stakes AI. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1\u201315. ACM, Yokohama (2021). https:\/\/doi.org\/10.1145\/3411764.3445518","DOI":"10.1145\/3411764.3445518"},{"key":"4_CR44","doi-asserted-by":"publisher","unstructured":"Schmidt, F.A.: The planetary stacking order of multilayered crowd-AI systems. In: Graham, M., Ferrari, F. (eds.) Digital Work in the Planetary Market, pp. 137\u2013156. The MIT Press (2022). https:\/\/doi.org\/10.7551\/mitpress\/13835.003.0012, https:\/\/direct.mit.edu\/books\/book\/5319\/chapter\/3800159\/The-Planetary-Stacking-Order-of-Multilayered-Crowd","DOI":"10.7551\/mitpress\/13835.003.0012"},{"key":"4_CR45","doi-asserted-by":"publisher","unstructured":"Touvron, H., et al.: LLaMA: open and efficient foundation language models (2023). https:\/\/doi.org\/10.48550\/arXiv.2302.13971, http:\/\/arxiv.org\/abs\/2302.13971","DOI":"10.48550\/arXiv.2302.13971"},{"key":"4_CR46","doi-asserted-by":"publisher","unstructured":"Veselovsky, V., Ribeiro, M.H., West, R.: Artificial artificial artificial intelligence: crowd workers widely use large language models for text production tasks (2023). https:\/\/doi.org\/10.48550\/arXiv.2306.07899, http:\/\/arxiv.org\/abs\/2306.07899","DOI":"10.48550\/arXiv.2306.07899"},{"key":"4_CR47","doi-asserted-by":"publisher","unstructured":"Wang, K., Hill, M., Knowles-Barley, S., Tikhonov, A., Litchfield, L., Bare, J.C.: Improving segmentation of breast arterial calcifications from digital mammography: good annotation is all you need. In: Zheng, Y., Kele\u015f, H.Y., Koniusz, P. (eds.) ACCV 2022. LNCS, vol. 13848, pp. 134\u2013150. Springer, Cham (2023). https:\/\/doi.org\/10.1007\/978-3-031-27066-6_10","DOI":"10.1007\/978-3-031-27066-6_10"},{"key":"4_CR48","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1007\/978-3-030-59710-8_72","volume-title":"Medical Image Computing and Computer Assisted Intervention \u2013 MICCAI 2020","author":"S Yu","year":"2020","unstructured":"Yu, S., et al.: Difficulty-aware glaucoma classification with multi-rater consensus modeling. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12261, pp. 741\u2013750. Springer, Cham (2020). https:\/\/doi.org\/10.1007\/978-3-030-59710-8_72"},{"key":"4_CR49","doi-asserted-by":"publisher","unstructured":"\u00d8rting, S.N., et al.: A survey of crowdsourcing in medical image analysis. Hum. Computation 7, 1\u201326 (2020). https:\/\/doi.org\/10.15346\/hc.v7i1.1, https:\/\/hcjournal.org\/index.php\/jhc\/article\/view\/111","DOI":"10.15346\/hc.v7i1.1"}],"container-title":["Lecture Notes in Computer Science","Computer Vision \u2013 ECCV 2024"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-73229-4_4","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,24]],"date-time":"2024-10-24T15:12:43Z","timestamp":1729782763000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-73229-4_4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,25]]},"ISBN":["9783031732287","9783031732294"],"references-count":49,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-73229-4_4","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,25]]},"assertion":[{"value":"25 October 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"ECCV","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"European Conference on Computer Vision","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Milan","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Italy","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"29 September 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"4 October 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"18","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"eccv2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/eccv2024.ecva.net\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}