{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:39:43Z","timestamp":1760060383395,"version":"build-2065373602"},"reference-count":61,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T00:00:00Z","timestamp":1756166400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Open Access Publication Fund of the Technische Universit\u00e4t Ilmenau"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["BDCC"],"abstract":"<jats:p>Crowdsourcing enables the acquisition of distributed human intelligence for solving tasks involving human judgments in scalable ways, with many use cases in various application areas accessing human intelligence. However, crowdworkers completing the tasks may have limited or no background knowledge about the tasks they solve due to the plethora of various tasks available. Therefore, the tasks\u2014even on a micro scale\u2014also need to include appropriate training for the crowdworkers to enable them to complete them successfully. However, training crowdworkers efficiently in a short time for complex tasks poses a challenge and remains an unresolved issue. This paper addresses this challenge by empirically comparing different training strategies for crowdworkers and evaluating their impact on the crowdworkers\u2019 task results. We perform comparisons between a basic training strategy, a strategy based on previous errors made by other crowdworkers, and the addition of instant feedback during training and task completion. Our results show that adding instant feedback during both the training phase and during the task yields more attention from the workers in difficult tasks and hence reduces errors and improves the results. We conclude that more attention is retained when the content of instant feedback includes information about mistakes made by other crowdworkers previously.<\/jats:p>","DOI":"10.3390\/bdcc9090220","type":"journal-article","created":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T14:43:18Z","timestamp":1756219398000},"page":"220","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Assessing the Influence of Feedback Strategies on Errors in Crowdsourced Annotation of Tumor Images"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5434-5464","authenticated-orcid":false,"given":"Jose Alejandro","family":"Libreros","sequence":"first","affiliation":[{"name":"User-Centric Analysis of Multimedia Data Research Group, Faculty of Electrical Engineering and Information Technology, Technische Universit\u00e4t Ilmenau, Gustav-Kirchhoff-Stra\u00dfe 1, 98693 Ilmenau, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5037-5279","authenticated-orcid":false,"given":"Edwin","family":"Gamboa","sequence":"additional","affiliation":[{"name":"ScaleHub GmbH, Heidbergstra\u00dfe 100, 22846 Norderstedt, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2380-2682","authenticated-orcid":false,"given":"Erik","family":"Henke","sequence":"additional","affiliation":[{"name":"Institute of Anatomy and Cell Biology, Faculty of Medicine, Julius-Maximilians-Universit\u00e4t W\u00fcrzburg, Koellikerstra\u00dfe 6, 97070 W\u00fcrzburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1359-363X","authenticated-orcid":false,"given":"Matthias","family":"Hirth","sequence":"additional","affiliation":[{"name":"User-Centric Analysis of Multimedia Data Research Group, Faculty of Electrical Engineering and Information Technology, Technische Universit\u00e4t Ilmenau, Gustav-Kirchhoff-Stra\u00dfe 1, 98693 Ilmenau, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Estell\u00e9s-Arolas, E., Navarro-Giner, R., and Gonz\u00e1lez-Ladr\u00f3n-de-Guevara, F. (2015). Crowdsourcing Fundamentals: Definition and Typology. Advances in Crowdsourcing, Springer.","DOI":"10.1007\/978-3-319-18341-1_3"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1177\/0165551512437638","article-title":"Towards an Integrated Crowdsourcing Definition","volume":"38","year":"2012","journal-title":"J. Inf. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1177\/0165551514550140","article-title":"A Taxonomy of Crowdsourcing Based on Task Complexity","volume":"40","author":"Nakatsu","year":"2014","journal-title":"J. Inf. Sci."},{"key":"ref_4","first-page":"543","article-title":"Income Security in the On-Demand Economy: Findings and Policy Lessons from a Survey of Crowdworkers 2016","volume":"37","author":"Berg","year":"2015","journal-title":"Comp. Labor Law Policy J."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1007\/s10462-016-9491-9","article-title":"Learning from Crowdsourced Labeled Data: A Survey","volume":"46","author":"Zhang","year":"2016","journal-title":"Artif. Intell. Rev."},{"key":"ref_6","first-page":"1","article-title":"Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research","volume":"18","author":"Vaughan","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e75","DOI":"10.1158\/0008-5472.CAN-17-0629","article-title":"The Digital Slide Archive: A Software Platform for Management, Integration, and Analysis of Histology for Cancer Research","volume":"77","author":"Gutman","year":"2017","journal-title":"Cancer Res."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"101356","DOI":"10.1016\/j.aei.2021.101356","article-title":"Recognition of Pedestrian Trajectories and Attributes with Computer Vision and Deep Learning Techniques","volume":"49","author":"Wong","year":"2021","journal-title":"Adv. Eng. Inform."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1080\/01615440.2023.2164879","article-title":"Applications of Machine Learning in Tabular Document Digitisation","volume":"56","author":"Dahl","year":"2023","journal-title":"Hist. Methods J. Quant. Interdiscip. Hist."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mufti, N., and Shah, S.A.A. (2021). Automatic Number Plate Recognition:A Detailed Survey of Relevant Algorithms. Sensors, 21.","DOI":"10.3390\/s21093028"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"6793","DOI":"10.1038\/s41467-022-34591-0","article-title":"Mapping Global Dynamics of Benchmark Creation and Saturation in Artificial Intelligence","volume":"13","author":"Ott","year":"2022","journal-title":"Nat. Commun."},{"key":"ref_12","first-page":"102315","article-title":"Retos y desaf\u00edos de la inteligencia artificial en la investigaci\u00f3n en salud","volume":"37","author":"Fors","year":"2022","journal-title":"Gac. Sanit."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"309","DOI":"10.7705\/biomedica.7242","article-title":"Retos de la inteligencia artificial y sus posibles soluciones desde la perspectiva de un editorialista humano","volume":"43","year":"2023","journal-title":"Biom\u00e9dica"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1007\/978-3-031-57982-0_17","article-title":"Mistakes Hold the Key: Reducing Errors in a Crowdsourced Tumor Annotation Task by Optimizing the Training Strategy","volume":"Volume 1877","author":"Ruiz","year":"2024","journal-title":"Human-Computer Interaction"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Pedraza, A., Bueno, G., Deniz, O., Crist\u00f3bal, G., Blanco, S., and Borrego-Ramos, M. (2017). Automated Diatom Classification (Part B): A Deep Learning Approach. Appl. Sci., 7.","DOI":"10.3390\/app7050460"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1007\/978-3-031-68323-7_31","article-title":"Visual Transformers Meet Convolutional Neural Networks: Providing Context for Convolution Layers in Semantic Segmentation of Remote Sensing Photovoltaic Imaging","volume":"Volume 14912","author":"Wrembel","year":"2024","journal-title":"Big Data Analytics and Knowledge Discovery"},{"key":"ref_17","unstructured":"Gamboa, E., Libreros, A., Hirth, M., and Dubiner, D. (2022, January 17\u201321). Human-AI Collaboration for Improving the Identification of Cars for Autonomous Driving. Proceedings of the CIKM Workshops, Atlanta, GA, USA."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"901","DOI":"10.1109\/TKDE.2016.2518669","article-title":"Challenges in Data Crowdsourcing","volume":"28","author":"Joglekar","year":"2016","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_19","unstructured":"Salminen, J. (2015). The Role of Collective Intelligence in Crowdsourcing Innovation, LUT University."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Tondello, G.F., Wehbe, R.R., Diamond, L., Busch, M., Marczewski, A., and Nacke, L.E. (2016, January 16\u201319). The Gamification User Types Hexad Scale. Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, Austin, TX, USA.","DOI":"10.1145\/2967934.2968082"},{"key":"ref_21","unstructured":"Vroom, V.H. (1964). Work and Motivation, Wiley."},{"key":"ref_22","first-page":"267","article-title":"Understanding the Relationships between Motivators and Effort in Crowdsourcing Marketplaces: A Nonlinear Analysis","volume":"35","author":"Sun","year":"2015","journal-title":"Int. J. Inf. Manag."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"103616","DOI":"10.1016\/j.im.2022.103616","article-title":"Motivating Participation in Crowdsourcing Contests: The Role of Instruction-Writing Strategy","volume":"59","author":"Yin","year":"2022","journal-title":"Inf. Manag."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"L\u00f3pez-P\u00e9rez, M., Amgad, M., Morales-\u00c1lvarez, P., Ruiz, P., Cooper, L.A.D., Molina, R., and Katsaggelos, A.K. (2021). Learning from Crowds in Digital Pathology Using Scalable Variational Gaussian Processes. Sci. Rep., 11.","DOI":"10.1038\/s41598-021-90821-3"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mehta, P., Sandfort, V., Gheysens, D., Braeckevelt, G.-J., Berte, J., and Summers, R.M. (2019, January 8\u201311). Segmenting the Kidney on CT Scans Via Crowdsourcing. Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), Venice, Italy.","DOI":"10.1109\/ISBI.2019.8759240"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"814","DOI":"10.1016\/j.jsurg.2018.10.007","article-title":"Crowdsourced Assessment of Inanimate Biotissue Drills: A Valid and Cost-Effective Way to Evaluate Surgical Trainees","volume":"76","author":"Rice","year":"2019","journal-title":"J. Surg. Educ."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1089\/end.2019.0626","article-title":"Objective Assessment and Standard Setting for Basic Flexible Ureterorenoscopy Skills Among Urology Trainees Using Simulation-Based Methods","volume":"34","author":"Goldenberg","year":"2020","journal-title":"J. Endourol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3461","DOI":"10.1093\/bioinformatics\/btz083","article-title":"Structured Crowdsourcing Enables Convolutional Segmentation of Histology Images","volume":"35","author":"Amgad","year":"2019","journal-title":"Bioinformatics"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1007\/s11548-018-1874-8","article-title":"Robust Navigation Support in Lowest Dose Image Setting","volume":"14","author":"Bui","year":"2019","journal-title":"Int. J. Comput. Assist. Radiol. Surg."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kandala, P.A., and Sivaswamy, J. (2017, January 26\u201329). Crowdsourced Annotations as an Additional Form of Data Augmentation for CAD Development. Proceedings of the 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR), Nanjing, China.","DOI":"10.1109\/ACPR.2017.6"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1089\/end.2018.0534","article-title":"Crowdsourced Assessment of Ureteroscopy with Laser Lithotripsy Video Feed Does Not Correlate with Trainee Experience","volume":"33","author":"Conti","year":"2019","journal-title":"J. Endourol."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Morozov, S.P., Gombolevskiy, V.A., Elizarov, A.B., Gusev, M.A., Novik, V.P., Prokudaylo, S.B., Bardin, A.S., Popov, E.V., Ledikhova, N.V., and Chernina, V.Y. (2021). A Simplified Cluster Model and a Tool Adapted for Collaborative Labeling of Lung Cancer CT Scans. Comput. Methods Programs Biomed., 206.","DOI":"10.1016\/j.cmpb.2021.106111"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Tolxdorff, T., Deserno, T.M., Handels, H., Maier, A., Maier-Hein, K.H., and Palm, C. (2020). Is Crowd-Algorithm Collaboration an Advanced Alternative to Crowd-Sourcing on Cytology Slides?. Bildverarbeitung f\u00fcr die Medizin 2020, Springer Fachmedien.","DOI":"10.1007\/978-3-658-29267-6"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"3599","DOI":"10.1038\/s41467-024-48010-z","article-title":"A Vascularized Breast Cancer Spheroid Platform for the Ranked Evaluation of Tumor Microenvironment-Targeted Drugs by Light Sheet Fluorescence Microscopy","volume":"15","author":"Ascheid","year":"2024","journal-title":"Nat. Commun."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1077","DOI":"10.1016\/j.jsurg.2021.02.004","article-title":"Crowdsourced Assessment of Surgical Skill Proficiency in Cataract Surgery","volume":"78","author":"Paley","year":"2021","journal-title":"J. Surg. Educ."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1284","DOI":"10.1109\/TMI.2018.2883237","article-title":"Crowdsourcing of Histological Image Labeling and Object Delineation by Medical Students","volume":"38","author":"Grote","year":"2019","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_37","unstructured":"Archambault, D., Purchase, H., and Ho\u00dffeld, T. (2015, January 22\u201327). Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing. Proceedings of the Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments, Dagstuhl Castle, Germany."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Difallah, D., Filatova, E., and Ipeirotis, P. (2018, January 5\u20139). Demographics and Dynamics of Mechanical Turk Workers. Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, Marina Del Rey, CA, USA.","DOI":"10.1145\/3159652.3159661"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1108\/INTR-12-2021-0897","article-title":"How Leaderboard Positions Shape Our Motivation: The Impact of Competence Satisfaction and Competence Frustration on Motivation in a Gamified Crowdsourcing Task","volume":"33","author":"Na","year":"2023","journal-title":"Internet Res."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"313","DOI":"10.3233\/WEB-190421","article-title":"Proposal of Grade Training Method for Quality Improvement in Microtask Crowdsourcing","volume":"17","author":"Ashikawa","year":"2019","journal-title":"Web Intell."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1287\/isre.2023.1232","article-title":"An Examination of the Dynamics of Crowdsourcing Contests: Role of Feedback Type","volume":"35","author":"Sanyal","year":"2024","journal-title":"Inf. Syst. Res."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1186\/s40249-020-0622-9","article-title":"Crowdsourcing in Health and Medical Research: A Systematic Review","volume":"9","author":"Wang","year":"2020","journal-title":"Infect. Dis. Poverty"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Stephanidis, C., Soares, M.M., Rosenzweig, E., Marcus, A., Yamamoto, S., Mori, H., Rau, P.-L.P., Meiselwitz, G., Fang, X., and Moallem, A. (2021, January 24\u201329). The Crowd Thinks Aloud: Crowdsourcing Usability Testing with the Thinking Aloud Method. Proceedings of the HCI International 2021\u2014Late Breaking Papers: Design and User Experience, Virtual Event.","DOI":"10.1007\/978-3-030-90238-4"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Dean, A., and Voss, D. (1999). Fractional Factorial Experiments. Design and Analysis of Experiments, Springer.","DOI":"10.1007\/b97673"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Dean, A., and Voss, D. (1999). Design and Analysis of Experiments, Springer. Springer Texts in Statistics.","DOI":"10.1007\/b97673"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1093\/biomet\/38.3-4.330","article-title":"On the Comparison of Several Mean Values: An Alternative Approach","volume":"38","author":"Welch","year":"1951","journal-title":"Biometrika"},{"key":"ref_47","unstructured":"Tukey, J.W. (1953). The Problem of Multiple Comparisons, Princeton University."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"2636","DOI":"10.1080\/03610918.2014.931971","article-title":"Overview of Friedman\u2019s Test and Post-Hoc Analysis","volume":"44","author":"Pereira","year":"2015","journal-title":"Commun. Stat.\u2014Simul. Comput."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1508","DOI":"10.1080\/03610918.2015.1006777","article-title":"A Modified Friedman Test for Randomized Complete Block Designs","volume":"46","author":"Xu","year":"2017","journal-title":"Commun. Stat.\u2014Simul. Comput."},{"key":"ref_50","first-page":"1","article-title":"Tukey\u2019s Honestly Significant Difference (HSD) Test","volume":"3","author":"Abdi","year":"2010","journal-title":"Encycl. Res. Des."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1093\/biomet\/52.3-4.591","article-title":"An Analysis of Variance Test for Normality (Complete Samples)","volume":"52","author":"Shapiro","year":"1965","journal-title":"Biometrika"},{"key":"ref_52","unstructured":"Campling, J. (2002). Analysis of Variance (ANOVA). SPSS for Social Scientists, Macmillan Education UK."},{"key":"ref_53","unstructured":"Nemenyi, P.B. (1963). Distribution-Free Multiple Comparisons, Princeton University."},{"key":"ref_54","unstructured":"Hollander, M., Wolfe, D.A., and Chicken, E. (2013). Nonparametric Statistical Methods, John Wiley & Sons."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Weiner, I.B., and Craighead, W.E. (2010). Kruskal-Wallis Test. The Corsini Encyclopedia of Psychology, Wiley.","DOI":"10.1002\/9780470479216"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1080\/01621459.1952.10483441","article-title":"Use of Ranks in One-Criterion Variance Analysis","volume":"47","author":"Kruskal","year":"1952","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","article-title":"Multiple Comparisons among Means","volume":"56","author":"Dunn","year":"1961","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1111\/j.1475-1313.2010.00815.x","article-title":"Statistical Guidelines for Clinical Studies of Human Vision","volume":"31","author":"Armstrong","year":"2011","journal-title":"Ophthalmic Physiol. Opt."},{"key":"ref_59","unstructured":"Hirth, M., Libreros, J., Gamboa, E., and Henke, E. (2024). Mistakes Hold the Key: Providing Instant Feedback Based on Errors to Crowdworkers in Tumor Annotation Imaging. Zenodo."},{"key":"ref_60","unstructured":"Keith, N. (2011). Learning through Errors in Training. Errors in Organizations, Routledge."},{"key":"ref_61","unstructured":"Frese, M., and Altmann, A. (1989). The Treatment of Errors in Learning and Training. Dev. Ski. Inf. Technol., 65."}],"container-title":["Big Data and Cognitive Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-2289\/9\/9\/220\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:33:21Z","timestamp":1760034801000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-2289\/9\/9\/220"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,26]]},"references-count":61,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["bdcc9090220"],"URL":"https:\/\/doi.org\/10.3390\/bdcc9090220","relation":{},"ISSN":["2504-2289"],"issn-type":[{"type":"electronic","value":"2504-2289"}],"subject":[],"published":{"date-parts":[[2025,8,26]]}}}