{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T05:35:05Z","timestamp":1769751305657,"version":"3.49.0"},"reference-count":30,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2020,3,6]],"date-time":"2020-03-06T00:00:00Z","timestamp":1583452800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"ERASMUS-MUNDUS","award":["Euroweb+"],"award-info":[{"award-number":["Euroweb+"]}]},{"DOI":"10.13039\/501100022397","name":"Govern de les Illes Balears","doi-asserted-by":"crossref","award":["PROCOE\/2\/2017"],"award-info":[{"award-number":["PROCOE\/2\/2017"]}],"id":[{"id":"10.13039\/501100022397","id-type":"DOI","asserted-by":"crossref"}]},{"name":"MINECO\/AEI\/ERDF, EU","award":["TIN2016-81143-R"],"award-info":[{"award-number":["TIN2016-81143-R"]}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Health Informatics J"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p> Computer-assisted algorithms for the analysis of medical images require human interactions to achieve satisfying results. Human-based computation and crowdsourcing offer a solution to this problem. We performed a systematic literature review of studies on crowdsourcing human-based computation for medical image analysis based on the guidelines proposed by Kitchenham and Charters. We identified 43 studies relevant to the objective of this research. We determined three primary purposes and problems that crowdsourcing human-based computation systems can solve. We found that the users provided five information types. We compared systems that use pre-, post-evaluation and quality control methods to select and filter the user inputs. We analyzed the metrics used for the evaluation of the crowdsourcing human-based computation system performance. Finally, we identified the most popular crowdsourcing human-based computation platforms with their advantages and disadvantages.Crowdsourcing human-based computation systems can successfully solve medical image analysis problems. However, the application of crowdsourcing human-based computation systems in this research area is still limited and more studies should be conducted to obtain generalizable results. We provided guidelines to practitioners and researchers based on the results obtained in this research. <\/jats:p>","DOI":"10.1177\/1460458220907435","type":"journal-article","created":{"date-parts":[[2020,3,6]],"date-time":"2020-03-06T12:56:00Z","timestamp":1583499360000},"page":"2446-2469","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":12,"title":["Crowdsourcing human-based computation for medical image analysis: A systematic literature review"],"prefix":"10.1177","volume":"26","author":[{"given":"Nata\u0161a","family":"Petrovi\u0107","sequence":"first","affiliation":[{"name":"University of Belgrade, Serbia"}]},{"given":"Gabriel","family":"Moy\u00e0-Alcover","sequence":"additional","affiliation":[]},{"given":"Javier","family":"Varona","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3312-5347","authenticated-orcid":false,"given":"Antoni","family":"Jaume-i-Cap\u00f3","sequence":"additional","affiliation":[{"name":"Universitat de les Illes Balears, Spain"}]}],"member":"179","published-online":{"date-parts":[[2020,3,6]]},"reference":[{"key":"bibr1-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.bioeng.2.1.315"},{"key":"bibr2-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2014.01.010"},{"key":"bibr3-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-010-9155-0"},{"key":"bibr4-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"bibr5-1460458220907435","doi-asserted-by":"publisher","DOI":"10.4103\/0971-6203.58777"},{"key":"bibr6-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1016\/j.compmedimag.2007.02.002"},{"key":"bibr7-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-019-00227-x"},{"key":"bibr8-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-bioeng-071516-044442"},{"key":"bibr9-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2009.03.008"},{"key":"bibr10-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2011.02.011"},{"key":"bibr11-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1016\/S1361-8415(00)00041-4"},{"issue":"1","key":"bibr12-1460458220907435","first-page":"1","volume":"9","author":"Fatima M","year":"2017","journal-title":"J Intell Learn Syst Appl"},{"key":"bibr13-1460458220907435","unstructured":"Vezhnevets V, Konouchine V. GrowCut\u2014interactive multi-label N-D image segmentation by cellular automata. In: Graphicon, 2005, pp. 150\u2013156, http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.59.8092&rep=rep1&type=pdf"},{"key":"bibr14-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2007.891796"},{"key":"bibr15-1460458220907435","doi-asserted-by":"crossref","unstructured":"Snow R, Connor BO, Jurafsky D, et al. Cheap and fast\u2014but is it good? Evaluation non-expert annotations for natural language tasks. In: Proceedings of the 2008 conference on empirical methods in natural language processing, 2008, pp. 254\u2013263, https:\/\/www.aclweb.org\/anthology\/D08-1027.pdf","DOI":"10.3115\/1613715.1613751"},{"key":"bibr16-1460458220907435","doi-asserted-by":"crossref","unstructured":"Von Ahn L. Human computation. In: 2008 IEEE 24th international conference on data engineering, 2008, pp. 1\u20132, https:\/\/doi.org\/10.1109\/ICDE.2008.4497403","DOI":"10.1109\/ICDE.2008.4497403"},{"key":"bibr17-1460458220907435","doi-asserted-by":"crossref","unstructured":"Quinn AJ, Bederson BB. Human computation: a survey and taxonomy of a growing field. In: Proceedings of the international conference on human factors in computing systems (CHI 2011), 2011, pp. 1403\u20131412, https:\/\/doi.org\/10.1145\/1978942.1979148","DOI":"10.1145\/1978942.1979148"},{"key":"bibr18-1460458220907435","unstructured":"Howe J. The rise of crowdsourcing. Wired Magazine 2006; 14(6): 1\u20134. https:\/\/www.wired.com\/2006\/06\/crowds (accessed 17 July 2018)."},{"key":"bibr19-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btt333"},{"key":"bibr20-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1017\/S1930297500002205"},{"key":"bibr21-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1038\/nature09304"},{"key":"bibr22-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1126\/science.1160379"},{"key":"bibr23-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33863-2_25"},{"key":"bibr24-1460458220907435","doi-asserted-by":"crossref","unstructured":"Von Ahn L, Liu R, Blum M. Peekaboom. In: Proceedings of the SIGCHI conference human factors in computing systems (CHI \u201906), 2006, https:\/\/doi.org\/10.1145\/1124772.1124782","DOI":"10.1145\/1124772.1124782"},{"key":"bibr25-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"bibr26-1460458220907435","doi-asserted-by":"crossref","unstructured":"Von Ahn L, Dabbish L. Labeling images with a computer game. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI \u201904), 2004, pp. 319\u2013326, https:\/\/doi.org\/10.1145\/985692.985733","DOI":"10.1145\/985692.985733"},{"key":"bibr27-1460458220907435","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.9330"},{"key":"bibr28-1460458220907435","volume":"45","author":"Kitchenham B","year":"2007","journal-title":"Engineering"},{"key":"bibr29-1460458220907435","unstructured":"Critical Appraisal Skills Programme. CASP systematic review checklist, 2017, http:\/\/www.casp-uk.net\/checklists (accessed 17 July 2018)."},{"key":"bibr30-1460458220907435","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.01.006"}],"container-title":["Health Informatics Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1460458220907435","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1460458220907435","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1460458220907435","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T13:12:33Z","timestamp":1740834753000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1460458220907435"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,6]]},"references-count":30,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["10.1177\/1460458220907435"],"URL":"https:\/\/doi.org\/10.1177\/1460458220907435","relation":{},"ISSN":["1460-4582","1741-2811"],"issn-type":[{"value":"1460-4582","type":"print"},{"value":"1741-2811","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,6]]}}}