{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T18:51:43Z","timestamp":1740163903431,"version":"3.37.3"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T00:00:00Z","timestamp":1652313600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T00:00:00Z","timestamp":1652313600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100014013","name":"UK Research and Innovation","doi-asserted-by":"publisher","award":["104689","104689","104689","104689","104689","104689","104689","104689"],"award-info":[{"award-number":["104689","104689","104689","104689","104689","104689","104689","104689"]}],"id":[{"id":"10.13039\/100014013","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Due to COVID-19 outbreaks, most school pupils have had to be home-schooled for long periods of time. Two editions of a web-based competition \u201cBeat the Pathologists\u201d for school age participants in the UK ran to fill up pupils\u2019 spare time after home-schooling and evaluate their ability on contributing to AI annotation. The two editions asked the participants to annotate different types of cells on Ki67 stained breast cancer images. The Main competition was at four levels with different level of complexity. We obtained annotations of four kinds of cells entered by school pupils and ground truth from expert pathologists. In this paper, we analyse school pupils\u2019 performance on differentiating different kinds of cells and compare their performance with two neural networks (AlexNet and VGG16). It was observed that children tend to get very good performance in tumour cell annotation with the best F1 measure 0.81 which is a metrics taking both false positives and false negatives into account. Low accuracy was achieved with F1 score 0.75 on positive non-tumour cells and 0.59 on negative non-tumour cells. Superior performance on non-tumour cell detection was achieved by neural networks. VGG16 with training from scratch achieved an F1 score over 0.70 in all cell categories and 0.92 in tumour cell detection. We conclude that non-experts like school pupils have the potential to contribute to large-scale labelling for AI algorithm development if sufficient training activities are organised. We hope that competitions like this can promote public interest in pathology and encourage participation by more non-experts for annotation.<\/jats:p>","DOI":"10.1038\/s41598-022-11782-9","type":"journal-article","created":{"date-parts":[[2022,5,12]],"date-time":"2022-05-12T10:33:06Z","timestamp":1652351586000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Lessons from a breast cell annotation competition series for school pupils"],"prefix":"10.1038","volume":"12","author":[{"given":"Wenqi","family":"Lu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Islam M.","family":"Miligy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fayyaz","family":"Minhas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Young Saeng","family":"Park","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David R. J.","family":"Snead","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emad A.","family":"Rakha","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Clare","family":"Verrill","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nasir","family":"Rajpoot","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,5,12]]},"reference":[{"key":"11782_CR1","unstructured":"Oak National Academy https:\/\/www.thenational.academy\/. Accessed 20 Jan 2021."},{"key":"11782_CR2","unstructured":"Life sciences: industrial strategy. A report to government from the life sciences sector. Office for Life Sciences 30th August 2017 https:\/\/www.gov.uk\/government\/publications\/life-sciences-industrial-strategy. Accessed 20 Jan 2021."},{"issue":"5","key":"11782_CR3","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1016\/S1470-2045(19)30154-8","volume":"20","author":"MKK Niazi","year":"2019","unstructured":"Niazi, M. K. K., Parwani, A. V. & Gurcan, M. N. Digital pathology and artificial intelligence. Lancet Oncol. 20(5), 253\u2013261 (2019).","journal-title":"Lancet Oncol."},{"issue":"8","key":"11782_CR4","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1016\/S2589-7500(20)30159-X","volume":"2","author":"L Pantanowitz","year":"2020","unstructured":"Pantanowitz, L. et al. An artificial intelligence algorithm for prostate cancer diagnosis in whole slide images of core needle biopsies: A blinded clinical validation and deployment study. Lancet Digit. Health. 2(8), 407\u2013416 (2020).","journal-title":"Lancet Digit. Health."},{"issue":"8","key":"11782_CR5","doi-asserted-by":"publisher","first-page":"1301","DOI":"10.1038\/s41591-019-0508-1","volume":"25","author":"G Campanella","year":"2019","unstructured":"Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25(8), 1301\u20131309 (2019).","journal-title":"Nat. Med."},{"issue":"2","key":"11782_CR6","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1002\/path.5310","volume":"249","author":"R Colling","year":"2019","unstructured":"Colling, R. et al. Artificial intelligence in digital pathology: A roadmap to routine use in clinical practice. J. Pathol. 249(2), 143\u2013150 (2019).","journal-title":"J. Pathol."},{"issue":"1110","key":"11782_CR7","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1136\/postgradmedj-2017-135309","volume":"94","author":"TW Lambert","year":"2018","unstructured":"Lambert, T. W., Smith, F. & Goldacre, M. J. Career specialty choices of UK medical graduates of 2015 compared with earlier cohorts: Questionnaire surveys. Postgrad. Med. J. 94(1110), 191\u2013197 (2018).","journal-title":"Postgrad. Med. J."},{"issue":"4","key":"11782_CR8","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1016\/j.humpath.2009.09.012","volume":"41","author":"JC Ford","year":"2010","unstructured":"Ford, J. C. If not, why not? Reasons why Canadian postgraduate trainees chose\u2014or did not choose\u2014to become pathologists. Hum. Pathol. 41(4), 566\u2013573 (2010).","journal-title":"Hum. Pathol."},{"issue":"6","key":"11782_CR9","doi-asserted-by":"publisher","first-page":"802","DOI":"10.1016\/j.humpath.2010.10.018","volume":"42","author":"T Hung","year":"2011","unstructured":"Hung, T., Jarvis-Selinger, S. & Ford, J. C. Residency choices by graduating medical students: why not pathology? Hum. Pathol. 42(6), 802\u2013807 (2011).","journal-title":"Hum. Pathol."},{"issue":"1","key":"11782_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1471-2482-10-32","volume":"10","author":"MJ Goldacre","year":"2010","unstructured":"Goldacre, M. J. et al. Early career choices and successful career progression in surgery in the UK: Prospective cohort studies. BMC Surg. 10(1), 1\u201311 (2010).","journal-title":"BMC Surg."},{"issue":"3","key":"11782_CR11","doi-asserted-by":"publisher","first-page":"446","DOI":"10.1002\/path.1886","volume":"208","author":"TW Lambert","year":"2006","unstructured":"Lambert, T. W. et al. Career choices for pathology: National surveys of graduates of 1974\u20132002 from UK medical schools. J. Pathol. 208(3), 446\u2013452 (2006).","journal-title":"J. Pathol."},{"key":"11782_CR12","unstructured":"The Pathologist. \u201cAverting a Histopathology Crisis.\u201d Accessed 22 Jan 2021. https:\/\/thepathologist.com\/inside-the-lab\/averting-a-histopathology-crisis."},{"key":"11782_CR13","unstructured":"The Royal College of Pathologists. \u201cArt of Pathology Competition.\u201d Accessed 22 Jan 2021. https:\/\/www.rcpath.org\/discover-pathology\/competitions\/art-of-pathology-competition.html."},{"issue":"11","key":"11782_CR14","doi-asserted-by":"publisher","first-page":"3000472","DOI":"10.1371\/journal.pbio.3000472","volume":"17","author":"F Khatib","year":"2019","unstructured":"Khatib, F. et al. Building de novo cryo-electron microscopy structures collaboratively with citizen scientists. PLoS Biol. 17(11), 3000472 (2019).","journal-title":"PLoS Biol."},{"key":"11782_CR15","unstructured":"Etch a cell: https:\/\/www.zooniverse.org\/projects\/h-spiers\/etch-a-cell"},{"issue":"7761","key":"11782_CR16","doi-asserted-by":"publisher","first-page":"390","DOI":"10.1038\/s41586-019-1274-4","volume":"570","author":"B Koepnick","year":"2019","unstructured":"Koepnick, B. et al. De novo protein design by citizen scientists. Nature 570(7761), 390\u2013394 (2019).","journal-title":"Nature"},{"issue":"3","key":"11782_CR17","doi-asserted-by":"publisher","first-page":"1179","DOI":"10.1111\/j.1365-2966.2008.13689.x","volume":"389","author":"CJ Lintott","year":"2008","unstructured":"Lintott, C. J. et al. Galaxy Zoo: Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Mon. Not. R. Astron. Soc. 389(3), 1179\u20131189 (2008).","journal-title":"Mon. Not. R. Astron. Soc."},{"key":"11782_CR18","unstructured":"\u201cList of Citizen Science Projects.\u201d In Wikipedia, January 18, 2021. https:\/\/en.wikipedia.org\/w\/index.php?title=List_of_citizen_science_projects&oldid=1001222831."},{"issue":"6","key":"11782_CR19","doi-asserted-by":"publisher","first-page":"298","DOI":"10.1890\/110294","volume":"10","author":"G Newman","year":"2012","unstructured":"Newman, G. et al. The future of citizen science: Emerging technologies and shifting paradigms. Front. Ecol. Environ. 10(6), 298\u2013304 (2012).","journal-title":"Front. Ecol. Environ."},{"issue":"4","key":"11782_CR20","doi-asserted-by":"publisher","first-page":"523","DOI":"10.1177\/1075547016642241","volume":"38","author":"I Ruiz-Mall\u00e9n","year":"2016","unstructured":"Ruiz-Mall\u00e9n, I. et al. Citizen science: Toward transformative learning. Sci. Commun. 38(4), 523\u2013534 (2016).","journal-title":"Sci. Commun."},{"issue":"5","key":"11782_CR21","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1177\/0963662519840934","volume":"28","author":"S Brouwer","year":"2019","unstructured":"Brouwer, S. & Hessels, L. K. Increasing research impact with citizen science: The influence of recruitment strategies on sample diversity. Public Underst. Sci. 28(5), 606\u2013621 (2019).","journal-title":"Public Underst. Sci."},{"issue":"1","key":"11782_CR22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5334\/cstp.244","volume":"5","author":"M Aristeidou","year":"2020","unstructured":"Aristeidou, M. & Herodotou, C. Online citizen science: A systematic review of effects on learning and scientific literacy. Citizen Sci. Theory Pract. 5(1), 1\u201312 (2020).","journal-title":"Citizen Sci. Theory Pract."},{"issue":"7726","key":"11782_CR23","doi-asserted-by":"publisher","first-page":"480","DOI":"10.1038\/d41586-018-07106-5","volume":"562","author":"A Irwin","year":"2018","unstructured":"Irwin, A. No PhDs needed: How citizen science is transforming research. Nature 562(7726), 480\u2013483 (2018).","journal-title":"Nature"},{"issue":"16","key":"11782_CR24","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1093\/femsle\/fnv129","volume":"362","author":"R Kelwick","year":"2015","unstructured":"Kelwick, R. et al. Promoting microbiology education through the iGEM synthetic biology competition. FEMS Microbiol. Lett. 362(16), 129 (2015).","journal-title":"FEMS Microbiol. Lett."},{"key":"11782_CR25","doi-asserted-by":"crossref","unstructured":"Irshad, H. et al. Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: evaluating experts, automated methods, and the crowd. In Pacific Symposium on Biocomputing Co-chairs. 294\u2013305 (2014).","DOI":"10.1142\/9789814644730_0029"},{"issue":"2","key":"11782_CR26","doi-asserted-by":"publisher","first-page":"87264","DOI":"10.1371\/journal.pone.0087264","volume":"9","author":"FY Li","year":"2014","unstructured":"Li, F. Y. et al. Prognostic value of Ki-67 in breast cancer patients with positive axillary lymph nodes: A retrospective cohort study. PLoS ONE 9(2), 87264 (2014).","journal-title":"PLoS ONE"},{"key":"11782_CR27","first-page":"1097","volume":"25","author":"A Krizhevsky","year":"2012","unstructured":"Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097\u20131105 (2012).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"11782_CR28","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Chan, W. & Jaitly, N. Very deep convolutional networks for end-to-end speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 4845\u20134849 (2017).","DOI":"10.1109\/ICASSP.2017.7953077"},{"key":"11782_CR29","doi-asserted-by":"crossref","unstructured":"Deng, J. et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248\u2013255 (2009).","DOI":"10.1109\/CVPR.2009.5206848"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-11782-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-11782-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-11782-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T03:15:56Z","timestamp":1669346156000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-11782-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,12]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["11782"],"URL":"https:\/\/doi.org\/10.1038\/s41598-022-11782-9","relation":{},"ISSN":["2045-2322"],"issn-type":[{"type":"electronic","value":"2045-2322"}],"subject":[],"published":{"date-parts":[[2022,5,12]]},"assertion":[{"value":"16 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 May 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"7792"}}