{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T16:16:07Z","timestamp":1780762567182,"version":"3.54.1"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,10,7]],"date-time":"2021-10-07T00:00:00Z","timestamp":1633564800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,10,7]],"date-time":"2021-10-07T00:00:00Z","timestamp":1633564800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Biology has become a prime area for the deployment of deep learning and artificial intelligence (AI), enabled largely by the massive data sets that the field can generate. Key to most AI tasks is the availability of a sufficiently large, labeled data set with which to train AI models. In the context of microscopy, it is easy to generate image data sets containing millions of cells and structures. However, it is challenging to obtain large-scale high-quality annotations for AI models. Here, we present HALS (Human-Augmenting Labeling System), a human-in-the-loop data labeling AI, which begins uninitialized and learns annotations from a human, in real-time. Using a multi-part AI composed of three deep learning models, HALS learns from just a few examples and immediately decreases the workload of the annotator, while increasing the quality of their annotations. Using a highly repetitive use-case\u2014annotating cell types\u2014and running experiments with seven pathologists\u2014experts at the microscopic analysis of biological specimens\u2014we demonstrate a manual work reduction of 90.60%, and an average data-quality boost of 4.34%, measured across four use-cases and two tissue stain types.<\/jats:p>","DOI":"10.1038\/s41746-021-00520-6","type":"journal-article","created":{"date-parts":[[2021,10,7]],"date-time":"2021-10-07T10:25:59Z","timestamp":1633602359000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":34,"title":["Biological data annotation via a human-augmenting AI-based labeling system"],"prefix":"10.1038","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2323-1357","authenticated-orcid":false,"given":"Douwe","family":"van der Wal","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Iny","family":"Jhun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9688-3167","authenticated-orcid":false,"given":"Israa","family":"Laklouk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6857-341X","authenticated-orcid":false,"given":"Jeff","family":"Nirschl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lara","family":"Richer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rebecca","family":"Rojansky","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Talent","family":"Theparee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua","family":"Wheeler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5047-9257","authenticated-orcid":false,"given":"J\u00f6rg","family":"Sander","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Felix","family":"Feng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Osama","family":"Mohamad","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Silvio","family":"Savarese","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Socher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1937-9682","authenticated-orcid":false,"given":"Andre","family":"Esteva","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,10,7]]},"reference":[{"key":"520_CR1","doi-asserted-by":"crossref","unstructured":"Beveridge, T. J., Lawrence, J. R. & Murray, R. G. E. Sampling and Staining for Light Microscopy. In Methods for General and Molecular Microbiology 19\u201333 (2007).","DOI":"10.1128\/9781555817497.ch2"},{"key":"520_CR2","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1007\/978-1-60761-849-2_4","volume":"691","author":"M Slaoui","year":"2011","unstructured":"Slaoui, M. & Fiette, L. Histopathology procedures: from tissue sampling to histopathological evaluation. Methods Mol. Biol. 691, 69\u201382 (2011).","journal-title":"Methods Mol. Biol."},{"key":"520_CR3","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1016\/j.media.2019.02.012","volume":"54","author":"M Veta","year":"2019","unstructured":"Veta, M. et al. Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge. Med. Image Anal. 54, 111\u2013121 (2019).","journal-title":"Med. Image Anal."},{"key":"520_CR4","doi-asserted-by":"publisher","first-page":"giy065","DOI":"10.1093\/gigascience\/giy065","volume":"7","author":"G Litjens","year":"2018","unstructured":"Litjens, G. et al. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. Gigascience 7, giy065 (2018).","journal-title":"Gigascience"},{"key":"520_CR5","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"520_CR6","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211\u2013252 (2015).","journal-title":"Int. J. Comput. Vis."},{"key":"520_CR7","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","volume":"42","author":"G Litjens","year":"2017","unstructured":"Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A. & Ciompi, F. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60\u201388 (2017).","journal-title":"Med. Image Anal."},{"key":"520_CR8","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1038\/s41746-019-0112-2","volume":"2","author":"K Nagpal","year":"2019","unstructured":"Nagpal, K. et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. NPJ Digit. Med. 2, 48 (2019).","journal-title":"NPJ Digit. Med."},{"key":"520_CR9","doi-asserted-by":"publisher","first-page":"101563","DOI":"10.1016\/j.media.2019.101563","volume":"58","author":"S Graham","year":"2019","unstructured":"Graham, S. et al. Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med. Image Anal. 58, 101563 (2019).","journal-title":"Med. Image Anal."},{"key":"520_CR10","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P. & Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. in Medical Image Computing and Computer-Assisted Intervention \u2013 MICCAI 2015 234\u2013241 (Springer International Publishing, 2015).","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"520_CR11","doi-asserted-by":"publisher","first-page":"792","DOI":"10.1016\/j.cell.2018.03.040","volume":"173","author":"EM Christiansen","year":"2018","unstructured":"Christiansen, E. M. et al. In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images. Cell 173, 792\u2013803.e19 (2018).","journal-title":"Cell"},{"key":"520_CR12","doi-asserted-by":"publisher","unstructured":"Verma, R. et al. Multi-organ Nuclei Segmentation and Classification Challenge 2020. https:\/\/doi.org\/10.13140\/RG.2.2.12290.02244\/1 (2020).","DOI":"10.13140\/RG.2.2.12290.02244\/1"},{"key":"520_CR13","unstructured":"Goodfellow, I. et al. Generative Adversarial Nets. in Advances in Neural Information Processing Systems (eds. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. & Weinberger, K. Q.) vol. 27, 2672\u20132680 (Curran Associates, Inc., 2014)."},{"key":"520_CR14","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115\u2013118 (2017).","journal-title":"Nature"},{"key":"520_CR15","doi-asserted-by":"crossref","unstructured":"Jing, L. & Tian, Y. Self-supervised visual feature learning with deep neural networks: a survey. IEEE Trans. Pattern Anal. Mach. Intell. PP (2020).","DOI":"10.1109\/TPAMI.2020.2992393"},{"key":"520_CR16","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-017-17204-5","volume":"7","author":"P Bankhead","year":"2017","unstructured":"Bankhead, P. et al. QuPath: open source software for digital pathology image analysis. Sci. Rep. 7, 16878 (2017).","journal-title":"Sci. Rep."},{"key":"520_CR17","doi-asserted-by":"publisher","first-page":"e2005970","DOI":"10.1371\/journal.pbio.2005970","volume":"16","author":"C McQuin","year":"2018","unstructured":"McQuin, C. et al. CellProfiler 3.0: Next-generation image processing for biology. PLoS Biol. 16, e2005970 (2018).","journal-title":"PLoS Biol."},{"key":"520_CR18","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-017-15092-3","volume":"7","author":"M Nalisnik","year":"2017","unstructured":"Nalisnik, M. et al. Interactive phenotyping of large-scale histology imaging data with HistomicsML. Sci. Rep. 7, 14588 (2017).","journal-title":"Sci. Rep."},{"key":"520_CR19","unstructured":"Smit, A. et al. MedSelect: selective labeling for medical image classification combining meta-learning with deep reinforcement learning. arXiv https:\/\/arxiv.org\/abs\/2103.14339 (2021)."},{"key":"520_CR20","doi-asserted-by":"crossref","unstructured":"Aubreville, M., Bertram, C., Klopfleisch, R. & Maier, A. SlideRunner. In Bildverarbeitung f\u00fcr die Medizin 309\u2013314 (Springer, 2018).","DOI":"10.1007\/978-3-662-56537-7_81"},{"key":"520_CR21","doi-asserted-by":"publisher","unstructured":"Gamper, J., Koohbanani, N. A., Benet, K., Khuram, A. & Rajpoot, N. PanNuke: An Open Pan-Cancer Histology Dataset for Nuclei Instance Segmentation and Classification. Digital Pathology 11\u201319 https:\/\/doi.org\/10.1007\/978-3-030-23937-4_2 (2019).","DOI":"10.1007\/978-3-030-23937-4_2"},{"key":"520_CR22","doi-asserted-by":"publisher","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https:\/\/doi.org\/10.1109\/cvpr.2016.90 (2016).","DOI":"10.1109\/cvpr.2016.90"},{"key":"520_CR23","unstructured":"Sener, O. & Savarese, S. Active Learning for Convolutional Neural Networks: A Core-Set Approach. in International Conference on Learning Representations https:\/\/arxiv.org\/abs\/1708.00489 (2018)."},{"key":"520_CR24","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1097\/PAP.0000000000000162","volume":"24","author":"S Hendry","year":"2017","unstructured":"Hendry, S. et al. Assessing tumor-infiltrating lymphocytes in solid tumors: a practical review for pathologists and proposal for a standardized method from the international immunooncology biomarkers working group: part 1: assessing the host immune response, tils in invasive breast carcinoma and ductal carcinoma in situ, metastatic tumor deposits and areas for further research. Adv. Anat. Pathol. 24, 235\u2013251 (2017).","journal-title":"Adv. Anat. Pathol."},{"key":"520_CR25","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1038\/modpathol.2013.134","volume":"27","author":"AJJ Smits","year":"2014","unstructured":"Smits, A. J. J. et al. The estimation of tumor cell percentage for molecular testing by pathologists is not accurate. Mod. Pathol. 27, 168\u2013174 (2014).","journal-title":"Mod. Pathol."},{"key":"520_CR26","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1097\/MOG.0b013e328352b5ef","volume":"28","author":"ES Dellon","year":"2012","unstructured":"Dellon, E. S. Eosinophilic esophagitis: diagnostic tests and criteria. Curr. Opin. Gastroenterol. 28, 382\u2013388 (2012).","journal-title":"Curr. Opin. Gastroenterol."},{"key":"520_CR27","doi-asserted-by":"publisher","first-page":"1061","DOI":"10.1200\/JCO.2016.69.4406","volume":"35","author":"MJ Ellis","year":"2017","unstructured":"Ellis, M. J. et al. Ki67 proliferation index as a tool for chemotherapy decisions during and after neoadjuvant aromatase inhibitor treatment of breast cancer: results from the american college of surgeons oncology group Z1031 Trial (Alliance). J. Clin. Oncol. 35, 1061\u20131069 (2017).","journal-title":"J. Clin. Oncol."},{"key":"520_CR28","unstructured":"Vanschoren, J. Meta-learning: a survey. arXiv https:\/\/arxiv.org\/abs\/1810.03548 (2018)."},{"key":"520_CR29","unstructured":"He, X., Zhao, K. & Chu, X. AutoML: A Survey of the State-of-the-Art. arXiv https:\/\/arxiv.org\/abs\/1908.00709 (2020)."},{"key":"520_CR30","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1038\/ng.2764","volume":"45","author":"JN Weinstein","year":"2013","unstructured":"Weinstein, J. N. et al. The Cancer Genome Atlas Pan-Cancer analysis project. Nat. Genet. 45, 1113\u20131120 (2013).","journal-title":"Nat. Genet."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00520-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00520-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00520-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T19:00:25Z","timestamp":1670094025000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00520-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,7]]},"references-count":30,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["520"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00520-6","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,7]]},"assertion":[{"value":"28 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 October 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A.E., D.W., S.S. and R.S. are or were employees of Salesforce.com, Inc., and may own CRM stock. The remaining authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"145"}}