{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T02:10:13Z","timestamp":1776132613143,"version":"3.50.1"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2023,3,21]],"date-time":"2023-03-21T00:00:00Z","timestamp":1679356800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Council of Scientific and Technological Development"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,4,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Motivation<\/jats:title><jats:p>Deep learning attained excellent results in digital pathology recently. A challenge with its use is that high quality, representative training datasets are required to build robust models. Data annotation in the domain is labor intensive and demands substantial time commitment from expert pathologists. Active learning (AL) is a strategy to minimize annotation. The goal is to select samples from the pool of unlabeled data for annotation that improves model accuracy. However, AL is a very compute demanding approach. The benefits for model learning may vary according to the strategy used, and it may be hard for a domain specialist to fine tune the solution without an integrated interface.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We developed a framework that includes a friendly user interface along with run-time optimizations to reduce annotation and execution time in AL in digital pathology. Our solution implements several AL strategies along with our diversity-aware data acquisition (DADA) acquisition function, which enforces data diversity to improve the prediction performance of a model. In this work, we employed a model simplification strategy [Network Auto-Reduction (NAR)] that significantly improves AL execution time when coupled with DADA. NAR produces less compute demanding models, which replace the target models during the AL process to reduce processing demands. An evaluation with a tumor-infiltrating lymphocytes classification application shows that: (i) DADA attains superior performance compared to state-of-the-art AL strategies for different convolutional neural networks (CNNs), (ii) NAR improves the AL execution time by up to 4.3\u00d7, and (iii) target models trained with patches\/data selected by the NAR reduced versions achieve similar or superior classification quality to using target CNNs for data selection.<\/jats:p><\/jats:sec><jats:sec><jats:title>Availability and implementation<\/jats:title><jats:p>Source code: https:\/\/github.com\/alsmeirelles\/DADA.<\/jats:p><\/jats:sec>","DOI":"10.1093\/bioinformatics\/btad138","type":"journal-article","created":{"date-parts":[[2023,3,18]],"date-time":"2023-03-18T07:34:33Z","timestamp":1679124873000},"source":"Crossref","is-referenced-by-count":6,"title":["Effective and efficient active learning for deep learning-based tissue image analysis"],"prefix":"10.1093","volume":"39","author":[{"given":"Andr\u00e9 L S","family":"Meirelles","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Bras\u00edlia , Bras\u00edlia 70910-900, Brazil"}]},{"given":"Tahsin","family":"Kurc","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Department, Stony Brook University , Stony Brook, NY 11794-8322, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8757-8339","authenticated-orcid":false,"given":"Jun","family":"Kong","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Statistics and Computer Science, Georgia State University , Atlanta, GA 30302-4110, USA"}]},{"given":"Renato","family":"Ferreira","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Universidade Federal de Minas Gerais , Belo Horizonte 31270-901, Brazil"}]},{"given":"Joel","family":"Saltz","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Department, Stony Brook University , Stony Brook, NY 11794-8322, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6289-3914","authenticated-orcid":false,"given":"George","family":"Teodoro","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Bras\u00edlia , Bras\u00edlia 70910-900, Brazil"},{"name":"Biomedical Informatics Department, Stony Brook University , Stony Brook, NY 11794-8322, USA"},{"name":"Department of Computer Science, Universidade Federal de Minas Gerais , Belo Horizonte 31270-901, Brazil"}]}],"member":"286","published-online":{"date-parts":[[2023,3,21]]},"reference":[{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"3461","DOI":"10.1093\/bioinformatics\/btz083","article-title":"Structured crowdsourcing enables convolutional segmentation of histology images","volume":"35","author":"Amgad","year":"2019","journal-title":"Bioinformatics"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1016\/j.coi.2013.03.004","article-title":"From the immune contexture to the immunoscore: the role of prognostic and predictive immune markers in cancer","volume":"25","author":"Angell","year":"2013","journal-title":"Curr Opin Immunol"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2015.12.002","article-title":"Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles","volume":"30","author":"Barker","year":"2016","journal-title":"Med Image Anal"},{"key":"2023040621282848000_","first-page":"9368","author":"Beluch","year":"2018"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"e1057388","DOI":"10.1080\/2162402X.2015.1057388","article-title":"Density of tumor-infiltrating lymphocytes correlates with extent of brain edema and overall survival time in patients with brain metastases","volume":"5","author":"Berghoff","year":"2016","journal-title":"OncoImmunology"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1007\/978-3-030-23937-4_3","volume-title":"European Congress on Digital Pathology","author":"Carse","year":"2019"},{"key":"2023040621282848000_","author":"Cheng"},{"key":"2023040621282848000_","first-page":"1607","author":"Ding","year":"2019"},{"key":"2023040621282848000_","first-page":"4510","author":"Ding","year":"2021"},{"key":"2023040621282848000_","volume-title":"Elementary Applied Statistics: For Students in Behavioral Science","author":"Freeman","year":"1965"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1007\/s10115-012-0507-8","article-title":"A survey on instance selection for active learning","volume":"35","author":"Fu","year":"2013","journal-title":"Knowl Inform Syst"},{"key":"2023040621282848000_","first-page":"1183","author":"Gal","year":"2017"},{"key":"2023040621282848000_","first-page":"3994","author":"Gong","year":"2021"},{"key":"2023040621282848000_","first-page":"770","author":"He","year":"2016"},{"key":"2023040621282848000_","first-page":"236","author":"Hossain","year":"2020"},{"key":"2023040621282848000_","author":"Houlsby","year":"2011"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"1491","DOI":"10.1016\/j.ajpath.2020.03.012","article-title":"Utilizing automated breast cancer detection to identify spatial distributions of tumor infiltrating lymphocytes in invasive breast cancer","volume":"190","author":"Le","year":"2020","journal-title":"Am J Pathol"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1109\/TIP.2018.2868382","article-title":"Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition","volume":"28","author":"Li","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"2023040621282848000_","first-page":"859","author":"Li","year":"2013"},{"key":"2023040621282848000_","first-page":"2790","author":"Lin","year":"2019"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"2525","DOI":"10.1109\/TPAMI.2018.2858232","article-title":"ThiNet: pruning CNN filters for a thinner net","volume":"41","author":"Luo","year":"2018","journal-title":"IEEE Trans Patt Anal Mach Intell"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"1604","DOI":"10.3389\/fmed.2022.894430","article-title":"Building efficient CNN architectures for histopathology images analysis: a case-study in tumor-infiltrating lymphocytes","volume":"9","author":"Meirelles","year":"2022","journal-title":"Front Med"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"106828","DOI":"10.1016\/j.cmpb.2022.106828","article-title":"Effective active learning in digital pathology: a case study in tumor infiltrating lymphocytes","volume":"220","author":"Meirelles","year":"2022","journal-title":"Comput Methods Programs Biomed"},{"key":"2023040621282848000_","first-page":"542","article-title":"Quick annotator: an open-source digital pathology based rapid image annotation tool","volume":"7","author":"Miao","year":"2021","journal-title":"J Pathol"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/s10555-011-9270-7","article-title":"Tumor immunosurveillance in human cancers","volume":"30","author":"Mlecnik","year":"2011","journal-title":"Cancer Metastasis Rev"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"103196","DOI":"10.1016\/j.dsp.2021.103196","article-title":"Artificial intelligence in computational pathology \u2013 challenges and future directions","volume":"119","author":"Morales","year":"2021","journal-title":"Dig Signal Process"},{"key":"2023040621282848000_","first-page":"1","article-title":"A survey of crowdsourcing in medical image analysis","volume-title":"Hum Comput","author":"\u00d8rting","year":"2019"},{"key":"2023040621282848000_","first-page":"235","volume-title":"Pattern Recognit Lett","author":"Osaku","year":"2021"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"14347","DOI":"10.1038\/s41598-019-50587-1","article-title":"ARA: accurate, reliable and active histopathological image classification framework with Bayesian deep learning","volume":"9","author":"R\u0105czkowski Mo\u017cejko","year":"2019","journal-title":"Sci Rep"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1016\/j.humpath.2018.05.017","article-title":"Evaluation of tumor-infiltrating lymphocytes using routine H&E slides predicts patient survival in resected non-small cell lung cancer","volume":"70","author":"Rakaee","year":"2018","journal-title":"Hum Pathol"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","DOI":"10.21037\/tlcr-20-591","article-title":"A narrative review of digital pathology and artificial intelligence: focusing on lung cancer","volume":"9","author":"Sakamoto","year":"2020","journal-title":"Translation Lung Cancer Res"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/j.celrep.2018.03.086","article-title":"Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images","volume":"23","author":"Saltz","year":"2018","journal-title":"Cell Rep"},{"key":"2023040621282848000_","first-page":"4510","author":"Sandler","year":"2018"},{"key":"2023040621282848000_","author":"Sener","year":"2018"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1007\/s11760-020-01760-x","article-title":"A dynamic CNN pruning method based on matrix similarity","volume":"15","author":"Shao","year":"2021","journal-title":"Signal, Image Video Process"},{"key":"2023040621282848000_","author":"Simonyan","year":"2014"},{"key":"2023040621282848000_","author":"Tan","year":"2019"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1038\/s41591-021-01343-4","article-title":"Deep learning in histopathology: the path to the clinic","volume":"27","author":"van der Laak","year":"2021","journal-title":"Nat Med"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"4291","DOI":"10.1093\/bioinformatics\/btab380","article-title":"HEAL: an automated deep learning framework for cancer histopathology image analysis","volume":"37","author":"Wang","year":"2021","journal-title":"Bioinformatics"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1007\/s11263-014-0781-x","article-title":"Multi-class active learning by uncertainty sampling with diversity maximization","volume":"113","author":"Yang","year":"2015","journal-title":"Int J Comput Vis"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/j.knosys.2019.02.013","article-title":"Multi-criteria active deep learning for image classification","volume":"172","author":"Yuan","year":"2019","journal-title":"Knowl-Based Syst"},{"key":"2023040621282848000_","doi-asserted-by":"crossref","first-page":"1323","DOI":"10.1109\/TASL.2009.2033421","article-title":"Active learning with sampling by uncertainty and density for data annotations","volume":"18","author":"Zhu","year":"2009","journal-title":"IEEE Trans Audio, Speech, Lang Process"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btad138\/49583778\/btad138.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/39\/4\/btad138\/49790978\/btad138.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/39\/4\/btad138\/49790978\/btad138.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,9]],"date-time":"2023-12-09T04:37:28Z","timestamp":1702096648000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btad138\/7082521"}},"subtitle":[],"editor":[{"given":"Hanchuan","family":"Peng","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2023,3,21]]},"references-count":42,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,4,3]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btad138","relation":{},"ISSN":["1367-4811"],"issn-type":[{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,4,1]]},"published":{"date-parts":[[2023,3,21]]},"article-number":"btad138"}}