{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T17:55:41Z","timestamp":1772301341186,"version":"3.50.1"},"reference-count":38,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2020,1,27]],"date-time":"2020-01-27T00:00:00Z","timestamp":1580083200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["CMUPERI\/TIC\/0028\/2014"],"award-info":[{"award-number":["CMUPERI\/TIC\/0028\/2014"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["wires.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["WIREs Data Min &amp; Knowl"],"published-print":{"date-parts":[[2020,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Active learning (AL) methods create an optimized labeled training set from unlabeled data. We introduce a novel online active deep learning method for medical image analysis. We extend our MedAL AL framework to present new results in this paper. A novel sampling method queries the unlabeled examples that maximize the average distance to all training set examples. Our online method enhances performance of its underlying baseline deep network. These novelties contribute to significant performance improvements, including improving the model's underlying deep network accuracy by 6.30%, using only 25% of the labeled dataset to achieve baseline accuracy, reducing backpropagated images during training by as much as 67%, and demonstrating robustness to class imbalance in binary and multiclass tasks.<\/jats:p><jats:p>This article is categorized under:<\/jats:p><jats:p>Technologies &gt; Machine Learning<\/jats:p><jats:p>Technologies &gt; Classification<\/jats:p><jats:p>Application Areas &gt; Health Care<\/jats:p>","DOI":"10.1002\/widm.1353","type":"journal-article","created":{"date-parts":[[2020,1,27]],"date-time":"2020-01-27T14:28:59Z","timestamp":1580135339000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["O\u2010MedAL: Online active deep learning for medical image analysis"],"prefix":"10.1002","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8524-997X","authenticated-orcid":false,"given":"Asim","family":"Smailagic","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]},{"given":"Pedro","family":"Costa","sequence":"additional","affiliation":[{"name":"INESC TEC  Porto Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1380-6620","authenticated-orcid":false,"given":"Alex","family":"Gaudio","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]},{"given":"Kartik","family":"Khandelwal","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7364-7284","authenticated-orcid":false,"given":"Mostafa","family":"Mirshekari","sequence":"additional","affiliation":[{"name":"Department of Civil and Environmental Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]},{"given":"Jonathon","family":"Fagert","sequence":"additional","affiliation":[{"name":"Department of Civil and Environmental Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]},{"given":"Devesh","family":"Walawalkar","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]},{"given":"Susu","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Civil and Environmental Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]},{"given":"Adrian","family":"Galdran","sequence":"additional","affiliation":[{"name":"INESC TEC  Porto Portugal"}]},{"given":"Pei","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]},{"given":"Aur\u00e9lio","family":"Campilho","sequence":"additional","affiliation":[{"name":"INESC TEC  Porto Portugal"},{"name":"Faculty of Engineering University of Porto  Portugal"}]},{"given":"Hae Young","family":"Noh","sequence":"additional","affiliation":[{"name":"Department of Civil and Environmental Engineering Carnegie Mellon University  Pittsburgh Pennsylvania"}]}],"member":"311","published-online":{"date-parts":[[2020,1,27]]},"reference":[{"key":"e_1_2_17_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tcs.2003.11.004"},{"key":"e_1_2_17_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2019.05.010"},{"key":"e_1_2_17_4_1","first-page":"255","article-title":"Online choice of active learning algorithms","volume":"5","author":"Baram Y.","year":"2003","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_17_5_1","doi-asserted-by":"crossref","unstructured":"Cai W. Zhang Y. &Zhou J.(2013). Maximizing expected model change for active learning in regression. In 2013IEEE 13th international conference on data mining(pp. 51\u201360). Dallas TX: IEEE.","DOI":"10.1109\/ICDM.2013.104"},{"key":"e_1_2_17_6_1","first-page":"705","article-title":"Active learning with statistical models","author":"Cohn D. A.","year":"1995","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_17_7_1","doi-asserted-by":"crossref","unstructured":"Costa P. Ara\u00fajo T. Aresta G. Galdran A. Mendon\u00e7a A. M. Smailagic A.andCampilho A.(2019) Eyewes: Weakly supervised pre\u2010trained convolutional neural networks for diabetic retinopathy detection. In2019 16th international conference on machine vision applications (MVA)(pp. 1\u20136). Tokyo Japan.","DOI":"10.23919\/MVA.2019.8757991"},{"key":"e_1_2_17_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2017.2759102"},{"key":"e_1_2_17_9_1","first-page":"353","article-title":"A general agnostic active learning algorithm","author":"Dasgupta S.","year":"2008","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_17_10_1","doi-asserted-by":"crossref","unstructured":"Ertekin S. Huang J. Bottou L. &Giles L.(2007). Learning on the border: Active learning in imbalanced data classification. InProceedings of the sixteenth ACM conference on conference on information and knowledge management CIKM '07 (pp. 127\u2013136). New York NY: ACM.http:\/\/doi.acm.org\/10.1145\/1321440.1321461.","DOI":"10.1145\/1321440.1321461"},{"key":"e_1_2_17_11_1","first-page":"483","volume-title":"Advances in Neural Information Processing Systems","author":"Freund Y.","year":"1993"},{"key":"e_1_2_17_12_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007330508534"},{"key":"e_1_2_17_13_1","doi-asserted-by":"crossref","unstructured":"Freytag A. Rodner E. &Denzler J.(2014). Selecting influential examples: Active learning with expected model output changes. InEuropean conference on computer vision(pp. 562\u2013577). Zuerich Switzerland: Springer.","DOI":"10.1007\/978-3-319-10593-2_37"},{"key":"e_1_2_17_14_1","unstructured":"Gal Y. Islam R. &Ghahramani Z. (2017). Deep bayesian active learning with image data. In D. Precup & Y. Whye teh (Eds.) Proceedings of the 34th International Conference on Machine Learning(vol. 70 pp. 1183\u20131192). Sydney Australia: PMLR.http:\/\/proceedings.mlr.press\/v70\/gal17a\/gal17a.pdf"},{"key":"e_1_2_17_15_1","unstructured":"Glorot X. &Bengio Y.(2010). Understanding the difficulty of training deep feedforward neural networks. InProceedings of the thirteenth international conference on artificial intelligence and statistics. (pp. 249\u2013256). Sardinia Italy."},{"key":"e_1_2_17_16_1","unstructured":"G\u00f3rriz M. Gir\u00f3\u2010i Nieto X. Carlier A. &Faure E.(2017). Cost\u2010effective active learning for melanoma segmentation. InML4H: Machine learning for health workshop at NIPS 2017 Long Beach CA."},{"key":"e_1_2_17_17_1","doi-asserted-by":"crossref","unstructured":"Gutman D. Codella N. C. Celebi E. Helba B. Marchetti M. Mishra N. &Halpern A.(2018). Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi) 2016 hosted by the international skin imaging collaboration (isic).2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI2018)(pp. 168\u2013172).https:\/\/doi.org\/10.1109\/ISBI.2018.8363547.","DOI":"10.1109\/ISBI.2018.8363547"},{"key":"e_1_2_17_18_1","unstructured":"He K. Zhang X. Ren S.andSun J.(2015)Deep residual learning for image recognition.CoRR abs\/1512.03385.http:\/\/arxiv.org\/abs\/1512.03385."},{"key":"e_1_2_17_19_1","doi-asserted-by":"crossref","DOI":"10.4103\/2153-3539.92034","article-title":"A comparison of sampling strategies for histological image analysis","volume":"2","author":"Homeyer A.","year":"2012","journal-title":"Journal of Pathology Informatics"},{"key":"e_1_2_17_20_1","unstructured":"Kingma D. P. &Ba J.(2014). Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980.https:\/\/ui.adsabs.harvard.edu\/abs\/2014arXiv1412.6980K."},{"key":"e_1_2_17_21_1","doi-asserted-by":"crossref","unstructured":"Lewis D. D.andCatlett J.(1994) Heterogeneous uncertainty sampling for supervised learning. InMachine learning proceedings 1994(pp. 148\u2013156). Elsevier.","DOI":"10.1016\/B978-1-55860-335-6.50026-X"},{"key":"e_1_2_17_22_1","doi-asserted-by":"crossref","unstructured":"Lewis D. D. &Gale W. A.(1994). A sequential algorithm for training text classifiers. InProceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval(pp. 3\u201312). Budapest Hungary: Springer\u2010Verlag New York Inc.","DOI":"10.1007\/978-1-4471-2099-5_1"},{"key":"e_1_2_17_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"e_1_2_17_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2015.2505672"},{"key":"e_1_2_17_25_1","doi-asserted-by":"crossref","unstructured":"Melville P. &Mooney R. J.(2004).Diverse ensembles for active learning. InProceedings of the twenty\u2010first international conference on Machine learning(p. 74). Banff Canada: ACM.","DOI":"10.1145\/1015330.1015385"},{"key":"e_1_2_17_26_1","unstructured":"Mikolov T. Chen K. Corrado G. &Dean J. (2013). Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781.http:\/\/arxiv.org\/abs\/1301.3781"},{"key":"e_1_2_17_27_1","first-page":"7008","article-title":"Active learning from peers","volume":"30","author":"Murugesan K.","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_17_28_1","doi-asserted-by":"crossref","unstructured":"Pace D. F. Dalca A. V. Geva T. Powell A. J. Moghari M. H. &Golland P.(2015). Interactive whole\u2010heart segmentation in congenital heart disease. InMedical image computing and computer\u2010assisted intervention\u2014MICCAI 2015 Lecture Notes in Computer Science(pp. 80\u201388).","DOI":"10.1007\/978-3-319-24574-4_10"},{"key":"e_1_2_17_29_1","doi-asserted-by":"publisher","DOI":"10.1088\/0004-637X\/744\/2\/192"},{"key":"e_1_2_17_30_1","unstructured":"Sculley D.(2007). Online active learning methods for fast label\u2010efficient spam filtering. InCEAS."},{"key":"e_1_2_17_31_1","unstructured":"Settles B.(2008).Curious machines: Active learning with structured instances. (Ph.D. thesis). University of Wisconsin\u2013Madison."},{"key":"e_1_2_17_32_1","doi-asserted-by":"crossref","unstructured":"Smailagic A. Costa P. Noh H. Y. Walawalkar D. Khandelwal K. Galdran A. Mirshekari\u2026Zhang P.(2018). Medal: Accurate and robust deep active learning for medical image analysis. In2018 17th IEEE international conference on machine learning and applications (ICMLA)(pp. 481\u2013488). Orlando FL: IEEE.","DOI":"10.1109\/ICMLA.2018.00078"},{"key":"e_1_2_17_33_1","doi-asserted-by":"crossref","unstructured":"Szegedy C. Vanhoucke V. Ioffe S. Shlens J. &Wojna Z.(2016). Rethinking the inception architecture for computer vision. InProceedings of the IEEE conference on computer vision and pattern recognition(pp. 2818\u20132826). LasVegas Nevada.","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_2_17_34_1","first-page":"227","article-title":"Comparison of different classifiers with active learning to support quality control in nucleus segmentation in pathology images","volume":"2017","author":"Wen S.","year":"2018","journal-title":"AMIA Summits on Translational Science Proceedings"},{"key":"e_1_2_17_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-014-0781-x"},{"key":"e_1_2_17_36_1","unstructured":"Zhang C. Bengio S. Hardt M. Recht B. &Vinyals O.(2016). Understanding deep learning requires rethinking generalization.arXiv preprint arXiv:1611.03530."},{"key":"e_1_2_17_37_1","doi-asserted-by":"crossref","unstructured":"Zhang C. Tavanapong W. Kijkul G. Wong J. deGroen P. C. &Oh J.(2018). Similarity\u2010based active learning for image classification under class imbalance. In2018 IEEE international conference on data mining (ICDM)(pp. 1422\u20131427). Sentosa Island Singapore: IEEE.https:\/\/doi.org\/10.1109\/ICDM.2018.00196","DOI":"10.1109\/ICDM.2018.00196"},{"key":"e_1_2_17_38_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/aac042"},{"key":"e_1_2_17_39_1","first-page":"7340","article-title":"Fine\u2010tuning convolutional neural networks for biomedical image analysis: Actively and incrementally","author":"Zhou Z.","year":"2017","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"}],"container-title":["WIREs Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fwidm.1353","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/widm.1353","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/widm.1353","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/wires.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/widm.1353","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T17:22:29Z","timestamp":1693329749000},"score":1,"resource":{"primary":{"URL":"https:\/\/wires.onlinelibrary.wiley.com\/doi\/10.1002\/widm.1353"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,27]]},"references-count":38,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,7]]}},"alternative-id":["10.1002\/widm.1353"],"URL":"https:\/\/doi.org\/10.1002\/widm.1353","archive":["Portico"],"relation":{},"ISSN":["1942-4787","1942-4795"],"issn-type":[{"value":"1942-4787","type":"print"},{"value":"1942-4795","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,27]]},"assertion":[{"value":"2019-06-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-13","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-01-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e1353"}}