{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T18:28:26Z","timestamp":1771525706636,"version":"3.50.1"},"reference-count":119,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,3,20]],"date-time":"2020-03-20T00:00:00Z","timestamp":1584662400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100012659","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61402311, 91846104, 61728205, 61876117, 61876217, 61603186"],"award-info":[{"award-number":["61402311, 91846104, 61728205, 61876117, 61876217, 61603186"]}],"id":[{"id":"10.13039\/501100012659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1115417"],"award-info":[{"award-number":["IIS-1115417"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2021,3,31]]},"abstract":"<jats:p>Image classification is a key task in image understanding, and multi-label image classification has become a popular topic in recent years. However, the success of multi-label image classification is closely related to the way of constructing a training set. As active learning aims to construct an effective training set through iteratively selecting the most informative examples to query labels from annotators, it was introduced into multi-label image classification. Accordingly, multi-label active learning is becoming an important research direction. In this work, we first review existing multi-label active learning algorithms for image classification. These algorithms can be categorized into two top groups from two aspects respectively: sampling and annotation. The most important component of multi-label active learning is to design an effective sampling strategy that actively selects the examples with the highest informativeness from an unlabeled data pool, according to various information measures. Thus, different informativeness measures are emphasized in this survey. Furthermore, this work also makes a deep investigation on existing challenging issues and future promises in multi-label active learning with a focus on four core aspects: example dimension, label dimension, annotation, and application extension.<\/jats:p>","DOI":"10.1145\/3379504","type":"journal-article","created":{"date-parts":[[2020,3,20]],"date-time":"2020-03-20T21:04:17Z","timestamp":1584738257000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":96,"title":["Multi-Label Active Learning Algorithms for Image Classification"],"prefix":"10.1145","volume":"53","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8759-7107","authenticated-orcid":false,"given":"Jian","family":"Wu","sequence":"first","affiliation":[{"name":"Soochow University, China and Human Longevity, Inc., San Diego, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Victor S.","family":"Sheng","sequence":"additional","affiliation":[{"name":"Texas Tech University, Lubbock, TX, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jing","family":"Zhang","sequence":"additional","affiliation":[{"name":"Nanjing University of Science and Technology, Nanjing, Jiangsu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hua","family":"Li","sequence":"additional","affiliation":[{"name":"Washington University in St. Louis, St. Louis, MO, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tetiana","family":"Dadakova","sequence":"additional","affiliation":[{"name":"Human Longevity, Inc., San Diego, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christine Leon","family":"Swisher","sequence":"additional","affiliation":[{"name":"Human Longevity, Inc., San Diego, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiming","family":"Cui","sequence":"additional","affiliation":[{"name":"Suzhou University of Science and Technology, Suzhou, Jiangsu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pengpeng","family":"Zhao","sequence":"additional","affiliation":[{"name":"Soochow University, Suzhou, Jiangsu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,3,20]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/170035.170072"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 17th International Conference on Machine Learning (ICML\u201900)","author":"Allwein Erin L.","year":"2000"},{"key":"e_1_2_1_3_1","first-page":"1","article-title":"Semi-supervised learning with an imperfect supervisor","volume":"13","author":"Amini Massih R.","year":"2005","journal-title":"Knowl. Inf. Syst."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022821128753"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972825.60"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11060-016-2312-9"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2004.03.009"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201911)","author":"Bucak Selcuk"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2016.2593343"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2072298.2072028"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10851-014-0511-6"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972788.37"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2011.119"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022673506211"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.295"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 12th International Conference on Machine Learning (ICML\u201995)","author":"Dagan Ido"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557053"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401920"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2011.12.073"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-00958-7_12"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-012-0507-8"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/972764.972766"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2006.16"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-016-5421-x"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2716262"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2015151169"},{"key":"e_1_2_1_27_1","first-page":"1","article-title":"A robust regularization path algorithm for v-support vector classification","volume":"1","author":"Gu Bin","year":"2016","journal-title":"IEEE Trans. Neur. Netw. Learn. Syst."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the International Conference on Artificial Intelligence (AAAI\u201915)","author":"Gu Bin","year":"2015"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2016.2544779"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICME.2017.8019412"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1656274.1656278"},{"key":"e_1_2_1_32_1","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe Jeff","year":"2006","journal-title":"Wired Mag."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1459359.1459379"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the International Conference on Neural Information Processing Systems. 892--900","author":"Huang Shengjun","year":"2010"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2014.2307881"},{"key":"e_1_2_1_36_1","unstructured":"Sheng Jun Huang Songcan Chen and Zhihua Zhou. 2015. Multi-label active learning: Query type matters. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI\u201915) 946--952.  Sheng Jun Huang Songcan Chen and Zhihua Zhou. 2015. Multi-label active learning: Query type matters. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI\u201915) 946--952."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2012.2188038"},{"key":"e_1_2_1_38_1","volume-title":"Active Learning-Based Multi-Label Image Classification","author":"Jiao Yang"},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Yang Jiao Pengpeng Zhao Jian Wu Yujie Shi and Zhiming Cui. 2014. A multicriterion query-based batch mode active learning technique. In Foundations of Intelligent Systems. 669--680.  Yang Jiao Pengpeng Zhao Jian Wu Yujie Shi and Zhiming Cui. 2014. A multicriterion query-based batch mode active learning technique. In Foundations of Intelligent Systems. 669--680.","DOI":"10.1007\/978-3-642-54924-3_63"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-14717-8_41"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206627"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2007.4408844"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI\u201913)","author":"Kawewong Aram","year":"2013"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1046\/j.1467-9884.2003.t01-2-00383_4.x"},{"key":"e_1_2_1_45_1","volume-title":"Soldatova","author":"King Ross D.","year":"2009"},{"key":"e_1_2_1_46_1","volume-title":"Oliver","author":"King Ross D.","year":"2004"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSP.2002.1003062"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN\u201992)","author":"Lang Kenneth","year":"1992"},{"key":"e_1_2_1_49_1","volume-title":"Shao Wen Yang, and Shou De Lin","author":"Lee Guang He","year":"2016"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201994)","author":"David"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2014.2381872"},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201913)","author":"Li Xin","year":"2013"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201903)","author":"Li Xiaoli","year":"2003"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the International Conference on Image Processing (ICIP\u201904)","volume":"4","author":"Li Xuchun","year":"2004"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2017.07.005"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the IEEE International Conference on Data Mining (ICDM\u201903)","author":"Liu B."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.88"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.169"},{"key":"e_1_2_1_59_1","first-page":"35","article-title":"Large-scale matrix factorization with missing data under additional constraints","volume":"40","author":"Mitra Kaushik","year":"2010","journal-title":"Ophthal. Res."},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the International Conference on Digital Image Computing: Techniques and Applications (DICTA\u201910)","author":"Nasierding Gulisong"},{"key":"e_1_2_1_61_1","unstructured":"Reports of International Data Corporation. 2010. Retrieved from http:\/\/www.idc.com\/prodserv\/prodserv.jsp.  Reports of International Data Corporation. 2010. Retrieved from http:\/\/www.idc.com\/prodserv\/prodserv.jsp."},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201908)","author":"Qi Guojun","year":"2008"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2008.218"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2018.01.033"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.08.001"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.3007048"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3161606"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/130385.130417"},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD\u201908)","author":"Sheng Victor S."},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI\u201919)","volume":"33","author":"Victor"},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the Irish Conference on Artificial Intelligence and Cognitive Science (AICS\u201908)","author":"Singh Mohan","year":"2008"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICMLA.2018.00078"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISWC.2008.4911590"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13064"},{"key":"e_1_2_1_77_1","volume-title":"Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI\u201910)","author":"Sun Yu Yin","year":"2010"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2011.2180916"},{"key":"e_1_2_1_79_1","volume-title":"Active Learning: Theory and Applications","author":"Tong Simon","year":"2001"},{"key":"e_1_2_1_80_1","first-page":"45","article-title":"Support vector machine active learning with applications to text classification","volume":"2","author":"Tong Simon","year":"2001","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.4018\/jdwm.2007070101"},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of the ECML\/PKDD Workshop on Mining Multidimensional Data (MMD\u201908)","author":"Tsoumakas Grigorios","year":"2008"},{"key":"e_1_2_1_83_1","volume-title":"Data Mining and Knowledge Discovery Handbook","author":"Tsoumakas Grigorios"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2021078"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTSP.2011.2139193"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2004.08.002"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.nicl.2019.101727"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/1899412.1899414"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2013.03.011"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2014.10.040"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123266.3123388"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1109\/TRPMS.2018.2872406"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISBI.2018.8363578"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2014.7026058"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24075-6_46"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1049\/iet-cvi.2016.0243"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2017.2652065"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICME.2016.7552899"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-014-2381-8"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1145\/2072298.2072019"},{"key":"e_1_2_1_101_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201912)","author":"Yan Shuicheng","year":"2012"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557119"},{"key":"e_1_2_1_103_1","volume-title":"Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI\u201917)","author":"Yang Lin"},{"key":"e_1_2_1_104_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201913)","author":"Yang Shu Jun","year":"2013"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2015.7351442"},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1145\/2671188.2749365"},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.2318508"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/2382936.2382962"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1145\/1081870.1081911"},{"key":"e_1_2_1_110_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201908)","author":"Zha Zheng Jun","year":"2008"},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2014.2302675"},{"key":"e_1_2_1_112_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV.2012.6163043"},{"key":"e_1_2_1_113_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2018.2890148"},{"key":"e_1_2_1_114_1","volume-title":"Proceedings of the IEEE International Conference on Multimedia and Expo (ICME\u201909)","author":"Zhang Xiaoyu","year":"2009"},{"key":"e_1_2_1_115_1","volume-title":"Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI\u201910)","author":"Zhang Yi","year":"2010"},{"key":"e_1_2_1_116_1","doi-asserted-by":"publisher","DOI":"10.1145\/2733373.2806298"},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","DOI":"10.5555\/2729770.2729812"},{"key":"e_1_2_1_118_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-009-0209-z"},{"key":"e_1_2_1_119_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553534"},{"key":"e_1_2_1_120_1","volume-title":"Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS\u201907)","author":"Zhou Zhi Hua","year":"2007"},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2011.10.002"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3379504","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3379504","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3379504","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:02:23Z","timestamp":1750197743000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3379504"}},"subtitle":["Overview and Future Promise"],"short-title":[],"issued":{"date-parts":[[2020,3,20]]},"references-count":119,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,3,31]]}},"alternative-id":["10.1145\/3379504"],"URL":"https:\/\/doi.org\/10.1145\/3379504","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,20]]},"assertion":[{"value":"2017-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-03-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}