{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T02:53:27Z","timestamp":1764557607906,"version":"3.41.0"},"reference-count":148,"publisher":"Association for Computing Machinery (ACM)","issue":"3-4","license":[{"start":{"date-parts":[[2021,9,3]],"date-time":"2021-09-03T00:00:00Z","timestamp":1630627200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001655","name":"German Academic Exchange Service","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001655","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Austrian Research Promotion Agency","award":["866880"],"award-info":[{"award-number":["866880"]}]},{"name":"Lower Austrian Research and Education Company","award":["FTI17-014 and FTI18-005"],"award-info":[{"award-number":["FTI17-014 and FTI18-005"]}]},{"DOI":"10.13039\/501100000038","name":"National Sciences and Engineering Research Council of Canada","doi-asserted-by":"crossref","award":["RGPIN-2014-06309"],"award-info":[{"award-number":["RGPIN-2014-06309"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["251654672\u2013TRR 161"],"award-info":[{"award-number":["251654672\u2013TRR 161"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2021,12,31]]},"abstract":"<jats:p>\n            Strategies for selecting the next data instance to label, in service of generating labeled data for machine learning, have been considered separately in the machine learning literature on active learning and in the visual analytics literature on human-centered approaches. We propose a unified design space for instance selection strategies to support detailed and fine-grained analysis covering both of these perspectives. We identify a concise set of 15 properties, namely measureable characteristics of datasets or of machine learning models applied to them, that cover most of the strategies in these literatures. To quantify these properties, we introduce Property Measures (PM) as fine-grained building blocks that can be used to formalize instance selection strategies. In addition, we present a taxonomy of PMs to support the description, evaluation, and generation of PMs across four dimensions: machine learning (ML)\n            <jats:italic>Model Output<\/jats:italic>\n            ,\n            <jats:italic>Instance Relations<\/jats:italic>\n            ,\n            <jats:italic>Measure Functionality<\/jats:italic>\n            , and\n            <jats:italic>Measure Valence<\/jats:italic>\n            . We also create computational infrastructure to support qualitative visual data analysis: a visual analytics explainer for PMs built around an implementation of PMs using cascades of eight atomic functions. It supports eight analysis tasks, covering the analysis of datasets and ML models using visual comparison within and between PMs and groups of PMs, and over time during the interactive labeling process. We iteratively refined the PM taxonomy, the explainer, and the task abstraction in parallel with each other during a two-year formative process, and show evidence of their utility through a summative evaluation with the same infrastructure. This research builds a formal baseline for the better understanding of the commonalities and differences of instance selection strategies, which can serve as the stepping stone for the synthesis of novel strategies in future work.\n          <\/jats:p>","DOI":"10.1145\/3439333","type":"journal-article","created":{"date-parts":[[2021,9,3]],"date-time":"2021-09-03T19:30:07Z","timestamp":1630697407000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["A Taxonomy of Property Measures to Unify Active Learning and Human-centered Approaches to Data Labeling"],"prefix":"10.1145","volume":"11","author":[{"given":"J\u00fcrgen","family":"Bernard","sequence":"first","affiliation":[{"name":"University of British Columbia, Vancouver, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marco","family":"Hutter","sequence":"additional","affiliation":[{"name":"TU Darmstadt, Darmstadt, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Sedlmair","sequence":"additional","affiliation":[{"name":"University of Stuttgart, Stuttgart, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthias","family":"Zeppelzauer","sequence":"additional","affiliation":[{"name":"St. P\u00f6lten University of Applied Sciences, St. P\u00f6lten, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tamara","family":"Munzner","sequence":"additional","affiliation":[{"name":"University of British Columbia, Vancouver, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,9,3]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_2_2_1_1","DOI":"10.1111\/cgf.13684"},{"doi-asserted-by":"publisher","key":"e_1_2_2_2_1","DOI":"10.1109\/ACCESS.2018.2870052"},{"volume-title":"Data Mining","author":"Aggarwal Charu C.","unstructured":"Charu C. Aggarwal . 2015. Outlier analysis . In Data Mining . Springer , 237\u2013263. Charu C. Aggarwal. 2015. Outlier analysis. In Data Mining. Springer, 237\u2013263.","key":"e_1_2_2_3_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_4_1","DOI":"10.1109\/TVCG.2015.2467618"},{"doi-asserted-by":"publisher","key":"e_1_2_2_5_1","DOI":"10.1109\/TVCG.2014.2346660"},{"doi-asserted-by":"publisher","key":"e_1_2_2_6_1","DOI":"10.1145\/2702123.2702509"},{"volume-title":"Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 1027\u20131035","author":"Arthur D.","unstructured":"D. Arthur and S. Vassilvitskii . 2007. k-means++: The advantages of careful seeding . In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 1027\u20131035 . D. Arthur and S. Vassilvitskii. 2007. k-means++: The advantages of careful seeding. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 1027\u20131035.","key":"e_1_2_2_7_1"},{"key":"e_1_2_2_8_1","volume-title":"et\u00a0al","author":"Arya Vijay","year":"2019","unstructured":"Vijay Arya , Rachel K. E. Bellamy , Pin-Yu Chen , Amit Dhurandhar , Michael Hind , Samuel C. Hoffman , Stephanie Houde , Q. Vera Liao , Ronny Luss , Aleksandra Mojsilovi\u0107 , et\u00a0al . 2019 . One explanation does not fit all: A toolkit and taxonomy of AI explainability techniques. arXiv:1909.03012. Retrieved from https:\/\/arxiv.org\/abs\/1909.03012. Vijay Arya, Rachel K. E. Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Q. Vera Liao, Ronny Luss, Aleksandra Mojsilovi\u0107, et\u00a0al. 2019. One explanation does not fit all: A toolkit and taxonomy of AI explainability techniques. arXiv:1909.03012. Retrieved from https:\/\/arxiv.org\/abs\/1909.03012."},{"doi-asserted-by":"publisher","key":"e_1_2_2_9_1","DOI":"10.1145\/1964897.1964906"},{"key":"e_1_2_2_10_1","volume-title":"Proceedings of the IEEE Pacific Visualization Symposium (PacificVis\u201916)","author":"Aupetit M.","year":"2016","unstructured":"M. Aupetit and M. Sedlmair . 2016. SepMe: 2002 new visual separation measures . In Proceedings of the IEEE Pacific Visualization Symposium (PacificVis\u201916) . 1\u20138. https:\/\/doi.org\/10.1109\/PACIFICVIS. 2016 .7465244 10.1109\/PACIFICVIS.2016.7465244 M. Aupetit and M. Sedlmair. 2016. SepMe: 2002 new visual separation measures. In Proceedings of the IEEE Pacific Visualization Symposium (PacificVis\u201916). 1\u20138. https:\/\/doi.org\/10.1109\/PACIFICVIS.2016.7465244"},{"doi-asserted-by":"publisher","key":"e_1_2_2_11_1","DOI":"10.1016\/j.image.2007.05.010"},{"volume-title":"Fuzzy Modeling for Control","author":"Babu\u0161ka Robert","unstructured":"Robert Babu\u0161ka . 2012. Fuzzy Modeling for Control . Vol. 12 . Springer Science & Business Media . Robert Babu\u0161ka. 2012. Fuzzy Modeling for Control. Vol. 12. Springer Science & Business Media.","key":"e_1_2_2_12_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_13_1","DOI":"10.1145\/1007730.1007735"},{"doi-asserted-by":"publisher","key":"e_1_2_2_14_1","DOI":"10.1145\/989863.989865"},{"doi-asserted-by":"publisher","key":"e_1_2_2_15_1","DOI":"10.1109\/TVCG.2016.2598467"},{"key":"e_1_2_2_16_1","volume-title":"Proceedings of the IEEE Visual Analytics Science and Technology (VAST\u201914)","author":"Behrisch M.","year":"2014","unstructured":"M. Behrisch , F. Korkmaz , Lin Shao , and T. Schreck . 2014. Feedback-driven interactive exploration of large multidimensional data supported by visual classifier . In Proceedings of the IEEE Visual Analytics Science and Technology (VAST\u201914) . 43\u201352. https:\/\/doi.org\/10.1109\/VAST. 2014 .7042480 10.1109\/VAST.2014.7042480 M. Behrisch, F. Korkmaz, Lin Shao, and T. Schreck. 2014. Feedback-driven interactive exploration of large multidimensional data supported by visual classifier. In Proceedings of the IEEE Visual Analytics Science and Technology (VAST\u201914). 43\u201352. https:\/\/doi.org\/10.1109\/VAST.2014.7042480"},{"key":"e_1_2_2_17_1","volume-title":"Proceedings of the Graphics, Patterns and Images (SIBGRAPI\u201918)","author":"Benato B. C.","year":"2018","unstructured":"B. C. Benato , A. C. Telea , and A. X. Falc\u00e3o . 2018. Semi-supervised learning with interactive label propagation guided by feature space projections . In Proceedings of the Graphics, Patterns and Images (SIBGRAPI\u201918) . 392\u2013399. https:\/\/doi.org\/10.1109\/SIBGRAPI. 2018 .00057 10.1109\/SIBGRAPI.2018.00057 B. C. Benato, A. C. Telea, and A. X. Falc\u00e3o. 2018. Semi-supervised learning with interactive label propagation guided by feature space projections. In Proceedings of the Graphics, Patterns and Images (SIBGRAPI\u201918). 392\u2013399. https:\/\/doi.org\/10.1109\/SIBGRAPI.2018.00057"},{"key":"e_1_2_2_18_1","volume-title":"Proceedings of the Conference on Visualization (EuroVis\u201918)","author":"Bernard J\u00fcrgen","year":"2018","unstructured":"J\u00fcrgen Bernard , Marco Hutter , Markus Lehmann , Martin M\u00fcller , Matthias Zeppelzauer , and Michael Sedlmair . 2018 . Learning from the best\u2014Visual analysis of a quasi-optimal data labeling strategy . In Proceedings of the Conference on Visualization (EuroVis\u201918) . Eurographics. https:\/\/doi.org\/10.2312\/eurovisshort. 20181085 10.2312\/eurovisshort.20181085 J\u00fcrgen Bernard, Marco Hutter, Markus Lehmann, Martin M\u00fcller, Matthias Zeppelzauer, and Michael Sedlmair. 2018. Learning from the best\u2014Visual analysis of a quasi-optimal data labeling strategy. In Proceedings of the Conference on Visualization (EuroVis\u201918). Eurographics. https:\/\/doi.org\/10.2312\/eurovisshort.20181085"},{"key":"e_1_2_2_19_1","volume-title":"Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201919)","author":"Bernard J\u00fcrgen","year":"2019","unstructured":"J\u00fcrgen Bernard , Marco Hutter , Christian Ritter , Markus Lehmann , Michael Sedlmair , and Matthias Zeppelzauer . 2019 . Visual analysis of degree-of-interest functions to support selection strategies for instance labeling . In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201919) . The Eurographics Association. https:\/\/doi.org\/10.2312\/eurova. 20191116 10.2312\/eurova.20191116 J\u00fcrgen Bernard, Marco Hutter, Christian Ritter, Markus Lehmann, Michael Sedlmair, and Matthias Zeppelzauer. 2019. Visual analysis of degree-of-interest functions to support selection strategies for instance labeling. In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201919). The Eurographics Association. https:\/\/doi.org\/10.2312\/eurova.20191116"},{"doi-asserted-by":"publisher","key":"e_1_2_2_20_1","DOI":"10.1109\/TVCG.2017.2744818"},{"key":"e_1_2_2_21_1","volume-title":"Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201920)","author":"Bernard J\u00fcrgen","year":"2020","unstructured":"J\u00fcrgen Bernard , Marco Hutter , Matthias Zeppelzauer , Michael Sedlmair , and Tamara Munzner . 2020 . SepEx: Visual analysis of class separation measures . In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201920) . The Eurographics Association. https:\/\/doi.org\/10.2312\/eurova. 20201079 10.2312\/eurova.20201079 J\u00fcrgen Bernard, Marco Hutter, Matthias Zeppelzauer, Michael Sedlmair, and Tamara Munzner. 2020. SepEx: Visual analysis of class separation measures. In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201920). The Eurographics Association. https:\/\/doi.org\/10.2312\/eurova.20201079"},{"doi-asserted-by":"publisher","key":"e_1_2_2_22_1","DOI":"10.1145\/2836034.2836035"},{"key":"e_1_2_2_23_1","first-page":"329","article-title":"User-based visual-interactive similarity definition for mixed data objects-concept and first implementation","volume":"22","author":"Bernard J\u00fcrgen","year":"2014","unstructured":"J\u00fcrgen Bernard , David Sessler , Tobias Ruppert , James Davey , Arjan Kuijper , and J\u00f6rn Kohlhammer . 2014 . User-based visual-interactive similarity definition for mixed data objects-concept and first implementation . J. WSCG 22 (2014), 329 \u2013 338 . J\u00fcrgen Bernard, David Sessler, Tobias Ruppert, James Davey, Arjan Kuijper, and J\u00f6rn Kohlhammer. 2014. User-based visual-interactive similarity definition for mixed data objects-concept and first implementation. J. WSCG 22 (2014), 329\u2013338.","journal-title":"J. WSCG"},{"doi-asserted-by":"crossref","unstructured":"J\u00fcrgen Bernard Matthias Zeppelzauer Markus Lehmann Martin M\u00fcller and Michael Sedlmair. 2018. Towards user-centered active learning algorithms. Comput. Graph. For. (2018). https:\/\/doi.org\/10.1111\/cgf.13406 10.1111\/cgf.13406","key":"#cr-split#-e_1_2_2_24_1.1","DOI":"10.1111\/cgf.13406"},{"doi-asserted-by":"crossref","unstructured":"J\u00fcrgen Bernard Matthias Zeppelzauer Markus Lehmann Martin M\u00fcller and Michael Sedlmair. 2018. Towards user-centered active learning algorithms. Comput. Graph. For. (2018). https:\/\/doi.org\/10.1111\/cgf.13406","key":"#cr-split#-e_1_2_2_24_1.2","DOI":"10.1111\/cgf.13406"},{"doi-asserted-by":"publisher","key":"e_1_2_2_25_1","DOI":"10.1007\/s00371-018-1500-3"},{"doi-asserted-by":"publisher","key":"e_1_2_2_26_1","DOI":"10.1145\/2133806.2133826"},{"doi-asserted-by":"publisher","key":"e_1_2_2_27_1","DOI":"10.1086\/228631"},{"doi-asserted-by":"publisher","key":"e_1_2_2_28_1","DOI":"10.1023\/A:1010933404324"},{"doi-asserted-by":"publisher","key":"e_1_2_2_29_1","DOI":"10.1145\/335191.335388"},{"doi-asserted-by":"publisher","key":"e_1_2_2_30_1","DOI":"10.1109\/VAST.2012.6400486"},{"unstructured":"\u00c1ngel Alexander Cabrera Fred Hohman Jason Lin and Duen Horng Chau. 2018. Interactive classification for deep learning interpretation. arXiv:1806.05660. Retrieved from https:\/\/arxiv.org\/abs\/1806.05660.  \u00c1ngel Alexander Cabrera Fred Hohman Jason Lin and Duen Horng Chau. 2018. Interactive classification for deep learning interpretation. arXiv:1806.05660. Retrieved from https:\/\/arxiv.org\/abs\/1806.05660.","key":"e_1_2_2_31_1"},{"key":"e_1_2_2_32_1","volume-title":"Article 15","author":"Chandola Varun","year":"2009","unstructured":"Varun Chandola , Arindam Banerjee , and Vipin Kumar . 2009. Anomaly detection: A survey. ACM Comput. Surv. 41, 3 , Article 15 ( 2009 ), 58 pages. https:\/\/doi.org\/10.1145\/1541880.1541882 10.1145\/1541880.1541882 Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM Comput. Surv. 41, 3, Article 15 (2009), 58 pages. https:\/\/doi.org\/10.1145\/1541880.1541882"},{"doi-asserted-by":"publisher","key":"e_1_2_2_33_1","DOI":"10.1016\/j.visinf.2019.03.002"},{"key":"e_1_2_2_34_1","volume-title":"Proceedings of the IEEE VIS Workshop on Evaluation of Interactive Visual Machine Learning Systems.","author":"Chegini Mohammad","year":"2019","unstructured":"Mohammad Chegini , J\u00fcrgen Bernard , Lin Shao , Alexei Sourin , Keith Andrews , and Tobias Schreck . 2019 . mVis in the Wild: Pre-Study of an interactive visual machine learning system for labelling . In Proceedings of the IEEE VIS Workshop on Evaluation of Interactive Visual Machine Learning Systems. Mohammad Chegini, J\u00fcrgen Bernard, Lin Shao, Alexei Sourin, Keith Andrews, and Tobias Schreck. 2019. mVis in the Wild: Pre-Study of an interactive visual machine learning system for labelling. In Proceedings of the IEEE VIS Workshop on Evaluation of Interactive Visual Machine Learning Systems."},{"key":"e_1_2_2_35_1","volume-title":"Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems. 2172\u20132180.","author":"Chen Xi","year":"2016","unstructured":"Xi Chen , Yan Duan , Rein Houthooft , John Schulman , Ilya Sutskever , and Pieter Abbeel . 2016 . Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems. 2172\u20132180. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems. 2172\u20132180."},{"doi-asserted-by":"publisher","key":"e_1_2_2_36_1","DOI":"10.1109\/INFVIS.2000.885092"},{"volume-title":"Proceedings of the 12th International Conference on Machine Learning. 108\u2013114","author":"John","unstructured":"John G. Cleary and Leonard E. Trigg. 1995. K*: An instance-based learner using an entropic distance measure . In Proceedings of the 12th International Conference on Machine Learning. 108\u2013114 . John G. Cleary and Leonard E. Trigg. 1995. K*: An instance-based learner using an entropic distance measure. In Proceedings of the 12th International Conference on Machine Learning. 108\u2013114.","key":"e_1_2_2_37_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_38_1","DOI":"10.1007\/BF00994018"},{"doi-asserted-by":"publisher","key":"e_1_2_2_39_1","DOI":"10.5951\/MT.53.1.0033"},{"unstructured":"Piotr Dabkowski and Yarin Gal. 2017. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems. 6967\u20136976.  Piotr Dabkowski and Yarin Gal. 2017. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems. 6967\u20136976.","key":"e_1_2_2_40_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_41_1","DOI":"10.1109\/TVCG.2012.128"},{"doi-asserted-by":"publisher","key":"e_1_2_2_42_1","DOI":"10.1109\/TVCG.2014.2346572"},{"doi-asserted-by":"publisher","key":"e_1_2_2_43_1","DOI":"10.1109\/TVCG.2010.184"},{"doi-asserted-by":"crossref","unstructured":"A. Dasgupta H. Wang N. O'Brien and S. Burrows. 2019. Separating the wheat from the chaff: Comparative visual cues for transparent diagnostics of competing models. IEEE Trans. Vis. Comput. Graph. (2019) 1-1. https:\/\/doi.org\/10.1109\/TVCG.2019.2934540 10.1109\/TVCG.2019.2934540","key":"#cr-split#-e_1_2_2_44_1.1","DOI":"10.1109\/TVCG.2019.2934540"},{"doi-asserted-by":"crossref","unstructured":"A. Dasgupta H. Wang N. O'Brien and S. Burrows. 2019. Separating the wheat from the chaff: Comparative visual cues for transparent diagnostics of competing models. IEEE Trans. Vis. Comput. Graph. (2019) 1-1. https:\/\/doi.org\/10.1109\/TVCG.2019.2934540","key":"#cr-split#-e_1_2_2_44_1.2","DOI":"10.1109\/TVCG.2019.2934540"},{"key":"e_1_2_2_45_1","first-page":"2","article-title":"A cluster separation measure","volume":"1","author":"Davies D. L.","year":"1979","unstructured":"D. L. Davies and D. W. Bouldin . 1979 . A cluster separation measure . IEEE Trans. Pattern Anal. Mach. Intell. 1 , 2 (Apr. 1979), 224\u2013227. https:\/\/doi.org\/10.1109\/TPAMI.1979.4766909 10.1109\/TPAMI.1979.4766909 D. L. Davies and D. W. Bouldin. 1979. A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell. 1, 2 (Apr. 1979), 224\u2013227. https:\/\/doi.org\/10.1109\/TPAMI.1979.4766909","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"doi-asserted-by":"publisher","key":"e_1_2_2_46_1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x"},{"doi-asserted-by":"crossref","unstructured":"Frederik L. Dennig Tom Polk Zudi Lin Tobias Schreck Hanspeter Pfister and Michael Behrisch. 2019. FDive: Learning relevance models using pattern-based similarity measures. arXiv:1907.12489. Retrieved from http:\/\/arxiv.org\/abs\/1907.12489.  Frederik L. Dennig Tom Polk Zudi Lin Tobias Schreck Hanspeter Pfister and Michael Behrisch. 2019. FDive: Learning relevance models using pattern-based similarity measures. arXiv:1907.12489. Retrieved from http:\/\/arxiv.org\/abs\/1907.12489.","key":"e_1_2_2_47_1","DOI":"10.1109\/VAST47406.2019.8986940"},{"key":"e_1_2_2_48_1","volume-title":"Stork","author":"Duda Richard O.","year":"2012","unstructured":"Richard O. Duda , Peter E. Hart , and David G . Stork . 2012 . Pattern Classification. John Wiley & Sons . Richard O. Duda, Peter E. Hart, and David G. Stork. 2012. Pattern Classification. John Wiley & Sons."},{"doi-asserted-by":"publisher","key":"e_1_2_2_49_1","DOI":"10.1080\/01969727408546059"},{"doi-asserted-by":"publisher","key":"e_1_2_2_50_1","DOI":"10.1109\/TVCG.2019.2944182"},{"key":"e_1_2_2_51_1","volume-title":"Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201919)","author":"Espadoto Mateus","year":"2019","unstructured":"Mateus Espadoto , Francisco Caio Maia Rodrigues , Nina S. T. Hirata , Roberto Hirata Jr ., and Alexandru C. Telea . 2019. Deep learning inverse multidimensional projections . In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201919) . The Eurographics Association. https:\/\/doi.org\/10.2312\/eurova. 2019 1118 10.2312\/eurova.20191118 Mateus Espadoto, Francisco Caio Maia Rodrigues, Nina S. T. Hirata, Roberto Hirata Jr., and Alexandru C. Telea. 2019. Deep learning inverse multidimensional projections. In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA\u201919). The Eurographics Association. https:\/\/doi.org\/10.2312\/eurova.20191118"},{"key":"e_1_2_2_52_1","volume-title":"Frey and Delbert Dueck","author":"Brendan","year":"2007","unstructured":"Brendan J. Frey and Delbert Dueck . 2007 . Clustering by passing messages between data points. Science 315, 5814 (2007), 972\u2013976. https:\/\/doi.org\/10.1126\/science.1136800 arXiv:https:\/\/science.sciencemag.org\/content\/315\/5814\/972.full.pdf. 10.1126\/science.1136800 Brendan J. Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. Science 315, 5814 (2007), 972\u2013976. https:\/\/doi.org\/10.1126\/science.1136800 arXiv:https:\/\/science.sciencemag.org\/content\/315\/5814\/972.full.pdf."},{"key":"e_1_2_2_53_1","volume-title":"Bayesian network classifiers. Mach. Learn. 29, 2 (01","author":"Friedman Nir","year":"1997","unstructured":"Nir Friedman , Dan Geiger , and Moises Goldszmidt . 1997. Bayesian network classifiers. Mach. Learn. 29, 2 (01 Nov. 1997 ), 131\u2013163. https:\/\/doi.org\/10.1023\/A:1007465528199 10.1023\/A:1007465528199 Nir Friedman, Dan Geiger, and Moises Goldszmidt. 1997. Bayesian network classifiers. Mach. Learn. 29, 2 (01 Nov. 1997), 131\u2013163. https:\/\/doi.org\/10.1023\/A:1007465528199"},{"key":"e_1_2_2_54_1","volume-title":"A survey on instance selection for active learning. Knowl. Inf. Syst. 35, 2 (01","author":"Fu Yifan","year":"2013","unstructured":"Yifan Fu , Xingquan Zhu , and Bin Li. 2013. A survey on instance selection for active learning. Knowl. Inf. Syst. 35, 2 (01 May 2013 ), 249\u2013283. https:\/\/doi.org\/10.1007\/s10115-012-0507-8 10.1007\/s10115-012-0507-8 Yifan Fu, Xingquan Zhu, and Bin Li. 2013. A survey on instance selection for active learning. Knowl. Inf. Syst. 35, 2 (01 May 2013), 249\u2013283. https:\/\/doi.org\/10.1007\/s10115-012-0507-8"},{"key":"e_1_2_2_55_1","volume-title":"Proceedings of the International Symposium on Information Theory (ISIT\u201904)","author":"Fuglede B.","year":"2004","unstructured":"B. Fuglede and F. Topsoe . 2004. Jensen-Shannon divergence and Hilbert space embedding . In Proceedings of the International Symposium on Information Theory (ISIT\u201904) . 31. https:\/\/doi.org\/10.1109\/ISIT. 2004 .1365067 10.1109\/ISIT.2004.1365067 B. Fuglede and F. Topsoe. 2004. Jensen-Shannon divergence and Hilbert space embedding. In Proceedings of the International Symposium on Information Theory (ISIT\u201904). 31. https:\/\/doi.org\/10.1109\/ISIT.2004.1365067"},{"key":"e_1_2_2_56_1","volume-title":"E. Pizetti and T. Salvemini. Libreria Eredi Virgilio Veschi","author":"Gini Corrado","year":"1912","unstructured":"Corrado Gini . 1912. Variabilit\u00e0 e mutabilit\u00e0. Reprinted in Memorie di metodologica statistica , E. Pizetti and T. Salvemini. Libreria Eredi Virgilio Veschi , Rome ( 1912 ). Corrado Gini. 1912. Variabilit\u00e0 e mutabilit\u00e0. Reprinted in Memorie di metodologica statistica, E. Pizetti and T. Salvemini. Libreria Eredi Virgilio Veschi, Rome (1912)."},{"volume-title":"Proceedings of the International Workshop on Computer Vision Meets Databases. ACM, 51\u201358","author":"Philippe","unstructured":"Philippe H. Gosselin and Matthieu Cord. 2004. A comparison of active classification methods for content-based image retrieval . In Proceedings of the International Workshop on Computer Vision Meets Databases. ACM, 51\u201358 . Philippe H. Gosselin and Matthieu Cord. 2004. A comparison of active classification methods for content-based image retrieval. In Proceedings of the International Workshop on Computer Vision Meets Databases. ACM, 51\u201358.","key":"e_1_2_2_57_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_58_1","DOI":"10.1145\/601858.601862"},{"key":"e_1_2_2_59_1","volume-title":"Data Mining: Concepts and Techniques","author":"Han Jiawei","year":"2012","unstructured":"Jiawei Han , Jian Pei , and Micheline Kamber . 2012 . Data Mining: Concepts and Techniques . Elsevier . https:\/\/doi.org\/10.1016\/C2009-0-61819-5 10.1016\/C2009-0-61819-5 Jiawei Han, Jian Pei, and Micheline Kamber. 2012. Data Mining: Concepts and Techniques. Elsevier. https:\/\/doi.org\/10.1016\/C2009-0-61819-5"},{"doi-asserted-by":"publisher","key":"e_1_2_2_60_1","DOI":"10.1109\/TVCG.2012.277"},{"doi-asserted-by":"publisher","key":"e_1_2_2_61_1","DOI":"10.1007\/978-3-319-46493-0_1"},{"unstructured":"Andreas Hinterreiter Peter Ruch Holger Stitz Martin Ennemoser J\u00fcrgen Bernard Hendrik Strobelt and Marc Streit. 2019. ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion. arXiv:cs.LG\/1910.00969  Andreas Hinterreiter Peter Ruch Holger Stitz Martin Ennemoser J\u00fcrgen Bernard Hendrik Strobelt and Marc Streit. 2019. ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion. arXiv:cs.LG\/1910.00969","key":"e_1_2_2_62_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_63_1","DOI":"10.1287\/moor.10.2.180"},{"doi-asserted-by":"publisher","key":"e_1_2_2_64_1","DOI":"10.1109\/VAST.2012.6400492"},{"doi-asserted-by":"publisher","key":"e_1_2_2_65_1","DOI":"10.1109\/TVCG.2018.2843369"},{"volume-title":"Proceedings of the Annual Conference on the World Wide Web. ACM, 633\u2013642","author":"Hoi Steven C. H.","unstructured":"Steven C. H. Hoi , Rong Jin , and Michael R. Lyu . 2006. Large-scale text categorization by batch mode active learning . In Proceedings of the Annual Conference on the World Wide Web. ACM, 633\u2013642 . https:\/\/doi.org\/10.1145\/1135777.1135870 10.1145\/1135777.1135870 Steven C. H. Hoi, Rong Jin, and Michael R. Lyu. 2006. Large-scale text categorization by batch mode active learning. In Proceedings of the Annual Conference on the World Wide Web. ACM, 633\u2013642. https:\/\/doi.org\/10.1145\/1135777.1135870","key":"e_1_2_2_66_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_67_1","DOI":"10.1016\/0893-6080(89)90020-8"},{"doi-asserted-by":"publisher","key":"e_1_2_2_68_1","DOI":"10.1145\/3038462.3038469"},{"doi-asserted-by":"publisher","key":"e_1_2_2_69_1","DOI":"10.1016\/j.patrec.2009.09.011"},{"doi-asserted-by":"publisher","key":"e_1_2_2_70_1","DOI":"10.5555\/1293951.1293954"},{"doi-asserted-by":"publisher","key":"e_1_2_2_71_1","DOI":"10.1007\/BF02418571"},{"key":"e_1_2_2_72_1","volume-title":"Principal Component Analysis","author":"Jolliffe I. T.","unstructured":"I. T. Jolliffe . 2002. Principal Component Analysis ( 3 rd ed.). Springer . I. T. Jolliffe. 2002. Principal Component Analysis (3rd ed.). Springer.","edition":"3"},{"doi-asserted-by":"publisher","key":"e_1_2_2_73_1","DOI":"10.1145\/2926720"},{"key":"e_1_2_2_74_1","volume-title":"Koyejo","author":"Kim Been","year":"2016","unstructured":"Been Kim , Rajiv Khanna , and Oluwasanmi O . Koyejo . 2016 . Examples are not enough, learn to criticize! Criticism for Interpretability. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc ., 2280\u20132288. Been Kim, Rajiv Khanna, and Oluwasanmi O. Koyejo. 2016. Examples are not enough, learn to criticize! Criticism for Interpretability. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc., 2280\u20132288."},{"doi-asserted-by":"publisher","key":"e_1_2_2_75_1","DOI":"10.1145\/3214366"},{"volume-title":"Proceedings of the Conference on Very Large Data Bases (VLDB\u201998)","author":"Edwin","unstructured":"Edwin M. Knorr and Raymond T. Ng. 1998. Algorithms for mining distance-based outliers in large datasets . In Proceedings of the Conference on Very Large Data Bases (VLDB\u201998) . Morgan Kaufmann, 392\u2013403. Edwin M. Knorr and Raymond T. Ng. 1998. Algorithms for mining distance-based outliers in large datasets. In Proceedings of the Conference on Very Large Data Bases (VLDB\u201998). Morgan Kaufmann, 392\u2013403.","key":"e_1_2_2_76_1"},{"key":"e_1_2_2_77_1","first-page":"83","article-title":"Sulla determinazione empirica di una legge di distribuzione","volume":"4","author":"Kolmogorov Andrey","year":"1933","unstructured":"Andrey Kolmogorov . 1933 . Sulla determinazione empirica di una legge di distribuzione . G. Ist. Ital. Attuari 4 (1933), 83 \u2013 91 . Andrey Kolmogorov. 1933. Sulla determinazione empirica di una legge di distribuzione. G. Ist. Ital. Attuari 4 (1933), 83\u201391.","journal-title":"G. Ist. Ital. Attuari"},{"unstructured":"Ksenia Konyushkova Raphael Sznitman and Pascal Fua. 2017. Learning active learning from data. In Advances in Neural Information Processing Systems. 4226\u20134236.  Ksenia Konyushkova Raphael Sznitman and Pascal Fua. 2017. Learning active learning from data. In Advances in Neural Information Processing Systems. 4226\u20134236.","key":"e_1_2_2_78_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_79_1","DOI":"10.1109\/TVCG.2014.2346751"},{"key":"e_1_2_2_80_1","volume-title":"Proceedings of the Workshop and Tutorial on Interactive Adaptive Learning. 2\u201314","author":"Kottke Daniel","year":"2017","unstructured":"Daniel Kottke , Adrian Calma , Denis Huseljic , Georg Krempl , and Bernhard Sick . 2017 . Challenges of reliable, realistic and comparable active learning evaluation . In Proceedings of the Workshop and Tutorial on Interactive Adaptive Learning. 2\u201314 . Daniel Kottke, Adrian Calma, Denis Huseljic, Georg Krempl, and Bernhard Sick. 2017. Challenges of reliable, realistic and comparable active learning evaluation. In Proceedings of the Workshop and Tutorial on Interactive Adaptive Learning. 2\u201314."},{"doi-asserted-by":"publisher","key":"e_1_2_2_81_1","DOI":"10.1145\/2858036.2858529"},{"doi-asserted-by":"publisher","key":"e_1_2_2_82_1","DOI":"10.1145\/1401890.1401946"},{"doi-asserted-by":"publisher","key":"e_1_2_2_83_1","DOI":"10.1007\/BF02289565"},{"doi-asserted-by":"publisher","key":"e_1_2_2_84_1","DOI":"10.1145\/3132169"},{"doi-asserted-by":"publisher","key":"e_1_2_2_85_1","DOI":"10.1214\/aoms\/1177729694"},{"doi-asserted-by":"publisher","key":"e_1_2_2_86_1","DOI":"10.1007\/s10994-005-0466-3"},{"key":"e_1_2_2_87_1","volume-title":"Deep learning. Nature 521, 7553","author":"LeCun Yann","year":"2015","unstructured":"Yann LeCun , Yoshua Bengio , and Geoffrey Hinton . 2015. Deep learning. Nature 521, 7553 ( 2015 ), 436\u2013444. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436\u2013444."},{"doi-asserted-by":"publisher","key":"e_1_2_2_88_1","DOI":"10.5555\/2858877.2858908"},{"unstructured":"Qimai Li Xiao-Ming Wu and Zhichao Guan. 2019. Generalized label propagation methods for semi-supervised learning. arXiv:1901.09993. Retrieved from https:\/\/arxiv.org\/abs\/1901.09993.  Qimai Li Xiao-Ming Wu and Zhichao Guan. 2019. Generalized label propagation methods for semi-supervised learning. arXiv:1901.09993. Retrieved from https:\/\/arxiv.org\/abs\/1901.09993.","key":"e_1_2_2_89_1"},{"key":"e_1_2_2_90_1","volume-title":"Proceedings of the IEEE Conference on Graphics, Patterns and Images (SIBGRAPI\u201918)","author":"Rodrigues F. C. M.","year":"2018","unstructured":"F. C. M. Rodrigues , R. Hirata , and A. C. Telea . 2018. Image-based visualization of classifier decision boundaries . In Proceedings of the IEEE Conference on Graphics, Patterns and Images (SIBGRAPI\u201918) . 353\u2013360. https:\/\/doi.org\/10.1109\/SIBGRAPI. 2018 .00052 10.1109\/SIBGRAPI.2018.00052 F. C. M. Rodrigues, R. Hirata, and A. C. Telea. 2018. Image-based visualization of classifier decision boundaries. In Proceedings of the IEEE Conference on Graphics, Patterns and Images (SIBGRAPI\u201918). 353\u2013360. https:\/\/doi.org\/10.1109\/SIBGRAPI.2018.00052"},{"key":"e_1_2_2_91_1","article-title":"Visualizing data using t-SNE","author":"van der Maaten Laurens","year":"2008","unstructured":"Laurens van der Maaten and Geoffrey Hinton . 2008 . Visualizing data using t-SNE . J. Mach. Learn. Res. 9 , ( Nov. 2008), 2579\u20132605. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, (Nov. 2008), 2579\u20132605.","journal-title":"J. Mach. Learn. Res. 9"},{"key":"e_1_2_2_92_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201998)","volume":"1","author":"Hiroshi Mamitsuka Naoki Abe","year":"1998","unstructured":"Naoki Abe Hiroshi Mamitsuka . 1998 . Query learning strategies using boosting and bagging . In Proceedings of the International Conference on Machine Learning (ICML\u201998) , Vol. 1 . Morgan Kaufmann. Naoki Abe Hiroshi Mamitsuka. 1998. Query learning strategies using boosting and bagging. In Proceedings of the International Conference on Machine Learning (ICML\u201998), Vol. 1. Morgan Kaufmann."},{"doi-asserted-by":"publisher","key":"e_1_2_2_93_1","DOI":"10.1109\/TVCG.2017.2744339"},{"key":"e_1_2_2_94_1","volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201998)","author":"McCallum Andrew","year":"1998","unstructured":"Andrew McCallum and Kamal Nigam . 1998 . Employing EM and pool-based active learning for text classification . In Proceedings of the International Conference on Machine Learning (ICML\u201998) . Morgan Kaufmann, San Francisco, CA, 350\u2013358. Andrew McCallum and Kamal Nigam. 1998. Employing EM and pool-based active learning for text classification. In Proceedings of the International Conference on Machine Learning (ICML\u201998). Morgan Kaufmann, San Francisco, CA, 350\u2013358."},{"volume-title":"Proceedings of the SIGKDD International Conference on Knowledge Discovery and Data Mining. Citeseer, 169\u2013178","author":"McCallum Andrew","unstructured":"Andrew McCallum , Kamal Nigam , and Lyle H. Ungar . 2000. Efficient clustering of high-dimensional data sets with application to reference matching . In Proceedings of the SIGKDD International Conference on Knowledge Discovery and Data Mining. Citeseer, 169\u2013178 . Andrew McCallum, Kamal Nigam, and Lyle H. Ungar. 2000. Efficient clustering of high-dimensional data sets with application to reference matching. In Proceedings of the SIGKDD International Conference on Knowledge Discovery and Data Mining. Citeseer, 169\u2013178.","key":"e_1_2_2_95_1"},{"volume-title":"Advances in Computers.","author":"Mitrovi\u0107 Dalibor","unstructured":"Dalibor Mitrovi\u0107 , Matthias Zeppelzauer , and Christian Breiteneder . 2010. Features for content-based audio retrieval . In Advances in Computers. Vol. 78 . Elsevier , 71\u2013150. Dalibor Mitrovi\u0107, Matthias Zeppelzauer, and Christian Breiteneder. 2010. Features for content-based audio retrieval. In Advances in Computers. Vol. 78. Elsevier, 71\u2013150.","key":"e_1_2_2_96_1"},{"key":"e_1_2_2_97_1","volume-title":"Gowayyed","author":"Momtaz Rana","year":"2013","unstructured":"Rana Momtaz , Nesma Mohssen , and Mohammad A . Gowayyed . 2013 . DWOF : A robust density-based outlier detection approach. In Pattern Recognition and Image Analysis. Springer , 517\u2013525. Rana Momtaz, Nesma Mohssen, and Mohammad A. Gowayyed. 2013. DWOF: A robust density-based outlier detection approach. In Pattern Recognition and Image Analysis. Springer, 517\u2013525."},{"doi-asserted-by":"publisher","key":"e_1_2_2_98_1","DOI":"10.1093\/comjnl\/26.4.354"},{"doi-asserted-by":"publisher","key":"e_1_2_2_101_1","DOI":"10.1109\/SIBGRAPI.2007.21"},{"key":"e_1_2_2_102_1","volume-title":"Moore","author":"Pelleg Dan","year":"2000","unstructured":"Dan Pelleg and Andrew W . Moore . 2000 . X-means : Extending K-means with efficient estimation of the number of clusters. In Proceedings of the Conference on Machine Learning. Morgan Kaufmann , 727\u2013734. Dan Pelleg and Andrew W. Moore. 2000. X-means: Extending K-means with efficient estimation of the number of clusters. In Proceedings of the Conference on Machine Learning. Morgan Kaufmann, 727\u2013734."},{"doi-asserted-by":"publisher","key":"e_1_2_2_103_1","DOI":"10.1109\/TVCG.2017.2744358"},{"doi-asserted-by":"publisher","key":"e_1_2_2_104_1","DOI":"10.1109\/TPAMI.2008.218"},{"doi-asserted-by":"publisher","key":"e_1_2_2_105_1","DOI":"10.1145\/342009.335437"},{"doi-asserted-by":"publisher","key":"e_1_2_2_106_1","DOI":"10.1177\/1473871617713337"},{"doi-asserted-by":"publisher","key":"e_1_2_2_107_1","DOI":"10.1109\/TVCG.2016.2598828"},{"doi-asserted-by":"publisher","key":"e_1_2_2_108_1","DOI":"10.1111\/j.1467-8659.2009.01694.x"},{"doi-asserted-by":"publisher","key":"e_1_2_2_109_1","DOI":"10.3390\/info10090280"},{"doi-asserted-by":"publisher","key":"e_1_2_2_110_1","DOI":"10.1016\/0377-0427(87)90125-7"},{"doi-asserted-by":"publisher","key":"e_1_2_2_111_1","DOI":"10.1109\/34.790428"},{"key":"e_1_2_2_112_1","volume-title":"Keim","author":"Schneidewind Jorn","year":"2006","unstructured":"Jorn Schneidewind , Mike Sips , and Daniel A . Keim . 2006 . Pixnostics : Towards measuring the value of visualization. In Proceedings of the IEEE Visual Analytics Science and Technology (VAST\u2019 06). 199\u2013206. Jorn Schneidewind, Mike Sips, and Daniel A. Keim. 2006. Pixnostics: Towards measuring the value of visualization. In Proceedings of the IEEE Visual Analytics Science and Technology (VAST\u201906). 199\u2013206."},{"doi-asserted-by":"publisher","key":"e_1_2_2_113_1","DOI":"10.5555\/2858877.2858899"},{"doi-asserted-by":"crossref","unstructured":"M. Sedlmair A. Tatu T. Munzner and M. Tory. 2012. A taxonomy of visual cluster separation factors. Comput. Graph. Forum 31 3pt4 (2012) 1335-1344. https:\/\/doi.org\/10.1111\/j.1467-8659.2012.03125.x 10.1111\/j.1467-8659.2012.03125.x","key":"#cr-split#-e_1_2_2_114_1.1","DOI":"10.1111\/j.1467-8659.2012.03125.x"},{"doi-asserted-by":"crossref","unstructured":"M. Sedlmair A. Tatu T. Munzner and M. Tory. 2012. A taxonomy of visual cluster separation factors. Comput. Graph. Forum 31 3pt4 (2012) 1335-1344. https:\/\/doi.org\/10.1111\/j.1467-8659.2012.03125.x","key":"#cr-split#-e_1_2_2_114_1.2","DOI":"10.1111\/j.1467-8659.2012.03125.x"},{"key":"e_1_2_2_115_1","volume-title":"Proceedings of the IEEE Conference on Data Mining Workshops (ICDMW\u201910)","author":"Seifert C.","year":"2010","unstructured":"C. Seifert and M. Granitzer . 2010. User-based active learning . In Proceedings of the IEEE Conference on Data Mining Workshops (ICDMW\u201910) . 418\u2013425. https:\/\/doi.org\/10.1109\/ICDMW. 2010 .181 10.1109\/ICDMW.2010.181 C. Seifert and M. Granitzer. 2010. User-based active learning. In Proceedings of the IEEE Conference on Data Mining Workshops (ICDMW\u201910). 418\u2013425. https:\/\/doi.org\/10.1109\/ICDMW.2010.181"},{"unstructured":"Ramprasaath R. Selvaraju Abhishek Das Ramakrishna Vedantam Michael Cogswell Devi Parikh and Dhruv Batra. 2016. Grad-CAM: Why did you say that? arXiv:1611.07450. Retrieved from https:\/\/arxiv.org\/abs\/1611.07450.  Ramprasaath R. Selvaraju Abhishek Das Ramakrishna Vedantam Michael Cogswell Devi Parikh and Dhruv Batra. 2016. Grad-CAM: Why did you say that? arXiv:1611.07450. Retrieved from https:\/\/arxiv.org\/abs\/1611.07450.","key":"e_1_2_2_116_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_118_1","DOI":"10.2200\/S00429ED1V01Y201207AIM018"},{"doi-asserted-by":"publisher","key":"e_1_2_2_119_1","DOI":"10.3115\/1613715.1613855"},{"unstructured":"Burr Settles Mark Craven and Soumya Ray. 2008. Multiple-instance active learning. In Advances in Neural Information Processing Systems. 1289\u20131296.   Burr Settles Mark Craven and Soumya Ray. 2008. Multiple-instance active learning. In Advances in Neural Information Processing Systems. 1289\u20131296.","key":"e_1_2_2_120_1"},{"volume-title":"Proceedings of the Workshop on Computer Learning Theory (COLT\u201992)","author":"Seung H. S.","unstructured":"H. S. Seung , M. Opper , and H. Sompolinsky . 1992. Query by committee . In Proceedings of the Workshop on Computer Learning Theory (COLT\u201992) . ACM, 287\u2013294. https:\/\/doi.org\/10.1145\/130385.130417 10.1145\/130385.130417 H. S. Seung, M. Opper, and H. Sompolinsky. 1992. Query by committee. In Proceedings of the Workshop on Computer Learning Theory (COLT\u201992). ACM, 287\u2013294. https:\/\/doi.org\/10.1145\/130385.130417","key":"e_1_2_2_121_1"},{"doi-asserted-by":"publisher","key":"e_1_2_2_122_1","DOI":"10.1002\/j.1538-7305.1948.tb01338.x"},{"key":"e_1_2_2_123_1","volume-title":"Measurement of diversity. Nature 163, 4148","author":"Simpson Edward H.","year":"1949","unstructured":"Edward H. Simpson . 1949. Measurement of diversity. Nature 163, 4148 ( 1949 ), 688. Edward H. Simpson. 1949. Measurement of diversity. Nature 163, 4148 (1949), 688."},{"doi-asserted-by":"publisher","key":"e_1_2_2_124_1","DOI":"10.1111\/j.1467-8659.2009.01467.x"},{"doi-asserted-by":"publisher","key":"e_1_2_2_125_1","DOI":"10.1214\/aoms\/1177730256"},{"doi-asserted-by":"publisher","key":"e_1_2_2_126_1","DOI":"10.1016\/j.ipm.2009.03.002"},{"doi-asserted-by":"publisher","key":"e_1_2_2_127_1","DOI":"10.1109\/TVCG.2017.2744158"},{"doi-asserted-by":"publisher","key":"e_1_2_2_128_1","DOI":"10.1109\/TVCG.2016.2598829"},{"key":"e_1_2_2_129_1","volume-title":"Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL\u201902)","author":"Tang Min","year":"2002","unstructured":"Min Tang , Xiaoqiang Luo , and Salim Roukos . 2002 . Active learning for statistical natural language parsing . In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL\u201902) . Association for Computational Linguistics, Stroudsburg, PA, 120\u2013127. https:\/\/doi.org\/10.3115\/1073083.1073105 10.3115\/1073083.1073105 Min Tang, Xiaoqiang Luo, and Salim Roukos. 2002. Active learning for statistical natural language parsing. In Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL\u201902). Association for Computational Linguistics, Stroudsburg, PA, 120\u2013127. https:\/\/doi.org\/10.3115\/1073083.1073105"},{"key":"e_1_2_2_130_1","volume-title":"Automated analytical methods to support visual exploration of high-dimensional data","author":"Tatu Andrada","year":"2010","unstructured":"Andrada Tatu , Georgia Albuquerque , Martin Eisemann , Peter Bak , Hogler Theisel , Marcus Magnor , and Daniel Keim . 2010. Automated analytical methods to support visual exploration of high-dimensional data .IEEE Trans. Vis. Comput. Graph . ( 2010 ), 1\u201314. https:\/\/doi.org\/10.1109\/TVCG.2010.242 10.1109\/TVCG.2010.242 Andrada Tatu, Georgia Albuquerque, Martin Eisemann, Peter Bak, Hogler Theisel, Marcus Magnor, and Daniel Keim. 2010. Automated analytical methods to support visual exploration of high-dimensional data.IEEE Trans. Vis. Comput. Graph. (2010), 1\u201314. https:\/\/doi.org\/10.1109\/TVCG.2010.242"},{"key":"e_1_2_2_131_1","volume-title":"Proceedings of the 2008 12th International Conference Information Visualisation. 373\u2013380","author":"Tominski C.","year":"2008","unstructured":"C. Tominski , G. Fuchs , and H. Schumann . 2008. Task-driven color coding . In Proceedings of the 2008 12th International Conference Information Visualisation. 373\u2013380 . https:\/\/doi.org\/10.1109\/IV. 2008 .24 10.1109\/IV.2008.24 C. Tominski, G. Fuchs, and H. Schumann. 2008. Task-driven color coding. In Proceedings of the 2008 12th International Conference Information Visualisation. 373\u2013380. https:\/\/doi.org\/10.1109\/IV.2008.24"},{"key":"e_1_2_2_132_1","volume-title":"Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2 (Mar","author":"Tong Simon","year":"2002","unstructured":"Simon Tong and Daphne Koller . 2002. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2 (Mar . 2002 ), 45\u201366. https:\/\/doi.org\/10.1162\/153244302760185243 10.1162\/153244302760185243 Simon Tong and Daphne Koller. 2002. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2 (Mar. 2002), 45\u201366. https:\/\/doi.org\/10.1162\/153244302760185243"},{"doi-asserted-by":"publisher","key":"e_1_2_2_133_1","DOI":"10.1109\/JSTSP.2011.2139193"},{"doi-asserted-by":"publisher","key":"e_1_2_2_134_1","DOI":"10.1109\/VAST.2011.6102453"},{"volume-title":"Proceedings of the Text REtrieval Conference (TREC\u201902)","author":"Vendrig Jeroen","unstructured":"Jeroen Vendrig , Ioannis Patras , Cees Snoek , Marcel Worring , Jurgen den Hartog , Stephan Raaijmakers , Jeroen van Rest , and David A . van Leeuwen. 2002. TREC feature extraction by active learning . In Proceedings of the Text REtrieval Conference (TREC\u201902) . Jeroen Vendrig, Ioannis Patras, Cees Snoek, Marcel Worring, Jurgen den Hartog, Stephan Raaijmakers, Jeroen van Rest, and David A. van Leeuwen. 2002. TREC feature extraction by active learning. In Proceedings of the Text REtrieval Conference (TREC\u201902).","key":"e_1_2_2_135_1"},{"key":"e_1_2_2_136_1","first-page":"12","article-title":"Cost-effective active learning for deep image classification","volume":"27","author":"Wang K.","year":"2017","unstructured":"K. Wang , D. Zhang , Y. Li , R. Zhang , and L. Lin . 2017 . Cost-effective active learning for deep image classification . IEEE Trans. Circ. Syst. Vid. Technol. 27 , 12 (Dec. 2017), 2591\u20132600. https:\/\/doi.org\/10.1109\/TCSVT.2016.2589879 10.1109\/TCSVT.2016.2589879 K. Wang, D. Zhang, Y. Li, R. Zhang, and L. Lin. 2017. Cost-effective active learning for deep image classification. IEEE Trans. Circ. Syst. Vid. Technol. 27, 12 (Dec. 2017), 2591\u20132600. https:\/\/doi.org\/10.1109\/TCSVT.2016.2589879","journal-title":"IEEE Trans. Circ. Syst. Vid. Technol."},{"doi-asserted-by":"publisher","key":"e_1_2_2_137_1","DOI":"10.1145\/1899412.1899414"},{"doi-asserted-by":"publisher","key":"e_1_2_2_138_1","DOI":"10.1109\/TVCG.2019.2934796"},{"doi-asserted-by":"publisher","key":"e_1_2_2_139_1","DOI":"10.1080\/01621459.1963.10500845"},{"doi-asserted-by":"crossref","unstructured":"J. Wexler M. Pushkarna T. Bolukbasi M. Wattenberg F. Vi\u00e9gas and J. Wilson. 2019. The What-If Tool: Interactive probing of machine learning models. IEEE Trans. Vis. Comput. Graph. (2019) 1-1. https:\/\/doi.org\/10.1109\/TVCG.2019.2934619 10.1109\/TVCG.2019.2934619","key":"#cr-split#-e_1_2_2_140_1.1","DOI":"10.1109\/TVCG.2019.2934619"},{"doi-asserted-by":"crossref","unstructured":"J. Wexler M. Pushkarna T. Bolukbasi M. Wattenberg F. Vi\u00e9gas and J. Wilson. 2019. The What-If Tool: Interactive probing of machine learning models. IEEE Trans. Vis. Comput. Graph. (2019) 1-1. https:\/\/doi.org\/10.1109\/TVCG.2019.2934619","key":"#cr-split#-e_1_2_2_140_1.2","DOI":"10.1109\/TVCG.2019.2934619"},{"key":"e_1_2_2_141_1","volume-title":"Proceedings of the IEEE Symposium on Information Visualization (InfoVis\u201915)","author":"Wilkinson Leland","year":"2005","unstructured":"Leland Wilkinson , Anushka Anand , and Robert L. Grossman . 2005. Graph-theoretic scagnostics . In Proceedings of the IEEE Symposium on Information Visualization (InfoVis\u201915) . 157\u2013164. https:\/\/doi.org\/10.1109\/INFOVIS. 2005 .14 10.1109\/INFOVIS.2005.14 Leland Wilkinson, Anushka Anand, and Robert L. Grossman. 2005. Graph-theoretic scagnostics. In Proceedings of the IEEE Symposium on Information Visualization (InfoVis\u201915). 157\u2013164. https:\/\/doi.org\/10.1109\/INFOVIS.2005.14"},{"doi-asserted-by":"publisher","key":"e_1_2_2_142_1","DOI":"10.1109\/TVCG.2017.2744878"},{"doi-asserted-by":"publisher","key":"e_1_2_2_143_1","DOI":"10.1109\/ICME.2006.262442"},{"doi-asserted-by":"publisher","key":"e_1_2_2_144_1","DOI":"10.1109\/ITCS.2009.230"},{"volume-title":"Proceedings of the IEEE ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM\u201913)","author":"Zhang J.","unstructured":"J. Zhang , X. Wu , and V. S. Sheng . 2013. A threshold method for Imbalanced Multiple Noisy Labeling . In Proceedings of the IEEE ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM\u201913) . 61\u201365. https:\/\/doi.org\/10.1145\/2492517.2492640 10.1145\/2492517.2492640 J. Zhang, X. Wu, and V. S. Sheng. 2013. A threshold method for Imbalanced Multiple Noisy Labeling. In Proceedings of the IEEE ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM\u201913). 61\u201365. https:\/\/doi.org\/10.1145\/2492517.2492640","key":"e_1_2_2_145_1"},{"key":"e_1_2_2_146_1","first-page":"5","article-title":"Active learning with imbalanced multiple noisy labeling","volume":"45","author":"Zhang J.","year":"2015","unstructured":"J. Zhang , X. Wu , and V. S. Shengs . 2015 . Active learning with imbalanced multiple noisy labeling . IEEE Trans. Cybernet. 45 , 5 (May 2015), 1095\u20131107. https:\/\/doi.org\/10.1109\/TCYB.2014.2344674 10.1109\/TCYB.2014.2344674 J. Zhang, X. Wu, and V. S. Shengs. 2015. Active learning with imbalanced multiple noisy labeling. IEEE Trans. Cybernet. 45, 5 (May 2015), 1095\u20131107. https:\/\/doi.org\/10.1109\/TCYB.2014.2344674","journal-title":"IEEE Trans. Cybernet."},{"doi-asserted-by":"crossref","unstructured":"Quanshi Zhang Yu Yang Haotian Ma and Ying Nian Wu. 2019. Interpreting cnns via decision trees. In Computer Vision and Pattern Recognition. 6261\u20136270.  Quanshi Zhang Yu Yang Haotian Ma and Ying Nian Wu. 2019. Interpreting cnns via decision trees. In Computer Vision and Pattern Recognition. 6261\u20136270.","key":"e_1_2_2_147_1","DOI":"10.1109\/CVPR.2019.00642"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3439333","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3439333","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:08Z","timestamp":1750197788000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3439333"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,3]]},"references-count":148,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2021,12,31]]}},"alternative-id":["10.1145\/3439333"],"URL":"https:\/\/doi.org\/10.1145\/3439333","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"type":"print","value":"2160-6455"},{"type":"electronic","value":"2160-6463"}],"subject":[],"published":{"date-parts":[[2021,9,3]]},"assertion":[{"value":"2019-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}