{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,30]],"date-time":"2026-05-30T00:22:25Z","timestamp":1780100545319,"version":"3.54.0"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2019,3,1]],"date-time":"2019-03-01T00:00:00Z","timestamp":1551398400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"DOD contract","award":["H98230-13-D-0056"],"award-info":[{"award-number":["H98230-13-D-0056"]}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["14-29404 and 15-64212"],"award-info":[{"award-number":["14-29404 and 15-64212"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"State of Maryland's MPower initiative"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2019,3,31]]},"abstract":"<jats:p>Deep learning has emerged as a powerful tool for feature-driven labeling of datasets. However, for it to be effective, it requires a large and finely labeled training dataset. Precisely labeling a large training dataset is expensive, time-consuming, and error prone. In this article, we present a visually driven deep-learning approach that starts with a coarsely labeled training dataset and iteratively refines the labeling through intuitive interactions that leverage the latent structures of the dataset. Our approach can be used to (a) alleviate the burden of intensive manual labeling that captures the fine nuances in a high-dimensional dataset by simple visual interactions, (b) replace a complicated (and therefore difficult to design) labeling algorithm by a simpler (but coarse) labeling algorithm supplemented by user interaction to refine the labeling, or (c) use low-dimensional features (such as the RGB colors) for coarse labeling and turn to higher-dimensional latent structures that are progressively revealed by deep learning, for fine labeling. We validate our approach through use cases on three high-dimensional datasets and a user study.<\/jats:p>","DOI":"10.1145\/3150977","type":"journal-article","created":{"date-parts":[[2019,3,4]],"date-time":"2019-03-04T13:31:47Z","timestamp":1551706307000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Enhancing Deep Learning with Visual Interactions"],"prefix":"10.1145","volume":"9","author":[{"given":"Eric","family":"Krokos","sequence":"first","affiliation":[{"name":"University of Maryland College Park, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hsueh-Chen","family":"Cheng","sequence":"additional","affiliation":[{"name":"University of Maryland College Park, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jessica","family":"Chang","sequence":"additional","affiliation":[{"name":"Department of Defense, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bohdan","family":"Nebesh","sequence":"additional","affiliation":[{"name":"Department of Defense, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Celeste Lyn","family":"Paul","sequence":"additional","affiliation":[{"name":"Department of Defense, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kirsten","family":"Whitley","sequence":"additional","affiliation":[{"name":"Department of Defense, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amitabh","family":"Varshney","sequence":"additional","affiliation":[{"name":"University of Maryland College Park, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,3]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916)","volume":"16","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , 2016 . TensorFlow: A system for large-scale machine learning . In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916) , Vol. 16 . 265--283. Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916), Vol. 16. 265--283."},{"key":"e_1_2_1_2_1","unstructured":"Shun-ichi Amari Andrzej Cichocki and Howard Hua Yang. 1996. A new learning algorithm for blind signal separation. In Advances in Neural Information Processing Systems. 757--763.   Shun-ichi Amari Andrzej Cichocki and Howard Hua Yang. 1996. A new learning algorithm for blind signal separation. In Advances in Neural Information Processing Systems. 757--763."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1978966"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.2594"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1080\/03081088508817681"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1162\/089976603321780317"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTARS.2014.2329330"},{"key":"e_1_2_1_8_1","article-title":"Deep-learning-assisted volume visualization","volume":"25","author":"Cheng Hsueh-Chien","year":"2018","unstructured":"Hsueh-Chien Cheng , Antonio Cardone , Somay Jain , Eric Krokos , Kedar Narayan , Sriram Subramaniam , and Amitabh Varshney . 2018 . Deep-learning-assisted volume visualization . IEEE Trans. Vis. Comput. Graph. 25 , 2 (22 Jan. 2018), 1--14. Hsueh-Chien Cheng, Antonio Cardone, Somay Jain, Eric Krokos, Kedar Narayan, Sriram Subramaniam, and Amitabh Varshney. 2018. Deep-learning-assisted volume visualization. IEEE Trans. Vis. Comput. Graph. 25, 2 (22 Jan. 2018), 1--14.","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2017.8296507"},{"key":"e_1_2_1_10_1","volume-title":"Keras: Deep learning library for theano and tensorflow.","author":"Chollet Fran\u00e7ois","year":"2015","unstructured":"Fran\u00e7ois Chollet , 2015 . Keras: Deep learning library for theano and tensorflow. Retrieved from https:\/\/keras.io\/k. Fran\u00e7ois Chollet, et al. 2015. Keras: Deep learning library for theano and tensorflow. Retrieved from https:\/\/keras.io\/k."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346572"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS\u201903)","volume":"1","author":"Dell\u2019Acqua Fabio","year":"2003","unstructured":"Fabio Dell\u2019Acqua , Paolo Gamba , and Alessio Ferrari . 2003 . Exploiting spectral and spatial information for classifying hyperspectral data in urban areas . In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS\u201903) , Vol. 1 . IEEE, 464--466. Fabio Dell\u2019Acqua, Paolo Gamba, and Alessio Ferrari. 2003. Exploiting spectral and spatial information for classifying hyperspectral data in urban areas. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS\u201903), Vol. 1. IEEE, 464--466."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207741"},{"key":"e_1_2_1_14_1","volume-title":"Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11 (Feb","author":"Erhan Dumitru","year":"2010","unstructured":"Dumitru Erhan , Yoshua Bengio , Aaron Courville , Pierre-Antoine Manzagol , Pascal Vincent , and Samy Bengio . 2010. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11 (Feb . 2010 ), 625--660. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11 (Feb. 2010), 625--660."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24775-3_5"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2006.100"},{"key":"e_1_2_1_17_1","volume-title":"Salakhutdinov","author":"Hinton Geoffrey E.","year":"2006","unstructured":"Geoffrey E. Hinton and Ruslan R . Salakhutdinov . 2006 . Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507. Geoffrey E. Hinton and Ruslan R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504--507."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723731"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.231"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/645326.649721"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2013.2264508"},{"key":"e_1_2_1_22_1","series-title":"Lecture Notes in Computer Science","volume-title":"On spectral graph drawing","author":"Koren Yehuda","unstructured":"Yehuda Koren . 2003. On spectral graph drawing . In Computing and Combinatorics, Tandy Warnow and Binhai Zhu (Eds.). Lecture Notes in Computer Science , Vol. 2697 . Springer , Berlin , 496--508. Yehuda Koren. 2003. On spectral graph drawing. In Computing and Combinatorics, Tandy Warnow and Binhai Zhu (Eds.). Lecture Notes in Computer Science, Vol. 2697. Springer, Berlin, 496--508."},{"key":"e_1_2_1_23_1","volume-title":"Virtual memory palaces: Immersion aids recall. Virtual Reality","author":"Krokos Eric","year":"2018","unstructured":"Eric Krokos , Catherine Plaisant , and Amitabh Varshney . 2018. Virtual memory palaces: Immersion aids recall. Virtual Reality ( 2018 ), 1--15. https:\/\/link.springer.com\/article\/10.1007\/s10055-018-0346-3 Eric Krokos, Catherine Plaisant, and Amitabh Varshney. 2018. Virtual memory palaces: Immersion aids recall. Virtual Reality (2018), 1--15. https:\/\/link.springer.com\/article\/10.1007\/s10055-018-0346-3"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2755492.3036685"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02289565"},{"key":"e_1_2_1_26_1","volume-title":"Signal Theory Methods in Multispectral Remote Sensing","author":"Landgrebe David A.","unstructured":"David A. Landgrebe . 2005. Signal Theory Methods in Multispectral Remote Sensing . Vol. 29 . John Wiley 8 Sons. David A. Landgrebe. 2005. Signal Theory Methods in Multispectral Remote Sensing. Vol. 29. John Wiley 8 Sons."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the Workshop on Challenges in Representation Learning (ICML\u201913)","volume":"3","author":"Lee Dong-Hyun","year":"2013","unstructured":"Dong-Hyun Lee . 2013 . Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks . In Proceedings of the Workshop on Challenges in Representation Learning (ICML\u201913) , Vol. 3 . 2. Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Proceedings of the Workshop on Challenges in Representation Learning (ICML\u201913), Vol. 3. 2."},{"key":"e_1_2_1_28_1","volume-title":"Computer Graphics Forum","author":"Liu Shusen","unstructured":"Shusen Liu , Bei Wang , P.-T. Bremer , and Valerio Pascucci . 2014. Distortion-guided structure-driven interactive exploration of high-dimensional data . In Computer Graphics Forum , Vol. 33 . Wiley Online Library , 101--110. Shusen Liu, Bei Wang, P.-T. Bremer, and Valerio Pascucci. 2014. Distortion-guided structure-driven interactive exploration of high-dimensional data. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 101--110."},{"key":"e_1_2_1_29_1","volume-title":"Computer Graphics Forum","author":"Liu Shusen","unstructured":"Shusen Liu , Bei Wang , Jayaraman J Thiagarajan , P.-T. Bremer , and Valerio Pascucci . 2015. Visual exploration of high-dimensional data through subspace analysis and dynamic projections . In Computer Graphics Forum , Vol. 34 . Wiley Online Library , 271--280. Shusen Liu, Bei Wang, Jayaraman J Thiagarajan, P.-T. Bremer, and Valerio Pascucci. 2015. Visual exploration of high-dimensional data through subspace analysis and dynamic projections. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 271--280."},{"key":"e_1_2_1_30_1","volume-title":"Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (Nov","author":"van der Maaten Laurens","year":"2008","unstructured":"Laurens van der Maaten and Geoffrey Hinton . 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (Nov . 2008 ), 2579--2605. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (Nov. 2008), 2579--2605."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 3rd Annual BRAIN Initiative Investigators Meeting.","author":"Matheny Jason","year":"2016","unstructured":"Jason Matheny . 2016 . Intelligence Advanced Research Projects Activity. (2016) . In Proceedings of the 3rd Annual BRAIN Initiative Investigators Meeting. Jason Matheny. 2016. Intelligence Advanced Research Projects Activity. (2016). In Proceedings of the 3rd Annual BRAIN Initiative Investigators Meeting."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 27th International Conference on Machine Learning (ICML\u201910)","author":"Nair Vinod","unstructured":"Vinod Nair and Geoffrey E. Hinton . 2010. Rectified linear units improve restricted boltzmann machines . In Proceedings of the 27th International Conference on Machine Learning (ICML\u201910) . 807--814. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML\u201910). 807--814."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the International Conference on Machine Learning. 2368--2376","author":"Pezeshki Mohammad","year":"2016","unstructured":"Mohammad Pezeshki , Linxi Fan , Philemon Brakel , Aaron Courville , and Yoshua Bengio . 2016 . Deconstructing the ladder network architecture . In Proceedings of the International Conference on Machine Learning. 2368--2376 . Mohammad Pezeshki, Linxi Fan, Philemon Brakel, Aaron Courville, and Yoshua Bengio. 2016. Deconstructing the ladder network architecture. In Proceedings of the International Conference on Machine Learning. 2368--2376."},{"key":"e_1_2_1_34_1","unstructured":"Antti Rasmus Mathias Berglund Mikko Honkala Harri Valpola and Tapani Raiko. 2015. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems. 3546--3554.   Antti Rasmus Mathias Berglund Mikko Honkala Harri Valpola and Tapani Raiko. 2015. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems. 3546--3554."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.290.5500.2323"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598495"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.868688"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2812115"},{"key":"e_1_2_1_39_1","first-page":"1345","article-title":"Modeling human motion using binary latent variables","volume":"20","author":"Taylor Graham W.","year":"2007","unstructured":"Graham W. Taylor , Geoffrey E. Hinton , and Sam T. Roweis . 2007 . Modeling human motion using binary latent variables . Adv. Neur. Inf. Process. Syst. 20 (2007), 1345 -- 1357 . Graham W. Taylor, Geoffrey E. Hinton, and Sam T. Roweis. 2007. Modeling human motion using binary latent variables. Adv. Neur. Inf. Process. Syst. 20 (2007), 1345--1357.","journal-title":"Adv. Neur. Inf. Process. Syst."},{"key":"e_1_2_1_40_1","volume-title":"Vlahavas","author":"Trohidis Konstantinos","year":"2008","unstructured":"Konstantinos Trohidis , Grigorios Tsoumakas , George Kalliris , and Ioannis P . Vlahavas . 2008 . Multi-label classification of music into emotions. In The International Society of Music Information Retrieval, Vol. 5 . Springer , 325--330. Konstantinos Trohidis, Grigorios Tsoumakas, George Kalliris, and Ioannis P. Vlahavas. 2008. Multi-label classification of music into emotions. In The International Society of Music Information Retrieval, Vol. 5. Springer, 325--330."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598470"},{"key":"e_1_2_1_42_1","first-page":"737","article-title":"Parametric mixture models for multi-labeled text","volume":"15","author":"Ueda Naonori","year":"2003","unstructured":"Naonori Ueda and Kazumi Saito . 2003 . Parametric mixture models for multi-labeled text . Advances in Neural Information Processing Systems 15 (2003), 737 -- 744 . Naonori Ueda and Kazumi Saito. 2003. Parametric mixture models for multi-labeled text. Advances in Neural Information Processing Systems 15 (2003), 737--744.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.2001.0499"},{"key":"e_1_2_1_44_1","volume-title":"How to use t-SNE effectively. Distill 5, 1","author":"Wattenberg Martin","year":"2016","unstructured":"Martin Wattenberg , Fernanda Vi\u00e9gas , and Ian Johnson . 2016. How to use t-SNE effectively. Distill 5, 1 ( 2016 ). Martin Wattenberg, Fernanda Vi\u00e9gas, and Ian Johnson. 2016. How to use t-SNE effectively. Distill 5, 1 (2016)."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642681"},{"key":"e_1_2_1_46_1","volume-title":"ADADELTA: An adaptive learning rate method. CoRR abs\/1212.5701.","author":"Zeiler Matthew D.","year":"2012","unstructured":"Matthew D. Zeiler . 2012 . ADADELTA: An adaptive learning rate method. CoRR abs\/1212.5701. Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. CoRR abs\/1212.5701."}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3150977","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3150977","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3150977","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:13:39Z","timestamp":1750212819000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3150977"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3]]},"references-count":46,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,3,31]]}},"alternative-id":["10.1145\/3150977"],"URL":"https:\/\/doi.org\/10.1145\/3150977","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,3]]},"assertion":[{"value":"2017-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}