{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T14:12:34Z","timestamp":1766067154557,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":20,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,4,20]],"date-time":"2020-04-20T00:00:00Z","timestamp":1587340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,4,20]]},"DOI":"10.1145\/3366423.3380306","type":"proceedings-article","created":{"date-parts":[[2020,5,4]],"date-time":"2020-05-04T08:11:44Z","timestamp":1588579904000},"page":"2432-2442","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["Towards Hybrid Human-AI Workflows for Unknown Unknown Detection"],"prefix":"10.1145","author":[{"given":"Anthony","family":"Liu","sequence":"first","affiliation":[{"name":"University of Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Santiago","family":"Guerra","sequence":"additional","affiliation":[{"name":"Universidad de Monterrey"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Isaac","family":"Fung","sequence":"additional","affiliation":[{"name":"University of Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gabriel","family":"Matute","sequence":"additional","affiliation":[{"name":"University of Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ece","family":"Kamar","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Walter","family":"Lasecki","sequence":"additional","affiliation":[{"name":"University of Michigan Computer Science and Engineering"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,4,20]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_3_2_1_1_1","DOI":"10.1145\/2034691.2034742"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_2_1","DOI":"10.1145\/2700832"},{"key":"e_1_3_2_1_3_1","volume-title":"Thirty-Second AAAI Conference on Artificial Intelligence.","author":"Bansal Gagan","year":"2018","unstructured":"Gagan Bansal and Daniel\u00a0 S Weld . 2018 . A coverage-based utility model for identifying unknown unknowns . In Thirty-Second AAAI Conference on Artificial Intelligence. Gagan Bansal and Daniel\u00a0S Weld. 2018. A coverage-based utility model for identifying unknown unknowns. In Thirty-Second AAAI Conference on Artificial Intelligence."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_4_1","DOI":"10.1609\/hcomp.v1i1.13091"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_5_1","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_3_2_1_6_1","volume-title":"Imagenet: A large-scale hierarchical image database.","author":"Deng Jia","year":"2009","unstructured":"Jia Deng , Wei Dong , Richard Socher , Li-Jia Li , Kai Li , and Li Fei-Fei . 2009 . Imagenet: A large-scale hierarchical image database. (2009). Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. (2009)."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_7_1","DOI":"10.1145\/2556288.2557011"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_8_1","DOI":"10.1016\/j.procs.2012.04.177"},{"key":"e_1_3_2_1_9_1","volume-title":"Stanford 1, 12","author":"Go Alec","year":"2009","unstructured":"Alec Go , Richa Bhayani , and Lei Huang . 2009. Twitter sentiment classification using distant supervision. CS224N Project Report , Stanford 1, 12 ( 2009 ). Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford 1, 12 (2009)."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_10_1","DOI":"10.1145\/2872427.2883037"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_11_1","DOI":"10.18653\/v1\/N18-2099"},{"unstructured":"Kaggle. 2013. Dogs vs cats dataset. dataset.https:\/\/www.kaggle.com\/c\/dogs-vs-cats\/data(2013).  Kaggle. 2013. Dogs vs cats dataset. dataset.https:\/\/www.kaggle.com\/c\/dogs-vs-cats\/data(2013).","key":"e_1_3_2_1_12_1"},{"unstructured":"B. Kim Wattenberg M. J. Gilmer Cai C. Wexler J. F. Viegas and R. Sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML (2018). arxiv:1711.11279  B. Kim Wattenberg M. J. Gilmer Cai C. Wexler J. F. Viegas and R. Sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). ICML (2018). arxiv:1711.11279","key":"e_1_3_2_1_13_1"},{"key":"e_1_3_2_1_14_1","volume-title":"Thirty-First AAAI Conference on Artificial Intelligence.","author":"Lakkaraju Himabindu","year":"2017","unstructured":"Himabindu Lakkaraju , Ece Kamar , Rich Caruana , and Eric Horvitz . 2017 . Identifying unknown unknowns in the open world: Representations and policies for guided exploration . In Thirty-First AAAI Conference on Artificial Intelligence. Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. 2017. Identifying unknown unknowns in the open world: Representations and policies for guided exploration. In Thirty-First AAAI Conference on Artificial Intelligence."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_15_1","DOI":"10.5555\/3091622.3091662"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_16_1","DOI":"10.1145\/2702123.2702416"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_17_1","DOI":"10.1145\/2783258.2783381"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_19_1","DOI":"10.3115\/1613715.1613751"},{"key":"e_1_3_2_1_20_1","volume-title":"International Conference on Learning Representations. http:\/\/arxiv.org\/abs\/1312","author":"Szegedy Christian","year":"2014","unstructured":"Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian Goodfellow , and Rob Fergus . 2014 . Intriguing properties of neural networks . In International Conference on Learning Representations. http:\/\/arxiv.org\/abs\/1312 .6199 Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations. http:\/\/arxiv.org\/abs\/1312.6199"},{"key":"e_1_3_2_1_21_1","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. 1220\u20131229","author":"Zaidan F","year":"2011","unstructured":"Omar\u00a0 F Zaidan and Chris Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. 1220\u20131229 . Omar\u00a0F Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. 1220\u20131229."}],"event":{"sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"],"acronym":"WWW '20","name":"WWW '20: The Web Conference 2020","location":"Taipei Taiwan"},"container-title":["Proceedings of The Web Conference 2020"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3366423.3380306","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3366423.3380306","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:33:11Z","timestamp":1750199591000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3366423.3380306"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,20]]},"references-count":20,"alternative-id":["10.1145\/3366423.3380306","10.1145\/3366423"],"URL":"https:\/\/doi.org\/10.1145\/3366423.3380306","relation":{},"subject":[],"published":{"date-parts":[[2020,4,20]]},"assertion":[{"value":"2020-04-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}