{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T00:05:26Z","timestamp":1783123526174,"version":"3.54.6"},"reference-count":70,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2024,4,23]],"date-time":"2024-04-23T00:00:00Z","timestamp":1713830400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Foundation for Science and Technology (FCT): INTERWHALE - Advancing Interactive Technology for Responsible Whale-Watching","award":["PTDC\/CCI-COM\/0450\/2020"],"award-info":[{"award-number":["PTDC\/CCI-COM\/0450\/2020"]}]},{"name":"Foundation for Science and Technology (FCT): MARE - The Marine and Environmental Sciences Centre","award":["UIDB\/04292\/2020"],"award-info":[{"award-number":["UIDB\/04292\/2020"]}]},{"name":"Foundation for Science and Technology (FCT): ARNET - Aquatic Research Network","award":["LA\/P\/0069\/2020"],"award-info":[{"award-number":["LA\/P\/0069\/2020"]}]},{"name":"Foundation for Science and Technology (FCT): PhD scholarship","award":["2022.09961.BD"],"award-info":[{"award-number":["2022.09961.BD"]}]},{"name":"EU Horizon Europe project CLIMAREST: Coastal Climate Resilience and Marine Restoration Tools for the Arctic Atlantic basin","award":["101093865"],"award-info":[{"award-number":["101093865"]}]},{"DOI":"10.13039\/501100002341","name":"Academy of Finland","doi-asserted-by":"crossref","award":["339614"],"award-info":[{"award-number":["339614"]}],"id":[{"id":"10.13039\/501100002341","id-type":"DOI","asserted-by":"crossref"}]},{"name":"European Social Fund via \u201cICT programme\u201d measure, Estonian Center of Excellence in ICT Research","award":["TK148 EXCITE"],"award-info":[{"award-number":["TK148 EXCITE"]}]},{"DOI":"10.13039\/501100004181","name":"Nokia Foundation","doi-asserted-by":"crossref","award":["20220138"],"award-info":[{"award-number":["20220138"]}],"id":[{"id":"10.13039\/501100004181","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>\n            AI-assisted interactive annotation is a powerful way to facilitate data annotation\u2014a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators operate under time and cognitive constraints, e.g., while detecting suspicious or dangerous activities from real-time surveillance feeds. Understanding how AI can assist annotators in these tasks and facilitate consistent annotation is paramount to ensure high performance for AI models trained on these data. We address this gap in interactive machine learning (IML) research, contributing an extensive investigation of the benefits, limitations, and challenges of AI-assisted annotation in dynamic application use cases. We address both the effects of AI on annotators and the effects of (AI) annotations on the performance of AI models trained on annotated data in real-time video annotations. We conduct extensive experiments that compare annotation performance at two annotator levels (expert and non-expert) and two interactive labeling techniques (with and without AI assistance). In a controlled study with\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(N=34\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            annotators and a follow-up study with 51,963 images and their annotation labels being input to the AI model, we demonstrate that the benefits of AI-assisted models are greatest for non-expert users and for cases where targets are only partially or briefly visible. The expert users tend to outperform or achieve similar performance as the AI model. Labels combining AI and expert annotations result in the best overall performance as the AI reduces overflow and latency in the expert annotations. We derive guidelines for the use of AI-assisted human annotation in real-time dynamic use cases.\n          <\/jats:p>","DOI":"10.1145\/3649457","type":"journal-article","created":{"date-parts":[[2024,2,29]],"date-time":"2024-02-29T12:26:35Z","timestamp":1709209595000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-time Video Streams"],"prefix":"10.1145","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4842-3622","authenticated-orcid":false,"given":"Marko","family":"Radeta","sequence":"first","affiliation":[{"name":"Wave Labs, MARE\/ARNET\/ARDITI, University of Madeira, Funchal, Portugal, University of Belgrade, Funchal, Belgrade, Serbia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0378-108X","authenticated-orcid":false,"given":"Ruben","family":"Freitas","sequence":"additional","affiliation":[{"name":"Wave Labs, MARE\/ARNET\/ARDITI, University of Madeira, Funchal, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5510-8527","authenticated-orcid":false,"given":"Claudio","family":"Rodrigues","sequence":"additional","affiliation":[{"name":"Wave Labs, MARE\/ARNET\/ARDITI, University of Madeira, Funchal, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6481-3559","authenticated-orcid":false,"given":"Agustin","family":"Zuniga","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Helsinki, Helsinki, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1011-8937","authenticated-orcid":false,"given":"Ngoc Thi","family":"Nguyen","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Helsinki, Helsinki, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4551-629X","authenticated-orcid":false,"given":"Huber","family":"Flores","sequence":"additional","affiliation":[{"name":"Institute of Computer Science, University of Tartu, Tartu, Estonia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8262-6434","authenticated-orcid":false,"given":"Petteri","family":"Nurmi","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Helsinki, Helsinki, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,4,23]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.3390\/S22114107"},{"issue":"1","key":"e_1_3_3_3_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3449163","article-title":"AI-assisted human labeling: Batching for efficiency without overreliance","volume":"5","author":"Ashktorab Zahra","year":"2021","unstructured":"Zahra Ashktorab, Michael Desmond, Josh Andres, Michael Muller, Narendra Nath Joshi, Michelle Brachman, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Christine T. Wolf, et\u00a0al. 2021. AI-assisted human labeling: Batching for efficiency without overreliance. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1\u201327.","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"issue":"1","key":"e_1_3_3_4_2","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1109\/TVCG.2017.2744818","article-title":"Comparing visual-interactive labeling with active learning: An experimental study","volume":"24","author":"Bernard J\u00fcrgen","year":"2017","unstructured":"J\u00fcrgen Bernard, Marco Hutter, Matthias Zeppelzauer, Dieter Fellner, and Michael Sedlmair. 2017. Comparing visual-interactive labeling with active learning: An experimental study. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2017), 298\u2013308.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"issue":"9","key":"e_1_3_3_5_2","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1007\/s00371-018-1500-3","article-title":"VIAL: A unified process for visual interactive labeling","volume":"34","author":"Bernard J\u00fcrgen","year":"2018","unstructured":"J\u00fcrgen Bernard, Matthias Zeppelzauer, Michael Sedlmair, and Wolfgang Aigner. 2018. VIAL: A unified process for visual interactive labeling. The Visual Computer 34, 9 (2018), 1189\u20131207.","journal-title":"The Visual Computer"},{"key":"e_1_3_3_6_2","doi-asserted-by":"crossref","first-page":"2167","DOI":"10.1007\/s00345-019-02711-z","article-title":"Systematic review of augmented reality in urological interventions: The evidences of an impact on surgical outcomes are yet to come","volume":"38","author":"Bertolo Riccardo","year":"2020","unstructured":"Riccardo Bertolo, Andrew Hung, Francesco Porpiglia, Pierluigi Bove, Mary Schleicher, and Prokar Dasgupta. 2020. Systematic review of augmented reality in urological interventions: The evidences of an impact on surgical outcomes are yet to come. World Journal of Urology 38 (2020), 2167\u20132176.","journal-title":"World Journal of Urology"},{"key":"e_1_3_3_7_2","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1109\/CRV.2014.35","volume-title":"2014 Canadian Conference on Computer and Robot Vision","author":"Beugeling Trevor","year":"2014","unstructured":"Trevor Beugeling and Alexandra Branzan-Albu. 2014. Computer vision-based identification of individual turtles using characteristic patterns of their plastrons. In 2014 Canadian Conference on Computer and Robot Vision. IEEE, 203\u2013210."},{"key":"e_1_3_3_8_2","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1613\/jair.606","article-title":"Identifying mislabeled training data","volume":"11","author":"Brodley Carla E.","year":"1999","unstructured":"Carla E. Brodley and Mark A. Friedl. 1999. Identifying mislabeled training data. Journal of Artificial Intelligence Research 11 (1999), 131\u2013167.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"e_1_3_3_9_2","volume-title":"Update on Abundance, Trends, and Migrations of Humpback Whales along the US West Coast","author":"Calambokidis John","year":"2017","unstructured":"John Calambokidis, Jay Barlow, Kiirsten Flynn, Elana Dobson, and Gretchen H. Steiger. 2017. Update on Abundance, Trends, and Migrations of Humpback Whales along the US West Coast. Technical Report SC\/A17\/NP\/13. International Whaling Commision."},{"key":"e_1_3_3_10_2","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.jembe.2013.12.010","article-title":"Automated marine turtle photograph identification using artificial neural networks, with application to green turtles","volume":"452","author":"Carter Steven J. B.","year":"2014","unstructured":"Steven J. B. Carter, Ian P. Bell, Jessica J. Miller, and Peter P. Gash. 2014. Automated marine turtle photograph identification using artificial neural networks, with application to green turtles. Journal of Experimental Marine Biology and Ecology 452 (2014), 105\u2013110.","journal-title":"Journal of Experimental Marine Biology and Ecology"},{"key":"e_1_3_3_11_2","doi-asserted-by":"crossref","first-page":"103697","DOI":"10.1016\/j.compedu.2019.103697","article-title":"A web-based collaborative reading annotation system with gamification mechanisms to improve reading performance","volume":"144","author":"Chen Chih-Ming","year":"2020","unstructured":"Chih-Ming Chen, Ming-Chaun Li, and Tze-Chun Chen. 2020. A web-based collaborative reading annotation system with gamification mechanisms to improve reading performance. Computers & Education 144 (2020), 103697.","journal-title":"Computers & Education"},{"key":"e_1_3_3_12_2","first-page":"1","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","author":"Choi Minsuk","year":"2019","unstructured":"Minsuk Choi, Cheonbok Park, Soyoung Yang, Yonggyu Kim, Jaegul Choo, and Sungsoo Ray Hong. 2019. AILA: Attentive interactive labeling assistant for document classification through attention-based deep neural networks. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1\u201312."},{"key":"e_1_3_3_13_2","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1145\/3397481.3450698","volume-title":"26th International Conference on Intelligent User Interfaces","author":"Desmond Michael","year":"2021","unstructured":"Michael Desmond, Michael Muller, Zahra Ashktorab, Casey Dugan, Evelyn Duesterwald, Kristina Brimijoin, Catherine Finegan-Dollak, Michelle Brachman, Aabhas Sharma, Narendra Nath Joshi, et\u00a0al. 2021. Increasing the speed and accuracy of data labeling through an AI assisted interface. In 26th International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 392\u2013401."},{"key":"e_1_3_3_14_2","first-page":"1","volume-title":"2017 26th International Conference on Computer Communication and Networks (ICCCN \u201917)","author":"Dodge Samuel","year":"2017","unstructured":"Samuel Dodge and Lina Karam. 2017. A study and comparison of human and deep learning recognition performance under visual distortions. In 2017 26th International Conference on Computer Communication and Networks (ICCCN \u201917). IEEE, 1\u20137."},{"key":"e_1_3_3_15_2","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1145\/3242587.3242596","volume-title":"Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology","author":"Felix Cristian","year":"2018","unstructured":"Cristian Felix, Aritra Dasgupta, and Enrico Bertini. 2018. The exploratory labeling assistant: Mixed-initiative label curation with large document collections. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA, 153\u2013164."},{"issue":"9","key":"e_1_3_3_16_2","doi-asserted-by":"crossref","first-page":"4224","DOI":"10.1109\/TII.2018.2822828","article-title":"Object classification using CNN-based fusion of vision and LIDAR in autonomous vehicle environment","volume":"14","author":"Gao Hongbo","year":"2018","unstructured":"Hongbo Gao, Bo Cheng, Jianqiang Wang, Keqiang Li, Jianhui Zhao, and Deyi Li. 2018. Object classification using CNN-based fusion of vision and LIDAR in autonomous vehicle environment. IEEE Transactions on Industrial Informatics 14, 9 (2018), 4224\u20134231.","journal-title":"IEEE Transactions on Industrial Informatics"},{"key":"e_1_3_3_17_2","first-page":"510","volume-title":"European Conference on Computer Vision","author":"Gao Mingfei","year":"2020","unstructured":"Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan \u00d6. Ar\u0131k, Larry S. Davis, and Tomas Pfister. 2020. Consistency-based semi-supervised active learning: Towards minimizing labeling cost. In European Conference on Computer Vision. Springer,510\u2013526."},{"key":"e_1_3_3_18_2","article-title":"Comparing deep neural networks against humans: Object recognition when the signal gets weaker","volume":"1706","author":"Geirhos Robert","year":"2017","unstructured":"Robert Geirhos, David H. J. Janssen, Heiko H. Sch\u00fctt, Jonas Rauber, Matthias Bethge, and Felix A. Wichmann. 2017. Comparing deep neural networks against humans: Object recognition when the signal gets weaker. CoRR abs\/1706.06969 (2017). arXiv:1706.06969http:\/\/arxiv.org\/abs\/1706.06969","journal-title":"CoRR"},{"key":"e_1_3_3_19_2","first-page":"535","volume-title":"Proceedings of the 15th ACM Conference on Computer and Communications Security (CCS \u201908)","author":"Golle Philippe","year":"2008","unstructured":"Philippe Golle. 2008. Machine learning attacks against the Asirra CAPTCHA. In Proceedings of the 15th ACM Conference on Computer and Communications Security (CCS \u201908). Association for Computing Machinery, New York, NY, USA, 535\u2013542. DOI:10.1145\/1455770.1455838"},{"key":"e_1_3_3_20_2","volume-title":"Insights into Jellyfish Distribution and Abundance Provided by a Platform of Opportunity","author":"Haberlin Michael Damien","year":"2010","unstructured":"Michael Damien Haberlin. 2010. Insights into Jellyfish Distribution and Abundance Provided by a Platform of Opportunity. Ph. D. Dissertation. NUI."},{"key":"e_1_3_3_21_2","first-page":"2014","volume-title":"Conference on Robot Learning","author":"III Donald Joseph Hejna","year":"2023","unstructured":"Donald Joseph Hejna III and Dorsa Sadigh. 2023. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning. PMLR, 2014\u20132025."},{"issue":"2","key":"e_1_3_3_22_2","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","article-title":"Interactive machine learning for health informatics: When do we need the human-in-the-loop?","volume":"3","author":"Holzinger Andreas","year":"2016","unstructured":"Andreas Holzinger. 2016. Interactive machine learning for health informatics: When do we need the human-in-the-loop? Brain Informatics 3, 2 (2016), 119\u2013131.","journal-title":"Brain Informatics"},{"key":"e_1_3_3_23_2","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1145\/302979.303030","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Horvitz Eric","year":"1999","unstructured":"Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 159\u2013166."},{"issue":"5","key":"e_1_3_3_24_2","first-page":"213","article-title":"A comparative study of underwater marine products detection based on YOLOv5 and underwater image enhancement","volume":"7","author":"Huang Guanxi","year":"2021","unstructured":"Guanxi Huang. 2021. A comparative study of underwater marine products detection based on YOLOv5 and underwater image enhancement. International Core Journal of Engineering 7, 5 (2021), 213\u2013221.","journal-title":"International Core Journal of Engineering"},{"issue":"3","key":"e_1_3_3_25_2","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1016\/j.pec.2014.11.007","article-title":"Self-evaluation and peer-feedback of medical students\u2019 communication skills using a web-based video annotation system. Exploring content and specificity","volume":"98","author":"Hulsman Robert L.","year":"2015","unstructured":"Robert L. Hulsman and Jane van der Vloodt. 2015. Self-evaluation and peer-feedback of medical students\u2019 communication skills using a web-based video annotation system. Exploring content and specificity. Patient Education and Counseling 98, 3 (2015), 356\u2013363.","journal-title":"Patient Education and Counseling"},{"issue":"4","key":"e_1_3_3_26_2","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1016\/j.compedu.2005.04.020","article-title":"A study of multimedia annotation of Web-based materials","volume":"48","author":"Hwang Wu-Yuin","year":"2007","unstructured":"Wu-Yuin Hwang, Chin-Yu Wang, and Mike Sharples. 2007. A study of multimedia annotation of Web-based materials. Computers & Education 48, 4 (2007), 680\u2013699.","journal-title":"Computers & Education"},{"key":"e_1_3_3_27_2","first-page":"268","volume-title":"2018 4th International Conference on Electrical Engineering and Information & Communication Technology (iCEEiCT \u201918)","author":"Islam Md Milon","year":"2018","unstructured":"Md Milon Islam and Muhammad Sheikh Sadi. 2018. Path hole detection to assist the visually impaired people in navigation. In 2018 4th International Conference on Electrical Engineering and Information & Communication Technology (iCEEiCT \u201918). IEEE, 268\u2013273."},{"issue":"1","key":"e_1_3_3_28_2","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1177\/0963721418801342","article-title":"Comparing the visual representations and performance of humans and deep neural networks","volume":"28","author":"Jacobs Robert A.","year":"2019","unstructured":"Robert A. Jacobs and Christopher J. Bates. 2019. Comparing the visual representations and performance of humans and deep neural networks. Current Directions in Psychological Science 28, 1 (2019), 34\u201339.","journal-title":"Current Directions in Psychological Science"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1101\/2020.08.03.234625"},{"issue":"9","key":"e_1_3_3_30_2","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1038\/s42256-019-0088-2","article-title":"The global landscape of AI ethics guidelines","volume":"1","author":"Jobin Anna","year":"2019","unstructured":"Anna Jobin, Marcello Ienca, and Effy Vayena. 2019. The global landscape of AI ethics guidelines. Nature Machine Intelligence 1, 9 (2019), 389\u2013399.","journal-title":"Nature Machine Intelligence"},{"key":"e_1_3_3_31_2","article-title":"Testing robustness against unforeseen adversaries","volume":"1908","author":"Kang Daniel","year":"2019","unstructured":"Daniel Kang, Yi Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardt. 2019. Testing robustness against unforeseen adversaries. CoRR abs\/1908.08016 (2019). arXiv:1908.08016http:\/\/arxiv.org\/abs\/1908.08016","journal-title":"CoRR"},{"key":"e_1_3_3_32_2","doi-asserted-by":"crossref","first-page":"3075","DOI":"10.1145\/2556288.2557238","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Kulesza Todd","year":"2014","unstructured":"Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, and Denis Charles. 2014. Structured labeling for facilitating concept evolution in machine learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 3075\u20133084."},{"key":"e_1_3_3_33_2","unstructured":"Alina Kuznetsova Hassan Rom Neil Alldrin Jasper Uijlings Ivan Krasin Jordi Pont-Tuset Shahab Kamali Stefan Popov Matteo Malloci Tom Duerig and Vittorio Ferrari. 2018. The Open Images Dataset V4: Unified Image Classification Object Detection and Visual Relationship Detection at Scale. arxiv:1811.00982 [cs.CV]"},{"issue":"11","key":"e_1_3_3_34_2","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun Yann","year":"1998","unstructured":"Yann LeCun, L\u00e9on Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11 (1998), 2278\u20132324.","journal-title":"Proceedings of the IEEE"},{"issue":"2","key":"e_1_3_3_35_2","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1109\/MCE.2016.2640698","article-title":"Deep learning for consumer devices and services: Pushing the limits for machine learning, artificial intelligence, and computer vision.","volume":"6","author":"Lemley Joe","year":"2017","unstructured":"Joe Lemley, Shabab Bazrafkan, and Peter Corcoran. 2017. Deep learning for consumer devices and services: Pushing the limits for machine learning, artificial intelligence, and computer vision. IEEE Consumer Electronics Magazine 6, 2 (2017), 48\u201356.","journal-title":"IEEE Consumer Electronics Magazine"},{"key":"e_1_3_3_36_2","first-page":"531","volume-title":"Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology","author":"Lin Shih-Chieh","year":"2018","unstructured":"Shih-Chieh Lin, Chang-Hong Hsu, Walter Talamonti, Yunqi Zhang, Steve Oney, Jason Mars, and Lingjia Tang. 2018. Adasa: A conversational in-vehicle digital assistant for advanced driver assistance features. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA, 531\u2013542."},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.oceaneng.2022.110555"},{"issue":"1","key":"e_1_3_3_38_2","first-page":"1","article-title":"DolFin: An innovative digital platform for studying Risso\u2019s dolphins in the Northern Ionian Sea (North-eastern Central Mediterranean)","volume":"8","author":"Maglietta Rosalia","year":"2018","unstructured":"Rosalia Maglietta, Vito Ren\u00f2, Giulia Cipriano, Carmelo Fanizza, Annalisa Milella, Ettore Stella, and Roberto Carlucci. 2018. DolFin: An innovative digital platform for studying Risso\u2019s dolphins in the Northern Ionian Sea (North-eastern Central Mediterranean). Scientific Reports 8, 1 (2018), 1\u201311.","journal-title":"Scientific Reports"},{"key":"e_1_3_3_39_2","doi-asserted-by":"crossref","first-page":"105132","DOI":"10.1016\/j.resconrec.2020.105132","article-title":"Recycling waste classification using optimized convolutional neural network","volume":"164","author":"Mao Wei-Lung","year":"2021","unstructured":"Wei-Lung Mao, Wei-Chun Chen, Chien-Tsung Wang, and Yu-Hao Lin. 2021. Recycling waste classification using optimized convolutional neural network. Resources, Conservation and Recycling 164 (2021), 105132.","journal-title":"Resources, Conservation and Recycling"},{"issue":"6","key":"e_1_3_3_40_2","doi-asserted-by":"crossref","first-page":"1236","DOI":"10.1093\/bib\/bbx044","article-title":"Deep learning for healthcare: Review, opportunities and challenges","volume":"19","author":"Miotto Riccardo","year":"2018","unstructured":"Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T. Dudley. 2018. Deep learning for healthcare: Review, opportunities and challenges. Briefings in Bioinformatics 19, 6 (2018), 1236\u20131246.","journal-title":"Briefings in Bioinformatics"},{"issue":"4","key":"e_1_3_3_41_2","doi-asserted-by":"crossref","first-page":"3005","DOI":"10.1007\/s10462-022-10246-w","article-title":"Human-in-the-loop machine learning: A state of the art","volume":"56","author":"Mosqueira-Rey Eduardo","year":"2023","unstructured":"Eduardo Mosqueira-Rey, Elena Hern\u00e1ndez-Pereira, David Alonso-R\u00edos, Jos\u00e9 Bobes-Bascar\u00e1n, and \u00c1ngel Fern\u00e1ndez-Leal. 2023. Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review 56, 4 (2023), 3005\u20133054.","journal-title":"Artificial Intelligence Review"},{"issue":"10","key":"e_1_3_3_42_2","first-page":"1345","article-title":"A survey on transfer learning","volume":"22","author":"Pan Sinno Jialin","year":"2009","unstructured":"Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2009), 1345\u20131359.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"e_1_3_3_43_2","doi-asserted-by":"crossref","first-page":"1158","DOI":"10.18653\/v1\/P16-1110","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Poursabzi-Sangdeh Forough","year":"2016","unstructured":"Forough Poursabzi-Sangdeh, Jordan Boyd-Graber, Leah Findlater, and Kevin Seppi. 2016. Alto: Active learning with topic overviews for speeding label induction and document labeling. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1158\u20131169."},{"key":"e_1_3_3_44_2","first-page":"679","volume-title":"9th International Conference on Design and Emotion","author":"Radeta M.","year":"2014","unstructured":"M. Radeta, Z. Shafieyoun, and M. Maiocchi. 2014. Affective timelines towards the primary-process emotions of movie watchers: Measurements based on self-annotation and affective neuroscience. In 9th International Conference on Design and Emotion, J. Salamanca, P. Desmet, A. Burbano, G. Ludden, and J. Maya (Eds.). Universidad de los Andes, Bogota, Colombia, 679\u2013688."},{"issue":"5","key":"e_1_3_3_45_2","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/MC.2022.3143087","article-title":"Deep learning and the oceans","volume":"55","author":"Radeta Marko","year":"2022","unstructured":"Marko Radeta, Agustin Zuniga, Naser Hossein Motlagh, Mohan Liyanage, Ruben Freitas, Moustafa Youssef, Sasu Tarkoma, Huber Flores, and Petteri Nurmi. 2022. Deep learning and the oceans. Computer 55, 5 (2022), 39\u201350.","journal-title":"Computer"},{"issue":"1","key":"e_1_3_3_46_2","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1109\/MSP.2017.2764116","article-title":"Deep learning for understanding faces: Machines may be just as good, or better, than humans","volume":"35","author":"Ranjan Rajeev","year":"2018","unstructured":"Rajeev Ranjan, Swami Sankaranarayanan, Ankan Bansal, Navaneeth Bodla, Jun-Cheng Chen, Vishal M. Patel, Carlos D. Castillo, and Rama Chellappa. 2018. Deep learning for understanding faces: Machines may be just as good, or better, than humans. IEEE Signal Processing Magazine 35, 1 (2018), 66\u201383.","journal-title":"IEEE Signal Processing Magazine"},{"issue":"1","key":"e_1_3_3_47_2","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1177\/0022487108328486","article-title":"Video annotation tools: Technologies to scaffold, structure, and transform teacher reflection","volume":"60","author":"Rich Peter J.","year":"2009","unstructured":"Peter J. Rich and Michael Hannafin. 2009. Video annotation tools: Technologies to scaffold, structure, and transform teacher reflection. Journal of Teacher Education 60, 1 (2009), 52\u201367.","journal-title":"Journal of Teacher Education"},{"key":"e_1_3_3_48_2","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1109\/ICDAR.2009.250","volume-title":"2009 10th International Conference on Document Analysis and Recognition","author":"Saund Eric","year":"2009","unstructured":"Eric Saund, Jing Lin, and Prateek Sarkar. 2009. Pixlabeler: User interface for pixel-level labeling of elements in document images. In 2009 10th International Conference on Document Analysis and Recognition. IEEE, 646\u2013650."},{"key":"e_1_3_3_49_2","first-page":"14","volume-title":"2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2 \u201918)","author":"Sheng Tao","year":"2018","unstructured":"Tao Sheng, Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Aleksic. 2018. A quantization-friendly separable convolution for mobilenets. In 2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2 \u201918). IEEE, 14\u201318."},{"key":"e_1_3_3_50_2","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1145\/1401890.1401965","volume-title":"Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Sheng Victor S.","year":"2008","unstructured":"Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, New York, NY, USA, 614\u2013622."},{"key":"e_1_3_3_51_2","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1145\/2069618.2069653","volume-title":"Proceedings of the 8th ACM Conference on Creativity and Cognition","author":"Singh Vikash","year":"2011","unstructured":"Vikash Singh, Celine Latulipe, Erin Carroll, and Danielle Lottridge. 2011. The choreographer\u2019s notebook: A video annotation system for dancers and choreographers. In Proceedings of the 8th ACM Conference on Creativity and Cognition. Association for Computing Machinery, New York, NY, USA, 197\u2013206."},{"key":"e_1_3_3_52_2","first-page":"280","volume-title":"10th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON \u201919)","author":"Sinha Debjyoti","year":"2019","unstructured":"Debjyoti Sinha and Mohamed El-Sharkawy. 2019. Thin MobileNet: An enhanced MobileNet architecture. In 10th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON \u201919). IEEE, 280\u2013285. DOI:10.1109\/UEMCON47517.2019.8993089"},{"key":"e_1_3_3_53_2","first-page":"656","volume-title":"Advances in Neural Information Processing Systems","author":"Socher Richard","year":"2012","unstructured":"Richard Socher, Brody Huval, Bharath Bath, Christopher D. Manning, and Andrew Y. Ng. 2012. Convolutional-recursive deep learning for 3d object classification. In Advances in Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, 656\u2013664."},{"key":"e_1_3_3_54_2","volume-title":"Workshop on Crowdsourcing for Search Evaluation (SIGIR \u201910)","author":"Soleymani Mohammad","year":"2010","unstructured":"Mohammad Soleymani and Martha Larson. 2010. Crowdsourcing for affective annotation of video: Development of a viewer-reported boredom corpus. In Workshop on Crowdsourcing for Search Evaluation (SIGIR \u201910). ACM."},{"key":"e_1_3_3_55_2","doi-asserted-by":"crossref","first-page":"558","DOI":"10.1145\/3301275.3302305","volume-title":"Proceedings of the 24th International Conference on Intelligent User Interfaces","author":"Song Jean Y.","year":"2019","unstructured":"Jean Y. Song, Stephan J. Lemmer, Michael Xieyang Liu, Shiyan Yan, Juho Kim, Jason J. Corso, and Walter S. Lasecki. 2019. Popup: Reconstructing 3D video using particle filtering to aggregate crowd responses. In Proceedings of the 24th International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 558\u2013569."},{"key":"e_1_3_3_56_2","first-page":"1","volume-title":"2007 IEEE Conference on Computer Vision and Pattern Recognition","author":"Tian Yuandong","year":"2007","unstructured":"Yuandong Tian, Wei Liu, Rong Xiao, Fang Wen, and Xiaoou Tang. 2007. A face annotation framework with partial clustering and interactive labeling. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1\u20138."},{"issue":"1","key":"e_1_3_3_57_2","first-page":"1","article-title":"Biological data annotation via a human-augmenting AI-based labeling system","volume":"4","author":"Wal Douwe van der","year":"2021","unstructured":"Douwe van der Wal, Iny Jhun, Israa Laklouk, Jeff Nirschl, Lara Richer, Rebecca Rojansky, Talent Theparee, Joshua Wheeler, J\u00f6rg Sander, Felix Feng, et\u00a0al. 2021. Biological data annotation via a human-augmenting AI-based labeling system. NPJ Digital Medicine 4, 1 (2021), 1\u20137.","journal-title":"NPJ Digital Medicine"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3579605"},{"issue":"2","key":"e_1_3_3_59_2","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1023\/B:VISI.0000013087.49260.fb","article-title":"Robust real-time face detection","volume":"57","author":"Viola Paul","year":"2004","unstructured":"Paul Viola and Michael J. Jones. 2004. Robust real-time face detection. International Journal of Computer Vision 57, 2 (2004), 137\u2013154.","journal-title":"International Journal of Computer Vision"},{"key":"e_1_3_3_60_2","unstructured":"Angelo Vittorio. 2018. Toolkit to Download and Visualize Single or Multiple Classes from the Huge Open Images v4 Dataset. https:\/\/github.com\/EscVM\/OIDv4_ToolKit"},{"key":"e_1_3_3_61_2","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1109\/EST.2010.31","volume-title":"2010 International Conference on Emerging Security Technologies","author":"Waharte Sonia","year":"2010","unstructured":"Sonia Waharte and Niki Trigoni. 2010. Supporting search and rescue operations with UAVs. In 2010 International Conference on Emerging Security Technologies. IEEE, 142\u2013147."},{"key":"e_1_3_3_62_2","first-page":"1","volume-title":"2020 14th International Conference on Ubiquitous Information Management and Communication (IMCOM \u201920)","author":"Wang Dylan","year":"2020","unstructured":"Dylan Wang, Melody Moh, and Teng-Sheng Moh. 2020. Using deep learning to solve Google reCAPTCHA v2\u2019s image challenges. In 2020 14th International Conference on Ubiquitous Information Management and Communication (IMCOM \u201920). IEEE, 1\u20135."},{"key":"e_1_3_3_63_2","doi-asserted-by":"crossref","first-page":"595","DOI":"10.1145\/3172944.3173003","volume-title":"23rd International Conference on Intelligent User Interfaces","author":"Wang Isaac","year":"2018","unstructured":"Isaac Wang, Pradyumna Narayana, Jesse Smith, Bruce Draper, Ross Beveridge, and Jaime Ruiz. 2018. Easel: Easy automatic segmentation event labeler. In 23rd International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 595\u2013599."},{"issue":"13","key":"e_1_3_3_64_2","doi-asserted-by":"crossref","first-page":"4350","DOI":"10.3390\/s21134350","article-title":"Confidence score: The forgotten dimension of object detection performance evaluation","volume":"21","author":"Wenkel Simon","year":"2021","unstructured":"Simon Wenkel, Khaled Alhazmi, Tanel Liiv, Saud Alrshoud, and Martin Simon. 2021. Confidence score: The forgotten dimension of object detection performance evaluation. Sensors 21, 13 (2021), 4350.","journal-title":"Sensors"},{"issue":"4","key":"e_1_3_3_65_2","first-page":"114","article-title":"Collaborative intelligence: Humans and AI are joining forces","volume":"96","author":"Wilson H. James","year":"2018","unstructured":"H. James Wilson and Paul R. Daugherty. 2018. Collaborative intelligence: Humans and AI are joining forces. Harvard Business Review 96, 4 (2018), 114\u2013123.","journal-title":"Harvard Business Review"},{"key":"e_1_3_3_66_2","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1016\/j.future.2022.05.014","article-title":"A survey of human-in-the-loop for machine learning","volume":"135","author":"Wu Xingjiao","year":"2022","unstructured":"Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2022. A survey of human-in-the-loop for machine learning. Future Generation Computer Systems 135 (2022), 364\u2013381.","journal-title":"Future Generation Computer Systems"},{"key":"e_1_3_3_67_2","first-page":"1164","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Xu Zhongwen","year":"2017","unstructured":"Zhongwen Xu, Linchao Zhu, and Yi Yang. 2017. Few-shot object recognition from machine-labeled web images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1164\u20131172."},{"key":"e_1_3_3_68_2","first-page":"516","volume-title":"Proceedings of the 9th IEEE International Conference on Computer Vision","author":"Yang Jie","year":"2003","unstructured":"Jie Yang et\u00a0al. 2003. Automatically labeling video data using multi-class active learning. In Proceedings of the 9th IEEE International Conference on Computer Vision. IEEE, 516\u2013523."},{"key":"e_1_3_3_69_2","first-page":"516","volume-title":"Proceedings of the 24th ACM International Conference on Multimedia","author":"Yu Jiahui","year":"2016","unstructured":"Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas Huang. 2016. Unitbox: An advanced object detection network. In Proceedings of the 24th ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, 516\u2013520."},{"key":"e_1_3_3_70_2","first-page":"9\u2013pp","volume-title":"7th IEEE International Symposium on Multimedia (ISM \u201905)","author":"Zhai Gang","year":"2005","unstructured":"Gang Zhai, Geoffrey C. Fox, Marlon Pierce, Wenjun Wu, and Hasan Bulut. 2005. eSports: Collaborative and synchronous video annotation system in grid computing environment. In 7th IEEE International Symposium on Multimedia (ISM \u201905). IEEE, 9\u2013pp."},{"key":"e_1_3_3_71_2","article-title":"Neural network distiller: A Python package for DNN compression research","volume":"1910","author":"Zmora Neta","year":"2019","unstructured":"Neta Zmora, Guy Jacob, Lev Zlotnik, Bar Elharar, and Gal Novik. 2019. Neural network distiller: A Python package for DNN compression research. CoRR abs\/1910.12232 (2019). arXiv:1910.12232http:\/\/arxiv.org\/abs\/1910.12232","journal-title":"CoRR"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649457","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3649457","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:17Z","timestamp":1750291397000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649457"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,23]]},"references-count":70,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3649457"],"URL":"https:\/\/doi.org\/10.1145\/3649457","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,23]]},"assertion":[{"value":"2023-04-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}