{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:45:07Z","timestamp":1776113107929,"version":"3.50.1"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,1,9]],"date-time":"2024-01-09T00:00:00Z","timestamp":1704758400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Network of European Data Scientists"},{"name":"Research and Innovation Staff Exchange"},{"name":"Marie Sk\u0142odowska-Curie Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>Quality-sensitive applications of machine learning (ML) require quality assurance (QA) by humans before the predictions of an ML model can be deployed. QA for ML (QA4ML) interfaces require users to view a large amount of data and perform many interactions to correct errors made by the ML model. An optimized user interface (UI) can significantly reduce interaction costs. While UI optimization can be informed by user studies evaluating design options, this approach is not scalable, because there are typically numerous small variations that can affect the efficiency of a QA4ML interface. Hence, we propose using simulation to evaluate and aid the optimization of QA4ML interfaces. In particular, we focus on simulating the combined effects of human intelligence in initiating appropriate interaction commands and machine intelligence in providing algorithmic assistance for accelerating QA4ML processes. As QA4ML is usually labor-intensive, we use the simulated task completion time as the metric for UI optimization under different interface and algorithm setups. We demonstrate the usage of this UI design method in several QA4ML applications.<\/jats:p>","DOI":"10.1145\/3594552","type":"journal-article","created":{"date-parts":[[2023,5,17]],"date-time":"2023-05-17T12:01:05Z","timestamp":1684324865000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Simulation-based Optimization of User Interfaces for Quality-assuring Machine Learning Model Predictions"],"prefix":"10.1145","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9035-0463","authenticated-orcid":false,"given":"Yu","family":"Zhang","sequence":"first","affiliation":[{"name":"University of Oxford, The United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6506-9522","authenticated-orcid":false,"given":"Martijn","family":"Tennekes","sequence":"additional","affiliation":[{"name":"Statistics Netherlands, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7845-9602","authenticated-orcid":false,"given":"Tim","family":"De Jong","sequence":"additional","affiliation":[{"name":"Statistics Netherlands, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9108-3539","authenticated-orcid":false,"given":"Lyana","family":"Curier","sequence":"additional","affiliation":[{"name":"Open University of the Netherlands, The Netherlands"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5310-8723","authenticated-orcid":false,"given":"Bob","family":"Coecke","sequence":"additional","affiliation":[{"name":"University of Oxford, The United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5320-5729","authenticated-orcid":false,"given":"Min","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Oxford, The United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2024,1,9]]},"reference":[{"key":"e_1_3_3_2_2","first-page":"295","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201997)","author":"Accot Johnny","year":"1997","unstructured":"Johnny Accot and Shumin Zhai. 1997. Beyond fitts\u2019 law: Models for trajectory-based HCI tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201997). ACM, New York, NY, 295\u2013302."},{"issue":"12","key":"e_1_3_3_3_2","doi-asserted-by":"crossref","first-page":"1703","DOI":"10.1109\/TVCG.2014.2346660","article-title":"Visual methods for analyzing probabilistic classification data","volume":"20","author":"Alsallakh Bilal","year":"2014","unstructured":"Bilal Alsallakh, Allan Hanbury, Helwig Hauser, Silvia Miksch, and Andreas Rauber. 2014. Visual methods for analyzing probabilistic classification data. IEEE Trans. Visual. Comput. Graph. 20, 12 (Dec. 2014), 1703\u20131712.","journal-title":"IEEE Trans. Visual. Comput. Graph."},{"key":"e_1_3_3_4_2","volume-title":"The Architecture of Cognition","author":"Anderson John R.","year":"1983","unstructured":"John R. Anderson. 1983. The Architecture of Cognition. Harvard University Press, Mahwah, NJ."},{"key":"e_1_3_3_5_2","volume-title":"Rules of the Mind","author":"Anderson John R.","year":"1993","unstructured":"John R. Anderson. 1993. Rules of the Mind. Lawrence Erlbaum Associates, Mahwah, NJ."},{"key":"e_1_3_3_6_2","first-page":"7","volume-title":"Proceedings of the CHI Workshop on Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV\u201908)","author":"Andrews Keith","year":"2008","unstructured":"Keith Andrews. 2008. Evaluation comes in many guises. In Proceedings of the CHI Workshop on Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV\u201908). 7\u20138."},{"key":"e_1_3_3_7_2","doi-asserted-by":"crossref","first-page":"1957","DOI":"10.1145\/3240508.3241916","volume-title":"Proceedings of the ACM International Conference on Multimedia (MM\u201918)","author":"Andriluka Mykhaylo","year":"2018","unstructured":"Mykhaylo Andriluka, Jasper R. R. Uijlings, and Vittorio Ferrari. 2018. Fluid annotation: A human-machine collaboration interface for full image annotation. In Proceedings of the ACM International Conference on Multimedia (MM\u201918). ACM, New York, NY, 1957\u20131966."},{"key":"e_1_3_3_8_2","first-page":"15","volume-title":"Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201911)","author":"Azzopardi Leif","year":"2011","unstructured":"Leif Azzopardi. 2011. The economics in interactive information retrieval. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201911). ACM, New York, NY, 15\u201324."},{"key":"e_1_3_3_9_2","first-page":"23","volume-title":"Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201913)","author":"Azzopardi Leif","year":"2013","unstructured":"Leif Azzopardi, Diane Kelly, and Kathy Brennan. 2013. How query cost affects search behavior. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201913). ACM, New York, NY, 23\u201332."},{"key":"e_1_3_3_10_2","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201907)","author":"Barkhuus Louise","year":"2007","unstructured":"Louise Barkhuus and Jennifer A. Rode. 2007. From mice to men\u201424 years of evaluation in CHI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201907). ACM, New York, NY, 16."},{"issue":"3","key":"e_1_3_3_11_2","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1111\/cgf.13973","article-title":"Classifier-guided visual correction of noisy labels for image classification tasks","volume":"39","author":"B\u00e4uerle A.","year":"2020","unstructured":"A. B\u00e4uerle, H. Neumann, and T. Ropinski. 2020. Classifier-guided visual correction of noisy labels for image classification tasks. Comput. Graph. Forum 39, 3 (June 2020), 195\u2013205.","journal-title":"Comput. Graph. Forum"},{"issue":"2","key":"e_1_3_3_12_2","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1207\/s15327051hci1102_3","article-title":"Quick and dirty GOMS: A case study of computed tomography interpretation","volume":"11","author":"Beard David V.","year":"1996","unstructured":"David V. Beard, Dana K. Smith, and Kevin M. Denelsbeck. 1996. Quick and dirty GOMS: A case study of computed tomography interpretation. Hum.-Comput. Interact. 11, 2 (June 1996), 157\u2013180.","journal-title":"Hum.-Comput. Interact."},{"issue":"3","key":"e_1_3_3_13_2","doi-asserted-by":"crossref","first-page":"1235","DOI":"10.1111\/j.1467-8659.2012.03116.x","article-title":"Vismon: Facilitating analysis of trade-offs, uncertainty, and sensitivity in fisheries management decision making","volume":"31","author":"Booshehrian Maryam","year":"2012","unstructured":"Maryam Booshehrian, Torsten M\u00f6ller, Randall M. Peterman, and Tamara Munzner. 2012. Vismon: Facilitating analysis of trade-offs, uncertainty, and sensitivity in fisheries management decision making. Comput. Graph. Forum 31, 3 (\u2009June 2012), 1235\u20131244.","journal-title":"Comput. Graph. Forum"},{"issue":"6","key":"e_1_3_3_14_2","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1109\/MCG.2020.3017064","article-title":"Challenges in evaluating interactive visual machine learning systems","volume":"40","author":"Boukhelifa N.","year":"2020","unstructured":"N. Boukhelifa, A. Bezerianos, R. Chang, C. Collins, S. Drucker, A. Endert, J. Hullman, C. North, M. Sedlmair, and Theresa-Marie Rhyne. 2020. Challenges in evaluating interactive visual machine learning systems. IEEE Comput. Graph. Appl. 40, 6 (2020), 88\u201396.","journal-title":"IEEE Comput. Graph. Appl."},{"key":"e_1_3_3_15_2","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1145\/2556288.2557253","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201914)","author":"Bryan Nicholas J.","year":"2014","unstructured":"Nicholas J. Bryan, Gautham J. Mysore, and Ge Wang. 2014. ISSE: An interactive source separation editor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201914). ACM, New York, NY, 257\u2013266."},{"issue":"7","key":"e_1_3_3_16_2","doi-asserted-by":"crossref","first-page":"396","DOI":"10.1145\/358886.358895","article-title":"The keystroke-level model for user performance time with interactive systems","volume":"23","author":"Card Stuart K.","year":"1980","unstructured":"Stuart K. Card, Thomas P. Moran, and Allen Newell. 1980. The keystroke-level model for user performance time with interactive systems. Commun. ACM 23, 7 (\u2009July 1980), 396\u2013410.","journal-title":"Commun. ACM"},{"key":"e_1_3_3_17_2","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1145\/1978942.1978967","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201911)","author":"Chau Duen Horng","year":"2011","unstructured":"Duen Horng Chau, Aniket Kittur, Jason I. Hong, and Christos Faloutsos. 2011. Apolo: Making sense of large network data by combining rich user interaction and machine learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201911). ACM, New York, NY, 167\u2013176."},{"issue":"9","key":"e_1_3_3_18_2","doi-asserted-by":"crossref","first-page":"3701","DOI":"10.1109\/TVCG.2021.3084694","article-title":"Interactive graph construction for graph-based semi-supervised learning","volume":"27","author":"Chen Changjian","year":"2021","unstructured":"Changjian Chen, Zhaowei Wang, Jing Wu, Xiting Wang, Lan-Zhe Guo, Yu-Feng Li, and Shixia Liu. 2021. Interactive graph construction for graph-based semi-supervised learning. IEEE Trans. Vis. Comput. Graph. 27, 9 (2021), 3701\u20133716.","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"e_1_3_3_19_2","first-page":"549","volume-title":"Proceedings of the International Conference on Intelligent User Interfaces (IUI\u201922)","author":"Chen Quan Ze","year":"2022","unstructured":"Quan Ze Chen, Tobias Schnabel, Besmira Nushi, and Saleema Amershi. 2022. HINT: Integration testing for AI-based features with humans in the loop. In Proceedings of the International Conference on Intelligent User Interfaces (IUI\u201922). ACM, New York, NY, 549\u2013565."},{"key":"e_1_3_3_20_2","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1007\/978-1-4613-3306-7_6","volume-title":"Manned Systems Design","author":"Chubb Gerald P.","year":"1981","unstructured":"Gerald P. Chubb. 1981. SAINT, A digital simulation language for the study of manned systems. In Manned Systems Design, Jan Moraal and Karl-Friedrich Kraiss (Eds.). Plenum Press, New York, NY, 153\u2013179."},{"key":"e_1_3_3_21_2","first-page":"367","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201907)","author":"Cui Jingyu","year":"2007","unstructured":"Jingyu Cui, Fang Wen, Rong Xiao, Yuandong Tian, and Xiaoou Tang. 2007. EasyAlbum: An interactive photo annotation system based on face clustering and re-ranking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201907). ACM, New York, NY, 367\u2013376."},{"key":"e_1_3_3_22_2","volume-title":"Human-Computer Interaction","author":"Dix Alan","year":"2004","unstructured":"Alan Dix, Janet Finlay, Gregory D. Abowd, and Russell Beale. 2004. Human-Computer Interaction. Pearson, London, UK."},{"issue":"2","key":"e_1_3_3_23_2","first-page":"8","article-title":"A review of user interface design for interactive machine learning","volume":"8","author":"Dudley John J.","year":"2018","unstructured":"John J. Dudley and Per Ola Kristensson. 2018. A review of user interface design for interactive machine learning. ACM Trans. Interact. Intell. Syst. 8, 2, Article 8 (\u2009June 2018), 37 pages.","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"e_1_3_3_24_2","first-page":"39","volume-title":"Proceedings of the International Conference on Intelligent User Interfaces (IUI\u201903)","author":"Fails Jerry Alan","year":"2003","unstructured":"Jerry Alan Fails and Dan R. Olsen. 2003. Interactive machine learning. In Proceedings of the International Conference on Intelligent User Interfaces (IUI\u201903). ACM, New York, NY, 39\u201345."},{"issue":"3","key":"e_1_3_3_25_2","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1037\/0096-3445.121.3.262","article-title":"The information capacity of the human motor system in controlling the amplitude of movement","volume":"121","author":"Fitts Paul M.","year":"1992","unstructured":"Paul M. Fitts. 1992. The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol.: Gen. 121, 3 (1992), 262\u2013269.","journal-title":"J. Exp. Psychol.: Gen."},{"key":"e_1_3_3_26_2","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1145\/1357054.1357061","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201908)","author":"Fogarty James","year":"2008","unstructured":"James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: Interactive concept learning in image search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201908). ACM, New York, NY, 29\u201338."},{"key":"e_1_3_3_27_2","first-page":"93","volume-title":"Proceedings of the International Conference on Intelligent User Interfaces (IUI\u201904)","author":"Gajos Krzysztof","year":"2004","unstructured":"Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically generating user interfaces. In Proceedings of the International Conference on Intelligent User Interfaces (IUI\u201904). ACM, New York, NY, 93\u2013100."},{"issue":"3","key":"e_1_3_3_28_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/cgf.13667","article-title":"V-Awake: A visual analytics approach for correcting sleep predictions from deep learning models","volume":"38","author":"Caballero Humberto S. Garcia","year":"2019","unstructured":"Humberto S. Garcia Caballero, Michel A. Westenberg, Binyam Gebre, and Jarke J. van Wijk. 2019. V-Awake: A visual analytics approach for correcting sleep predictions from deep learning models. Comput. Graph. Forum 38, 3 (2019), 1\u201312.","journal-title":"Comput. Graph. Forum"},{"key":"e_1_3_3_29_2","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1145\/191666.191782","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201994)","author":"Gong Richard","year":"1994","unstructured":"Richard Gong and David Kieras. 1994. A validation of the GOMS model methodology in the development of a specialized, commercial software application. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201994). ACM, New York, NY, 351\u2013357."},{"issue":"3","key":"e_1_3_3_30_2","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1207\/s15327051hci1303_2","article-title":"Damaged merchandise? A review of experiments that compare usability evaluation methods","volume":"13","author":"Gray Wayne D.","year":"1998","unstructured":"Wayne D. Gray and Marilyn C. Salzman. 1998. Damaged merchandise? A review of experiments that compare usability evaluation methods. Hum.-Comput. Interact. 13, 3 (Sept. 1998), 203\u2013261.","journal-title":"Hum.-Comput. Interact."},{"key":"e_1_3_3_31_2","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1145\/989863.989881","volume-title":"Proceedings of the Working Conference on Advanced Visual Interfaces (AVI\u201904)","author":"Guiard Yves","year":"2004","unstructured":"Yves Guiard, Michel Beaudouin-Lafon, Julien Bastin, Dennis Pasveer, and Shumin Zhai. 2004. View size and pointing difficulty in multi-scale navigation. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI\u201904). ACM, New York, NY, 117\u2013124."},{"key":"e_1_3_3_32_2","first-page":"23","volume-title":"Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST\u201912)","author":"H\u00f6ferlin Benjamin","year":"2012","unstructured":"Benjamin H\u00f6ferlin, Rudolf Netzel, Markus H\u00f6ferlin, Daniel Weiskopf, and Gunther Heidemann. 2012. Inter-active learning of ad-hoc classifiers for video visual analytics. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST\u201912). IEEE, Los Alamitos, CA, 23\u201332."},{"issue":"4","key":"e_1_3_3_33_2","doi-asserted-by":"crossref","first-page":"648","DOI":"10.1109\/JPROC.2008.916364","article-title":"Active learning for interactive multimedia retrieval","volume":"96","author":"Huang Thomas S.","year":"2008","unstructured":"Thomas S. Huang, Charlie K. Dagli, Shyamsundar Rajaram, Edward Y. Chang, Michael I. Mandel, Graham E. Poliner, and Daniel P. W. Ellis. 2008. Active learning for interactive multimedia retrieval. Proc. IEEE 96, 4 (April 2008), 648\u2013667.","journal-title":"Proc. IEEE"},{"issue":"4","key":"e_1_3_3_34_2","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1145\/235833.236054","article-title":"The GOMS family of user interface analysis techniques: Comparison and contrast","volume":"3","author":"John Bonnie E.","year":"1996","unstructured":"Bonnie E. John and David E. Kieras. 1996. The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Trans. Comput.-Hum. Interact. 3, 4 (Dec. 1996), 320\u2013351.","journal-title":"ACM Trans. Comput.-Hum. Interact."},{"issue":"4","key":"e_1_3_3_35_2","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1145\/235833.236050","article-title":"Using GOMS for user interface design and evaluation: Which technique?","volume":"3","author":"John Bonnie E.","year":"1996","unstructured":"Bonnie E. John and David E. Kieras. 1996. Using GOMS for user interface design and evaluation: Which technique? ACM Trans. Comput.-Hum. Interact. 3, 4 (Dec. 1996), 287\u2013319.","journal-title":"ACM Trans. Comput.-Hum. Interact."},{"key":"e_1_3_3_36_2","series-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","first-page":"251","author":"John Bonnie E.","year":"1992","unstructured":"Bonnie E. John and Alonso H. Vera. 1992. A GOMS analysis of a graphic machine-paced, highly interactive task. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (1992) (CHI\u201992). ACM, New York, NY, 251\u2013258."},{"issue":"1","key":"e_1_3_3_37_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3487065","article-title":"Finding AI\u2019s faults with AAR\/AI: An empirical study","volume":"12","author":"Khanna Roli","year":"2022","unstructured":"Roli Khanna, Jonathan Dodge, Andrew Anderson, Rupika Dikkala, Jed Irvine, Zeyad Shureih, Kin-Ho Lam, Caleb R. Matthews, Zhengxian Lin, Minsuk Kahng, Alan Fern, and Margaret Burnett. 2022. Finding AI\u2019s faults with AAR\/AI: An empirical study. ACM Trans. Interact. Intell. Syst. 12, 1, Article 1 (March 2022), 33 pages.","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"e_1_3_3_38_2","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1016\/B978-0-444-70536-5.50012-9","volume-title":"Handbook of Human-Computer Interaction","author":"Kieras David E.","year":"1988","unstructured":"David E. Kieras. 1988. Towards a practical GOMS model methodology for user interface design. In Handbook of Human-Computer Interaction, Martin G. Helander (Ed.). Elsevier, Amsterdam, The Netherlands, 135\u2013157."},{"key":"e_1_3_3_39_2","doi-asserted-by":"crossref","first-page":"3875","DOI":"10.1145\/2556288.2557324","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201914)","author":"Kieras David E.","year":"2014","unstructured":"David E. Kieras and Anthony J. Hornof. 2014. Towards accurate and practical predictive models of active-vision-based visual search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201914). ACM, New York, NY, 3875\u20133884."},{"key":"e_1_3_3_40_2","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1145\/985692.985705","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201904)","author":"Kieras David E.","year":"2004","unstructured":"David E. Kieras and Thomas P. Santoro. 2004. Computational GOMS modeling of a complex team task: Lessons learned. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201904). ACM, New York, NY, 97\u2013104."},{"issue":"3","key":"e_1_3_3_41_2","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1145\/264645.264658","article-title":"Predictive engineering models based on the EPIC architecture for a multimodal high-performance human-computer interaction task","volume":"4","author":"Kieras David E.","year":"1997","unstructured":"David E. Kieras, Scott D. Wood, and David E. Meyer. 1997. Predictive engineering models based on the EPIC architecture for a multimodal high-performance human-computer interaction task. ACM Trans. Comput.-Hum. Interact. 4, 3 (Sept. 1997), 230\u2013275.","journal-title":"ACM Trans. Comput.-Hum. Interact."},{"issue":"5","key":"e_1_3_3_42_2","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1109\/TSMC.1981.4308690","article-title":"A display design and evaluation study using task network models","volume":"11","author":"Kraiss Karl-Friedrich","year":"1981","unstructured":"Karl-Friedrich Kraiss. 1981. A display design and evaluation study using task network models. IEEE Trans. Syst. Man Cybernet. 11, 5 (May 1981), 339\u2013351.","journal-title":"IEEE Trans. Syst. Man Cybernet."},{"key":"e_1_3_3_43_2","first-page":"235","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201990)","author":"Lewis Clayton","year":"1990","unstructured":"Clayton Lewis, Peter G. Polson, Cathleen Wharton, and John Rieman. 1990. Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201990). ACM, New York, NY, 235\u2013242."},{"key":"e_1_3_3_44_2","volume-title":"Task-Centered User Interface Design","author":"Lewis Clayton","year":"1993","unstructured":"Clayton Lewis and John Rieman. 1993. Task-Centered User Interface Design."},{"issue":"1","key":"e_1_3_3_45_2","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1109\/TVCG.2018.2864843","article-title":"An interactive method to improve crowdsourced annotations","volume":"25","author":"Liu Shixia","year":"2019","unstructured":"Shixia Liu, Changjian Chen, Yafeng Lu, Fangxin Ouyang, and Bin Wang. 2019. An interactive method to improve crowdsourced annotations. IEEE Trans. Vis. Comput. Graph. 25, 1 (\u2009Jan. 2019), 235\u2013245.","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"e_1_3_3_46_2","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1145\/142750.142794","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201992)","author":"MacKenzie I. Scott","year":"1992","unstructured":"I. Scott MacKenzie and William Buxton. 1992. Extending fitts\u2019 law to two-dimensional tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201992). ACM, New York, NY, 219\u2013226."},{"key":"e_1_3_3_47_2","first-page":"249","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201990)","author":"Nielsen Jakob","year":"1990","unstructured":"Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201990). ACM, New York, NY, 249\u2013256."},{"key":"e_1_3_3_48_2","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780198799603.001.0001","volume-title":"Computational Interaction","author":"Oulasvirta Antti","year":"2018","unstructured":"Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes (Eds.). 2018. Computational Interaction. Oxford University Press, Oxford, UK."},{"issue":"2","key":"e_1_3_3_49_2","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1080\/10447318.2016.1220729","article-title":"Using GOMS and NASA-TLX to evaluate human-computer interaction process in interactive segmentation","volume":"33","author":"Ramkumar Anjana","year":"2017","unstructured":"Anjana Ramkumar, Pieter Jan Stappers, Wiro J. Niessen, Sonja Adebahr, Tanja Schimek-Jasch, Ursula Nestle, and Yu Song. 2017. Using GOMS and NASA-TLX to evaluate human-computer interaction process in interactive segmentation. Int. J. Hum.-Comput. Interact. 33, 2 (2017), 123\u2013134.","journal-title":"Int. J. Hum.-Comput. Interact."},{"issue":"5","key":"e_1_3_3_50_2","first-page":"36","article-title":"An introduction to meta-evaluation","volume":"2","author":"Scriven Michael","year":"1969","unstructured":"Michael Scriven. 1969. An introduction to meta-evaluation. Educ. Prod. Rep. 2, 5 (Feb. 1969), 36\u201338.","journal-title":"Educ. Prod. Rep."},{"key":"e_1_3_3_51_2","doi-asserted-by":"crossref","DOI":"10.1201\/9781410615862","volume-title":"The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications","author":"Sears Andrew","year":"2007","unstructured":"Andrew Sears and Julie A. Jacko (Eds.). 2007. The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. CRC Press, Boca Raton, FL."},{"key":"e_1_3_3_52_2","first-page":"1","volume-title":"Proceedings of the NIPS Workshop on Cost-Sensitive Learning","author":"Settles Burr","year":"2008","unstructured":"Burr Settles, Mark Craven, and Lewis Friedland. 2008. Active learning with real annotation costs. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning. 1\u201310."},{"key":"e_1_3_3_53_2","volume-title":"On the Mode of Communication of Cholera","author":"Snow John","year":"1855","unstructured":"John Snow. 1855. On the Mode of Communication of Cholera. John Churchill, London, UK."},{"issue":"1","key":"e_1_3_3_54_2","first-page":"1064","article-title":"explAIner: A visual analytics framework for interactive and explainable machine learning","volume":"26","author":"Spinner Thilo","year":"2020","unstructured":"Thilo Spinner, Udo Schlegel, Hanna Sch\u00e4fer, and Mennatallah El-Assady. 2020. explAIner: A visual analytics framework for interactive and explainable machine learning. IEEE Trans. Vis. Comput. Graph. 26, 1 (2020), 1064\u20131074.","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"issue":"1","key":"e_1_3_3_55_2","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1177\/016327877800100102","article-title":"Meta evaluation: An overview","volume":"1","author":"Stufflebeam Daniel L.","year":"1978","unstructured":"Daniel L. Stufflebeam. 1978. Meta evaluation: An overview. Eval. Health Profess. 1, 1 (April 1978), 17\u201343.","journal-title":"Eval. Health Profess."},{"issue":"4","key":"e_1_3_3_56_2","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1016\/j.intcom.2007.02.002","article-title":"Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition","volume":"19","author":"Suh Bongwon","year":"2007","unstructured":"Bongwon Suh and Benjamin B. Bederson. 2007. Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition. Interact. Comput. 19, 4 (\u2009July 2007), 524\u2013544.","journal-title":"Interact. Comput."},{"issue":"4","key":"e_1_3_3_57_2","first-page":"29","article-title":"Towards optimizing human labeling for interactive image tagging","volume":"9","author":"Tang Jinhui","year":"2013","unstructured":"Jinhui Tang, Qiang Chen, Meng Wang, Shuicheng Yan, Tat-Seng Chua, and Ramesh Jain. 2013. Towards optimizing human labeling for interactive image tagging. ACM Trans. Multimedia Comput. Commun. Appl. 9, 4, Article 29 (Aug. 2013), 18 pages.","journal-title":"ACM Trans. Multimedia Comput. Commun. Appl."},{"key":"e_1_3_3_58_2","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201921)","author":"Todi Kashyap","year":"2021","unstructured":"Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta. 2021. Adapting user interfaces with model-based reinforcement learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201921). ACM, New York, NY, USA, Article 573, 13 pages."},{"issue":"1","key":"e_1_3_3_59_2","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1109\/TVCG.2019.2934630","article-title":"Tac-simur: Tactic-based simulative visual analytics of table tennis","volume":"26","author":"Wang Jiachen","year":"2020","unstructured":"Jiachen Wang, Kejian Zhao, Dazhen Deng, Anqi Cao, Xiao Xie, Zheng Zhou, Hui Zhang, and Yingcai Wu. 2020. Tac-simur: Tactic-based simulative visual analytics of table tennis. IEEE Trans. Vis. Comput. Graph. 26, 1 (\u2009Jan. 2020), 407\u2013417.","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"issue":"1","key":"e_1_3_3_60_2","doi-asserted-by":"crossref","first-page":"709","DOI":"10.1109\/TVCG.2017.2744218","article-title":"iTTVis: Interactive visualization of table tennis data","volume":"24","author":"Wu Yingcai","year":"2018","unstructured":"Yingcai Wu, Ji Lan, Xinhuan Shu, Chenyang Ji, Kejian Zhao, Jiachen Wang, and Hui Zhang. 2018. iTTVis: Interactive visualization of table tennis data. IEEE Trans. Vis. Comput. Graph. 24, 1 (\u2009Jan. 2018), 709\u2013718.","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"e_1_3_3_61_2","first-page":"57","volume-title":"Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST\u201919)","author":"Xiang Shouxing","year":"2019","unstructured":"Shouxing Xiang, Xi Ye, Jiazhi Xia, Jing Wu, Yang Chen, and Shixia Liu. 2019. Interactive correction of mislabeled training data. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST\u201919). IEEE, Los Alamitos, CA, 57\u201368."},{"key":"e_1_3_3_62_2","volume-title":"Proceedings of the IEEE VIS Workshop on Evaluation of Interactive Visual Machine Learning Systems","author":"Zhang Yu","year":"2019","unstructured":"Yu Zhang, Bob Coecke, and Min Chen. 2019. On the cost of interactions in interactive visual machine learning. In Proceedings of the IEEE VIS Workshop on Evaluation of Interactive Visual Machine Learning Systems. IEEE, Los Alamitos, CA, 5."},{"issue":"3","key":"e_1_3_3_63_2","first-page":"18","article-title":"MI3: Machine-initiated intelligent interaction for interactive classification and data reconstruction","volume":"11","author":"Zhang Yu","year":"2021","unstructured":"Yu Zhang, Bob Coecke, and Min Chen. 2021. MI3: Machine-initiated intelligent interaction for interactive classification and data reconstruction. ACM Trans. Interact. Intell. Syst. 11, 3\u20134, Article 18 (Aug. 2021), 34 pages.","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"e_1_3_3_64_2","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201922)","author":"Zhang Yu","year":"2022","unstructured":"Yu Zhang, Yun Wang, Haidong Zhang, Bin Zhu, Siming Chen, and Dongmei Zhang. 2022. OneLabeler: A flexible system for building data labeling tools. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI\u201922). ACM, New York, NY."},{"key":"e_1_3_3_65_2","first-page":"685","volume-title":"Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201915)","author":"Zhang Yinan","year":"2015","unstructured":"Yinan Zhang and Chengxiang Zhai. 2015. Information retrieval as card playing: A formal model for optimizing interactive retrieval interface. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR\u201915). ACM, New York, NY, 685\u2013694."}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3594552","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3594552","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:07Z","timestamp":1750183747000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3594552"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,9]]},"references-count":64,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3594552"],"URL":"https:\/\/doi.org\/10.1145\/3594552","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,9]]},"assertion":[{"value":"2022-10-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-02","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}