{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T03:33:55Z","timestamp":1777692835663,"version":"3.51.4"},"reference-count":26,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2018,6,13]],"date-time":"2018-06-13T00:00:00Z","timestamp":1528848000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2018,6,30]]},"abstract":"<jats:p>\n            People are not infallible consistent \u201coracles\u201d: their confidence in decision-making may vary significantly between tasks and over time. We have previously reported the benefits of using an interface and algorithms that\n            <jats:italic>explicitly<\/jats:italic>\n            captured and exploited users\u2019 confidence: error rates were reduced by up to 50% for an industrial multi-class learning problem; and the number of interactions required in a design-optimisation context was reduced by 33%. Having access to users\u2019 confidence judgements could significantly benefit intelligent interactive systems in industry, in areas such as intelligent tutoring systems and in health care. There are many reasons for wanting to capture information about confidence\n            <jats:italic>implicitly<\/jats:italic>\n            . Some are ergonomic, but others are more \u201csocial\u201d\u2014such as wishing to understand (and possibly take account of) users\u2019 cognitive state without interrupting them.\n          <\/jats:p>\n          <jats:p>We investigate the hypothesis that users\u2019 confidence can be accurately predicted from measurements of their behaviour. Eye-tracking systems were used to capture users\u2019 gaze patterns as they undertook a series of visual decision tasks, after each of which they reported their confidence on a 5-point Likert scale. Subsequently, predictive models were built using \u201cconventional\u201d machine learning approaches for numerical summary features derived from users\u2019 behaviour. We also investigate the extent to which the deep learning paradigm can reduce the need to design features specific to each application by creating \u201cgaze maps\u201d\u2014visual representations of the trajectories and durations of users\u2019 gaze fixations\u2014and then training deep convolutional networks on these images.<\/jats:p>\n          <jats:p>Treating the prediction of user confidence as a two-class problem (confident\/not confident), we attained classification accuracy of 88% for the scenario of new users on known tasks, and 87% for known users on new tasks. Considering the confidence as an ordinal variable, we produced regression models with a mean absolute error of \u22480.7 in both cases. Capturing just a simple subset of non-task-specific numerical features gave slightly worse, but still quite high accuracy (e.g., MAE \u2248 1.0). Results obtained with gaze maps and convolutional networks are competitive, despite not having access to longer-term information about users and tasks, which was vital for the \u201csummary\u201d feature sets. This suggests that the gaze-map-based approach forms a viable, transferable alternative to handcrafting features for each different application. These results provide significant evidence to confirm our hypothesis, and offer a way of substantially improving many interactive artificial intelligence applications via the addition of cheap non-intrusive hardware and computationally cheap prediction algorithms.<\/jats:p>","DOI":"10.1145\/3185524","type":"journal-article","created":{"date-parts":[[2018,6,13]],"date-time":"2018-06-13T15:30:12Z","timestamp":1528903812000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Predicting User Confidence During Visual Decision Making"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7908-1859","authenticated-orcid":false,"given":"Jim","family":"Smith","sequence":"first","affiliation":[{"name":"University of the West of England, Bristol, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Phil","family":"Legg","sequence":"additional","affiliation":[{"name":"University of the West of England, Bristol, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Milos","family":"Matovic","sequence":"additional","affiliation":[{"name":"University of the West of England, Bristol, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kristofer","family":"Kinsey","sequence":"additional","affiliation":[{"name":"University of the West of England, Bristol, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,6,13]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"M. Abadi A. Agarwal P. Barham E. Brevdo Z. Chen C. Citro G. S. Corrado A. Davis J. Dean M. Devin S. Ghemawat I. Goodfellow A. Harp G. Irving M. Isard Y. Jia R. Jozefowicz L. Kaiser M. Kudlur J. Levenberg D. Man\u00e9 R. Monga S. Moore D. Murray C. Olah M. Schuster J. Shlens B. Steiner I. Sutskever K. Talwar P. Tucker V. Vanhoucke V. Vasudevan F. Vi\u00e9gas O. Vinyals P. Warden M. Wattenberg M. Wicke Y. Yu and X. Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Retrieved from http:\/\/tensorflow.org\/.  M. Abadi A. Agarwal P. Barham E. Brevdo Z. Chen C. Citro G. S. Corrado A. Davis J. Dean M. Devin S. Ghemawat I. Goodfellow A. Harp G. Irving M. Isard Y. Jia R. Jozefowicz L. Kaiser M. Kudlur J. Levenberg D. Man\u00e9 R. Monga S. Moore D. Murray C. Olah M. Schuster J. Shlens B. Steiner I. Sutskever K. Talwar P. Tucker V. Vanhoucke V. Vasudevan F. Vi\u00e9gas O. Vinyals P. Warden M. Wattenberg M. Wicke Y. Yu and X. Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Retrieved from http:\/\/tensorflow.org\/."},{"key":"e_1_2_1_2_1","volume-title":"Individual consistency in the accuracy and distribution of confidence judgments. Cognition 146 (Jan","author":"Ais Joaqu\u00edn","year":"2016","unstructured":"Joaqu\u00edn Ais , Ariel Zylberberg , Pablo Barttfeld , and Mariano Sigman . 2016. Individual consistency in the accuracy and distribution of confidence judgments. Cognition 146 (Jan . 2016 ), 377--386. Joaqu\u00edn Ais, Ariel Zylberberg, Pablo Barttfeld, and Mariano Sigman. 2016. Individual consistency in the accuracy and distribution of confidence judgments. Cognition 146 (Jan. 2016), 377--386."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2838739.2838753"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1076-6332(03)80499-3"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.606"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2006.04.023"},{"key":"e_1_2_1_8_1","unstructured":"F. Chollet et al. 2015. Keras. https:\/\/github.com\/fchollet\/keras.  F. Chollet et al. 2015. Keras. https:\/\/github.com\/fchollet\/keras."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780198524793.001.0001"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2015.04.006"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864349.1864395"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 24th International Florida Artificial Intelligence Research Society Conference. Association for the Advancement of Artificial Intelligence, USA, 31--36","author":"Jraidi I.","unstructured":"I. Jraidi , M. Chaouachi , and C. Frasson . 2011. Automatic detection of user\u2019s uncertainty in problem solving task: A multimodal approach . In Proceedings of the 24th International Florida Artificial Intelligence Research Society Conference. Association for the Advancement of Artificial Intelligence, USA, 31--36 . I. Jraidi, M. Chaouachi, and C. Frasson. 2011. Automatic detection of user\u2019s uncertainty in problem solving task: A multimodal approach. In Proceedings of the 24th International Florida Artificial Intelligence Research Society Conference. Association for the Advancement of Artificial Intelligence, USA, 31--36."},{"key":"e_1_2_1_13_1","first-page":"219","article-title":"Student\u2019s uncertainty modeling through a multimodal sensor-based approach","volume":"16","author":"Jraidi I.","year":"2013","unstructured":"I. Jraidi and C. Frasson . 2013 . Student\u2019s uncertainty modeling through a multimodal sensor-based approach . Educ. Technol. Soc. 16 , 1 (2013), 219 -- 230 . I. Jraidi and C. Frasson. 2013. Student\u2019s uncertainty modeling through a multimodal sensor-based approach. Educ. Technol. Soc. 16, 1 (2013), 219--230.","journal-title":"Educ. Technol. Soc."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/THS.2015.7446229"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.207"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2009.2025025"},{"key":"e_1_2_1_17_1","unstructured":"S. W. McQuiggan S. Lee and J. C. Lester. 2006. Predicting user physiological response for interactive environments: An inductive approach. In AIIDE. AAAI Press Stanford CA 60--65. http:\/\/www.aaai.org\/Papers\/AIIDE\/2006\/AIIDE06-014.pdf.   S. W. McQuiggan S. Lee and J. C. Lester. 2006. Predicting user physiological response for interactive environments: An inductive approach. In AIIDE. AAAI Press Stanford CA 60--65. http:\/\/www.aaai.org\/Papers\/AIIDE\/2006\/AIIDE06-014.pdf."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2009.03.007"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1037\/0278-7393.14.1.85"},{"key":"e_1_2_1_20_1","unstructured":"H. Raghavan O. Madani and R. Jones. 2006. Active learning with feedback on both features and instances. J. Mach. Learn. Res. 7 (Aug. 2006) 1655--1686.   H. Raghavan O. Madani and R. Jones. 2006. Active learning with feedback on both features and instances. J. Mach. Learn. Res. 7 (Aug. 2006) 1655--1686."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11409-010-9059-z"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-014-0094-2"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2633043"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2007.09.009"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/9781119078845.ch11"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.2114\/jpa.24.81"},{"key":"e_1_2_1_27_1","volume-title":"Wechsler Abbreviated Scale of Intelligence","author":"Wechsler D.","unstructured":"D. Wechsler . 1999. Wechsler Abbreviated Scale of Intelligence . The Psychological Corporation: Harcourt Brace and Company , New York. D. Wechsler. 1999. Wechsler Abbreviated Scale of Intelligence. The Psychological Corporation: Harcourt Brace and Company, New York."}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3185524","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3185524","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:26:37Z","timestamp":1750213597000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3185524"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,13]]},"references-count":26,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,6,30]]}},"alternative-id":["10.1145\/3185524"],"URL":"https:\/\/doi.org\/10.1145\/3185524","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,6,13]]},"assertion":[{"value":"2016-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-06-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}