{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T18:17:53Z","timestamp":1781547473478,"version":"3.54.5"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T00:00:00Z","timestamp":1667174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"ICT Centre for Excellence Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2022,10,31]]},"abstract":"<jats:p>User trust plays a key role in determining whether autonomous computer applications are relied upon. It will play a key role in the acceptance of emerging AI applications such as optimisation. Two important factors known to affect trust are system transparency, i.e., how well the user understands how the system works, and system performance. However, in the case of optimisation, it is difficult for the end-user to understand the underlying algorithms or to judge the quality of the solution. Through two controlled user studies, we explore whether the user is better able to calibrate their trust in the system when: (a) They are provided feedback on the system operation in the form of visualisation of intermediate solutions and their quality; (b) They can interactively explore the solution space by modifying the solution returned by the system. We found that showing intermediate solutions can lead to over-trust, while interactive exploration leads to more accurately calibrated trust.<\/jats:p>","DOI":"10.1145\/3503461","type":"journal-article","created":{"date-parts":[[2022,7,22]],"date-time":"2022-07-22T11:15:01Z","timestamp":1658488501000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":28,"title":["Increasing User Trust in Optimisation through Feedback and Interaction"],"prefix":"10.1145","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8937-7095","authenticated-orcid":false,"given":"Jie","family":"Liu","sequence":"first","affiliation":[{"name":"Monash University, Caulfield East, Victoria, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9813-0377","authenticated-orcid":false,"given":"Kim","family":"Marriott","sequence":"additional","affiliation":[{"name":"Monash University, Caulfield East, Victoria, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9076-9571","authenticated-orcid":false,"given":"Tim","family":"Dwyer","sequence":"additional","affiliation":[{"name":"Monash University, Caulfield East, Victoria, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3357-6498","authenticated-orcid":false,"given":"Guido","family":"Tack","sequence":"additional","affiliation":[{"name":"Monash University, Caulfield East, Victoria, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,1,6]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174156"},{"key":"e_1_3_3_3_2","first-page":"209","volume-title":"Proceedings of the AAAI\/IAAI","author":"Anderson David","year":"2000","unstructured":"David Anderson, Emily Anderson, Neal Lesh, Joe Marks, Brian Mirtich, David Ratajczak, and Kathy Ryall. 2000. Human-guided simple search. In Proceedings of the AAAI\/IAAI. 209\u2013216."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.12.012"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2007.03.002"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1080\/14639220500535301"},{"key":"e_1_3_3_7_2","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1145\/2501988.2502024","volume-title":"Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology","author":"Bailly Gilles","year":"2013","unstructured":"Gilles Bailly, Antti Oulasvirta, Timo K\u00f6tzing, and Sabrina Hoppe. 2013. Menuoptimizer: Interactive optimization of menu systems. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology. ACM, 331\u2013342."},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2614803"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.1998.0224"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/1921427.1921443"},{"key":"e_1_3_3_11_2","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1145\/1240624.1240739","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Butler Keith A.","year":"2007","unstructured":"Keith A. Butler, Jiajie Zhang, Chris Esposito, Ali Bahrami, Ron Hebron, and David Kieras. 2007. Work-centered design: A case study of a mixed-initiative scheduler. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 747\u2013756."},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2788613"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2010.10.005"},{"key":"e_1_3_3_14_2","volume-title":"Proceedings of the Command and Control Research and Technology Symposium","author":"Cohen Marvin S.","year":"1998","unstructured":"Marvin S. Cohen, Raja Parasuraman, and Jared T. Freeman. 1998. Trust in decision aids: A model and its training implications. In Proceedings of the Command and Control Research and Technology Symposium. Citeseer."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1071-5819(03)00041-7"},{"key":"e_1_3_3_16_2","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1145\/642611.642713","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Cosley Dan","year":"2003","unstructured":"Dan Cosley, Shyong K. Lam, Istvan Albert, Joseph A. Konstan, and John Riedl. 2003. Is seeing believing? How recommender system interfaces affect users\u2019 opinions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 585\u2013592."},{"key":"e_1_3_3_17_2","volume-title":"Fleet Management and Logistics","author":"Crainic Teodor G.","year":"2012","unstructured":"Teodor G. Crainic and Gilbert Laporte. 2012. Fleet Management and Logistics. Springer Science & Business Media."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2011.10.001"},{"key":"e_1_3_3_19_2","first-page":"2752","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"33","author":"\u010cyras Kristijonas","year":"2019","unstructured":"Kristijonas \u010cyras, Dimitrios Letsios, Ruth Misener, and Francesca Toni. 2019. Argumentation for explainable scheduling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2752\u20132759."},{"issue":"2","key":"e_1_3_3_20_2","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1177\/1555343411410160","article-title":"Adaptive aiding of human-robot teaming: Effects of imperfect automation on performance, trust, and workload","volume":"5","author":"Visser Ewart de","year":"2011","unstructured":"Ewart de Visser and Raja Parasuraman. 2011. Adaptive aiding of human-robot teaming: Effects of imperfect automation on performance, trust, and workload. Journal of Cognitive Engineering and Decision Making 5, 2 (2011), 209\u2013231.","journal-title":"Journal of Cognitive Engineering and Decision Making"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1071-5819(03)00038-7"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jksus.2010.03.002"},{"key":"e_1_3_3_23_2","volume-title":"Discovering Statistics Using R","author":"Field Andy","year":"2012","unstructured":"Andy Field, Jeremy Miles, and Zo\u00eb Field. 2012. Discovering Statistics Using R. Sage publications."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/302979.303001"},{"key":"e_1_3_3_25_2","unstructured":"Gecode Team. 2016. Gecode: A Generic Constraint Development Environment. Retrieved from http:\/\/www.gecode.org."},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.2307\/30036519"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/1378773.1378804"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645761"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.techfore.2015.12.014"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/358916.358995"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1177\/0018720814547570"},{"issue":"6","key":"e_1_3_3_32_2","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1109\/MIS.2009.124","article-title":"The dynamics of trust in cyberdomains","author":"Hoffman Robert R.","year":"2009","unstructured":"Robert R. Hoffman, John D. Lee, David D. Woods, Nigel Shadbolt, Janet Miller, and Jeffrey M. Bradshaw. 2009. The dynamics of trust in cyberdomains. IEEE Intelligent Systems6 (2009), 5\u201311.","journal-title":"IEEE Intelligent Systems"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1207\/S15327566IJCE0401_04"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753529"},{"key":"e_1_3_3_35_2","volume-title":"Interactive and Interpretable Machine Learning Models for Human Machine Collaboration","author":"Kim Been","year":"2015","unstructured":"Been Kim. 2015. Interactive and Interpretable Machine Learning Models for Human Machine Collaboration. Ph.D. Dissertation. Massachusetts Institute of Technology."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0953-5438(97)00037-4"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.4236\/iim.2012.43010"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1080\/00140139208967392"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50.30392"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1080\/10864415.2001.11044227"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1214\/15-AOAS848"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2010.01.004"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236386.3241340"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744418"},{"issue":"2","key":"e_1_3_3_45_2","first-page":"1764","article-title":"Supporting the problem-solving loop: Designing highly interactive optimisation systems","volume":"27","author":"Liu Jie","year":"2020","unstructured":"Jie Liu, Tim Dwyer, Guido Tack, Samuel Gratzl, and Kim Marriott. 2020. Supporting the problem-solving loop: Designing highly interactive optimisation systems. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 1764\u20131774.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"e_1_3_3_46_2","first-page":"382","article-title":"What is trust? A conceptual analysis and an interdisciplinary model","author":"McKnight D. Harrison","year":"2000","unstructured":"D. Harrison McKnight and Norman L. Chervany. 2000. What is trust? A conceptual analysis and an interdisciplinary model. AMCIS 2000 Proceedings (2000), 382.","journal-title":"AMCIS 2000 Proceedings"},{"key":"e_1_3_3_47_2","doi-asserted-by":"crossref","first-page":"1009","DOI":"10.1109\/ICSMC.2011.6083801","volume-title":"Proceedings of the 2011 IEEE International Conference on Systems, Man, and Cybernetics","author":"Meignan David","year":"2011","unstructured":"David Meignan, Jean-Marc Frayret, and Gilles Pesant. 2011. An interactive heuristic approach for the P-forest problem. In Proceedings of the 2011 IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 1009\u20131013."},{"issue":"3","key":"e_1_3_3_48_2","first-page":"17","article-title":"A review and taxonomy of interactive optimization methods in operations research","volume":"5","author":"Meignan David","year":"2015","unstructured":"David Meignan, Sigrid Knust, Jean-Marc Frayret, Gilles Pesant, and Nicolas Gaud. 2015. A review and taxonomy of interactive optimization methods in operations research. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 3 (2015), 17.","journal-title":"ACM Transactions on Interactive Intelligent Systems (TiiS)"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1037\/1076-898X.6.1.44"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0020-7373(87)80013-5"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1080\/00140139608964474"},{"key":"e_1_3_3_52_2","first-page":"529","volume-title":"Proceedings of the Principles and Practice of Constraint Programming \u2013 CP 2007","author":"Nethercote Nicholas","year":"2007","unstructured":"Nicholas Nethercote, Peter J. Stuckey, Ralph Becket, Sebastian Brand, Gregory J. Duck, and Guido Tack. 2007. MiniZinc: Towards a standard CP modelling language. In Proceedings of the Principles and Practice of Constraint Programming \u2013 CP 2007, Christian Bessi\u00e8re (Ed.). Springer, Berlin, 529\u2013543."},{"key":"e_1_3_3_53_2","first-page":"445","volume-title":"Proceedings of the International Conference on Principles and Practice of Constraint Programming","author":"O\u2019Callaghan Barry","year":"2005","unstructured":"Barry O\u2019Callaghan, Barry O\u2019Sullivan, and Eugene C. Freuder. 2005. Generating corrective explanations for interactive constraint satisfaction. In Proceedings of the International Conference on Principles and Practice of Constraint Programming. Springer, 445\u2013459."},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1518\/001872097778543886"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240746"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-010-9253-3"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744198"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2015.2467551"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300750"},{"key":"e_1_3_3_61_2","volume-title":"A Meta-analysis of Factors Influencing the Development of Trust in Automation: Implications for Human-robot Interaction","author":"Schaefer Kristin E.","year":"2014","unstructured":"Kristin E. Schaefer, Deborah R. Billings, James L. Szalma, Jeffrey K. Adams, Tracy L. Sanders, Jessie Y. Chen, and Peter A. Hancock. 2014. A Meta-analysis of Factors Influencing the Development of Trust in Automation: Implications for Human-robot Interaction. Technical Report. Army Research Lab Aberdeen Proving Ground Md Human Research and Engineering Directorate."},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1177\/0018720816634228"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/503376.503405"},{"key":"e_1_3_3_64_2","first-page":"646","volume-title":"Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics.","author":"Shibuya Mitsuhiro","year":"1999","unstructured":"Mitsuhiro Shibuya, Hajime Kita, and Shigenobu Kobayashi. 1999. Integration of multi-objective and interactive genetic algorithms and its application to animation design. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 646\u2013651."},{"issue":"2","key":"e_1_3_3_65_2","first-page":"47","article-title":"Building trust in artificial intelligence, machine learning, and robotics","volume":"31","author":"Siau Keng","year":"2018","unstructured":"Keng Siau and Weiyu Wang. 2018. Building trust in artificial intelligence, machine learning, and robotics. Cutter Business Technology Journal 31, 2 (2018), 47\u201353.","journal-title":"Cutter Business Technology Journal"},{"issue":"2","key":"e_1_3_3_66_2","doi-asserted-by":"crossref","first-page":"292","DOI":"10.1016\/j.radonc.2007.06.020","article-title":"A new concept for interactive radiotherapy planning with multicriteria optimization: First clinical evaluation","volume":"85","author":"Thieke Christian","year":"2007","unstructured":"Christian Thieke, Karl-Heinz K\u00fcfer, Michael Monz, Alexander Scherrer, Fernando Alonso, Uwe Oelfke, Peter E. Huber, J\u00fcrgen Debus, and Thomas Bortfeld. 2007. A new concept for interactive radiotherapy planning with multicriteria optimization: First clinical evaluation. Radiotherapy and Oncology 85, 2 (2007), 292\u2013298.","journal-title":"Radiotherapy and Oncology"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.5555\/505847"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1137\/1.9780898718515"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1177\/0018720812443825"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3503461","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3503461","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:18Z","timestamp":1750186938000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3503461"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,31]]},"references-count":68,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2022,10,31]]}},"alternative-id":["10.1145\/3503461"],"URL":"https:\/\/doi.org\/10.1145\/3503461","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,31]]},"assertion":[{"value":"2019-06-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}