{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T15:00:43Z","timestamp":1776092443597,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":43,"publisher":"ACM","license":[{"start":{"date-parts":[[2025,4,25]],"date-time":"2025-04-25T00:00:00Z","timestamp":1745539200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,4,26]]},"DOI":"10.1145\/3706598.3714063","type":"proceedings-article","created":{"date-parts":[[2025,4,24]],"date-time":"2025-04-24T03:30:09Z","timestamp":1745465409000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Underspecified Human Decision Experiments Considered Harmful"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6826-3550","authenticated-orcid":false,"given":"Jessica","family":"Hullman","sequence":"first","affiliation":[{"name":"Computer Science, Northwestern University, Evanston, Illinois, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7668-2800","authenticated-orcid":false,"given":"Alex","family":"Kale","sequence":"additional","affiliation":[{"name":"Computer Science, Data Science Institute, University of Chicago, Chicago, Illinois, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5505-6819","authenticated-orcid":false,"given":"Jason","family":"Hartline","sequence":"additional","affiliation":[{"name":"Computer Science, Northwestern University, Evanston, Illinois, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,4,25]]},"reference":[{"key":"e_1_3_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300760"},{"key":"e_1_3_3_3_3_2","unstructured":"Rohan Alur Manish Raghavan and Devavrat Shah. 2024. Distinguishing the Indistinguishable: Human Expertise in Algorithmic Prediction. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2402.00793 (2024)."},{"key":"e_1_3_3_3_4_2","volume-title":"Applying quadratic scoring rule transparently in multiple choice settings: A note","author":"Artinger Florian","year":"2010","unstructured":"Florian Artinger, Filippos Exadaktylos, Hannes Koppel, and Lauri S\u00e4\u00e4ksvuori. 2010. Applying quadratic scoring rule transparently in multiple choice settings: A note. Technical Report. Jena Economic Research Papers."},{"key":"e_1_3_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445717"},{"key":"e_1_3_3_3_6_2","doi-asserted-by":"crossref","unstructured":"Dirk Bergemann and Stephen Morris. 2019. Information design: A unified perspective. Journal of Economic Literature 57 1 (2019) 44\u201395.","DOI":"10.1257\/jel.20181489"},{"key":"e_1_3_3_3_7_2","doi-asserted-by":"crossref","unstructured":"Zana Bu\u00e7inca Maja\u00a0Barbara Malaya and Krzysztof\u00a0Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5 CSCW1 (2021) 1\u201321.","DOI":"10.1145\/3449287"},{"key":"e_1_3_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICHI.2015.26"},{"key":"e_1_3_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300234"},{"key":"e_1_3_3_3_10_2","doi-asserted-by":"crossref","unstructured":"Colin Camerer. 1995. Individual decision making. Handbook of experimental economics (1995).","DOI":"10.1515\/9780691213255-010"},{"key":"e_1_3_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174216"},{"key":"e_1_3_3_3_12_2","volume-title":"Advances in Neural Information Processing Systems","author":"Corvelo\u00a0Benz Nina","year":"2024","unstructured":"Nina Corvelo\u00a0Benz and Manuel Rodriguez. 2024. Human-aligned calibration for ai-assisted decision making. In Advances in Neural Information Processing Systems , Vol.\u00a036."},{"key":"e_1_3_3_3_13_2","doi-asserted-by":"crossref","unstructured":"Berkeley\u00a0J Dietvorst Joseph\u00a0P Simmons and Cade Massey. 2015. Algorithm aversion: people erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General 144 1 (2015) 114.","DOI":"10.1037\/xge0000033"},{"key":"e_1_3_3_3_14_2","doi-asserted-by":"crossref","unstructured":"Evanthia Dimara and John Stasko. 2021. A critical reflection on visualization research: Where do decision making tasks hide? IEEE Transactions on Visualization and Computer Graphics 28 1 (2021) 1128\u20131138.","DOI":"10.1109\/TVCG.2021.3114813"},{"key":"e_1_3_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173718"},{"key":"e_1_3_3_3_16_2","doi-asserted-by":"crossref","unstructured":"Riccardo Fogliato Alexandra Chouldechova and Zachary Lipton. 2021. The impact of algorithmic risk assessments on human predictions and its analysis via crowdsourcing studies. Proceedings of the ACM on Human-Computer Interaction 5 CSCW2 (2021) 1\u201324.","DOI":"10.1145\/3479572"},{"key":"e_1_3_3_3_17_2","doi-asserted-by":"crossref","unstructured":"Daniel\u00a0G Goldstein and David Rothschild. 2014. Lay understanding of probability distributions. Judgment and Decision making 9 1 (2014) 1\u201314.","DOI":"10.1017\/S1930297500004940"},{"key":"e_1_3_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287563"},{"key":"e_1_3_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3630106.3658901"},{"key":"e_1_3_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376454"},{"key":"e_1_3_3_3_21_2","doi-asserted-by":"crossref","unstructured":"Jessica Hullman Matthew Kay Yea-Seul Kim and Samana Shrestha. 2017. Imagining replications: Graphical prediction & discrete visualizations improve recall & estimation of effect uncertainty. IEEE transactions on visualization and computer graphics 24 1 (2017) 446\u2013456.","DOI":"10.1109\/TVCG.2017.2743898"},{"key":"e_1_3_3_3_22_2","doi-asserted-by":"crossref","unstructured":"Jessica Hullman Paul Resnick and Eytan Adar. 2015. Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PloS one 10 11 (2015) e0142444.","DOI":"10.1371\/journal.pone.0142444"},{"key":"e_1_3_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_3_3_3_24_2","doi-asserted-by":"crossref","unstructured":"Alex Kale Matthew Kay and Jessica Hullman. 2020. Visual reasoning strategies for effect size judgments and decisions. IEEE transactions on visualization and computer graphics 27 2 (2020) 272\u2013282.","DOI":"10.1109\/TVCG.2020.3030335"},{"key":"e_1_3_3_3_25_2","doi-asserted-by":"crossref","unstructured":"Yea-Seul Kim Paula Kayongo Madeleine Grunde-McLaughlin and Jessica Hullman. 2020. Bayesian-assisted inference from visualized data. IEEE Transactions on Visualization and Computer Graphics 27 2 (2020) 989\u2013999.","DOI":"10.1109\/TVCG.2020.3028984"},{"key":"e_1_3_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300912"},{"key":"e_1_3_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207678"},{"key":"e_1_3_3_3_28_2","unstructured":"Vivian Lai Chacha Chen Q\u00a0Vera Liao Alison Smith-Renner and Chenhao Tan. 2021. Towards a science of human-ai decision making: a survey of empirical studies. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2112.11471 (2021)."},{"key":"e_1_3_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594087"},{"key":"e_1_3_3_3_30_2","doi-asserted-by":"crossref","unstructured":"Benjamin\u00a0Anders Levinstein. 2017. A pragmatist\u2019s guide to epistemic utility. Philosophy of Science 84 4 (2017) 613\u2013638.","DOI":"10.1086\/693444"},{"key":"e_1_3_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490486.3538338"},{"key":"e_1_3_3_3_32_2","doi-asserted-by":"crossref","unstructured":"Han Liu Vivian Lai and Chenhao Tan. 2021. Understanding the effect of out-of-distribution examples and interactive explanations on human-ai decision making. Proceedings of the ACM on Human-Computer Interaction 5 CSCW2 (2021) 1\u201345.","DOI":"10.1145\/3479552"},{"key":"e_1_3_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i5.20469"},{"key":"e_1_3_3_3_34_2","unstructured":"C.\u00a0Thi Nguyen. 2023. Value Collapse. Presented as the Annual Cardiff Lecture of the Royal Institute of Philosophy. https:\/\/royalinstitutephilosophy.org\/event\/value-collapse\/"},{"key":"e_1_3_3_3_35_2","doi-asserted-by":"crossref","unstructured":"Lace\u00a0M Padilla Grace Hansen Ian\u00a0T Ruginski Heidi\u00a0S Kramer William\u00a0B Thompson and Sarah\u00a0H Creem-Regehr. 2015. The influence of different graphical displays on nonexpert decision making under uncertainty. Journal of Experimental Psychology: Applied 21 1 (2015) 37.","DOI":"10.1037\/xap0000037"},{"key":"e_1_3_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3581641.3584033"},{"key":"e_1_3_3_3_37_2","doi-asserted-by":"crossref","unstructured":"Daniel Read. 2005. Monetary incentives what are they good for? Journal of Economic Methodology 12 2 (2005) 265\u2013276.","DOI":"10.1080\/13501780500086180"},{"key":"e_1_3_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_3_3_39_2","volume-title":"The foundations of statistics","author":"Savage Leonard\u00a0J","year":"1972","unstructured":"Leonard\u00a0J Savage. 1972. The foundations of statistics. Courier Corporation."},{"key":"e_1_3_3_3_40_2","unstructured":"Katie Steele and H\u00a0Orri Stef\u00e1nsson. 2015. Decision theory. (2015)."},{"key":"e_1_3_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1515\/9781400829460"},{"key":"e_1_3_3_3_42_2","unstructured":"Yifan Wu Ziyang Guo Michails Mamakos Jason Hartline and Jessica Hullman. 2023. The Rational Agent Benchmark for Data Visualization. IEEE Transactions on Visualization and Computer Graphics (forthcoming) (2023)."},{"key":"e_1_3_3_3_43_2","unstructured":"Dongping Zhang Jason Hartline and Jessica Hullman. 2023. Designing Shared Information Displays for Agents of Varying Strategic Sophistication. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2310.10858 (2023)."},{"key":"e_1_3_3_3_44_2","doi-asserted-by":"crossref","unstructured":"Alexandra Zytek Dongyu Liu Rhema Vaithianathan and Kalyan Veeramachaneni. 2021. Sibyl: Understanding and addressing the usability challenges of machine learning in high-stakes decision making. IEEE Transactions on Visualization and Computer Graphics 28 1 (2021) 1161\u20131171.","DOI":"10.1109\/TVCG.2021.3114864"}],"event":{"name":"CHI 2025: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '25","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3706598.3714063","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3706598.3714063","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,4]],"date-time":"2025-07-04T05:15:53Z","timestamp":1751606153000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3706598.3714063"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,25]]},"references-count":43,"alternative-id":["10.1145\/3706598.3714063","10.1145\/3706598"],"URL":"https:\/\/doi.org\/10.1145\/3706598.3714063","relation":{},"subject":[],"published":{"date-parts":[[2025,4,25]]},"assertion":[{"value":"2025-04-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}