{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T00:03:36Z","timestamp":1793145816398,"version":"4.5.0"},"reference-count":97,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T00:00:00Z","timestamp":1753142400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T00:00:00Z","timestamp":1753142400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2333553"],"award-info":[{"award-number":["2333553"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008047","name":"Carnegie Mellon University","doi-asserted-by":"publisher","award":["Small Undergraduate Research Grant"],"award-info":[{"award-number":["Small Undergraduate Research Grant"]}],"id":[{"id":"10.13039\/100008047","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008047","name":"Carnegie Mellon University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100008047","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mem Cogn"],"published-print":{"date-parts":[[2026,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    The rise of Large Language Model (LLM) chatbots, such as ChatGPT and Gemini, has revolutionized how we access information. These LLMs can answer a wide array of questions on nearly any topic. When humans answer questions, especially difficult or uncertain questions, they often accompany their responses with metacognitive confidence judgments indicating their belief in their accuracy. LLMs are certainly capable of providing confidence judgments, but it is currently unclear how accurate these confidence judgments are. To fill this gap in the literature, the present studies investigate the capability of LLMs to quantify uncertainty through confidence judgments. We compare the absolute and relative accuracy of confidence judgments made by four LLMs (ChatGPT, Bard\/Gemini, Sonnet, Haiku) and human participants in both domains of aleatory uncertainty\u2014NFL predictions (Study 1;\n                    <jats:italic>n<\/jats:italic>\n                    = 502) and Oscar predictions (Study 2;\n                    <jats:italic>n<\/jats:italic>\n                    = 109)\u2014and domains of epistemic uncertainty\u2014Pictionary performance (Study 3;\n                    <jats:italic>n<\/jats:italic>\n                    = 164), Trivia questions (Study 4;\n                    <jats:italic>n<\/jats:italic>\n                    = 110), and questions about life at a university (Study 5;\n                    <jats:italic>n<\/jats:italic>\n                    = 110). We find several commonalities between LLMs and humans, such as achieving similar levels of absolute and relative metacognitive accuracy (although LLMs tend to be slightly more accurate on both dimensions). Like humans, we also find that LLMs tend to be overconfident. However, we find that, unlike humans, LLMs\u2014especially ChatGPT and Gemini\u2014often fail to adjust their confidence judgments based on past performance, highlighting a key metacognitive limitation.\n                  <\/jats:p>","DOI":"10.3758\/s13421-025-01755-4","type":"journal-article","created":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T13:43:33Z","timestamp":1753191813000},"page":"375-400","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Quantifying uncert-AI-nty: Testing the accuracy of LLMs\u2019 confidence judgments"],"prefix":"10.3758","volume":"54","author":[{"given":"Trent N.","family":"Cash","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel M.","family":"Oppenheimer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sara","family":"Christie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mira","family":"Devgan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,7,22]]},"reference":[{"issue":"3","key":"1755_CR1","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1080\/13546783.2023.2259550","volume":"30","author":"R Ackerman","year":"2024","unstructured":"Ackerman, R., & Levontin, L. (2024). Mindset effects on the regulation of thinking time in problem-solving. Thinking & Reasoning, 30(3), 479\u2013508. https:\/\/doi.org\/10.1080\/13546783.2023.2259550","journal-title":"Thinking & Reasoning"},{"issue":"8","key":"1755_CR2","doi-asserted-by":"publisher","first-page":"607","DOI":"10.1016\/j.tics.2017.05.004","volume":"21","author":"R Ackerman","year":"2017","unstructured":"Ackerman, R., & Thompson, V. A. (2017). Meta-reasoning: Monitoring and control of thinking and reasoning. Trends in Cognitive Sciences, 21(8), 607\u2013617. https:\/\/doi.org\/10.1016\/j.tics.2017.05.004","journal-title":"Trends in Cognitive Sciences"},{"issue":"6","key":"1755_CR3","doi-asserted-by":"publisher","first-page":"1187","DOI":"10.3758\/s13423-012-0305-z","volume":"19","author":"R Ackerman","year":"2012","unstructured":"Ackerman, R., & Zalmanov, H. (2012). The persistence of the fluency\u2013confidence association in problem solving. Psychonomic Bulletin & Review, 19(6), 1187\u20131192. https:\/\/doi.org\/10.3758\/s13423-012-0305-z","journal-title":"Psychonomic Bulletin & Review"},{"key":"1755_CR4","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1016\/j.cognition.2015.10.006","volume":"146","author":"J Ais","year":"2016","unstructured":"Ais, J., Zylberberg, A., Barttfeld, P., & Sigman, M. (2016). Individual consistency in the accuracy and distribution of confidence judgments. Cognition, 146, 377\u2013386. https:\/\/doi.org\/10.1016\/j.cognition.2015.10.006","journal-title":"Cognition"},{"issue":"3","key":"1755_CR5","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1177\/1088868309341564","volume":"13","author":"AL Alter","year":"2009","unstructured":"Alter, A. L., & Oppenheimer, D. M. (2009). Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review, 13(3), 219\u2013235. https:\/\/doi.org\/10.1177\/1088868309341564","journal-title":"Personality and Social Psychology Review"},{"key":"1755_CR6","doi-asserted-by":"publisher","unstructured":"Bandarkar, L., Liang, D., Muller, B., Artetxe, M., Shukla, S. N., Husa, D., . . . Khabsa, M. (2024). The Belebele Benchmark: A parallel reading comprehension dataset in 122 language variants. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 749\u2013775. https:\/\/doi.org\/10.18653\/v1\/2024.acl-long.44","DOI":"10.18653\/v1\/2024.acl-long.44"},{"key":"1755_CR7","doi-asserted-by":"publisher","unstructured":"Banerjee, D., Singh, P., Avadhanam, A., & Srivastava, S. (2023). Benchmarking LLM powered chatbots: Methods and metrics (Version 1). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2308.04624","DOI":"10.48550\/ARXIV.2308.04624"},{"issue":"4","key":"1755_CR8","doi-asserted-by":"publisher","first-page":"412","DOI":"10.3758\/BF03205299","volume":"55","author":"JV Baranski","year":"1994","unstructured":"Baranski, J. V., & Petrusic, W. M. (1994). The calibration and resolution of confidence in perceptual judgments. Perception & Psychophysics, 55(4), 412\u2013428. https:\/\/doi.org\/10.3758\/BF03205299","journal-title":"Perception & Psychophysics"},{"key":"1755_CR9","doi-asserted-by":"publisher","unstructured":"Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6), Article e2218523120. https:\/\/doi.org\/10.1073\/pnas.2218523120","DOI":"10.1073\/pnas.2218523120"},{"issue":"1","key":"1755_CR10","doi-asserted-by":"publisher","first-page":"140","DOI":"10.1111\/nyas.15007","volume":"1525","author":"R Bommasani","year":"2023","unstructured":"Bommasani, R., Liang, P., & Lee, T. (2023). Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525(1), 140\u2013146. https:\/\/doi.org\/10.1111\/nyas.15007","journal-title":"Annals of the New York Academy of Sciences"},{"key":"1755_CR11","first-page":"65","volume-title":"Metacognition, motivation, and understanding","author":"A Brown","year":"1987","unstructured":"Brown, A. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In F. E. Weinert & R. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 65\u2013116). Erlbaum."},{"key":"1755_CR12","doi-asserted-by":"publisher","unstructured":"Brown, N. B. (2024). Enhancing trust in LLMs: Algorithms for comparing and interpreting LLMs (Version 1). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2406.01943","DOI":"10.48550\/ARXIV.2406.01943"},{"issue":"4","key":"1755_CR13","doi-asserted-by":"publisher","first-page":"490","DOI":"10.1037\/mac0000202","volume":"13","author":"TN Cash","year":"2024","unstructured":"Cash, T. N., & Oppenheimer, D. M. (2024). Generative chatbots ain\u2019t experts: Exploring cognitive and metacognitive limitations that hinder expertise in generative chatbots. Journal of Applied Research in Memory and Cognition, 13(4), 490\u2013494. https:\/\/doi.org\/10.1037\/mac0000202","journal-title":"Journal of Applied Research in Memory and Cognition"},{"issue":"2","key":"1755_CR14","doi-asserted-by":"publisher","first-page":"429","DOI":"10.3758\/MC.36.2.429","volume":"36","author":"AD Castel","year":"2008","unstructured":"Castel, A. D. (2008). Metacognition and learning about primacy and recency effects in free recall: The utilization of intrinsic and extrinsic cues when making judgments of learning. Memory & Cognition, 36(2), 429\u2013437. https:\/\/doi.org\/10.3758\/MC.36.2.429","journal-title":"Memory & Cognition"},{"key":"1755_CR15","doi-asserted-by":"publisher","unstructured":"Chen, Y., Liu, T. X., Shan, Y., & Zhong, S. (2023). The emergence of economic rationality of GPT. Proceedings of the National Academy of Sciences, 120(51), Article e2316205120. https:\/\/doi.org\/10.1073\/pnas.2316205120","DOI":"10.1073\/pnas.2316205120"},{"key":"1755_CR16","doi-asserted-by":"publisher","DOI":"10.1007\/978-81-322-3972-7_19","author":"KR Chowdhary","year":"2020","unstructured":"Chowdhary, K. R. (2020). Fundamentals of artificial intelligence. Springer. https:\/\/doi.org\/10.1007\/978-81-322-3972-7_19","journal-title":"Springer"},{"key":"1755_CR17","doi-asserted-by":"publisher","unstructured":"Colombatto, C., & Fleming, S. M. (2025). Illusions of confidence in artificial systems. PsyArXiv Preprints. https:\/\/doi.org\/10.31234\/osf.io\/mjx2v_v2","DOI":"10.31234\/osf.io\/mjx2v_v2"},{"key":"1755_CR18","doi-asserted-by":"publisher","first-page":"34390","DOI":"10.1109\/ACCESS.2024.3370437","volume":"12","author":"I Deandres-Tame","year":"2024","unstructured":"Deandres-Tame, I., Tolosana, R., Vera-Rodriguez, R., Morales, A., Fierrez, J., & Ortega-Garcia, J. (2024). How good is ChatGPT at face biometrics? A first look into recognition, soft biometrics, and explainability. IEEE Access, 12, 34390\u201334401. https:\/\/doi.org\/10.1109\/ACCESS.2024.3370437","journal-title":"IEEE Access"},{"issue":"3","key":"1755_CR19","doi-asserted-by":"publisher","first-page":"837","DOI":"10.2307\/2531595","volume":"44","author":"ER DeLong","year":"1988","unstructured":"DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837\u2013854. https:\/\/doi.org\/10.2307\/2531595","journal-title":"Biometrics"},{"issue":"5","key":"1755_CR20","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1037\/0033-295X.85.5.395","volume":"85","author":"HJ Einhorn","year":"1978","unstructured":"Einhorn, H. J., & Hogarth, R. M. (1978). Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85(5), 395\u2013416. https:\/\/doi.org\/10.1037\/0033-295X.85.5.395","journal-title":"Psychological Review"},{"key":"1755_CR21","doi-asserted-by":"publisher","unstructured":"Elyoseph, Z., Hadar-Shoval, D., Asraf, K., & Lvovsky, M. (2023). ChatGPT outperforms humans in emotional awareness evaluations. Frontiers in Psychology, 14, Article 1199058. https:\/\/doi.org\/10.3389\/fpsyg.2023.1199058","DOI":"10.3389\/fpsyg.2023.1199058"},{"key":"1755_CR22","doi-asserted-by":"publisher","unstructured":"Federiakin, D., Molerov, D., Zlatkin-Troitschanskaia, O., & Maur, A. (2024). Prompt engineering as a new 21st century skill. Frontiers in Education, 9, Article 1366434. https:\/\/doi.org\/10.3389\/feduc.2024.1366434","DOI":"10.3389\/feduc.2024.1366434"},{"key":"1755_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.learninstruc.2014.01.001","volume":"32","author":"B Finn","year":"2014","unstructured":"Finn, B., & Metcalfe, J. (2014). Overconfidence in children\u2019s multi-trial judgments of learning. Learning and Instruction, 32, 1\u20139. https:\/\/doi.org\/10.1016\/j.learninstruc.2014.01.001","journal-title":"Learning and Instruction"},{"issue":"4","key":"1755_CR24","doi-asserted-by":"publisher","first-page":"567","DOI":"10.1007\/s10648-015-9313-7","volume":"27","author":"B Finn","year":"2015","unstructured":"Finn, B., & Tauber, S. K. (2015). When confidence Is not a signal of knowing: How students\u2019 experiences and beliefs about processing fluency can lead to miscalibrated confidence. Educational Psychology Review, 27(4), 567\u2013586. https:\/\/doi.org\/10.1007\/s10648-015-9313-7","journal-title":"Educational Psychology Review"},{"issue":"10","key":"1755_CR25","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1037\/0003-066X.34.10.906","volume":"34","author":"JH Flavell","year":"1979","unstructured":"Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive\u2013developmental inquiry. American Psychologist, 34(10), 906\u2013911. https:\/\/doi.org\/10.1037\/0003-066X.34.10.906","journal-title":"American Psychologist"},{"key":"1755_CR26","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1146\/annurev-psych-022423-032425","volume":"75","author":"SM Fleming","year":"2024","unstructured":"Fleming, S. M. (2024). Metacognition and Confidence: A Review and Synthesis. Annual Review of Psychology, 75, 241\u2013268. https:\/\/doi.org\/10.1146\/annurev-psych-022423-032425","journal-title":"Annual Review of Psychology"},{"key":"1755_CR27","doi-asserted-by":"publisher","unstructured":"Fleming, S. M., & Lau, H. C. (2014). How to measure metacognition. Frontiers in Human Neuroscience, 8(JULY). https:\/\/doi.org\/10.3389\/fnhum.2014.00443","DOI":"10.3389\/fnhum.2014.00443"},{"key":"1755_CR28","doi-asserted-by":"publisher","unstructured":"Fleming, S. M., Massoni, S., Gajdos, T., & Vergnaud, J.-C. (2016). Metacognition about the past and future: Quantifying common and distinct influences on prospective and retrospective judgments of self-performance. Neuroscience of Consciousness, 2016(1), Article niw018. https:\/\/doi.org\/10.1093\/nc\/niw018","DOI":"10.1093\/nc\/niw018"},{"key":"1755_CR29","first-page":"21","volume-title":"Perspectives on thinking, judging, and decision making","author":"CR Fox","year":"2011","unstructured":"Fox, C. R., & \u00dclk\u00fcmen, G. (2011). Distinguishing two dimensions of uncertainty. In W. Brun, G. Keren, G. Kirkeb\u00f8en, & H. Montgomery (Eds.), Perspectives on thinking, judging, and decision making (pp. 21\u201335). Universitetsforlaget."},{"key":"1755_CR30","doi-asserted-by":"crossref","unstructured":"Groot, T., & Valdenegro-Toro, M. (2024). Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. arXiv. http:\/\/arxiv.org\/abs\/2405.02917","DOI":"10.18653\/v1\/2024.trustnlp-1.13"},{"issue":"10","key":"1755_CR31","doi-asserted-by":"publisher","first-page":"833","DOI":"10.1038\/s43588-023-00527-x","volume":"3","author":"T Hagendorff","year":"2023","unstructured":"Hagendorff, T., Fabi, S., & Kosinski, M. (2023). Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT. Nature Computational Science, 3(10), 833\u2013838. https:\/\/doi.org\/10.1038\/s43588-023-00527-x","journal-title":"Nature Computational Science"},{"key":"1755_CR32","doi-asserted-by":"publisher","unstructured":"Han, Y., & Dunning, D. (2024). Metaknowledge of experts versus nonexperts: Do experts know better what they do and do not know? Journal of Behavioral Decision Making, 37(2), Article e2375. https:\/\/doi.org\/10.1002\/bdm.2375","DOI":"10.1002\/bdm.2375"},{"issue":"2","key":"1755_CR33","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1016\/S1364-6613(97)01014-0","volume":"1","author":"N Harvey","year":"1997","unstructured":"Harvey, N. (1997). Confidence in judgment. Trends in Cognitive Sciences, 1(2), 78\u201382. https:\/\/doi.org\/10.1016\/S1364-6613(97)01014-0","journal-title":"Trends in Cognitive Sciences"},{"issue":"1","key":"1755_CR34","doi-asserted-by":"publisher","first-page":"108","DOI":"10.3758\/s13428-018-1125-5","volume":"51","author":"PA Higham","year":"2019","unstructured":"Higham, P. A., & Higham, D. P. (2019). New improved gamma: Enhancing the accuracy of Goodman\u2013Kruskal\u2019s gamma using ROC curves. Behavior Research Methods, 51(1), 108\u2013125. https:\/\/doi.org\/10.3758\/s13428-018-1125-5","journal-title":"Behavior Research Methods"},{"issue":"4","key":"1755_CR35","doi-asserted-by":"publisher","first-page":"1524","DOI":"10.3758\/s13423-022-02066-4","volume":"29","author":"L Hildenbrand","year":"2022","unstructured":"Hildenbrand, L., & Sanchez, C. A. (2022). Metacognitive accuracy across cognitive and physical task domains. Psychonomic Bulletin & Review, 29(4), 1524\u20131530. https:\/\/doi.org\/10.3758\/s13423-022-02066-4","journal-title":"Psychonomic Bulletin & Review"},{"issue":"6245","key":"1755_CR36","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1126\/science.aaa8685","volume":"349","author":"J Hirschberg","year":"2015","unstructured":"Hirschberg, J., & Manning, C. D. (2015). Advances in natural language processing. Science, 349(6245), 261\u2013266. https:\/\/doi.org\/10.1126\/science.aaa8685","journal-title":"Science"},{"key":"1755_CR37","doi-asserted-by":"publisher","unstructured":"Horton, J. (2023). Large Language Models as simulated economic agents: What can we learn from Homo Silicus? [Working Paper 31122]. National Bureau of Economic Research. https:\/\/doi.org\/10.3386\/w31122","DOI":"10.3386\/w31122"},{"issue":"2","key":"1755_CR38","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1136\/amiajnl-2011-000291","volume":"19","author":"X Jiang","year":"2012","unstructured":"Jiang, X., Osl, M., Kim, J., & Ohno-Machado, L. (2012). Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19(2), 263\u2013274. https:\/\/doi.org\/10.1136\/amiajnl-2011-000291","journal-title":"Journal of the American Medical Informatics Association"},{"key":"1755_CR39","doi-asserted-by":"publisher","unstructured":"Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2020). How can we know when language models know? On the calibration of language models for question answering (Version 2). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2012.00955","DOI":"10.48550\/ARXIV.2012.00955"},{"issue":"7364","key":"1755_CR40","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1038\/nature10384","volume":"477","author":"DDP Johnson","year":"2011","unstructured":"Johnson, D. D. P., & Fowler, J. H. (2011). The evolution of overconfidence. Nature, 477(7364), 317\u2013320. https:\/\/doi.org\/10.1038\/nature10384","journal-title":"Nature"},{"key":"1755_CR41","doi-asserted-by":"publisher","unstructured":"Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., . . . Kaplan, J. (2022). Language models (mostly) know what they know (Version 4). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2207.05221","DOI":"10.48550\/ARXIV.2207.05221"},{"issue":"4","key":"1755_CR42","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1037\/0096-3445.126.4.349","volume":"126","author":"A Koriat","year":"1997","unstructured":"Koriat, A. (1997). Monitoring one\u2019s own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4), 349\u2013370. https:\/\/doi.org\/10.1037\/0096-3445.126.4.349","journal-title":"Journal of Experimental Psychology: General"},{"key":"1755_CR43","doi-asserted-by":"publisher","unstructured":"Kuhn, L., Gal, Y., & Farquhar, S. (2023). Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation (Version 3). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2302.09664","DOI":"10.48550\/ARXIV.2302.09664"},{"key":"1755_CR44","doi-asserted-by":"publisher","unstructured":"Kumar, A., Liang, P., & Ma, T. (2019). Verified uncertainty calibration (Version 2). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.1909.10155","DOI":"10.48550\/ARXIV.1909.10155"},{"key":"1755_CR45","doi-asserted-by":"publisher","unstructured":"Lee, M. H. J., Montgomery, J. M., & Lai, C. K. (2024). Large Language Models portray socially subordinate groups as more homogeneous, consistent with a bias observed in humans. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 1321\u20131340. https:\/\/doi.org\/10.1145\/3630106.3658975","DOI":"10.1145\/3630106.3658975"},{"key":"1755_CR46","unstructured":"Li, Y., Huang, Y., Lin, Y., Wu, S., Wan, Y., & Sun, L. (2024). I think, therefore I am: Benchmarking awareness of Large Language Models using AwareBench. arXiv. http:\/\/arxiv.org\/abs\/2401.17882"},{"key":"1755_CR47","doi-asserted-by":"publisher","unstructured":"Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., . . . Koreeda, Y. (2022). Holistic evaluation of language models. arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2211.09110","DOI":"10.48550\/ARXIV.2211.09110"},{"key":"1755_CR48","doi-asserted-by":"publisher","unstructured":"Lin, S., Hilton, J., & Evans, O. (2022). Teaching models to express their uncertainty in words (Version 2). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2205.14334","DOI":"10.48550\/ARXIV.2205.14334"},{"key":"1755_CR49","doi-asserted-by":"publisher","unstructured":"Long, R. (2023). Introspective capabilities in large language models. Journal of Consciousness Studies, 30(9\/10), 143\u2013153. https:\/\/doi.org\/10.53765\/20512201.30.9.143","DOI":"10.53765\/20512201.30.9.143"},{"key":"1755_CR50","doi-asserted-by":"publisher","unstructured":"Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., . . . Gao, J. (2023). MathVista: Evaluating mathematical reasoning of foundation models in visual contexts (Version 3). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2310.02255","DOI":"10.48550\/ARXIV.2310.02255"},{"key":"1755_CR51","unstructured":"Major League Baseball. (2025). Stats. https:\/\/www.mlb.com\/stats\/2024"},{"issue":"1","key":"1755_CR52","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1016\/j.concog.2011.09.021","volume":"21","author":"B Maniscalco","year":"2012","unstructured":"Maniscalco, B., & Lau, H. (2012). A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Consciousness and Cognition, 21(1), 422\u2013430. https:\/\/doi.org\/10.1016\/j.concog.2011.09.021","journal-title":"Consciousness and Cognition"},{"issue":"2","key":"1755_CR53","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1037\/0033-295X.115.2.502","volume":"115","author":"DA Moore","year":"2008","unstructured":"Moore, D. A., & Healy, P. J. (2008). The trouble with overconfidence. Psychological Review, 115(2), 502\u2013517. https:\/\/doi.org\/10.1037\/0033-295X.115.2.502","journal-title":"Psychological Review"},{"key":"1755_CR54","doi-asserted-by":"publisher","unstructured":"Moore, D. A., & Schatz, D. (2017). The three faces of overconfidence. Social and Personality Psychology Compass, 11(8), Article e12331. https:\/\/doi.org\/10.1111\/spc3.12331","DOI":"10.1111\/spc3.12331"},{"issue":"6","key":"1755_CR55","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1037\/xlm0000213","volume":"42","author":"K Murayama","year":"2016","unstructured":"Murayama, K., Blake, A. B., Kerr, T., & Castel, A. D. (2016). When enough is not enough: Information overload and metacognitive decisions to stop studying information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(6), 914\u2013924. https:\/\/doi.org\/10.1037\/xlm0000213","journal-title":"Journal of Experimental Psychology: Learning, Memory, and Cognition"},{"issue":"5","key":"1755_CR56","doi-asserted-by":"publisher","first-page":"544","DOI":"10.1136\/amiajnl-2011-000464","volume":"18","author":"PM Nadkarni","year":"2011","unstructured":"Nadkarni, P. M., Ohno-Machado, L., & Chapman, W. W. (2011). Natural language processing: An introduction. Journal of the American Medical Informatics Association, 18(5), 544\u2013551. https:\/\/doi.org\/10.1136\/amiajnl-2011-000464","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"1","key":"1755_CR57","doi-asserted-by":"publisher","first-page":"25","DOI":"10.4097\/kja.21209","volume":"75","author":"FS Nahm","year":"2022","unstructured":"Nahm, F. S. (2022). Receiver operating characteristic curve: Overview and practical use for clinicians. Korean Journal of Anesthesiology, 75(1), 25\u201336. https:\/\/doi.org\/10.4097\/kja.21209","journal-title":"Korean Journal of Anesthesiology"},{"issue":"3","key":"1755_CR58","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1037\/0033-295X.84.3.231","volume":"84","author":"RE Nisbett","year":"1977","unstructured":"Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231\u2013259. https:\/\/doi.org\/10.1037\/0033-295X.84.3.231","journal-title":"Psychological Review"},{"key":"1755_CR59","doi-asserted-by":"publisher","unstructured":"Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V., & Daneshjou, R. (2023). Large language Models propagate race-based medicine. Npj Digital Medicine, 6(1), Article 195. https:\/\/doi.org\/10.1038\/s41746-023-00939-z","DOI":"10.1038\/s41746-023-00939-z"},{"key":"1755_CR60","unstructured":"OpenAI. (2024). How ChatGPT and our foundation models are developed. https:\/\/help.openai.com\/en\/articles\/7842364-how-chatgpt-and-our-foundation-models-are-developed"},{"issue":"6","key":"1755_CR61","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.tics.2008.02.014","volume":"12","author":"DM Oppenheimer","year":"2008","unstructured":"Oppenheimer, D. M. (2008). The secret life of fluency. Trends in Cognitive Sciences, 12(6), 237\u2013241. https:\/\/doi.org\/10.1016\/j.tics.2008.02.014","journal-title":"Trends in Cognitive Sciences"},{"key":"1755_CR62","unstructured":"Pictionary Generator. (2024). https:\/\/randomwordgenerator.com\/pictionary.php"},{"issue":"1","key":"1755_CR63","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/s11409-008-9030-4","volume":"4","author":"S Pieschl","year":"2009","unstructured":"Pieschl, S. (2009). Metacognitive calibration\u2014An extended conceptualization and potential applications. Metacognition and Learning, 4(1), 3\u201331. https:\/\/doi.org\/10.1007\/s11409-008-9030-4","journal-title":"Metacognition and Learning"},{"issue":"4","key":"1755_CR64","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1016\/j.jesp.2006.05.011","volume":"43","author":"E Pronin","year":"2007","unstructured":"Pronin, E., & Kugler, M. B. (2007). Valuing thoughts, ignoring behavior: The introspection illusion as a source of the bias blind spot. Journal of Experimental Social Psychology, 43(4), 565\u2013578. https:\/\/doi.org\/10.1016\/j.jesp.2006.05.011","journal-title":"Journal of Experimental Social Psychology"},{"key":"1755_CR65","doi-asserted-by":"publisher","unstructured":"Renze, M. (2024). The effect of sampling temperature on problem solving in Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2024, 7346\u20137356. https:\/\/doi.org\/10.18653\/v1\/2024.findings-emnlp.432","DOI":"10.18653\/v1\/2024.findings-emnlp.432"},{"key":"1755_CR66","doi-asserted-by":"publisher","unstructured":"Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & M\u00fcller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(1), Article 77. https:\/\/doi.org\/10.1186\/1471-2105-12-77","DOI":"10.1186\/1471-2105-12-77"},{"issue":"6","key":"1755_CR67","doi-asserted-by":"publisher","first-page":"6258","DOI":"10.3758\/s13428-024-02357-9","volume":"56","author":"C Rodriguez","year":"2024","unstructured":"Rodriguez, C., & Oppenheimer, D. M. (2024). Creating a Bot-tleneck for malicious AI: Psychological methods for bot detection. Behavior Research Methods, 56(6), 6258\u20136275. https:\/\/doi.org\/10.3758\/s13428-024-02357-9","journal-title":"Behavior Research Methods"},{"key":"1755_CR68","unstructured":"Roose, K. (2023, March 8). How does ChatGPT really work? The New York Times. https:\/\/www.nytimes.com\/2023\/03\/28\/technology\/ai-chatbots-chatgpt-bing-bard-llm.html"},{"key":"1755_CR69","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","volume":"61","author":"J Schmidhuber","year":"2015","unstructured":"Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85\u2013117. https:\/\/doi.org\/10.1016\/j.neunet.2014.09.003","journal-title":"Neural Networks"},{"key":"1755_CR70","unstructured":"Schoenegger, P., Park, P. S., Karger, E., & Tetlock, P. E. (2024). AI-augmented predictions: LLM assistants improve human forecasting accuracy. arXiv. http:\/\/arxiv.org\/abs\/2402.07862"},{"issue":"1","key":"1755_CR71","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1007\/s11409-008-9031-3","volume":"4","author":"G Schraw","year":"2009","unstructured":"Schraw, G. (2009). A conceptual analysis of five measures of metacognitive monitoring. Metacognition and Learning, 4(1), 33\u201345. https:\/\/doi.org\/10.1007\/s11409-008-9031-3","journal-title":"Metacognition and Learning"},{"key":"1755_CR72","doi-asserted-by":"publisher","DOI":"10.1037\/xap0000525","author":"TM Seale-Carlisle","year":"2024","unstructured":"Seale-Carlisle, T. M., Grabman, J. H., Dobolyi, D. G., & Dodson, C. S. (2024). A comparison between numeric confidence ratings and verbal confidence statements. Journal of Experimental Psychology: Applied. https:\/\/doi.org\/10.1037\/xap0000525","journal-title":"Journal of Experimental Psychology: Applied."},{"issue":"7987","key":"1755_CR73","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1038\/s41586-023-06647-8","volume":"623","author":"M Shanahan","year":"2023","unstructured":"Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models. Nature, 623(7987), 493\u2013498. https:\/\/doi.org\/10.1038\/s41586-023-06647-8","journal-title":"Nature"},{"issue":"2","key":"1755_CR74","first-page":"165","volume":"132","author":"WE Shields","year":"2005","unstructured":"Shields, W. E., Smith, J. D., Guttmannova, K., & Washburn, D. A. (2005). Confidence judgments by humans and rhesus monkeys. The Journal of General Psychology, 132(2), 165\u2013186.","journal-title":"The Journal of General Psychology"},{"key":"1755_CR75","doi-asserted-by":"publisher","unstructured":"Shrivastava, V., Liang, P., & Kumar, A. (2023). Llamas know what GPTs don\u2019t show: Surrogate models for confidence estimation (Version 1). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2311.08877","DOI":"10.48550\/ARXIV.2311.08877"},{"key":"1755_CR76","doi-asserted-by":"publisher","unstructured":"Siedlecka, M., Paulewicz, B., & Wierzcho\u0144, M. (2016). But I was so sure! Metacognitive judgments are less accurate given prospectively than retrospectively. Frontiers in Psychology, 7. https:\/\/doi.org\/10.3389\/fpsyg.2016.00218","DOI":"10.3389\/fpsyg.2016.00218"},{"issue":"4","key":"1755_CR77","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1037\/0096-3445.124.4.391","volume":"124","author":"JD Smith","year":"1995","unstructured":"Smith, J. D., Schull, J., Strote, J., McGee, K., Egnor, R., & Erb, L. (1995). The uncertain response in bottlenosed dolphin (Tursiops truncatus). Journal of Experimental Psychology: General, 124(4), 391\u2013408. https:\/\/doi.org\/10.1037\/0096-3445.124.4.391","journal-title":"Journal of Experimental Psychology: General"},{"key":"1755_CR78","doi-asserted-by":"publisher","unstructured":"Spangher, L., Li, T., Arnold, W. F., Masiewicki, N., Dotiwalla, X., Parusmathi, R., Grabowski, P., . . . Gruhl, D. (2024). Project MPG: Towards a generalized performance benchmark for LLM capabilities (Version 1). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2410.22368","DOI":"10.48550\/ARXIV.2410.22368"},{"key":"1755_CR79","doi-asserted-by":"publisher","unstructured":"Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., . . . Smyth, P. (2025). What large language models know and what people think they know. Nature Machine Intelligence, 7, 221\u2013231. https:\/\/doi.org\/10.1038\/s42256-024-00976-7","DOI":"10.1038\/s42256-024-00976-7"},{"key":"1755_CR80","doi-asserted-by":"publisher","unstructured":"Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., . . . Becchio, C. (2024). Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7), 1285\u20131295. https:\/\/doi.org\/10.1038\/s41562-024-01882-z","DOI":"10.1038\/s41562-024-01882-z"},{"key":"1755_CR81","doi-asserted-by":"publisher","first-page":"3132","DOI":"10.1016\/j.procs.2024.09.359","volume":"246","author":"T Takasuka","year":"2024","unstructured":"Takasuka, T., Takano, Y. H., Watanabe, S., & Kumoi, G. (2024). Recognition performance validation of weather map images by ChatGPT. Procedia Computer Science, 246, 3132\u20133141. https:\/\/doi.org\/10.1016\/j.procs.2024.09.359","journal-title":"Procedia Computer Science"},{"key":"1755_CR82","doi-asserted-by":"publisher","unstructured":"Talboy, A. N., & Fuller, E. (2023). Challenging the appearance of machine intelligence: Cognitive bias in LLMs and best practices for Adoption. arXiv. https:\/\/doi.org\/10.48550\/arXiv.2304.01358","DOI":"10.48550\/arXiv.2304.01358"},{"issue":"3","key":"1755_CR83","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1016\/j.cogpsych.2011.06.001","volume":"63","author":"VA Thompson","year":"2011","unstructured":"Thompson, V. A., Prowse Turner, J. A., & Pennycook, G. (2011). Intuition, reason, and metacognition. Cognitive Psychology, 63(3), 107\u2013140. https:\/\/doi.org\/10.1016\/j.cogpsych.2011.06.001","journal-title":"Cognitive Psychology"},{"issue":"2","key":"1755_CR84","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.cognition.2012.09.012","volume":"128","author":"VA Thompson","year":"2013","unstructured":"Thompson, V. A., Turner, J. A. P., Pennycook, G., Ball, L. J., Brack, H., Ophir, Y., & Ackerman, R. (2013). The role of answer fluency and perceptual fluency as metacognitive cues for initiating analytic thinking. Cognition, 128(2), 237\u2013251. https:\/\/doi.org\/10.1016\/j.cognition.2012.09.012","journal-title":"Cognition"},{"key":"1755_CR85","doi-asserted-by":"publisher","unstructured":"Tian, D., Jiang, S., Zhang, L., Lu, X., & Xu, Y. (2024). The role of large language models in medical image processing: A narrative review. Quantitative Imaging in Medicine and Surgery, 14(1), 1108\u20131121. https:\/\/doi.org\/10.21037\/qims-23-892","DOI":"10.21037\/qims-23-892"},{"key":"1755_CR86","doi-asserted-by":"publisher","unstructured":"Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., . . . Manning, C. D. (2023). Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback (Version 2). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2305.14975","DOI":"10.48550\/ARXIV.2305.14975"},{"key":"1755_CR87","doi-asserted-by":"publisher","unstructured":"Wan, Y., Pu, G., Sun, J., Garimella, A., Chang, K.-W., & Peng, N. (2023). \u201cKelly is a warm person, Joseph is a role model\u201d: Gender biases in LLM-generated reference letters. Findings of the Association for Computational Linguistics: EMNLP 2023, 3730\u20133748. https:\/\/doi.org\/10.18653\/v1\/2023.findings-emnlp.243","DOI":"10.18653\/v1\/2023.findings-emnlp.243"},{"key":"1755_CR88","doi-asserted-by":"publisher","unstructured":"Wang, C. (2023). Calibration in deep learning: A survey of the state-of-the-art (Version 3). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2308.01222","DOI":"10.48550\/ARXIV.2308.01222"},{"key":"1755_CR89","doi-asserted-by":"publisher","unstructured":"Wang, Y., & Zhao, Y. (2023). Metacognitive prompting improves understanding in large language models (Version 4). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2308.05342","DOI":"10.48550\/ARXIV.2308.05342"},{"issue":"3","key":"1755_CR90","doi-asserted-by":"publisher","first-page":"387","DOI":"10.3758\/BF03210798","volume":"4","author":"RF West","year":"1997","unstructured":"West, R. F., & Stanovich, K. E. (1997). The domain specificity and generality of overconfidence: Individual differences in performance estimation bias. Psychonomic Bulletin & Review, 4(3), 387\u2013392. https:\/\/doi.org\/10.3758\/BF03210798","journal-title":"Psychonomic Bulletin & Review"},{"key":"1755_CR91","doi-asserted-by":"publisher","unstructured":"White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., . . . Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT (Version 1). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2302.11382","DOI":"10.48550\/ARXIV.2302.11382"},{"key":"1755_CR92","doi-asserted-by":"crossref","unstructured":"Wolfram, S. (2023). What is ChatGPT doing: ... And what does it work? Wolfram Media, Inc.","DOI":"10.31855\/bc47ee6b-75c"},{"issue":"5","key":"1755_CR93","doi-asserted-by":"publisher","first-page":"1122","DOI":"10.1109\/JAS.2023.123618","volume":"10","author":"T Wu","year":"2023","unstructured":"Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q.-L., & Tang, Y. (2023). A brief overview of ChatGPT: The history, status quo and potential future development. IEEE\/CAA Journal of Automatica Sinica, 10(5), 1122\u20131136. https:\/\/doi.org\/10.1109\/JAS.2023.123618","journal-title":"IEEE\/CAA Journal of Automatica Sinica"},{"key":"1755_CR94","doi-asserted-by":"publisher","unstructured":"Xiao, Y., Liang, P. P., Bhatt, U., Neiswanger, W., Salakhutdinov, R., & Morency, L.-P. (2022). Uncertainty quantification with pre-trained language models: A large-scale empirical analysis (Version 2). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2210.04714","DOI":"10.48550\/ARXIV.2210.04714"},{"key":"1755_CR95","doi-asserted-by":"publisher","unstructured":"Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2023). Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs (Version 2). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2306.13063","DOI":"10.48550\/ARXIV.2306.13063"},{"key":"1755_CR96","doi-asserted-by":"publisher","unstructured":"Yu, J., Wang, X., Tu, S., Cao, S., Zhang-Li, D., Lv, X., Peng, H., . . . Li, J. (2023). KoLA: Carefully benchmarking world knowledge of Large Language Models (Version 3). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2306.09296","DOI":"10.48550\/ARXIV.2306.09296"},{"key":"1755_CR97","doi-asserted-by":"publisher","unstructured":"Zhu, J.-Q., & Griffiths, T. L. (2024). Incoherent probability judgments in Large Language Models (Version 1). arXiv. https:\/\/doi.org\/10.48550\/ARXIV.2401.16646","DOI":"10.48550\/ARXIV.2401.16646"}],"container-title":["Memory &amp; Cognition"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13421-025-01755-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13421-025-01755-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13421-025-01755-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T18:03:59Z","timestamp":1772561039000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.3758\/s13421-025-01755-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,22]]},"references-count":97,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,2]]}},"alternative-id":["1755"],"URL":"https:\/\/doi.org\/10.3758\/s13421-025-01755-4","relation":{},"ISSN":["0090-502X","1532-5946"],"issn-type":[{"value":"0090-502X","type":"print"},{"value":"1532-5946","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,22]]},"assertion":[{"value":"17 June 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 July 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Approval was obtained from the Institutional Review Board at Carnegie Mellon University (Study #2017_00000367). The procedures used in this study adhere to the tenets of the Declaration of Helsinki.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}},{"value":"Informed consent was obtained from all individual participants included in the study.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Patients consented to the publication of their anonymized data.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The data and materials for all studies are available [\n                      \n                      ]. Study 2 [\n                      \n                      ], Study 3 [\n                      \n                      ], Study 4 [\n                      \n                      ], and Study 5 [\n                      \n                      ] were preregistered on AsPredicted.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Open practices statement"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interests\/Competing interests"}},{"value":"Studies 2, 3, 4, and 5 were preregistered (see links in manuscript). Data,\n                      \n                      , and analysis code for this manuscript are available online (\n                      \n                      ). This study was funded in part by a Doctoral Dissertation Research Improvement Grant from the National Science Foundation (Award #2333553) and a Small Undergraduate Research Grant from Carnegie Mellon University. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of our grantors. We declare no conflicts of interest. We acknowledge the support of the research assistants, friends, and family members who made this research possible.","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Author note"}}]}}