{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T02:58:21Z","timestamp":1792810701444,"version":"4.5.0"},"reference-count":41,"publisher":"Cambridge University Press (CUP)","issue":"4","license":[{"start":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T00:00:00Z","timestamp":1715904000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Polit. Anal."],"published-print":{"date-parts":[[2024,10]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Large language models (LLMs) offer new research possibilities for social scientists, but their potential as \u201csynthetic data\u201d is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different \u201cpersonas\u201d and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016\u20132020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.<\/jats:p>","DOI":"10.1017\/pan.2024.5","type":"journal-article","created":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T04:33:40Z","timestamp":1715920420000},"page":"401-416","source":"Crossref","is-referenced-by-count":182,"title":["Synthetic Replacements for Human Survey Data? The Perils of Large Language Models"],"prefix":"10.1017","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3334-2416","authenticated-orcid":false,"given":"James","family":"Bisbee","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua D.","family":"Clinton","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cassy","family":"Dorff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brenton","family":"Kenkel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer M.","family":"Larson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2024,5,17]]},"reference":[{"key":"S1047198724000056_r33","unstructured":"Shapiro, W. 2019. \u201cThe Polling Industry Is in Crisis.\u201d https:\/\/newrepublic.com\/article\/154124\/polling-industry-crisis."},{"key":"S1047198724000056_r38","unstructured":"Washington Post. 2023. \u201cAI Chatbots Are Learning to Hold More Natural Conversations.\u201d https:\/\/www.washingtonpost.com\/technology\/interactive\/2023\/ai-chatbot-learning\/."},{"key":"S1047198724000056_r3","first-page":"1","article-title":"Out of One, Many: Using Language Models to Simulate Human Samples","author":"Argyle","year":"2023","journal-title":"Political Analysis"},{"key":"S1047198724000056_r12","unstructured":"Clinton, J. D. , et al. 2021a. \u201cAmerican Association of Public Opinion Research Task Force on Pre-Election Polling: An Evaluation of the 2020 General Election Polls.\u201d https:\/\/aapor.org\/wp-content\/uploads\/2022\/11\/Task-Force-on-2020-Pre-Election-Polling_Executive-Summary.pdf."},{"key":"S1047198724000056_r5","doi-asserted-by":"publisher","DOI":"10.1111\/ajps.12274"},{"key":"S1047198724000056_r16","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfz003"},{"key":"S1047198724000056_r14","unstructured":"DeBell, M. , and Krosnick, J. A. . 2009. \u201cComputing Weights for American National Election Study Survey Data.\u201d ANES Technical Report Series, No. nes012427."},{"key":"S1047198724000056_r37","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-polisci-050517-114628"},{"key":"S1047198724000056_r1","unstructured":"Abdulhai, M. , Serapio-Garcia, G. , Crepy, C. , Valter, D. , Canny, J. and Jaques, N. . 2023. \u201cMoral Foundations of Large Language Models\u201d. Preprint, arXiv:2310.15337"},{"key":"S1047198724000056_r18","doi-asserted-by":"publisher","DOI":"10.1511\/2014.111.460"},{"key":"S1047198724000056_r28","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfv045"},{"key":"S1047198724000056_r2","unstructured":"Aher, G. V. , Arriaga, R. I. and Kalai, A. T. . 2023. \u201cUsing Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies\u201d. In International Conference on Machine Learning 337\u2013371. PMLR."},{"key":"S1047198724000056_r24","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-polisci-051117-073034"},{"key":"S1047198724000056_r35","unstructured":"T\u00f6rnberg, P. 2023. \u201cChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning.\u201d Preprint, arXiv:2304.06588."},{"key":"S1047198724000056_r17","first-page":"127","article-title":"Poststratification into Many Categories Using Hierarchical Logistic Regression","volume":"23","author":"Gelman","year":"1997","journal-title":"Survey Methodology"},{"key":"S1047198724000056_r20","doi-asserted-by":"crossref","unstructured":"Gilardi, F. , Alizadeh, M. , and Kubli, M. . 2023. \u201cChatGPT outperforms crowd workers for text-annotation tasks\u201d. In Proceedings of the National Academy of Sciences, 120(30), e2305016120.","DOI":"10.1073\/pnas.2305016120"},{"key":"S1047198724000056_r6","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"S1047198724000056_r22","unstructured":"Graham, D. A. 2023. \u201cThe Polling Crisis Is a Catastrophe for American Democracy\u2014theatlantic.com.\u201d https:\/\/www.theatlantic.com\/ideas\/archive\/2020\/11\/polling-catastrophe\/616986\/."},{"key":"S1047198724000056_r9","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.c3nlp-1.7"},{"key":"S1047198724000056_r25","volume-title":"What Low Response Rates Mean for Telephone Surveys","author":"Keeter","year":"2017"},{"key":"S1047198724000056_r39","unstructured":"Wei, J. , Tay, Y. , Bommasani, R. , Raffel, C. , Zoph, B. , Borgeaud, S. , Yogatama, D. , Bosma, M. , Zhou, D. , Metzler, D. , and Chi, E.H. . 2022. \u201cEmergent Abilities of Large Language Models.\u201d Preprint, arXiv:2206.07682."},{"key":"S1047198724000056_r27","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-5907.2008.00360.x"},{"key":"S1047198724000056_r36","first-page":"1","volume-title":"The Oxford Handbook of Polling and Polling Methods","author":"Tucker","year":"2017"},{"key":"S1047198724000056_r29","doi-asserted-by":"publisher","DOI":"10.1007\/s11127-023-01097-2"},{"key":"S1047198724000056_r10","doi-asserted-by":"publisher","DOI":"10.1093\/pan\/mpu021"},{"key":"S1047198724000056_r26","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfx047"},{"key":"S1047198724000056_r4","volume-title":"Breaking the Social Media Prism: How to Make Our Platforms Less Polarizing","author":"Bail","year":"2022"},{"key":"S1047198724000056_r8","doi-asserted-by":"publisher","DOI":"10.7910\/DVN\/VPN481"},{"key":"S1047198724000056_r15","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2023.04.008"},{"key":"S1047198724000056_r40","unstructured":"White, J. , Fu, Q. , Hays, S. , Sandborn, M. , Olea, C. , Gilbert, H. , Elnashar, A. , Spencer-Smith, J. , and Schmidt, D.C. . 2023. \u201cA Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.\u201d Preprint, arXiv:2302.11382."},{"key":"S1047198724000056_r7","doi-asserted-by":"publisher","DOI":"10.1017\/S0003055419000480"},{"key":"S1047198724000056_r23","doi-asserted-by":"publisher","DOI":"10.3386\/w31122"},{"key":"S1047198724000056_r41","unstructured":"Wu, P. Y. , Tucker, J. A. , Nagler, J. , and Messing, S. . 2023. \u201cLarge Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting.\u201d Preprint, arXiv:2303.12057."},{"key":"S1047198724000056_r21","doi-asserted-by":"publisher","DOI":"10.1017\/S0003055423000035"},{"key":"S1047198724000056_r32","unstructured":"Santurkar, S. , Durmus, E. , Ladhak, F. , Lee, C. , Liang, P. , and Hashimoto, T. . 2023. \u201cWhose Opinions Do Language Models Reflect?\u201d In International Conference on Machine Learning 29971\u201330004. PMLR."},{"key":"S1047198724000056_r11","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfac011"},{"key":"S1047198724000056_r13","unstructured":"Cowen, T. 2022. \u201cChatGPT AI Could Make Democracy Even More Messy.\u201d https:\/\/www.bloomberg.com\/opinion\/articles\/2022-12-06\/chatgpt-ai-could-make-democracy-even-more-messy."},{"key":"S1047198724000056_r31","doi-asserted-by":"publisher","DOI":"10.3390\/socsci12030148"},{"key":"S1047198724000056_r30","unstructured":"OpenAI. 2021. \u201cChatGPT 3.5 Turbo.\u201d https:\/\/openai.com\/blog\/chat-gpt-3-5-turbo\/."},{"key":"S1047198724000056_r19","doi-asserted-by":"publisher","DOI":"10.1111\/ajps.12004"},{"key":"S1047198724000056_r34","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-023-01295-4"}],"container-title":["Political Analysis"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1047198724000056","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,16]],"date-time":"2025-06-16T01:39:41Z","timestamp":1750037981000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1047198724000056\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,17]]},"references-count":41,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,10]]}},"alternative-id":["S1047198724000056"],"URL":"https:\/\/doi.org\/10.1017\/pan.2024.5","relation":{},"ISSN":["1047-1987","1476-4989"],"issn-type":[{"value":"1047-1987","type":"print"},{"value":"1476-4989","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,17]]}}}