{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T21:20:24Z","timestamp":1780694424286,"version":"3.54.1"},"reference-count":51,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T00:00:00Z","timestamp":1775606400000},"content-version":"vor","delay-in-days":97,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns about issue bias, where an LLM tends to present just one perspective on a given issue, which in turn may influence how users think about this issue. So far, it has not been possible to measure which issue biases LLMs manifest in real user interactions, making it difficult to address the risks from biased LLMs. Therefore, we create IssueBench: a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance, which we construct based on 3.9k templates (e.g., \u201cwrite a blog about\u201d) and 212 political issues (e.g., \u201cAI regulation\u201d) from real user interactions. Using IssueBench, we show that issue biases are common and persistent in 10 state-of-the-art LLMs. We also show that biases are very similar across models, and that all models align more with US Democrat than Republican voter opinion on a subset of issues. IssueBench can easily be adapted to include other issues, templates, or tasks. By enabling robust and realistic measurement, we hope that IssueBench can bring a new quality of evidence to ongoing discussions about LLM biases and how to address them.<\/jats:p>","DOI":"10.1162\/tacl.a.626","type":"journal-article","created":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T13:13:15Z","timestamp":1775653995000},"page":"318-340","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance"],"prefix":"10.1162","volume":"14","author":[{"given":"Paul","family":"R\u00f6ttger","sequence":"first","affiliation":[{"name":"Bocconi University, Italy"},{"name":"University of Oxford, UK paul.rottger@oii.ox.ac.uk"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Musashi","family":"Hinck","sequence":"additional","affiliation":[{"name":"Intel Labs, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Valentin","family":"Hofmann","sequence":"additional","affiliation":[{"name":"Allen Institute for AI, USA"},{"name":"University of Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kobi","family":"Hackenburg","sequence":"additional","affiliation":[{"name":"University of Oxford, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Valentina","family":"Pyatkin","sequence":"additional","affiliation":[{"name":"Allen Institute for AI, USA"},{"name":"University of Washington, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faeze","family":"Brahman","sequence":"additional","affiliation":[{"name":"Allen Institute for AI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dirk","family":"Hovy","sequence":"additional","affiliation":[{"name":"Bocconi University, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2026,4,1]]},"reference":[{"key":"2026040809130691900_bib1","article-title":"Training a helpful and harmless assistant with reinforcement learning from human feedback","author":"Bai","year":"2022","journal-title":"arXiv preprint arXiv:2204.05862"},{"key":"2026040809130691900_bib2","doi-asserted-by":"publisher","first-page":"11142","DOI":"10.18653\/v1\/2024.acl-long.600","article-title":"Measuring political bias in large language models: What is said and how it is said","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Bang","year":"2024"},{"key":"2026040809130691900_bib3","article-title":"Large language models reflect the ideology of their creators","author":"Buyl","year":"2024","journal-title":"arXiv preprint arXiv:2410.18417"},{"key":"2026040809130691900_bib4","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1007\/978-3-642-37456-2_14","article-title":"Density-based clustering based on hierarchical density estimates","volume-title":"Pacific-Asia Conference on Knowledge Discovery and Data Mining","author":"Campello","year":"2013"},{"key":"2026040809130691900_bib5","doi-asserted-by":"publisher","first-page":"17140","DOI":"10.18653\/v1\/2024.emnlp-main.952","article-title":"How susceptible are large language models to ideological manipulation?","volume-title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing","author":"Chen","year":"2024"},{"key":"2026040809130691900_bib6","article-title":"The Llama 3 herd of models","author":"Dubey","year":"2024","journal-title":"arXiv preprint arXiv:2407.21783"},{"key":"2026040809130691900_bib7","article-title":"Measuring the persuasiveness of language models","author":"Durmus","year":"2024","journal-title":"Anthropic.com - last accessed 06.09.2025"},{"key":"2026040809130691900_bib8","article-title":"Towards measuring the representation of subjective global opinions in language models","volume-title":"First Conference on Language Modeling","author":"Durmus","year":"2024"},{"key":"2026040809130691900_bib9","doi-asserted-by":"publisher","first-page":"31684","DOI":"10.18653\/v1\/2025.acl-long.1529","article-title":"Only a little to the left: A theory-grounded measure of political bias in large language models","volume-title":"Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Faulborn","year":"2025"},{"key":"2026040809130691900_bib10","doi-asserted-by":"publisher","first-page":"11737","DOI":"10.18653\/v1\/2023.acl-long.656","article-title":"From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Feng","year":"2023"},{"key":"2026040809130691900_bib11","article-title":"Support for legal abortion is widespread in many places, especially in europe","author":"Fetterolf","year":"2024","journal-title":"Pew Research - last accessed 06.09.2025"},{"key":"2026040809130691900_bib12","article-title":"Position: Political neutrality in AI is impossible\u2014but here is how to approximate it","volume-title":"Forty-second International Conference on Machine Learning Position Paper Track","author":"Fisher","year":"2025"},{"key":"2026040809130691900_bib13","doi-asserted-by":"publisher","DOI":"10.1093\/actrade\/9780199551378.001.0001","volume-title":"Information: A Very Short Introduction","author":"Floridi","year":"2010"},{"key":"2026040809130691900_bib14","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2023.1232003","article-title":"Revisiting the political biases of chatGPT","volume":"6","author":"Fujimoto","year":"2023","journal-title":"Frontiers in Artificial Intelligence"},{"key":"2026040809130691900_bib15","doi-asserted-by":"publisher","first-page":"9004","DOI":"10.18653\/v1\/2024.emnlp-main.508","article-title":"On the relationship between truth and political bias in language models","volume-title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing","author":"Fulay","year":"2024"},{"key":"2026040809130691900_bib16","article-title":"About one-in-five U.S. adults know someone who goes by a gender-neutral pronoun","author":"Geiger","year":"2019","journal-title":"Pew Research - last accessed 06.09.2025"},{"issue":"2","key":"2026040809130691900_bib17","doi-asserted-by":"publisher","first-page":"pgae034","DOI":"10.1093\/pnasnexus\/pgae034","article-title":"How persuasive is AI-generated propaganda?","volume":"3","author":"Goldstein","year":"2024","journal-title":"PNAS Nexus"},{"issue":"10","key":"2026040809130691900_bib18","doi-asserted-by":"publisher","first-page":"e2413443122","DOI":"10.1073\/pnas.2413443122","article-title":"Scaling language model size yields diminishing returns for single-message political persuasion","volume":"122","author":"Hackenburg","year":"2025","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2026040809130691900_bib19","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.4316084","article-title":"The political ideology of conversational AI: Converging evidence on chatgpt\u2019s pro-environmental, left- libertarian orientation","author":"Hartmann","year":"2023","journal-title":"arXiv preprint arXiv: 2301.01768"},{"key":"2026040809130691900_bib20","doi-asserted-by":"publisher","first-page":"6155","DOI":"10.18653\/v1\/2023.findings-emnlp.410","article-title":"GlotLID: Language identification for low-resource languages","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2023","author":"Kargaran","year":"2023"},{"key":"2026040809130691900_bib21","article-title":"The PRISM alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models","volume-title":"The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track","author":"Kirk","year":"2024"},{"key":"2026040809130691900_bib22","article-title":"Deepseek-v3 technical report","author":"Liu","year":"2024","journal-title":"arXiv preprint arXiv:2412.19437"},{"issue":"1","key":"2026040809130691900_bib23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1057\/s41599-025-04465-z","article-title":"\u201cTurning right\u201d? An experimental study on the political value shift in large language models","volume":"12","author":"Liu","year":"2025","journal-title":"Humanities and Social Sciences Communications"},{"key":"2026040809130691900_bib24","doi-asserted-by":"publisher","first-page":"137","DOI":"10.18653\/v1\/2025.acl-long.7","article-title":"Bias in language models: Beyond trick tests and towards RUTEd evaluation","volume-title":"Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Lum","year":"2025"},{"key":"2026040809130691900_bib25","article-title":"Utility engineering: Analyzing and controlling emergent value systems in AIs","author":"Mazeika","year":"2025","journal-title":"arXiv preprint arXiv:2502.08640"},{"issue":"11","key":"2026040809130691900_bib26","doi-asserted-by":"publisher","first-page":"205","DOI":"10.21105\/joss.00205","article-title":"hdbscan: Hierarchical density based clustering","volume":"2","author":"McInnes","year":"2017","journal-title":"Journal of Open Source Software"},{"key":"2026040809130691900_bib27","doi-asserted-by":"publisher","first-page":"15185","DOI":"10.18653\/v1\/2024.findings-emnlp.891","article-title":"Are large language models consistent over value-laden questions?","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2024","author":"Moore","year":"2024"},{"key":"2026040809130691900_bib28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11127-023-01097-2","article-title":"More human than human: Measuring chatgpt political bias","author":"Motoki","year":"2023","journal-title":"Public Choice"},{"key":"2026040809130691900_bib29","article-title":"2 OLMo 2 Furious","author":"OLMo","year":"2024","journal-title":"arXiv preprint arXiv:2501.00656"},{"key":"2026040809130691900_bib30","unstructured":"OpenAI. 2024. Model spec. OpenAI Website - last accessed 06.09.2025."},{"key":"2026040809130691900_bib31","doi-asserted-by":"publisher","first-page":"2375","DOI":"10.18653\/v1\/2023.emnlp-main.146","article-title":"The shifted and the overlooked: A task-oriented investigation of user-GPT interactions","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Ouyang","year":"2023"},{"key":"2026040809130691900_bib32","doi-asserted-by":"publisher","first-page":"4244","DOI":"10.18653\/v1\/2024.emnlp-main.244","article-title":"Hidden persuaders: LLMs\u2019 political leaning and their influence on voters","volume-title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing","author":"Potter","year":"2024"},{"key":"2026040809130691900_bib33","unstructured":"Qwen. 2024. Qwen2.5: A party of foundation models. Qwen Team Blog - last accessed 06.09.2025."},{"key":"2026040809130691900_bib34","doi-asserted-by":"publisher","first-page":"3982","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence-BERT: Sentence embeddings using Siamese BERT-networks","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Reimers","year":"2019"},{"issue":"2","key":"2026040809130691900_bib35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s42001-025-00376-w","article-title":"Assessing political bias in large language models","volume":"8","author":"Rettenberger","year":"2025","journal-title":"Journal of Computational Social Science"},{"key":"2026040809130691900_bib36","unstructured":"Reuters. 2024. Openai says chatgpt\u2019s weekly users have grown to 200 million. Reuters - last accessed 06.09.2025."},{"key":"2026040809130691900_bib37","doi-asserted-by":"publisher","first-page":"15295","DOI":"10.18653\/v1\/2024.acl-long.816","article-title":"Political compass or spinning arrow? Towards more meaningful evaluations for values and opinions in large language models","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"R\u00f6ttger","year":"2024"},{"key":"2026040809130691900_bib38","doi-asserted-by":"publisher","first-page":"175","DOI":"10.18653\/v1\/2022.naacl-main.13","article-title":"Two contrasting data annotation paradigms for subjective NLP tasks","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"R\u00f6ttger","year":"2022"},{"issue":"3","key":"2026040809130691900_bib39","doi-asserted-by":"publisher","first-page":"148","DOI":"10.3390\/socsci12030148","article-title":"The political biases of chatgpt","volume":"12","author":"Rozado","year":"2023","journal-title":"Social Sciences"},{"issue":"7","key":"2026040809130691900_bib40","doi-asserted-by":"publisher","first-page":"e0306621","DOI":"10.1371\/journal.pone.0306621","article-title":"The political preferences of LLMs","volume":"19","author":"Rozado","year":"2024","journal-title":"PLOS ONE"},{"key":"2026040809130691900_bib41","article-title":"Measuring political preferences in ai systems: An integrative approach","author":"Rozado","year":"2025","journal-title":"arXiv preprint arXiv:2503.10649"},{"issue":"1","key":"2026040809130691900_bib42","doi-asserted-by":"publisher","first-page":"7115633","DOI":"10.1155\/2024\/7115633","article-title":"The self-perception and political biases of chatgpt","volume":"2024","author":"Rutinowski","year":"2024","journal-title":"Human Behavior and Emerging Technologies"},{"key":"2026040809130691900_bib43","first-page":"29971","article-title":"Whose opinions do language models reflect?","volume-title":"International Conference on Machine Learning","author":"Santurkar","year":"2023"},{"key":"2026040809130691900_bib44","article-title":"Benchmarks as microscopes: A call for model metrology","volume-title":"First Conference on Language Modeling","author":"Saxon","year":"2024"},{"key":"2026040809130691900_bib45","doi-asserted-by":"publisher","first-page":"251","DOI":"10.18653\/v1\/2024.emnlp-main.16","article-title":"Systematic biases in LLM simulations of debates","volume-title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing","author":"Taubenfeld","year":"2024"},{"key":"2026040809130691900_bib46","doi-asserted-by":"publisher","first-page":"4420","DOI":"10.18653\/v1\/2024.findings-emnlp.255","article-title":"Quantifying generative media bias with a corpus of real-world and generated news articles","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2024","author":"Trhl\u00edk","year":"2024"},{"key":"2026040809130691900_bib47","article-title":"What the data says about Americans\u2019 views of climate change","author":"Tyson","year":"2023","journal-title":"Pew Research - last accessed 06.09.2025"},{"key":"2026040809130691900_bib48","article-title":"Measuring perceived slant in large language models through user evaluations","author":"Westwood","year":"2025","journal-title":"Stanford Graduate Business School, Working Paper No. 4262"},{"key":"2026040809130691900_bib49","doi-asserted-by":"publisher","first-page":"17085","DOI":"10.18653\/v1\/2024.findings-emnlp.995","article-title":"LLM tropes: Revealing fine-grained values and opinions in large language models","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2024","author":"Wright","year":"2024"},{"key":"2026040809130691900_bib50","article-title":"(inthe)wildchat: 570k chatGPT interaction logs in the wild","volume-title":"The Twelfth International Conference on Learning Representations","author":"Zhao","year":"2024"},{"key":"2026040809130691900_bib51","article-title":"LMSYS-chat-1m: A large-scale real-world LLM conversation dataset","volume-title":"The Twelfth International Conference on Learning Representations","author":"Zheng","year":"2024"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.626\/2593115\/tacl.a.626.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.626\/2593115\/tacl.a.626.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T13:13:21Z","timestamp":1775654001000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/TACL.a.626\/136157\/IssueBench-Millions-of-Realistic-Prompts-for"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":51,"URL":"https:\/\/doi.org\/10.1162\/tacl.a.626","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}