{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T11:17:22Z","timestamp":1770895042400,"version":"3.50.1"},"reference-count":25,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2025,10,1]],"date-time":"2025-10-01T00:00:00Z","timestamp":1759276800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002509","name":"Bisa Research Grant of Keimyung University","doi-asserted-by":"publisher","award":["20240535"],"award-info":[{"award-number":["20240535"]}],"id":[{"id":"10.13039\/501100002509","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.mdpi.com"],"crossmark-restriction":true},"short-container-title":["Publications"],"abstract":"<jats:p>This study evaluates digital object identifier (DOI) hallucination in large language model (LLM)-generated scholarly citations, with a focus on systematic geographic disparities. To conduct this study, we systematically evaluated four LLMs (GPT-4o-mini, Claude-3-haiku, Gemini-2.0-flash-lite, and DeepSeek V3) using standardized information behavior prompts across ten countries with diverse income levels. The models generated 3451 citations, which we validated using the CrossRef API. The results showed that DOI hallucination follows systematic patterns influenced by model choice, geographic context, and publication recency. Hallucination rates exceeded 80% in lower-income countries and increased sharply for publications from the 2020s across all regions. Fabricated citations\u2014citations that appear structurally complete but contain invalid DOIs\u2014were especially prevalent in countries such as India and Bangladesh. Model-specific factors showed the strongest association with hallucination, followed by income level and publication period. These findings raise concerns about the epistemic reliability of LLM-generated scholarly references and underscore the need for region-aware training, real-time DOI validation, and robust verification protocols in academic contexts.<\/jats:p>","DOI":"10.3390\/publications13040049","type":"journal-article","created":{"date-parts":[[2025,10,1]],"date-time":"2025-10-01T09:37:21Z","timestamp":1759311441000},"page":"49","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Geographic Variation in LLM DOI Fabrication: Cross-Country Analysis of Citation Accuracy Across Four Large Language Models"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7314-1726","authenticated-orcid":false,"given":"Eungi","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Library and Information Science, Keimyung University, 1095 Dalgubeoldaero, Dalseo-Gu, Daegu 42601, Republic of Korea"}]},{"given":"Frankline","family":"Kipchumba","sequence":"additional","affiliation":[{"name":"Department of Library and Information Science, Keimyung University, 1095 Dalgubeoldaero, Dalseo-Gu, Daegu 42601, Republic of Korea"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1581-9210","authenticated-orcid":false,"given":"Sein","family":"Min","sequence":"additional","affiliation":[{"name":"Department of Library and Information Science, Keimyung University, 1095 Dalgubeoldaero, Dalseo-Gu, Daegu 42601, Republic of Korea"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1108\/00220410810844150","article-title":"What do citation counts measure? A review of studies on citing behavior","volume":"64","author":"Bornmann","year":"2008","journal-title":"Journal of Documentation"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1515\/dx-2024-0033","article-title":"Computerized diagnostic decision support systems\u2013A comparative performance study of Isabel Pro vs. ChatGPT4","volume":"11","author":"Bridges","year":"2024","journal-title":"Diagnosis"},{"key":"ref_3","unstructured":"Chavarro, D., Alperin, J. P., and Willinsky, J. (2025). A caminho da indexa\u00e7\u00e3o universal: OpenAlex e open journal systems. SciELO, preprints."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1186\/s41077-025-00350-6","article-title":"Artificial intelligence-assisted academic writing: Recommendations for ethical use","volume":"10","author":"Cheng","year":"2025","journal-title":"Advances in Simulation"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Daquino, M., Peroni, S., Shotton, D., Colavizza, G., Ghavimi, B., Lauscher, A., and Zumstein, P. (2020). The OpenCitations data model. International semantic web conference, Springer International Publishing.","DOI":"10.1007\/978-3-030-62466-8_28"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1093\/llc\/fqaa011","article-title":"Citational politics: Quantifying the influence of gender on citation in digital scholarship in the humanities","volume":"36","author":"Earhart","year":"2021","journal-title":"Digital Scholarship in the Humanities"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1080\/09500790408668312","article-title":"Goodhart\u2019s law and performance indicators in higher education","volume":"18","author":"Elton","year":"2004","journal-title":"Evaluation & Research in Education"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Frosolini, A., Catarzi, L., Benedetti, S., Latini, L., Chisci, G., Franz, L., Gennaro, P., and Gabriele, G. (2024). The role of Large Language Models (LLMs) in providing triage for maxillofacial trauma cases: A preliminary study. Diagnostics, 14, (8).","DOI":"10.3390\/diagnostics14080839"},{"key":"ref_9","first-page":"12","article-title":"Accuracy of references in the kenya eye health journal volume 1 number 1","volume":"2","author":"Gathama","year":"2025","journal-title":"Kenya Eye Health Journal"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1162\/qss_a_00022","article-title":"Crossref: The sustainable source of community-owned scholarly metadata","volume":"1","author":"Hendricks","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1093\/applin\/20.3.341","article-title":"Academic attribution: Citation and the construction of disciplinary knowledge","volume":"20","author":"Hyland","year":"1999","journal-title":"Applied Linguistics"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1016\/j.jclinepi.2012.09.015","article-title":"Citation bias favoring statistically significant studies was present in medical research","volume":"66","author":"Jannot","year":"2013","journal-title":"Journal of Clinical Epidemiology"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Jedrzejczak, W. W., Skarzynski, P. H., Raj-Koziak, D., Sanfins, M. D., Hatzopoulos, S., and Kochanek, K. (2024). ChatGPT for tinnitus information and support: Response accuracy and retest after three and six months. Brain Sciences, 14, (5).","DOI":"10.3390\/brainsci14050465"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kumari, K. S., Shree, S. U., and Calarany, C. (, January August). Dynamic region-aware fine-tuning: Enhancing geographic inclusivity in large language models. 2024 IEEE 16th International Conference on Advanced Communication Technology, Enshi, China.","DOI":"10.1109\/CICN63059.2024.10847353"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"e70057","DOI":"10.1111\/tgis.70057","article-title":"Operationalizing geographic diversity for the evaluation of AI-generated content","volume":"29","author":"Liu","year":"2025","journal-title":"Transactions in GIS"},{"key":"ref_16","unstructured":"Manvi, R., Khanna, S., Burke, M., Lobell, D., and Ermon, S. (2024). Large language models are geographically biased. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Moayeri, M., Tabassi, E., and Feizi, S. (, January June). WorldBench: Quantifying geographic disparities in LLM factual recall. The 2024 ACM Conference on Fairness, Accountability, and Transparency, Rio de Janeiro, Brazil.","DOI":"10.1145\/3630106.3658967"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e52935","DOI":"10.2196\/52935","article-title":"Evaluation of large language model performance and reliability for citations and references in scholarly writing: Cross-disciplinary study","volume":"26","author":"Mugaanyi","year":"2024","journal-title":"Journal of Medical Internet Research"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"915","DOI":"10.12688\/f1000research.154408.2","article-title":"Publisher preferences for a journal transparency tool: A modified three-round Delphi study","volume":"13","author":"Ng","year":"2025","journal-title":"F1000Research"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"26626","DOI":"10.1038\/s41598-024-77790-z","article-title":"A mixed-methods survey and focus group study to understand researcher and clinician preferences for a Journal Transparency Tool","volume":"14","author":"Ng","year":"2024","journal-title":"Scientific Reports"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"13117","DOI":"10.1007\/s11069-025-07344-7","article-title":"Assessing WildfireGPT: A comparative analysis of AI models for quantitative wildfire spread prediction","volume":"121","author":"Ramesh","year":"2025","journal-title":"Natural Hazards"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3224","DOI":"10.1021\/acs.jchemed.4c00193","article-title":"Using ChatGPT for method development and green chemistry education in upper-level laboratory courses","volume":"101","author":"Ruff","year":"2024","journal-title":"Journal of Chemical Education"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"85","DOI":"10.3138\/jsp-2017-0017","article-title":"Publication history: A double-DOI-based method for storing and\/or monitoring information about published and corrected academic literature","volume":"53","author":"Nazarovets","year":"2022","journal-title":"Journal of Scholarly Publishing"},{"key":"ref_24","unstructured":"World Bank (2025, September 08). World Bank Country and Lending Groups. World bank data help desk. Available online: https:\/\/datahelpdesk.worldbank.org\/knowledgebase\/articles\/906519-world-bank-country-and-lending-groups."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e72998","DOI":"10.2196\/72998","article-title":"Citation accuracy challenges posed by large language models","volume":"11","author":"Zhang","year":"2025","journal-title":"JMIR Medical Education"}],"container-title":["Publications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2304-6775\/13\/4\/49\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,1]],"date-time":"2025-10-01T10:06:03Z","timestamp":1759313163000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2304-6775\/13\/4\/49"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,1]]},"references-count":25,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["publications13040049"],"URL":"https:\/\/doi.org\/10.3390\/publications13040049","relation":{},"ISSN":["2304-6775"],"issn-type":[{"value":"2304-6775","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,1]]}}}