{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T16:12:29Z","timestamp":1774627949269,"version":"3.50.1"},"reference-count":55,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T00:00:00Z","timestamp":1734998400000},"content-version":"vor","delay-in-days":358,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,12,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Although commonsense reasoning is greatly shaped by cultural and geographical factors, previous studies have predominantly centered on cultures grounded in the English language, potentially resulting in an Anglocentric bias. In this paper, we introduce IndoCulture, aimed at understanding the influence of geographical factors on language model reasoning ability, with a specific emphasis on the diverse cultures found within eleven Indonesian provinces. In contrast to prior work that has relied on templates (Yin et al., 2022) and online scrapping (Fung et al., 2024), we create IndoCulture by asking local people to manually develop a cultural context and plausible options, across a set of predefined topics. Evaluation of 27 language models reveals several insights: (1) the open-weight Llama\u20133 is competitive with GPT\u20134, while other open-weight models struggle, with accuracies below 50%; (2) there is a general pattern of models generally performing better for some provinces, such as Bali and West Java, and less well for others; and (3) the inclusion of location context enhances performance, especially for larger models like GPT\u20134, emphasizing the significance of geographical context in commonsense reasoning.1<\/jats:p>","DOI":"10.1162\/tacl_a_00726","type":"journal-article","created":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T16:39:15Z","timestamp":1735058355000},"page":"1703-1719","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":9,"title":["IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces"],"prefix":"10.1162","volume":"12","author":[{"given":"Fajri","family":"Koto","sequence":"first","affiliation":[{"name":"Department of Natural Language Processing, MBZUAI, UAE. fajri.koto@mbzuai.ac.ae"}]},{"given":"Rahmad","family":"Mahendra","sequence":"additional","affiliation":[{"name":"Universitas Indonesia, Indonesia. rahmad.mahendra@cs.ui.ac.id"},{"name":"RMIT University, Australia"}]},{"given":"Nurul","family":"Aisyah","sequence":"additional","affiliation":[{"name":"Quantic School of Business and Technology, USA"}]},{"given":"Timothy","family":"Baldwin","sequence":"additional","affiliation":[{"name":"Department of Natural Language Processing, MBZUAI, UAE"},{"name":"The University of Melbourne, Australia"}]}],"member":"281","published-online":{"date-parts":[[2024,12,18]]},"reference":[{"key":"2024122416390488400_bib1","doi-asserted-by":"crossref","first-page":"7226","DOI":"10.18653\/v1\/2022.acl-long.500","article-title":"One country, 700+ languages: NLP challenges for underrepresented languages and dialects in Indonesia","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Aji","year":"2022"},{"key":"2024122416390488400_bib2","volume-title":"Gestures: The do\u2019s and Taboos of Body Language Around the World","author":"Axtell","year":"1998"},{"key":"2024122416390488400_bib3","doi-asserted-by":"publisher","first-page":"7432","DOI":"10.1609\/aaai.v34i05.6239","article-title":"PIQA: Reasoning about physical commonsense in natural language","volume-title":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020","author":"Bisk","year":"2020"},{"key":"2024122416390488400_bib4","doi-asserted-by":"publisher","first-page":"8875","DOI":"10.18653\/v1\/2021.emnlp-main.699","article-title":"IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Cahyawijaya","year":"2021"},{"key":"2024122416390488400_bib5","doi-asserted-by":"publisher","first-page":"1173","DOI":"10.18653\/v1\/D19-1109","article-title":"Commonsense knowledge mining from pretrained models","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Davison","year":"2019"},{"key":"2024122416390488400_bib6","article-title":"The Llama 3 herd of models","author":"Dubey","year":"2024","journal-title":"arXiv preprint arXiv:2407.21783"},{"key":"2024122416390488400_bib7","doi-asserted-by":"publisher","first-page":"15217","DOI":"10.18653\/v1\/2023.emnlp-main.941","article-title":"NORMSAGE: Multi-lingual multi-cultural norm discovery from conversations on-the-fly","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Yi","year":"2023"},{"key":"2024122416390488400_bib8","article-title":"Massively multi-cultural knowledge acquisition & lm benchmarking","author":"Yi","year":"2024","journal-title":"arXiv preprint arXiv:2402.09369"},{"key":"2024122416390488400_bib9","doi-asserted-by":"crossref","DOI":"10.17323\/978-5-7598-2337-7","volume-title":"Essential Concepts in Sociology","author":"Giddens","year":"2021"},{"key":"2024122416390488400_bib10","doi-asserted-by":"publisher","first-page":"6997","DOI":"10.18653\/v1\/2022.acl-long.482","article-title":"Challenges and strategies in cross-cultural NLP","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Hershcovich","year":"2022"},{"key":"2024122416390488400_bib11","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.18653\/v1\/2023.findings-acl.67","article-title":"Towards reasoning in large language models: A survey","volume-title":"Findings of the Association for Computational Linguistics: ACL 2023","author":"Huang","year":"2023"},{"key":"2024122416390488400_bib12","article-title":"Merak-7b: The LLM for Bahasa Indonesia","author":"Ichsan","year":"2023","journal-title":"Hugging Face Repository"},{"key":"2024122416390488400_bib13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.760","article-title":"Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Koto","year":"2023"},{"key":"2024122416390488400_bib14","doi-asserted-by":"publisher","first-page":"8","DOI":"10.18653\/v1\/2022.csrr-1.2","article-title":"Cloze evaluation for deeper understanding of commonsense stories in Indonesian","volume-title":"Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022)","author":"Koto","year":"2022"},{"key":"2024122416390488400_bib15","doi-asserted-by":"publisher","first-page":"5622","DOI":"10.18653\/v1\/2024.findings-acl.334","article-title":"ArabicMMLU: Assessing massive multitask language understanding in Arabic","volume-title":"Findings of the Association for Computational Linguistics ACL 2024","author":"Koto","year":"2024"},{"key":"2024122416390488400_bib16","article-title":"The winograd schema challenge","volume-title":"Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning","author":"Levesque","year":"2012"},{"key":"2024122416390488400_bib17","article-title":"Bactrian-X: A multilingual replicable instruction-following model with low-rank adaptation","author":"Li","year":"2023","journal-title":"arXiv preprint arXiv:2305.15011"},{"key":"2024122416390488400_bib18","doi-asserted-by":"publisher","first-page":"11260","DOI":"10.18653\/v1\/2024.findings-acl.671","article-title":"CMMLU: Measuring massive multitask language understanding in Chinese","volume-title":"Findings of the Association for Computational Linguistics ACL 2024","author":"Li","year":"2024"},{"key":"2024122416390488400_bib19","doi-asserted-by":"crossref","first-page":"6862","DOI":"10.18653\/v1\/2020.emnlp-main.557","article-title":"Birds have four legs?! NumerSense: Probing numerical commonsense knowledge of pre-trained language models","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Lin","year":"2020"},{"key":"2024122416390488400_bib20","first-page":"9019","article-title":"Few-shot learning with multilingual generative language models","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Xi","year":"2022"},{"key":"2024122416390488400_bib21","doi-asserted-by":"publisher","first-page":"2016","DOI":"10.18653\/v1\/2024.naacl-long.112","article-title":"Are multilingual LLMs culturally-diverse reasoners? An investigation into multicultural proverbs and sayings","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Liu","year":"2024"},{"key":"2024122416390488400_bib22","article-title":"LLM360: Towards fully transparent open-source LLMs","author":"Liu","year":"2023","journal-title":"arXiv preprint arXiv:2312.06550"},{"key":"2024122416390488400_bib23","volume-title":"Sociology: Fourteenth Edition","author":"Macionis","year":"2012"},{"key":"2024122416390488400_bib24","doi-asserted-by":"publisher","first-page":"1384","DOI":"10.18653\/v1\/2022.emnlp-main.90","article-title":"Language models of code are few-shot commonsense learners","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Madaan","year":"2022"},{"key":"2024122416390488400_bib25","doi-asserted-by":"publisher","first-page":"10511","DOI":"10.18653\/v1\/2021.emnlp-main.821","article-title":"IndoNLI: A natural language inference dataset for Indonesian","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Mahendra","year":"2021"},{"key":"2024122416390488400_bib26","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/j.copsyc.2015.07.001","article-title":"Cultural evolution: Integrating psychology, evolution and culture","volume":"7","author":"Mesoudi","year":"2016","journal-title":"Current Opinion in Psychology"},{"key":"2024122416390488400_bib27","doi-asserted-by":"publisher","first-page":"839","DOI":"10.18653\/v1\/N16-1098","article-title":"A corpus and cloze evaluation for deeper understanding of commonsense stories","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Mostafazadeh","year":"2016"},{"key":"2024122416390488400_bib28","doi-asserted-by":"publisher","first-page":"15991","DOI":"10.18653\/v1\/2023.acl-long.891","article-title":"Crosslingual generalization through multitask finetuning","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Muennighoff","year":"2023"},{"key":"2024122416390488400_bib29","article-title":"BLEnD: A benchmark for LLMs on everyday knowledge in diverse cultures and languages","author":"Myung","year":"2024","journal-title":"arXiv preprint arXiv:2406.09948"},{"key":"2024122416390488400_bib30","doi-asserted-by":"publisher","first-page":"1907","DOI":"10.1145\/3543507.3583535","article-title":"Extracting cultural commonsense knowledge at scale","volume-title":"Proceedings of the ACM Web Conference 2023","author":"Nguyen","year":"2023"},{"key":"2024122416390488400_bib31","doi-asserted-by":"publisher","first-page":"294","DOI":"10.18653\/v1\/2024.acl-demos.28","article-title":"SeaLLMs - large language models for Southeast Asia","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)","author":"Nguyen","year":"2024"},{"key":"2024122416390488400_bib32","article-title":"GPT-4 technical report","volume":"abs\/2303.08774","author":"OpenAI","year":"2023","journal-title":"ArXiv"},{"key":"2024122416390488400_bib33","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024122416390488400_bib34","article-title":"Komodo: A linguistic expedition into Indonesia\u2019s regional languages","author":"Owen","year":"2024","journal-title":"arXiv preprint arXiv:2403.09362"},{"key":"2024122416390488400_bib35","doi-asserted-by":"publisher","first-page":"2362","DOI":"10.18653\/v1\/2020.emnlp-main.185","article-title":"XCOPA: A multilingual dataset for causal commonsense reasoning","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Ponti","year":"2020"},{"key":"2024122416390488400_bib36","doi-asserted-by":"publisher","first-page":"6:1\u20136:9","DOI":"10.1145\/3326467.3326487","article-title":"Budayakb: Extraction of cultural heritage entities from heterogeneous formats","volume-title":"Proceedings of the 9th International Conference on Web Intelligence, Mining and Semantics, WIMS 2019","author":"Putra","year":"2019"},{"key":"2024122416390488400_bib37","article-title":"Can LLM generate culturally relevant commonsense QA data? Case study in Indonesian and Sundanese","author":"Putri","year":"2024","journal-title":"arXiv e-prints"},{"key":"2024122416390488400_bib38","first-page":"7066","article-title":"TIMEDIAL: Temporal commonsense reasoning in dialog","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Qin","year":"2021"},{"key":"2024122416390488400_bib39","article-title":"Choice of plausible alternatives: An evaluation of commonsense causal reasoning","volume-title":"2011 AAAI Spring Symposium Series","author":"Roemmele","year":"2011"},{"issue":"9","key":"2024122416390488400_bib40","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1145\/3474381","article-title":"Winogrande: An adversarial Winograd schema challenge at scale","volume":"64","author":"Sakaguchi","year":"2021","journal-title":"Communications of the ACM"},{"key":"2024122416390488400_bib41","doi-asserted-by":"publisher","first-page":"4463","DOI":"10.18653\/v1\/D19-1454","article-title":"Social IQa: Commonsense reasoning about social interactions","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Sap","year":"2019"},{"key":"2024122416390488400_bib42","article-title":"Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models","author":"Sengupta","year":"2023","journal-title":"arXiv preprint arXiv:2308.16149"},{"key":"2024122416390488400_bib43","doi-asserted-by":"publisher","first-page":"2842","DOI":"10.18653\/v1\/2022.findings-acl.224","article-title":"Good night at 4 pm?! Time expressions in different cultures","volume-title":"Findings of the Association for Computational Linguistics: ACL 2022","author":"Shwartz","year":"2022"},{"key":"2024122416390488400_bib44","article-title":"SEA-LION (Southeast Asian languages in one network): A family of large language models for Southeast Asia","author":"Singapore","year":"2023"},{"issue":"2","key":"2024122416390488400_bib45","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1093\/applin\/4.2.91","article-title":"Cross-cultural pragmatic failure","volume":"4","author":"Thomas","year":"1983","journal-title":"Applied Linguistics"},{"key":"2024122416390488400_bib46","article-title":"Llama 2: Open foundation and fine-tuned chat models","author":"Touvron","year":"2023","journal-title":"arXiv preprint arXiv:2307.09288"},{"key":"2024122416390488400_bib47","doi-asserted-by":"publisher","first-page":"7407","DOI":"10.18653\/v1\/2024.findings-acl.441","article-title":"\u201cMy answer is C\u201d: First-token probabilities do not match text answers in instruction-tuned language models","volume-title":"Findings of the Association for Computational Linguistics ACL 2024","author":"Wang","year":"2024"},{"key":"2024122416390488400_bib48","doi-asserted-by":"publisher","first-page":"1404","DOI":"10.18653\/v1\/2024.naacl-long.77","article-title":"COPAL-ID: Indonesian language reasoning with local culture and nuances","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","author":"Wibowo","year":"2024"},{"key":"2024122416390488400_bib49","volume-title":"Keywords: A Vocabulary of Culture and Society","author":"Williams","year":"2014"},{"key":"2024122416390488400_bib50","doi-asserted-by":"publisher","first-page":"815","DOI":"10.18653\/v1\/2023.eacl-main.57","article-title":"NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages","volume-title":"Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics","author":"Winata","year":"2023"},{"key":"2024122416390488400_bib51","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2024122416390488400_bib52","doi-asserted-by":"publisher","first-page":"2039","DOI":"10.18653\/v1\/2022.emnlp-main.132","article-title":"GeoMLAMA: Geo-diverse commonsense probing on multilingual pre-trained language models","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Da","year":"2022"},{"issue":"1","key":"2024122416390488400_bib53","doi-asserted-by":"publisher","first-page":"1","DOI":"10.21831\/jss.v13i1.16966","article-title":"Multiculturalism in globalization era: History and challenge for Indonesia","volume":"13","author":"Zarbaliyev","year":"2017","journal-title":"Journal of Social Studies (JSS)"},{"key":"2024122416390488400_bib54","doi-asserted-by":"publisher","first-page":"4791","DOI":"10.18653\/v1\/P19-1472","article-title":"HellaSwag: Can a machine really finish your sentence?","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Zellers","year":"2019"},{"key":"2024122416390488400_bib55","doi-asserted-by":"publisher","first-page":"7756","DOI":"10.18653\/v1\/2023.acl-long.429","article-title":"NormBank: A knowledge bank of situational social norms","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Ziems","year":"2023"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00726\/2487346\/tacl_a_00726.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00726\/2487346\/tacl_a_00726.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T16:39:21Z","timestamp":1735058361000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00726\/125984\/IndoCulture-Exploring-Geographically-Influenced"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":55,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00726","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}