{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T05:35:28Z","timestamp":1793079328002,"version":"4.5.0"},"reference-count":32,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2026,10,19]],"date-time":"2026-10-19T00:00:00Z","timestamp":1792368000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,10,19]],"date-time":"2026-10-19T00:00:00Z","timestamp":1792368000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Committee of Science of the Ministry of Science and Higher Education of the Republic of Kazakhstan","award":["BR27101389"],"award-info":[{"award-number":["BR27101389"]}],"id":[{"id":"https:\/\/ror.org\/05tyne317","id-type":"ROR","asserted-by":"crossref"}]},{"award":["BR27101389"],"award-info":[{"award-number":["BR27101389"]}],"id":[{"id":"https:\/\/ror.org\/05tyne317","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Legal question answering is usually evaluated as if the law were static. In Kazakhstan, it changed twice within one year: a new Constitution replaced the 1995 text on 1 July 2026, and a new Tax Code replaced its predecessor on 1 January 2026, both reusing article numbers for different norms. We release a dated corpus of 41 Kazakhstani acts parsed to article level in both official languages (12,235 Russian and 11,973 Kazakh articles) and a benchmark of 150 question-answer pairs in seven layers, two of which score declining to answer and asking for a date. Two practising lawyers confirmed every pair. Comparing a no-retrieval baseline, retrieval-augmented generation and a tool-using agent, we find that retrieval rather than generation accounts for most citation error, measured against each pair\u2019s source articles: given the gold article, the model cites it in 92% of cases, and retrieval alone reduces citations of non-existent norms from 31% to zero. The agent adds nothing on this measure and abstains far less often. Dense retrieval improves citation recall but nearly doubles the repealed norms presented as current; date-aware ranking removes them for every retriever, although under dense retrieval it also removes three historically required citations. The retrieval findings are reproduced on two open models.<\/jats:p>","DOI":"10.3390\/info17101042","type":"journal-article","created":{"date-parts":[[2026,10,19]],"date-time":"2026-10-19T16:33:08Z","timestamp":1792427588000},"page":"1042","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["A Dated Corpus and a Seven-Layer Benchmark for Kazakhstani Legal Question Answering"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9423-717X","authenticated-orcid":false,"given":"Nurbol","family":"Beisov","sequence":"first","affiliation":[{"name":"Department of Computer Science, Faculty of Information Technology, Farabi University, Almaty 050040, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nurlykhan","family":"Kalzhanov","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Information Technology, Farabi University, Almaty 050040, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arman","family":"Yeleussinov","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Information Technology, Farabi University, Almaty 050040, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4621-6646","authenticated-orcid":false,"given":"Maxatbek","family":"Satymbekov","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Information Technology, Farabi University, Almaty 050040, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zholdas","family":"Buribayev","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Information Technology, Farabi University, Almaty 050040, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7782-409X","authenticated-orcid":false,"given":"Yerbol","family":"Alimkulov","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Information Technology, Farabi University, Almaty 050040, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ualikhan","family":"Akhatov","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Information Technology, Farabi University, Almaty 050040, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3060-8549","authenticated-orcid":false,"given":"Yernur","family":"Oxikbayev","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Faculty of Information Technology, Farabi University, Almaty 050040, Kazakhstan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,10,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Guha, N., Nyarko, J., Ho, D.E., R\u00e9, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., and Rockmore, D.N. (2023, January 10\u201316). LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. Proceedings of the Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track, New Orleans, LA, USA.","DOI":"10.52202\/075280-1915"},{"key":"ref_2","unstructured":"Pipitone, N., and Houir Alami, G. (2024). LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain. arXiv."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1093\/jla\/laae003","article-title":"Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models","volume":"16","author":"Dahl","year":"2024","journal-title":"J. Leg. Anal."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1111\/jels.12413","article-title":"Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools","volume":"22","author":"Magesh","year":"2025","journal-title":"J. Empir. Leg. Stud."},{"key":"ref_5","unstructured":"Han, J., Burgess, P., and Shareghi, E. (2024). Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study. arXiv."},{"key":"ref_6","unstructured":"Hu, Y., Gan, L., Xiao, W., Kuang, K., and Wu, F. (2025, January 19\u201324). Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering. Proceedings of the 31st International Conference on Computational Linguistics (COLING), Abu Dhabi, United Arab Emirates."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Soudani, H., Kanoulas, E., and Hasibi, F. (2024, January 9\u201312). Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge. Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP \u201924), Tokyo, Japan.","DOI":"10.1145\/3673791.3698415"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Louis, A., van Dijck, G., and Spanakis, G. (2024, January 20\u201327). Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v38i20.30232"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Gao, T., Yen, H., Yu, J., and Chen, D. (2023, January 6\u201310). Enabling Large Language Models to Generate Text with Citations. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.","DOI":"10.18653\/v1\/2023.emnlp-main.398"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., and Androutsopoulos, I. (2020, January 16\u201320). LEGAL-BERT: The Muppets straight out of Law School. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Online.","DOI":"10.18653\/v1\/2020.findings-emnlp.261"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"\u0160avelka, Z., and Ashley, K.D. (2023). Unreasonable Effectiveness of Large Language Models in Zero-Shot Semantic Annotation of Legal Texts. Front. Artif. Intell., 6.","DOI":"10.3389\/frai.2023.1279794"},{"key":"ref_12","unstructured":"Westermann, H., Mee\u00f9s, S., Godet, M., Troussel, A., Tan, J., Savelka, J., and Benyekhlef, K. (2023, January 23). Bridging the Gap: Mapping Layperson Narratives to Legal Issues with Language Models. Proceedings of the 6th Workshop on Automated Semantic Analysis of Information in Legal Text (ASAIL 2023), Braga, Portugal."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1162\/tacl_a_00459","article-title":"Time-Aware Language Models as Temporal Knowledge Bases","volume":"10","author":"Dhingra","year":"2022","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_14","unstructured":"Chen, W., Wang, X., and Wang, W.Y. (2021, January 6). A Dataset for Answering Time-Sensitive Questions. Proceedings of the Thirty-fifth Conference on Neural Information Processing Systems, Datasets and Benchmarks Track (Round 2), Virtual."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhang, M.J.Q., and Choi, E. (2021, January 7\u201311). SituatedQA: Incorporating Extra-Linguistic Contexts into QA. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Virtual.","DOI":"10.18653\/v1\/2021.emnlp-main.586"},{"key":"ref_16","unstructured":"Prior, M., Schultz, A., and Grabmair, M. (2026, January 8\u201312). Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering. Proceedings of the International Conference on Artificial Intelligence and Law (ICAIL), Singapore."},{"key":"ref_17","unstructured":"Cymbler, R., Guez, D., and Fabre, L. (2026, January 10). Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law. Proceedings of the ICML 2026 Workshop on AI for Law (AI4Law), Seoul, Republic of Korea."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1162\/tacl_a_00754","article-title":"Know Your Limits: A Survey of Abstention in Large Language Models","volume":"13","author":"Wen","year":"2025","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhang, T., Qin, P., Deng, Y., Huang, C., Lei, W., Liu, J., Jin, D., Liang, H., and Chua, T. (2024, January 11\u201316). CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand.","DOI":"10.18653\/v1\/2024.acl-long.578"},{"key":"ref_20","unstructured":"Chen, J., Lin, H., Han, X., and Sun, L. (2024, January 20\u201327). Benchmarking Large Language Models in Retrieval-Augmented Generation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Tleubayeva, A., Mansurova, A., Shomanov, A., Makhambetova, Z., and Sarisaray Boluk, P. (2026). KazakhLawCorpus: A Large-Scale Metadata Corpus for Kazakh Legislation. Data, 11.","DOI":"10.3390\/data11090217"},{"key":"ref_22","unstructured":"olzhasAl (2026, October 15). adilet-legal-qa-kz: Question\u2013Answer Pairs Generated from 28 Normative Legal Acts of the Republic of Kazakhstan. Hugging Face Dataset. Available online: https:\/\/huggingface.co\/datasets\/olzhasAl\/adilet-legal-qa-kz."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Rakhimova, D., Turarbek, A., Karyukin, V., Sarsenbayeva, A., and Alieyev, R. (2025). Legal AI in Low-Resource Languages: Building and Evaluating QA Systems for Kazakh Legislation. Computers, 14.","DOI":"10.3390\/computers14090354"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Yeleussinov, A., Buribayev, Z., Beisov, N., Kalzhanov, N., Satymbekov, M., Akhatov, U., and Alimkulov, Y. (2026). Fine-Tuning Qwen3 Models for the Legal Domain of Kazakhstan: A Comparative Study of LoRA-Adapted Models for Bilingual Legal Question Answering. Appl. Sci., 16.","DOI":"10.3390\/app16136777"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Satymbekov, M., Yeleussinov, A., Buribayev, Z., Beisov, N., Kalzhanov, N., Alimkulov, Y., and Zhumadilova, N. (2026). Detecting Contradictions in Bilingual Legislation: A Systematic Review and Conceptual Architecture. Information, 17.","DOI":"10.3390\/info17090931"},{"key":"ref_26","unstructured":"Togmanov, M., Mukhituly, N., Turmakhan, D., Mansurov, J., Goloburda, M., Sakip, A., Xie, Z., Wang, Y., Syzdykov, B., and Laiyk, N. (August, January 27). KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), Vienna, Austria."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Goloburda, M., Laiyk, N., Turmakhan, D., Wang, Y., Togmanov, M., Mansurov, J., Sametov, A., Mukhituly, N., Wang, M., and Orel, D. (2025). Qor\u011fau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contexts. arXiv.","DOI":"10.18653\/v1\/2025.findings-acl.507"},{"key":"ref_28","unstructured":"Koto, F., Joshi, R., Mukhituly, N., Wang, Y., Xie, Z., Pal, R., Orel, D., Mullah, P., Turmakhan, D., and Goloburda, M. (2025, January 7\u201310). Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting. Proceedings of the Conference on Language Modeling (COLM), Montreal, QC, Canada."},{"key":"ref_29","unstructured":"Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K\u00fcttler, H., Lewis, M., Yih, W., and Rockt\u00e4schel, T. (2020, January 6\u201312). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024). BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings through Self-Knowledge Distillation. arXiv.","DOI":"10.18653\/v1\/2024.findings-acl.137"},{"key":"ref_31","unstructured":"BAAI (2026, October 15). BGE-Reranker-v2-M3 Model Card and Documentation. Hugging Face, 2024. Available online: https:\/\/huggingface.co\/BAAI\/bge-reranker-v2-m3."},{"key":"ref_32","unstructured":"Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023, January 1\u20135). ReAct: Synergizing Reasoning and Acting in Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/17\/10\/1042\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T05:27:16Z","timestamp":1793078836000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/17\/10\/1042"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,19]]},"references-count":32,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["info17101042"],"URL":"https:\/\/doi.org\/10.3390\/info17101042","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,19]]}}}