{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T09:02:38Z","timestamp":1784710958929,"version":"3.55.0"},"reference-count":67,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T00:00:00Z","timestamp":1783555200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Accurately evaluating the degree of legalization of international instruments is analytically valuable for assessing the level of institutionalization of interstate cooperative arrangements. However, conventional assessment approaches rely heavily on specialized legal expertise, and the inherent efficiency constraints of manual analysis make systematic evaluation across large-scale instrument corpora exceedingly difficult.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>To address the lack of automated scoring tools in this domain, this study introduces a hybrid retrieval-augmented generation (RAG) scoring framework based on chain-of-thought (CoT) prompting. Based on the legalization conceptual framework proposed by Abbott and Snidal, this study constructs a five-level coding standard that covers the three dimensions of obligation, precision, and delegation, and accordingly designs a stepwise binary decision process to develop the corresponding CoT prompt template. In parallel, this study constructs a vector-indexed clause database comprising 2,611 expert-annotated samples from the ASEAN instrument corpus, providing retrieval-augmented semantic references for the scoring task. The retrieval mechanism incorporates an adaptive quality-weighted ranking strategy and employs an additional pre-trained model to perform secondary filtering of candidate results. To evaluate the effectiveness of the scoring framework, two independent test sets were constructed from ASEAN instruments and documents from other major regional organizations, respectively, containing 254 and 255 expert-annotated samples.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>GPT-5.2 and GPT-4o, based on this architecture, significantly outperform the baseline prompting strategy, traditional machine learning methods (TF-IDF+LR), and the fine-tuned pre-trained language model (Legal-BERT) on the international instrument scoring task. Among all evaluated models, GPT-5.2 exhibits the strongest overall performance. Based on the averaged outcomes of three independent runs, the model attained QWK scores of 0.806 and 0.788, and MAE scores of 0.089 and 0.084, on the ASEAN instrument test set and the test set of other major regional organizations, respectively, demonstrating a substantially high degree of agreement with human expert ratings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>These findings indicate that integrating CoT prompting with the RAG pipeline improves LLM performance in scoring the degree of legalization of international legal documents. This provides methodological and technical foundations for large-scale institutional empirical analysis in ASEAN and broader legal systems, and points toward future extensions of the framework to other regional and multilateral legal orders.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frai.2026.1784200","type":"journal-article","created":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T05:36:48Z","timestamp":1783575408000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["The legalization of international instruments: a hybrid RAG scoring framework based on chain-of-thought prompting"],"prefix":"10.3389","volume":"9","author":[{"given":"Yan","family":"Chen","sequence":"first","affiliation":[{"name":"Faculty of Law, Universiti Kebangsaan Malaysia","place":["Bangi, Malaysia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zihua","family":"Zeng","sequence":"additional","affiliation":[{"name":"School of Computer Science, The University of Sydney","place":["Sydney, NSW, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhamad Sayuti","family":"Hassan","sequence":"additional","affiliation":[{"name":"Faculty of Law, Universiti Kebangsaan Malaysia","place":["Bangi, Malaysia"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,7,9]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1162\/002081800551271","article-title":"The concept of legalization","volume":"54","author":"Abbott","year":"2000","journal-title":"Int. Organ."},{"key":"B2","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1162\/002081800551280","article-title":"Hard and soft law in international governance","volume":"54","author":"Abbott","year":"2000","journal-title":"Int. Organ."},{"key":"B3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1111\/j.1468-2478.2009.00575.x","article-title":"Delegating differences: bilateral investment treaties and bargaining over dispute resolution provisions","volume":"54","author":"Allee","year":"2010","journal-title":"Int. Stud. Q."},{"key":"B4","doi-asserted-by":"publisher","first-page":"238","DOI":"10.1177\/0539018412437110","article-title":"Legalization in international relations: a conceptual analysis","volume":"51","author":"B\u00e9langer","year":"2012","journal-title":"Soc. Sci. Inf."},{"key":"B5","doi-asserted-by":"publisher","first-page":"919","DOI":"10.1093\/isq\/sqaa053","article-title":"Dispute by design? Legalization, backlash, and the drafting of investment agreements","volume":"64","author":"Berge","year":"2020","journal-title":"Int. Stud. Q."},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.4443471","article-title":"GPT-4's law school grades: con law C, crim C-, law and econ C, partnership tax B, property B-, tax B","author":"Blair-Stanek","year":"2023","journal-title":"SSRN [Preprint]"},{"key":"B7","first-page":"21","article-title":"\u201cMulti-label classification of legislative text into EuroVoc,\u201d","volume-title":"Legal Knowledge and Information Systems","author":"Boella","year":"2012"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.31724\/rihjj.48.1.1","article-title":"A corpus-based study of deontic modality in legal discourse","volume":"48","author":"Boginskaya","year":"2022","journal-title":"Rasprave Cas. Inst. Hrvatsk. J. Jezikoslovlje"},{"key":"B9","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1007\/s10784-018-9391-3","article-title":"The self-selection of democracies into treaty design: insights from international environmental agreements","volume":"18","author":"B\u00f6hmelt","year":"2018","journal-title":"Int. Environ. Agreements Polit. Law Econ."},{"key":"B10","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.48550\/arXiv.2005.14165","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B11","doi-asserted-by":"publisher","first-page":"140","DOI":"10.1162\/isec_a_00392","article-title":"The stopping power of norms: saturation bombing, civilian immunity, and US attitudes toward the laws of war","volume":"45","author":"Carpenter","year":"2020","journal-title":"Int. Secur."},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.261","article-title":"LEGAL-BERT: the muppets straight out of law school","author":"Chalkidis","year":"2020","journal-title":"arXiv [Preprint]"},{"key":"B13","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1177\/0047117819830469","article-title":"W(h)ither the Association of South East Asian Nations (ASEAN)? w(h)ither constructivism? fixity of norms and the ASEAN Way","volume":"33","author":"Collins","year":"2019","journal-title":"Int. Relat."},{"key":"B14","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1093\/jla\/laae003","article-title":"Large legal fictions: profiling legal hallucinations in large language models","volume":"16","author":"Dahl","year":"2024","journal-title":"J. Leg. Anal."},{"key":"B15","first-page":"87","article-title":"\u201cMachine learning versus knowledge based classification of legal texts,\u201d","volume-title":"Legal Knowledge and Information Systems","author":"de Maat","year":"2010"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1007\/s10506-024-09411-z","article-title":"Applicability of large language models and generative models for legal case judgement summarization","volume":"33","author":"Deroy","year":"2025","journal-title":"Artif. Intell. Law"},{"key":"B17","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1162\/002081800551262","article-title":"Introduction: legalization and world politics","volume":"54","author":"Goldstein","year":"2000","journal-title":"Int. Organ."},{"key":"B18","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1093\/pan\/mps028","article-title":"Text as data: the promise and pitfalls of automatic content analysis methods for political texts","volume":"21","author":"Grimmer","year":"2013","journal-title":"Polit. Anal."},{"key":"B19","doi-asserted-by":"publisher","first-page":"44123","DOI":"10.52202\/075280-1915","article-title":"LegalBench: a collaboratively built benchmark for measuring legal reasoning in large language models","volume":"36","author":"Guha","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B20","doi-asserted-by":"crossref","first-page":"14982","DOI":"10.18653\/v1\/2024.findings-acl.890","article-title":"\u201cFound in the middle: calibrating positional attention bias improves long context utilization,\u201d","volume-title":"Findings Assoc Comput Linguist ACL 2024","author":"Hsieh","year":"2024"},{"key":"B21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"B22","volume-title":"Consent and Commitment in the World Community: The Classification and Analysis of International Agreement, Vol. 22","author":"Johnston","year":"2023"},{"key":"B23","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1162\/isec.2007.32.1.148","article-title":"Making process, not progress: ASEAN and the evolving East Asian regional order","volume":"32","author":"Jones","year":"2007","journal-title":"Int. Secur."},{"key":"B24","doi-asserted-by":"crossref","first-page":"6769","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"\u201cDense passage retrieval for open-domain question answering,\u201d","author":"Karpukhin","year":"2020","journal-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"},{"key":"B25","first-page":"337","article-title":"Legalization and norm internalization: an empirical study of international human rights commitments eliciting public support for compliance","volume":"7","author":"Kim","year":"2019","journal-title":"Penn. St. J. Law Int. Aff."},{"key":"B26","doi-asserted-by":"publisher","first-page":"27","DOI":"10.17323\/1996-7845-2019-03-02","article-title":"Sanctions and international law","volume":"14","author":"K\u00f6chler","year":"2019","journal-title":"Int. Organ. Res. J."},{"key":"B27","doi-asserted-by":"publisher","first-page":"151","DOI":"10.2307\/27592225","article-title":"When, what, and why do states choose to delegate?","volume":"71","author":"Koremenos","year":"2008","journal-title":"Law Contemp. Probl."},{"key":"B28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.14710\/ip.v7i1.48592","article-title":"Mengapa rezim internasional gagal? analisis legalisasi \u201cLima Poin Konsensus ASEAN\u201d tentang Myanmar pasca kudeta militer 2021","volume":"7","author":"Kusumawardhana","year":"2022","journal-title":"Indones. Perspect."},{"key":"B29","doi-asserted-by":"publisher","first-page":"840","DOI":"10.1080\/09692290.2016.1231130","article-title":"The domestic battle over the design of non-trade issues in preferential trade agreements","volume":"23","author":"Lechner","year":"2016","journal-title":"Rev. Int. Polit. Econ."},{"key":"B30","doi-asserted-by":"publisher","first-page":"9459","DOI":"10.48550\/arXiv.2005.11401","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.4474552","article-title":"A computational approach and dataset for analysing international investment treaties","author":"Lie","year":"2023","journal-title":"SSRN [Preprint]"},{"key":"B32","doi-asserted-by":"publisher","first-page":"18","DOI":"10.3390\/e23010018","article-title":"Explainable AI: a review of machine learning interpretability methods","volume":"23","author":"Linardatos","year":"2020","journal-title":"Entropy"},{"key":"B33","doi-asserted-by":"publisher","first-page":"157","DOI":"10.1162\/tacl_a_00638","article-title":"Lost in the middle: how language models use long contexts","volume":"12","author":"Liu","year":"2024","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"B34","author":"Looijenga","year":"2024","journal-title":"RechtBERT: training a Dutch legal BERT model to enhance legaltech"},{"key":"B35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/03050629.2017.1311258","article-title":"Learning and diffusion in international investment agreements","volume":"43","author":"Manger","year":"2013","journal-title":"Int. Interact."},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.5220\/0013191800003890","article-title":"GAIus: combining GenAI with legal clauses retrieval for knowledge-based assistant","author":"Matak","year":"2025","journal-title":"arXiv [Preprint]"},{"key":"B37","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1007\/s10784-021-09545-4","article-title":"Environmental agreement under the non-interference principle: the case of ASEAN agreement on transboundary haze pollution","volume":"22","author":"Muhammad","year":"2022","journal-title":"Int. Environ. Agreements Polit. Law Econ."},{"key":"B38","doi-asserted-by":"crossref","DOI":"10.1355\/9789814786751","volume-title":"Does ASEAN Matter?: A View From Within","author":"Natalegawa","year":"2018"},{"key":"B39","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1017\/S0020818307070130","article-title":"Mercenaries: strong norm, weak law","volume":"61","author":"Percy","year":"2007","journal-title":"Int. Organ."},{"key":"B40","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/irap\/lcae007","article-title":"The delegation of discretionary power in international agreements: new comparative evidence from the Association of Southeast Asian Nations","volume":"25","author":"Pham","year":"2025","journal-title":"Int. Relat. Asia Pac."},{"key":"B41","doi-asserted-by":"publisher","first-page":"1020592","DOI":"10.3389\/frai.2023.1020592","article-title":"The assessment list for trustworthy artificial intelligence: a review and recommendations","volume":"6","author":"Radclyffe","year":"2023","journal-title":"Front. Artif. Intell."},{"key":"B42","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"B43","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1093\/jel\/eqw015","article-title":"The 2015 Paris Agreement: interplay between hard, soft and non-obligations","volume":"28","author":"Rajamani","year":"2016","journal-title":"J. Environ. Law"},{"key":"B44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/BF03372160","article-title":"Estimation of latent ability using a response pattern of graded scores","volume":"34","author":"Samejima","year":"1969","journal-title":"Psychometrika"},{"key":"B45","doi-asserted-by":"publisher","first-page":"1279794","DOI":"10.3389\/frai.2023.1279794","article-title":"The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts","volume":"6","author":"Savelka","year":"2023","journal-title":"Front. Artif. Intell."},{"key":"B46","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1080\/17449057.2023.2207876","article-title":"What's law got to do with it? how the degree of legalization affects the durability of post-conflict autonomy agreements","volume":"23","author":"Schulte","year":"2024","journal-title":"Ethnopolitics"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2507.07748","article-title":"When large language models meet law: dual-lens taxonomy, technical advances, and ethical governance","author":"Shao","year":"2025","journal-title":"arXiv [Preprint]."},{"key":"B48","doi-asserted-by":"publisher","first-page":"021008","DOI":"10.1115\/1.4052293","article-title":"Engineering knowledge graph from patent database","volume":"22","author":"Siddharth","year":"2022","journal-title":"J. Comput. Inf. Sci. Eng."},{"key":"B49","doi-asserted-by":"publisher","first-page":"112410","DOI":"10.1016\/j.knosys.2024.112410","article-title":"Retrieval augmented generation using engineering design knowledge","volume":"303","author":"Siddharth","year":"2024","journal-title":"Knowl. Based Syst."},{"key":"B50","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1016\/j.sheji.2025.06.002","article-title":"Data-driven innovation for trustworthy AI","volume":"11","author":"Siddharth","year":"2025","journal-title":"She Ji J. Des. Econ. Innov."},{"key":"B51","doi-asserted-by":"publisher","first-page":"380","DOI":"10.1007\/s10462-025-11359-8","article-title":"A survey of classification tasks and approaches for legal contracts","volume":"58","author":"Singh","year":"2025","journal-title":"Artif. Intell. Rev."},{"key":"B52","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1109\/MC.2022.3212022","article-title":"Wasabi: a conceptual model for trustworthy artificial intelligence","volume":"56","author":"Singh","year":"2023","journal-title":"Computer"},{"key":"B53","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1111\/eulj.12090","article-title":"Soft law in the European Union\u2014the changing nature of EU law","volume":"21","author":"Terpan","year":"2015","journal-title":"Eur. Law J."},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.10097","article-title":"LegalPro-BERT: classification of legal provisions by fine-tuning BERT large language model","author":"Tewari","year":"2024","journal-title":"arXiv [Preprint]"},{"key":"B55","doi-asserted-by":"publisher","first-page":"e0298380","DOI":"10.1371\/journal.pone.0298380","article-title":"An NLP-based novel approach for assessing national influence in clause dissemination across bilateral investment treaties","volume":"19","author":"Uddin","year":"2024","journal-title":"PLoS ONE"},{"key":"B56","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.03533","article-title":"Text embeddings by weakly-supervised contrastive pre-training","author":"Wang","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B57","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.04398","article-title":"Chain-of-table: evolving tables in the reasoning chain for table understanding","author":"Wang","year":"2024","journal-title":"arXiv [Preprint]"},{"key":"B58","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1093\/icsidreview\/siab049","article-title":"Conciliation and mediation in investor-state dispute settlement provisions: a quantitative and qualitative analysis","volume":"38","author":"Weeramantry","year":"2023","journal-title":"ICSID Rev. Foreign Invest. Law J."},{"key":"B59","doi-asserted-by":"publisher","first-page":"24824","DOI":"10.52202\/068431-1800","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B60","first-page":"445","article-title":"\u201cCBR-RAG: case-based reasoning for retrieval augmented generation in LLMs for legal question answering,\u201d","volume-title":"International Conference on Case-Based Reasoning","author":"Wiratunga","year":"2024"},{"key":"B61","doi-asserted-by":"publisher","first-page":"6","DOI":"10.17304\/ijil.vol22.1.1726","article-title":"Legal instrument under the \u201cASEAN Way\u201d: the case of ASEAN Convention Against Trafficking in Persons","volume":"22","author":"Wirawan","year":"2024","journal-title":"Indones. J. Int. Law"},{"key":"B62","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/j.aiopen.2021.06.003","article-title":"Lawformer: a pre-trained language model for Chinese legal long documents","volume":"2","author":"Xiao","year":"2021","journal-title":"AI Open"},{"key":"B63","doi-asserted-by":"publisher","first-page":"11809","DOI":"10.52202\/075280-0517","article-title":"Tree of thoughts: deliberate problem solving with large language models","volume":"36","author":"Yao","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B64","doi-asserted-by":"crossref","first-page":"7577","DOI":"10.18653\/v1\/2024.findings-emnlp.445","article-title":"\u201cCan large language models grasp legal theories? Enhance legal reasoning with insights from multi-agent collaboration,\u201d","volume-title":"Findings Assoc Comput Linguist EMNLP 2024","author":"Yuan","year":"2024"},{"key":"B65","doi-asserted-by":"crossref","first-page":"6144","DOI":"10.63317\/2yywa664nb4b","article-title":"\u201cEnhancing zero-shot chain-of-thought reasoning in large language models through logic,\u201d","volume-title":"Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC\u2013COLING 2024)","author":"Zhao","year":"2024"},{"key":"B66","first-page":"159","article-title":"\u201cWhen does pretraining help? Assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings,\u201d","author":"Zheng","year":"2021","journal-title":"Proceedings of the 18th International Conference on Artificial Intelligence and Law"},{"key":"B67","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.08734","article-title":"Thread of thought: unraveling chaotic contexts","author":"Zhou","year":"2023","journal-title":"arXiv [Preprint]"}],"updated-by":[{"DOI":"10.3389\/frai.2026.1932998","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T00:00:00Z","timestamp":1784678400000}}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1784200\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T08:26:11Z","timestamp":1784708771000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1784200\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,9]]},"references-count":67,"alternative-id":["10.3389\/frai.2026.1784200"],"URL":"https:\/\/doi.org\/10.3389\/frai.2026.1784200","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,9]]},"article-number":"1784200"}}