{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T07:47:42Z","timestamp":1780472862504,"version":"3.54.1"},"reference-count":17,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,1,12]],"date-time":"2024-01-12T00:00:00Z","timestamp":1705017600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,12]],"date-time":"2024-01-12T00:00:00Z","timestamp":1705017600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["DGECR-2022-00369 and RGPIN-2022-03469"],"award-info":[{"award-number":["DGECR-2022-00369 and RGPIN-2022-03469"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100013373","name":"Alberta Machine Intelligence Institute","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100013373","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100009192","name":"Alberta Innovates","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100009192","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Rev Socionetwork Strat"],"published-print":{"date-parts":[[2024,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We summarize the 10th Competition on Legal Information Extraction and Entailment. In this tenth edition, the competition included four tasks on case law and statute law. The case law component includes an information retrieval task (Task 1), and the confirmation of an entailment relation between an existing case and a selected unseen case (Task 2). The statute law component includes an information retrieval task (Task 3), and an entailment\/question-answering task based on retrieved civil code statutes (Task 4). Participation was open to any group based on any approach. Ten different teams participated in the case law competition tasks, most of them in more than one task. We received results from 8 teams for Task 1 (22 runs) and seven teams for Task 2 (18 runs). On the statute law task, there were 9 different teams participating, most in more than one task. 6 teams submitted a total of 16 runs for Task 3, and 9 teams submitted a total of 26 runs for Task 4. We describe the variety of approaches, our official evaluation, and analysis of our data and submission results.<\/jats:p>","DOI":"10.1007\/s12626-023-00152-0","type":"journal-article","created":{"date-parts":[[2024,1,12]],"date-time":"2024-01-12T08:02:45Z","timestamp":1705046565000},"page":"27-47","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Overview and Discussion of the Competition on Legal Information, Extraction\/Entailment (COLIEE) 2023"],"prefix":"10.1007","volume":"18","author":[{"given":"Randy","family":"Goebel","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoshinobu","family":"Kano","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4486-9738","authenticated-orcid":false,"given":"Mi-Young","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Juliano","family":"Rabelo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ken","family":"Satoh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masaharu","family":"Yoshioka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,1,12]]},"reference":[{"key":"152_CR1","unstructured":"Bilgin, O., Fields, L., Jr., A. L., Marji, Z., Nighojkar, A., Steinle, S., & Licato, J. (2023) Amhr lab 2023 coliee competition approach. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR2","unstructured":"Bui, Q. M., Do, D. T., Le, N. K., Nguyen, D. H., Nguyen, K. V. H., Anh, T. P. N., & Nguyen, M. L. (2023). Jnlp $$@$$coliee-2023: Data argumentation and large language model for legal case retrieval and entailment. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR3","unstructured":"Custeau, M., & Inkpen, D. (2023). Individual models can perform better than agreement-based ensembles. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR4","unstructured":"Debbarma, R., Prawar, P., Chakraborty, A., & Bedathur, S. (2023) Iitdli: Legal case retrieval based on lexical models. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR5","unstructured":"Li, H., Su, W., Wang, C., Wu, Y., Ai, Q., & Liu, Y. (2023). Thuir$$@$$coliee 2023: Incorporating structural knowledge into pre-trained language models for legal case retrieval. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR6","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692"},{"key":"152_CR7","unstructured":"Nguyen, C., & Nguyen, M. L. (2023). Captain at coliee 2023: Efficient methods for legal information retrieval and entailment tasks. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR8","doi-asserted-by":"publisher","unstructured":"Nogueira, R., Jiang, Z., Pradeep, R., & Lin, J. (2020). Document ranking with a pretrained sequence-to-sequence model. In Findings of the association for computational linguistics: EMNLP 2020, pp. 708\u2013718. Association for Computational Linguistics, Online. https:\/\/doi.org\/10.18653\/v1\/2020.findings-emnlp.63. https:\/\/aclanthology.org\/2020.findings-emnlp.63.","DOI":"10.18653\/v1\/2020.findings-emnlp.63"},{"key":"152_CR9","unstructured":"Novaes, L. P., Vianna, D., & da\u00a0Silva, A. (2023). A topic-based approach for the legal case retrieval task. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR10","unstructured":"Onaga, T., Fujita, M., & Kano, Y. (2023). Japanese legal bar problem solver focusing on person names. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR11","unstructured":"Rabelo, J., Kim, M. Y., & Goebel, R. (2023). Transformer-based legal information extraction. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"key":"152_CR12","doi-asserted-by":"publisher","unstructured":"Rabelo, J., Kim, M. Y., Goebel, R., Yoshioka, M., Kano, Y., & Satoh, K. (2021). COLIEE 2020: Methods for legal document retrieval and entailment, pp. 196\u2013210. https:\/\/doi.org\/10.1007\/978-3-030-79942-7_13.","DOI":"10.1007\/978-3-030-79942-7_13"},{"issue":"4","key":"152_CR13","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1561\/1500000019","volume":"3","author":"S Robertson","year":"2009","unstructured":"Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333\u2013389. https:\/\/doi.org\/10.1561\/1500000019","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"152_CR14","doi-asserted-by":"crossref","unstructured":"Vuong, T. H. Y., Nguyen, H. L., Nguyen, T. M., Nguyen, H. T., Nguyen, T. B., & Nguyen, H. T. (2023). Nowj at coliee 2023 - multi-task and ensemble approaches in legal information processing. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)","DOI":"10.1007\/s12626-024-00157-3"},{"key":"152_CR15","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/978-3-319-93794-6_1","volume-title":"New frontiers in artificial intelligence","author":"M Yoshioka","year":"2018","unstructured":"Yoshioka, M. (2018). Analysis of coliee information retrieval task data. In S. Arai, K. Kojima, K. Mineshima, D. Bekki, K. Satoh, & Y. Ohta (Eds.), New frontiers in artificial intelligence (pp. 5\u201319). Cham: Springer International Publishing."},{"key":"152_CR16","unstructured":"Yoshioka, M., & Aoki, Y. (2023). Hukb at coliee 2023 statute law task. In Workshop of the tenth competition on legal information extraction\/entailment (COLIEE\u20192023) in the 19th international conference on artificial intelligence and law (ICAIL)"},{"issue":"1","key":"152_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/978-3-031-02130-5","volume":"1","author":"C Zhai","year":"2008","unstructured":"Zhai, C. (2008). Statistical language models for information retrieval. Synthesis Lectures on Human Language Technologies, 1(1), 1\u2013141.","journal-title":"Synthesis Lectures on Human Language Technologies"}],"container-title":["The Review of Socionetwork Strategies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12626-023-00152-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12626-023-00152-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12626-023-00152-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,18]],"date-time":"2024-04-18T10:31:27Z","timestamp":1713436287000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12626-023-00152-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,12]]},"references-count":17,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,4]]}},"alternative-id":["152"],"URL":"https:\/\/doi.org\/10.1007\/s12626-023-00152-0","relation":{},"ISSN":["2523-3173","1867-3236"],"issn-type":[{"value":"2523-3173","type":"print"},{"value":"1867-3236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,12]]},"assertion":[{"value":"1 September 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 November 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 January 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}