{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:18:22Z","timestamp":1784179102648,"version":"3.55.0"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T00:00:00Z","timestamp":1644883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Rev Socionetwork Strat"],"published-print":{"date-parts":[[2022,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We summarize the 8th Competition on Legal Information Extraction and Entailment. In this edition, the competition included five tasks on case law and statute law. The case law component includes an information retrieval Task (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component includes an information retrieval Task (Task 3), an entailment\/question answering task based on retrieved civil code statutes (Task 4) and an entailment\/question answering task without retrieved civil code statutes (Task 5). Participation was open to any group based on any approach. Eight different teams participated in the case law competition tasks, most of them in more than one task. We received results from six teams for Task 1 (16 runs) and 6 teams for Task 2 (17 runs). On the statute law task, there were eight different teams participating, most in more than one task. Six teams submitted a total of 18 runs for Task 3, 6 teams submitted a total of 18 runs for Task 4, and 4 teams submitted a total of 12 runs for Task 5. Here we summarize the approaches, our official evaluation, and analysis on our data and submission results.<\/jats:p>","DOI":"10.1007\/s12626-022-00105-z","type":"journal-article","created":{"date-parts":[[2022,2,15]],"date-time":"2022-02-15T18:05:11Z","timestamp":1644948311000},"page":"111-133","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":69,"title":["Overview and Discussion of the Competition on Legal Information Extraction\/Entailment (COLIEE) 2021"],"prefix":"10.1007","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2982-3401","authenticated-orcid":false,"given":"Juliano","family":"Rabelo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Randy","family":"Goebel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mi-Young","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoshinobu","family":"Kano","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masaharu","family":"Yoshioka","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ken","family":"Satoh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,2,15]]},"reference":[{"key":"105_CR1","unstructured":"Althammer, S., Askari, A., Verberne, S., & Hanbury, A. (2021). Dossier@coliee 2021: Leveraging dense retrieval and summarization-based re-ranking for case law retrieval. In: Proceedings of the COLIEE Workshop in ICAIL."},{"key":"105_CR2","doi-asserted-by":"publisher","first-page":"143","DOI":"10.5626\/JCSE.2009.3.3.143","volume":"3","author":"P Banerjee","year":"2009","unstructured":"Banerjee, P., & Han, H. (2009). Language modeling approaches to information retrieval. JCSE, 3, 143\u2013164. https:\/\/doi.org\/10.5626\/JCSE.2009.3.3.143.","journal-title":"JCSE"},{"key":"105_CR3","unstructured":"Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners."},{"key":"105_CR4","unstructured":"Clark, K., Luong, M.T., Le, Q.V., & Manning, C.D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. CoRR. http:\/\/arxiv.org\/abs\/1910.10683"},{"key":"105_CR5","unstructured":"Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR. http:\/\/arxiv.org\/abs\/1810.04805"},{"key":"105_CR6","unstructured":"Fujita, M., Kiyota, N., & Kano, Y. (2021). Predicate\u2019s argument resolver and entity abstraction for legal question answering: Kis teams at coliee 2021 shared task. In: Proceedings of the COLIEE Workshop in ICAIL."},{"key":"105_CR7","unstructured":"He, P., Liu, X., Gao, J., & Chen, W. (2020). Deberta: Decoding-enhanced BERT with disentangled attention. CoRR. https:\/\/arxiv.org\/abs\/2006.03654"},{"key":"105_CR8","doi-asserted-by":"crossref","unstructured":"Karpukhin, V., Oguz, B., Min, S., Wu, L., Edunov, S., Chen, D., & Yih, W. (2020). Dense passage retrieval for open-domain question answering. CoRR. https:\/\/arxiv.org\/abs\/2004.04906","DOI":"10.18653\/v1\/2020.emnlp-main.550"},{"key":"105_CR9","unstructured":"Kim, M.Y., Rabelo, J., & Goebel, R. (2021). Bm25 and transformer-based legal information extraction and entailment. In: Proceedings of the COLIEE Workshop in ICAIL."},{"key":"105_CR10","unstructured":"Kusner, M.J., Sun, Y., Kolkin, N.I., & Weinberger, K.Q. (2015). From word embeddings to document distances. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML\u201915, pp. 957\u2013966. JMLR.org."},{"key":"105_CR11","unstructured":"Li, J., Zhao, X., Liu, J., Wen, J., & Yang, M. (2021). Siat@coliee-2021: Combining statistics recall and semantic ranking for legal case retrieval and entailment. In: Proceedings of the COLIEE Workshop in ICAIL."},{"key":"105_CR12","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach"},{"key":"105_CR13","unstructured":"Ma, Y., Shao, Y., Liu, B., Liu, Y., Zhang, M., & Ma, S. (2021). Retrieving legal cases from a large-scale candidate corpus. In: Proceedings of the 18th International conference on Artificial Intelligence and Law (ICAIL.)"},{"key":"105_CR14","unstructured":"Nguyen, H.T., Nguyen, P.M., Vuong, T.H.Y., Bui, Q.M., Nguyen, C.M., Dang, B.T., Tran, V., Nguyen, M.L., & Satoh, K. (2021). Jnlp team: Deep learning approaches for legal processing tasks in coliee 2021. In: Proceedings of the COLIEE Workshop in ICAIL."},{"key":"105_CR15","unstructured":"Nguyen, H.T., Tran, V., Minh, N.L., Nguyen, M.P., Vuong, T.H.Y., Bui, M.Q., Nguyen, M.C., Dang, B., & Satoh, K. (2021). Paralaw nets - cross-lingual sentence-level pretraining for legal text processing. In: Proceedings of the COLIEE Workshop in ICAIL"},{"key":"105_CR16","doi-asserted-by":"publisher","unstructured":"Rabelo, J., Kim, M.Y., Goebel, R., Yoshioka, M., Kano, Y., & Satoh, K. (2021). COLIEE 2020: Methods for legal document retrieval and entailment, pp. 196\u2013210 . https:\/\/doi.org\/10.1007\/978-3-030-79942-7_13","DOI":"10.1007\/978-3-030-79942-7_13"},{"key":"105_CR17","unstructured":"Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P.J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR. http:\/\/arxiv.org\/abs\/1910.10683"},{"key":"105_CR18","doi-asserted-by":"crossref","unstructured":"Reimers, N., & Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3973\u20133983.","DOI":"10.18653\/v1\/D19-1410"},{"key":"105_CR19","doi-asserted-by":"crossref","unstructured":"Rosa, G.M., Rodrigues, R.C., de\u00a0Alencar\u00a0Lotufo, R., & Nogueira, R. (2021). To tune or not to tune? zero-shot models for legal case entailment. In: Proceedings of the 18th International conference on Artificial Intelligence and Law (ICAIL).","DOI":"10.1145\/3462757.3466103"},{"key":"105_CR20","unstructured":"Rosa, G.M., Rodrigues, R.C., Lotufo, R., & Nogueira, R. (2021). Yes, bm25 is a strong baseline for legal case retrieval. In: Proceedings of the 18th International conference on Artificial Intelligence and Law (ICAIL)."},{"key":"105_CR21","unstructured":"Schilder, F., Chinnappa, D., Madan, K., Harmouche, J., Vold, A., Bretz, H., & Hudzina, J. (2021). A pentapus grapples with legal reasoning. In: Proceedings of the COLIEE Workshop in ICAIL."},{"key":"105_CR22","unstructured":"Shao, H.L., Chen, Y.C., & Huang, S.C. (2020). BERT-based ensemble model for the statute law retrieval and legal information entailment. In: The Proceedings of the 14th International Workshop on Juris-Informatics (JURISIN2020), pp. 223\u2013234. The Japanese Society of Artificial Intelligence."},{"key":"105_CR23","unstructured":"Strohman, T., Metzler, D., Turtle, H., & Croft, W.B. (2005). Indri: A language model-based search engine for complex queries. In: Proceedings of the International Conference on Intelligent Analysis, pp. 2\u20136."},{"key":"105_CR24","doi-asserted-by":"crossref","unstructured":"Trivedi, H., Kwon, H., Khot, T., Sabharwal, A., & Balasubramanian, N. (2019). Repurposing entailment for multi-hop question answering tasks. In: Proc. of NAACL.","DOI":"10.18653\/v1\/N19-1302"},{"key":"105_CR25","doi-asserted-by":"crossref","unstructured":"Wehnert, S., Sudhi, V., Dureja, S., Kutty, L., Shahania, S., & Luca, E.W.D. (2021). Legal norm retrieval with variations of the bert model combined with tf-idf vectorization. In: Proceedings of the 18th International conference on Artificial Intelligence and Law (ICAIL).","DOI":"10.1145\/3462757.3466104"},{"key":"105_CR26","doi-asserted-by":"crossref","unstructured":"Yoshioka, M., Aoki, Y., & Suzuki, Y. (2021). Bert-based ensemble methods with data augmentation for legal textual entailment in coliee statute law task. In: Proceedings of the COLIEE Workshop in ICAIL.","DOI":"10.1145\/3462757.3466105"},{"key":"105_CR27","doi-asserted-by":"crossref","unstructured":"Yoshioka, M., Suzuki, Y., & Aoki, Y. (2021). Bert-based ensemble methods for information retrieval and legal textual entailment in coliee statute law task. In: Proceedings of the COLIEE Workshop in ICAIL.","DOI":"10.1145\/3462757.3466105"}],"container-title":["The Review of Socionetwork Strategies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12626-022-00105-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12626-022-00105-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12626-022-00105-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,24]],"date-time":"2022-04-24T04:03:40Z","timestamp":1650773020000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12626-022-00105-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,15]]},"references-count":27,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,4]]}},"alternative-id":["105"],"URL":"https:\/\/doi.org\/10.1007\/s12626-022-00105-z","relation":{},"ISSN":["2523-3173","1867-3236"],"issn-type":[{"value":"2523-3173","type":"print"},{"value":"1867-3236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,15]]},"assertion":[{"value":"13 September 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 January 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 February 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}