{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T16:53:36Z","timestamp":1771606416920,"version":"3.50.1"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T00:00:00Z","timestamp":1687910400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T00:00:00Z","timestamp":1687910400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["949316"],"award-info":[{"award-number":["949316"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Artif Intell Law"],"published-print":{"date-parts":[[2024,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>With the ever-growing accessibility of case law online, it has become challenging to manually identify case law relevant to one\u2019s legal issue. In the Netherlands, the planned increase in the online publication of case law is expected to exacerbate this challenge. In this paper, we tried to predict whether court decisions are cited by other courts or not after being published, thus in a way distinguishing between more and less authoritative cases. This type of system may be used to process the large amounts of available data by filtering out large quantities of non-authoritative decisions, thus helping legal practitioners and scholars to find relevant decisions more easily, and drastically reducing the time spent on preparation and analysis. For the Dutch Supreme Court, the match between our prediction and the actual data was relatively strong (with a Matthews Correlation Coefficient of 0.60). Our results were less successful for the Council of State and the district courts (MCC scores of 0.26 and 0.17, relatively). We also attempted to identify the most informative characteristics of a decision. We found that a completely explainable model, consisting only of handcrafted metadata features, performs almost as well as a less well-explainable system based on all text of the decision.\n<\/jats:p>","DOI":"10.1007\/s10506-023-09368-5","type":"journal-article","created":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T07:02:21Z","timestamp":1687935741000},"page":"807-837","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Predicting citations in Dutch case law with natural language processing"],"prefix":"10.1007","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4036-5904","authenticated-orcid":false,"given":"Iris","family":"Schepers","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2972-8447","authenticated-orcid":false,"given":"Masha","family":"Medvedeva","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1904-8122","authenticated-orcid":false,"given":"Michelle","family":"Bruijn","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0434-1526","authenticated-orcid":false,"given":"Martijn","family":"Wieling","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5762-8697","authenticated-orcid":false,"given":"Michel","family":"Vols","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,6,28]]},"reference":[{"key":"9368_CR1","doi-asserted-by":"publisher","DOI":"10.1017\/9781316761380","volume-title":"Artificial intelligence and legal analytics: new tools for law practice in the digital age","author":"KD Ashley","year":"2017","unstructured":"Ashley KD (2017) Artificial intelligence and legal analytics: new tools for law practice in the digital age. Cambridge University Press"},{"issue":"5","key":"9368_CR2","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/scientificamerican0503-60","volume":"288","author":"AL Barab\u00e1si","year":"2003","unstructured":"Barab\u00e1si AL, Bonabeau E (2003) Scale-free networks. Sci Am 288(5):60\u201369","journal-title":"Sci Am"},{"key":"9368_CR3","doi-asserted-by":"crossref","unstructured":"Chalkidis I, Androutsopoulos I, Aletras N (2019) Neural legal judgment prediction in English. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp 4317\u20134323","DOI":"10.18653\/v1\/P19-1424"},{"key":"9368_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12864-019-6413-7","volume":"21","author":"D Chicco","year":"2020","unstructured":"Chicco D, Jurman G (2020) The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC Genomics 21:1\u201313","journal-title":"BMC Genomics"},{"issue":"2","key":"9368_CR5","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1093\/jiel\/jgx011","volume":"20","author":"M Derl\u00e9n","year":"2017","unstructured":"Derl\u00e9n M, Lindholm J (2017) Is it good law? network analysis and the cjeu\u2019s internal market jurisprudence. J Int Econ Law 20(2):257\u2013277","journal-title":"J Int Econ Law"},{"issue":"1","key":"9368_CR6","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.socnet.2007.05.001","volume":"30","author":"JH Fowler","year":"2008","unstructured":"Fowler JH, Jeon S (2008) The authority of Supreme Court precedent. Soc Netw 30(1):16\u201330","journal-title":"Soc Netw"},{"key":"9368_CR7","unstructured":"Gholamy A, Kreinovich V, Kosheleva O (2018) Why 70\/30 or 80\/20 relation between training and testing sets: a pedagogical explanation. Int J Intell Technol Appl Stat 11(2)"},{"key":"9368_CR8","doi-asserted-by":"crossref","unstructured":"Hernandez\u00a0Serrano PV, Moodley K, Van\u00a0Dijck G, Dumontier M (2020) Sleeping beauties in case law. In: Legal Knowledge and Information Systems, IOS Press, pp 231\u2013234","DOI":"10.3233\/FAIA200871"},{"issue":"4","key":"9368_CR9","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0174698","volume":"12","author":"DM Katz","year":"2017","unstructured":"Katz DM, Bommarito MJ, Blackman J (2017) A general approach for predicting the behavior of the Supreme Court of the United States. PloS one 12(4):e0174698","journal-title":"PloS one"},{"key":"9368_CR10","unstructured":"Kaur A, Bozic B (2019) Convolutional neural network-based automatic prediction of judgments of the European Court of Human Rights. In: Irish Conference on Artificial Intelligence and Cognitive Science"},{"issue":"24","key":"9368_CR11","doi-asserted-by":"publisher","first-page":"7426","DOI":"10.1073\/pnas.1424329112","volume":"112","author":"Q Ke","year":"2015","unstructured":"Ke Q, Ferrara E, Radicchi F, Flammini A (2015) Defining and identifying sleeping beauties in science. Proc Nat Acad Sci 112(24):7426\u20137431","journal-title":"Proc Nat Acad Sci"},{"key":"9368_CR12","unstructured":"Kuppevelt D, Dijck G (2017) Answering legal research questions about dutch case law with network analysis and visualization. In: Legal Knowledge and Information Systems: JURIX 2017: The Thirtieth Annual Conference, IOS Press, vol 302, p\u00a095"},{"issue":"1","key":"9368_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s41109-018-0110-3","volume":"4","author":"JC Leit\u00e3o","year":"2019","unstructured":"Leit\u00e3o JC, Lehmann S, Olsen HP (2019) Quantifying long-term impact of court decisions. Appl Netw Sci 4(1):1\u201315","journal-title":"Appl Netw Sci"},{"issue":"2","key":"9368_CR14","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1007\/s10506-019-09243-2","volume":"27","author":"M Lippi","year":"2019","unstructured":"Lippi M, Pa\u0142ka P, Contissa G, Lagioia F, Micklitz HW, Sartor G, Torroni P (2019) CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service. Artifi Intell Law 27(2):117\u2013139","journal-title":"Artifi Intell Law"},{"key":"9368_CR15","doi-asserted-by":"crossref","unstructured":"Medvedeva M, Vols M, Wieling M (2020) Using machine learning to predict decisions of the European Court of Human Rights. Artif Intell Law pp 1\u201330","DOI":"10.1007\/s10506-019-09255-y"},{"key":"9368_CR16","doi-asserted-by":"crossref","unstructured":"Medvedeva M, Dam T, Wieling M, Vols M (2021a) Automatically identifying eviction cases and outcomes within case law of Dutch courts of first instance. In: Legal Knowledge and Information Systems, IOS Press, pp 13\u201322","DOI":"10.3233\/FAIA210312"},{"key":"9368_CR17","unstructured":"Medvedeva M, \u00dcst\u00fcn A, Xu X, Vols M, Wieling M (2021b) Automatic judgement forecasting for pending applications of the European Court of Human Rights. In: ASAIL\/LegalAIIA ICAIL"},{"key":"9368_CR18","doi-asserted-by":"crossref","unstructured":"Medvedeva M, Wieling M, Vols M (2022) Rethinking the field of automatic prediction of court decisions. Artif Intell Law pp 1\u201318","DOI":"10.1007\/s10506-021-09306-3"},{"key":"9368_CR19","doi-asserted-by":"crossref","unstructured":"Moens MF, Uyttendaele C, Dumortier J (1997) Abstracting of legal cases: The SALOMON experience. In: Proceedings of the 6th international conference on Artificial intelligence and law, pp 114\u2013122","DOI":"10.1145\/261618.261643"},{"key":"9368_CR20","doi-asserted-by":"crossref","unstructured":"Mones E, Sapie zy\u0144ski P, Thordal S, Olsen HP, Lehmann S, (2021) Emergence of network effects and predictability in the judicial system. Sci Rep 11(1):1\u201310","DOI":"10.1038\/s41598-021-82430-x"},{"key":"9368_CR21","unstructured":"O\u2019Sullivan C, Beel J (2019) Predicting the outcome of judicial decisions made by the European Court of Human Rights. 27th AIAI Irish Conference on Artificial Intelligence and Cognitive Science URL https:\/\/ceur-ws.org\/Vol-2563\/aics_26.pdf"},{"key":"9368_CR22","doi-asserted-by":"crossref","unstructured":"Pandya V (2019) Automatic text summarization of legal cases: A hybrid approach. In: CS & IT Conference Proceedings, CS & IT Conference Proceedings, vol 9 (10)","DOI":"10.5121\/csit.2019.91004"},{"key":"9368_CR23","volume-title":"The relevance of the network approach to European (case) law: reflection and evidence","author":"U Sadl","year":"2020","unstructured":"Sadl U, Tarissan F (2020) The relevance of the network approach to European (case) law: reflection and evidence. Oxford University Press"},{"key":"9368_CR24","doi-asserted-by":"crossref","unstructured":"Sartor G, Santin P, Audrito D, Sulis E, Di\u00a0Caro L (2023) Automated extraction and representation of citation network: A cjeu case-study. In: Advances in Conceptual Modeling: ER 2022 Workshops, CMLS, EmpER, and JUSMOD, Hyderabad, India, October 17\u201320, 2022, Proceedings, Springer, pp 102\u2013111","DOI":"10.1007\/978-3-031-22036-4_10"},{"key":"9368_CR25","doi-asserted-by":"crossref","unstructured":"Sleimi A, Sannier N, Sabetzadeh M, Briand L, Dann J (2018) Automated extraction of semantic legal metadata using natural language processing. In: 2018 IEEE 26th International Requirements Engineering Conference (RE), IEEE, pp 124\u2013135","DOI":"10.1109\/RE.2018.00022"},{"key":"9368_CR26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10664-020-09901-z","volume":"26","author":"A Sleimi","year":"2021","unstructured":"Sleimi A, Sannier N, Sabetzadeh M, Briand L, Ceci M, Dann J (2021) An automated framework for the extraction of semantic legal metadata from legal texts. Emp Softw Eng 26:1\u201350","journal-title":"Emp Softw Eng"},{"key":"9368_CR27","doi-asserted-by":"crossref","unstructured":"Spinosa P, Giardiello G, Cherubini M, Marchi S, Venturi G, Montemagni S (2009) NLP-based metadata extraction for legal text consolidation. In: Proceedings of the 12th International Conference on Artificial Intelligence and Law, pp 40\u201349","DOI":"10.1145\/1568234.1568240"},{"key":"9368_CR28","doi-asserted-by":"crossref","unstructured":"Strickson B, De\u00a0La\u00a0Iglesia B (2020) Legal judgement prediction for UK courts. In: Proceedings of the 2020 The 3rd International Conference on Information Science and System, pp 204\u2013209","DOI":"10.1145\/3388176.3388183"},{"key":"9368_CR29","unstructured":"\u015eulea OM, Zampieri M, Malmasi S, Vela M, P\u00a0Dinu L, van Genabith J (2017a) Exploring the use of text classification in the legal domain. In: Proceedings of 2nd Workshop on Automated Semantic Analysis of Information in Legal Texts (ASAIL), London, United Kingdom"},{"key":"9368_CR30","doi-asserted-by":"publisher","unstructured":"\u015eulea OM, Zampieri M, Vela M, van Genabith J (2017b) Predicting the law area and decisions of French Supreme Court cases. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, INCOMA Ltd., pp 716\u2013722, https:\/\/doi.org\/10.26615\/978-954-452-049-6_092,","DOI":"10.26615\/978-954-452-049-6_092"},{"key":"9368_CR31","first-page":"95","volume":"250","author":"M Van Opijnen","year":"2012","unstructured":"Van Opijnen M (2012) Citation analysis and beyond: in search of indicators measuring case law importance. JURIX 250:95\u2013104","journal-title":"JURIX"},{"key":"9368_CR32","doi-asserted-by":"crossref","unstructured":"Van\u00a0Opijnen M (2013) A model for automated rating of case law. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Law, pp 140\u2013149","DOI":"10.1145\/2514601.2514617"},{"key":"9368_CR33","unstructured":"Van\u00a0Opijnen M (2016) Towards a global importance indicator for court decisions. In: Legal Knowledge and Information Systems: JURIX 2016: The Twenty-Ninth Annual Conference, IOS Press, vol 294, p 155"},{"key":"9368_CR34","unstructured":"Van\u00a0Opijnen M (2018) Legal(ly) linked data. over de noodzaak tot betere standaardisatie van juridisch bronmateriaal (legal (ly) linked data about the need for better standardisation of legal sources). Computerrecht 51"},{"issue":"5","key":"9368_CR35","doi-asserted-by":"publisher","first-page":"988","DOI":"10.1109\/72.788640","volume":"10","author":"VN Vapnik","year":"1999","unstructured":"Vapnik VN (1999) An overview of statistical learning theory. IEEE Trans Neural Netw 10(5):988\u2013999","journal-title":"IEEE Trans Neural Netw"},{"key":"9368_CR36","unstructured":"Verheugt J (2020) Inleiding in het Nederlandse recht. Uitgeverij De Zuidas"},{"key":"9368_CR37","doi-asserted-by":"crossref","unstructured":"Virtucio MBL, Aborot JA, Abonita JKC, Avi\u00f1ante RS, Copino RJB, Neverida MP, Osiana VO, Peramo EC, Syjuco JG, Tan GBA (2018) Predicting decisions of the Philippine Supreme Court using natural language processing and machine learning. In: 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), IEEE, vol\u00a02, pp 130\u2013135","DOI":"10.1109\/COMPSAC.2018.10348"},{"key":"9368_CR38","volume-title":"Legal Research: One Hundred Questions and Answers","author":"M Vols","year":"2021","unstructured":"Vols M (2021) Legal Research: One Hundred Questions and Answers. Eleven"},{"key":"9368_CR39","unstructured":"Vols M (2021b) Methodologische kansen en beperkingen van gebruik van statistiek en machine learning bij de bestudering van rechtspraak. In: Methoden van systematische rechtspraakanalyse: Tussen juridische dogmatiek en data science, Boom Juridisch, pp 125\u2013141"},{"key":"9368_CR40","unstructured":"Whalen R (2016) Legal networks: The promises and challenges of legal network analysis. Mich St L Rev p 539"},{"key":"9368_CR41","doi-asserted-by":"crossref","unstructured":"Whalen R (2020) Computational Legal Studies: The Promise and Challenge of Data-Driven Research. Edward Elgar Publishing","DOI":"10.4337\/9781788977456"},{"key":"9368_CR42","doi-asserted-by":"crossref","unstructured":"Winkels R, de\u00a0Ruyter J (2011) Survival of the fittest: network analysis of Dutch Supreme Court cases. In: International Workshop on AI Approaches to the Complexity of Legal Systems, Springer, pp 106\u2013115","DOI":"10.1007\/978-3-642-35731-2_7"},{"issue":"1","key":"9368_CR43","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10115-007-0114-2","volume":"14","author":"X Wu","year":"2008","unstructured":"Wu X, Kumar V, Quinlan JR, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Philip SY et al (2008) Top 10 algorithms in data mining. Knowl Inf Syst 14(1):1\u201337","journal-title":"Knowl Inf Syst"},{"key":"9368_CR44","doi-asserted-by":"crossref","unstructured":"Zhong H, Guo Z, Tu C, Xiao C, Liu Z, Sun M (2018) Legal judgment prediction via topological learning. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 3540\u20133549","DOI":"10.18653\/v1\/D18-1390"},{"key":"9368_CR45","volume-title":"Introduction to comparative law","author":"K Zweigert","year":"1998","unstructured":"Zweigert K, K\u00f6tz H (1998) Introduction to comparative law, 3rd edn. Oxford University Press, USA","edition":"3"}],"container-title":["Artificial Intelligence and Law"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10506-023-09368-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10506-023-09368-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10506-023-09368-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,31]],"date-time":"2024-07-31T18:50:12Z","timestamp":1722451812000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10506-023-09368-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,28]]},"references-count":45,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,9]]}},"alternative-id":["9368"],"URL":"https:\/\/doi.org\/10.1007\/s10506-023-09368-5","relation":{},"ISSN":["0924-8463","1572-8382"],"issn-type":[{"value":"0924-8463","type":"print"},{"value":"1572-8382","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,28]]},"assertion":[{"value":"12 June 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 June 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}