{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T04:19:01Z","timestamp":1772252341986,"version":"3.50.1"},"reference-count":57,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2017,1,29]],"date-time":"2017-01-29T00:00:00Z","timestamp":1485648000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>\u201cPublic legal information from all countries and international institutions is part of the common heritage of humanity. Maximizing access to this information promotes justice and the rule of law\u201d. In accordance with the aforementioned declaration on free access to law by legal information institutes of the world, a plethora of legal information is available through the Internet, while the provision of legal information has never before been easier. Given that law is accessed by a much wider group of people, the majority of whom are not legally trained or qualified, diversification techniques should be employed in the context of legal information retrieval, as to increase user satisfaction. We address the diversification of results in legal search by adopting several state of the art methods from the web search, network analysis and text summarization domains. We provide an exhaustive evaluation of the methods, using a standard dataset from the common law domain that we objectively annotated with relevance judgments for this purpose. Our results: (i) reveal that users receive broader insights across the results they get from a legal information retrieval system; (ii) demonstrate that web search diversification techniques outperform other approaches (e.g., summarization-based, graph-based methods) in the context of legal diversification; and (iii) offer balance boundaries between reinforcing relevant documents or sampling the information space around the legal query.<\/jats:p>","DOI":"10.3390\/a10010022","type":"journal-article","created":{"date-parts":[[2017,1,30]],"date-time":"2017-01-30T11:36:30Z","timestamp":1485776190000},"page":"22","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Evaluation of Diversification Techniques for Legal Information Retrieval"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0679-5516","authenticated-orcid":false,"given":"Marios","family":"Koniaris","sequence":"first","affiliation":[{"name":"Knowledge and Database Systems Laboratory, Divison of Computer Science, School of Electrical and Computer Engineering, National Technical University of Athens, Iroon Polytechniou 9, Zographou Campus, 15780 Athens, Greece"}]},{"given":"Ioannis","family":"Anagnostopoulos","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Biomedical Informatics, School of Sciences, University of Thessaly, Papassiopoulou 2-4, 35131 Lamia, Greece"}]},{"given":"Yannis","family":"Vassiliou","sequence":"additional","affiliation":[{"name":"Knowledge and Database Systems Laboratory, Divison of Computer Science, School of Electrical and Computer Engineering, National Technical University of Athens, Iroon Polytechniou 9, Zographou Campus, 15780 Athens, Greece"}]}],"member":"1968","published-online":{"date-parts":[[2017,1,29]]},"reference":[{"key":"ref_1","first-page":"1977","article-title":"Legal diversification","volume":"113","author":"Alces","year":"2013","journal-title":"Columbia Law Rev."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Koniaris, M., Anagnostopoulos, I., and Vassiliou, Y. (2016, January 16\u201318). Diversifying the Legal Order. Proceedings of the IFIP International Conference on Artificial Intelligence Applications and Innovations, Thessaloniki, Greece.","DOI":"10.1007\/978-3-319-44944-9_44"},{"key":"ref_3","unstructured":"Carbonell, J., and Goldstein, J. (1988, January 24\u201328). The use of MMR, diversity-based reranking for reordering documents and producing summaries. Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, Melbourne, Australia."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gollapudi, S., and Sharma, A. (2009, January 20\u201324). An Axiomatic Approach for Result Diversification. Proceedings of the 18th International Conference on World Wide Web, Madrid, Spain.","DOI":"10.1145\/1526709.1526761"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1613\/jair.1523","article-title":"LexRank: Graph-based lexical centrality as salience in text summarization","volume":"22","author":"Erkan","year":"2004","journal-title":"J. Artif. Intell. Res."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.ipm.2008.06.004","article-title":"Biased LexRank: Passage retrieval using random walks with question-based priors","volume":"45","author":"Otterbacher","year":"2009","journal-title":"Inf. Process. Manag."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mei, Q., Guo, J., and Radev, D. (2010, January 25\u201328). Divrank: The interplay of prestige and diversity in information networks. Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA.","DOI":"10.1145\/1835804.1835931"},{"key":"ref_8","unstructured":"Zhu, X., Goldberg, A.B., Van Gael, J., and Andrzejewski, D. (2007, January 22\u201327). Improving Diversity in Ranking using Absorbing Random Walks. Proceedings of the Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2007, Rochester, NY, USA."},{"key":"ref_9","unstructured":"Hand, D.J., Mannila, H., and Smyth, P. (2001). Principles of Data Mining, MIT Press."},{"key":"ref_10","unstructured":"Wong, S.M., and Raghavan, V.V. (1984, January 2\u20136). Vector space model of information retrieval: A reevaluation. Proceedings of the 7th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Cambridge, UK."},{"key":"ref_11","unstructured":"Page, L., Brin, S., Motwani, R., and Winograd, T. (1999). The PageRank Citation Ranking: Bringing Order to the Web, Technical Report for Stanford InfoLab."},{"key":"ref_12","unstructured":"Galgani, F., Compton, P., and Hoffmann, A. (2012, January 22). Combining different summarization techniques for legal text. Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data, Avignon, France."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1561\/1500000009","article-title":"Test Collection Based Evaluation of Information Retrieval Systems","volume":"4","author":"Sanderson","year":"2010","journal-title":"Found. Trends\u00ae Inf. Retr."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1145\/1670564.1670572","article-title":"Redundancy, diversity and interdependent document relevance","volume":"Volume 43","author":"Radlinski","year":"2009","journal-title":"ACM SIGIR Forum"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., B\u00fcttcher, S., and MacKinnon, I. (2008, January 20\u201324). Novelty and diversity in information retrieval evaluation. Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore.","DOI":"10.1145\/1390334.1390446"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Chapelle, O., Metlzer, D., Zhang, Y., and Grinspan, P. (2009, January 2\u20136). Expected reciprocal rank for graded relevance. Proceedings of the 18th ACM conference on Information and Knowledge Management\u2014CIKM \u201909, Hong Kong, China.","DOI":"10.1145\/1645953.1646033"},{"key":"ref_17","unstructured":"Zhai, C.X., Cohen, W.W., and Lafferty, J. (August, January 28). Beyond independent relevance. Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval, Toronto, ON, Canada."},{"key":"ref_18","first-page":"993","article-title":"Latent dirichlet allocation","volume":"3","author":"Blei","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Singh, J., Nejdl, W., and Anand, A. (2016, January 13\u201317). History by Diversity: Helping Historians Search News Archives. Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, Carrboro, NC, USA.","DOI":"10.1145\/2854946.2854959"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10844-014-0328-1","article-title":"Algorithms and criteria for diversification of news article comments","volume":"44","author":"Giannopoulos","year":"2015","journal-title":"J. Intell. Inf. Syst."},{"key":"ref_21","unstructured":"Cheng, S., Arvanitis, A., Chrobak, M., and Hristidis, V. (2014, January 24\u201328). Multi-Query Diversification in Microblogging Posts. Proceedings of the 17th International Conference on Extending Database Technology (EDBT), Athens, Greece."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Koniaris, M., Giannopoulos, G., Sellis, T., and Vasileiou, Y. (2014, January 12\u201314). Diversifying microblog posts. Proceedings of the International Conference on Web Information Systems Engineering, Thessaloniki, Greece.","DOI":"10.1007\/978-3-319-11746-1_14"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Song, K., Tian, Y., Gao, W., and Huang, T. (2006, January 23\u201327). Diversifying the image retrieval results. Proceedings of the 14th ACM International Conference on Multimedia, Santa Barbara, CA, USA.","DOI":"10.1145\/1180639.1180789"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Ziegler, C.N., McNee, S.M., Konstan, J.A., and Lausen, G. (2005, January 10\u201314). Improving recommendation lists through topic diversification. Proceedings of the 14th International Conference on World Wide Web, Chiba, Japan.","DOI":"10.1145\/1060745.1060754"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Raman, K., Shivaswamy, P., and Joachims, T. (2012, January 12\u201316). Online learning to diversify from implicit feedback. Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Beijing, China.","DOI":"10.1145\/2339530.2339642"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Makris, C., Plegas, Y., Stamatiou, Y.C., Stavropoulos, E.C., and Tsakalidis, A.K. (2014, January 1\u20134). Reducing Redundant Information in Search Results Employing Approximation Algorithms. Proceedings of the International Conference on Database and Expert Systems Applications, Munich, Germany.","DOI":"10.1007\/978-3-319-10085-2_22"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhang, B., Li, H., Liu, Y., Ji, L., Xi, W., Fan, W., Chen, Z., and Ma, W.Y. (2005, January 15\u201319). Improving web search results using affinity graph. Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Salvador, Brazil.","DOI":"10.1145\/1076034.1076120"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chen, H., and Karger, D.R. (2006, January 6\u201310). Less is more: probabilistic models for retrieving fewer relevant documents. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, USA.","DOI":"10.1145\/1148170.1148245"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cronen-Townsend, S., and Croft, W.B. (2002, January 24\u201327). Quantifying Query Ambiguity. Proceedings of the Second International Conference on Human Language Technology Research, San Diego, CA, USA.","DOI":"10.3115\/1289189.1289266"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1500000040","article-title":"Search Result Diversification","volume":"9","author":"Santos","year":"2015","journal-title":"Found. Trends\u00ae Inf. Retr."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1145\/1860702.1860709","article-title":"Search result diversification","volume":"39","author":"Drosou","year":"2010","journal-title":"ACM SIGMOD Rec."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Santos, R.L., Macdonald, C., and Ounis, I. (2010, January 1\u20135). Exploiting query reformulations for web search result diversification. Proceedings of the 19th International Conference on World Wide Web, Hong Kong, China.","DOI":"10.1145\/1772690.1772780"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Agrawal, R., Gollapudi, S., Halverson, A., and Ieong, S. (2009, January 9\u201311). Diversifying search results. Proceedings of the second ACM international conference on web search and data mining, Barcelona, Spain.","DOI":"10.1145\/1498759.1498766"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Hu, S., Dou, Z., Wang, X., Sakai, T., and Wen, J.R. (2015, January 19\u201323). Search Result Diversification Based on Hierarchical Intents. Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, Melbourne, Australia.","DOI":"10.1145\/2806416.2806455"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1137\/S0036144503424786","article-title":"A survey of eigenvector methods for web information retrieval","volume":"47","author":"Langville","year":"2005","journal-title":"SIAM Rev."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1023\/A:1011297104922","article-title":"Innovative techniques for legal text retrieval","volume":"9","author":"Moens","year":"2001","journal-title":"Artif. Intell. Law"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Biagioli, C., Francesconi, E., Passerini, A., Montemagni, S., and Soria, C. (2005, January 6\u201311). Automatic semantics extraction in law documents. Proceedings of the 10th international conference on Artificial intelligence and law, Bologna, Italy.","DOI":"10.1145\/1165485.1165506"},{"key":"ref_38","unstructured":"Mencia, E.L., and F\u00fcrnkranz, J. (2008). Machine Learning and Knowledge Discovery in Databases, Springer."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Grabmair, M., Ashley, K.D., Chen, R., Sureshkumar, P., Wang, C., Nyberg, E., and Walker, V.R. (2015, January 8\u201312). Introducing LUIMA: an experiment in legal conceptual retrieval of vaccine injury decisions using a UIMA type system and tools. Proceedings of the 15th International Conference on Artificial Intelligence and Law, San Diego, CA, USA.","DOI":"10.1145\/2746090.2746096"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/s10506-009-9075-y","article-title":"Improving legal information retrieval using an ontological framework","volume":"17","author":"Saravanan","year":"2009","journal-title":"Artif. Intelli. Law"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1007\/s10506-007-9029-1","article-title":"Advanced lexical ontologies and hybrid knowledge based systems: First steps to a dynamic legal electronic commentary","volume":"15","author":"Schweighofer","year":"2007","journal-title":"Artif. Intell. Law"},{"key":"ref_42","unstructured":"Sagri, M.T., and Tiscornia, D. (2003, January 1\u20135). Metadata for content description in legal information. Proceedings of the 14th International Workshop on Database and Expert Systems Applications, Prague, Czech Republic."},{"key":"ref_43","unstructured":"Klein, M.C., Van Steenbergen, W., Uijttenbroek, E.M., Lodder, A.R., and van Harmelen, F. (2006, January 8). Thesaurus-based Retrieval of Case Law. Proceedings of the 2006 conference on Legal Knowledge and Information Systems: JURIX 2006: The Nineteenth Annual Conference, Paris, France."},{"key":"ref_44","unstructured":"Hoekstra, R., Breuker, J., di Bello, M., and Boer, A. (2007, January 4). The LKIF Core ontology of basic legal concepts. Proceedings of the 2nd Workshop on Legal Ontologies and Artificial Intelligence Techniques (LOAIT 2007), Stanford, CA, USA."},{"key":"ref_45","unstructured":"Farzindar, A., and Lapalme, G. (2004). Text Summarization Branches Out Workshop Held in Conjunction with ACL, Association for Computational Linguistics (ACL)."},{"key":"ref_46","unstructured":"Farzindar, A., and Lapalme, G. (2004, January 8\u201310). Letsum, an automatic legal text summarizing system. Proceedings of the Legal Knowledge and Information Systems. JURIX 2004: The Seventeenth Annual Conference, Berlin, Germany."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1748","DOI":"10.1016\/j.ipm.2007.01.005","article-title":"Summarizing court decisions","volume":"43","author":"Moens","year":"2007","journal-title":"Inf. Process. Manag."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Aktolga, E., Ros, I., and Assogba, Y. (2011, January 24\u201328). Detecting outlier sections in us congressional legislation. Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval\u2014SIGIR \u201911, Beijing, China.","DOI":"10.1145\/2009916.2009951"},{"key":"ref_49","first-page":"121","article-title":"Citation networks in the law","volume":"10","author":"Marx","year":"1970","journal-title":"Jurimetr. J."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Van Opijnen, M. (2012, January 17\u201319). Citation Analysis and Beyond: In Search of Indicators Measuring Case Law Importance. Proceedings of the Legal Knowledge and Information Systems-JURIX 2012: The Twenty-Fifth Annual Conference, Amsterdam, The Netherlands.","DOI":"10.3233\/978-1-61499-167-0-95"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.socnet.2007.05.001","article-title":"The Authority of Supreme Court precedent","volume":"30","author":"Fowler","year":"2008","journal-title":"Soc. Netw."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1093\/pan\/mpm011","article-title":"Network Analysis and the Law: Measuring the Legal Importance of Precedents at the U.S. Supreme Court","volume":"15","author":"Fowler","year":"2006","journal-title":"Political Anal."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Galgani, F., Compton, P., and Hoffmann, A. (2012, January 3\u20137). Citation based summarisation of legal texts. Proceedings of the PRICAI 2012: Trends in Artificial Intelligence, Kuching, Malaysia.","DOI":"10.1007\/978-3-642-32695-0_6"},{"key":"ref_54","unstructured":"Koniaris, M., Anagnostopoulos, I., and Vassiliou, Y. (arXiv, 2015). Network Analysis in the Legal Domain: A complex model for European Union legal sources, arXiv."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Lettieri, N., Altamura, A., Faggiano, A., and Malandrino, D. (2016). A computational approach for the experimental study of EU case law: analysis and implementation. Soc. Netw. Anal. Min., 6.","DOI":"10.1007\/s13278-016-0365-6"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Koniaris, M., Anagnostopoulos, I., and Vassiliou, Y. (2016, January 8\u201310). Multi-dimension Diversification in Legal Information Retrieval. Proceedings of the International Conference on Web Information Systems Engineering, Shanghai, China.","DOI":"10.1007\/978-3-319-48740-3_12"},{"key":"ref_57","unstructured":"Wittfoth, A., Chung, P., Greenleaf, G., and Mowbray, A. AustLII\u2019s Point-in-Time Legislation System: A generic PiT system for presenting legislation. Available online: http:\/\/portsea.austlii.edu.au\/pit\/papers\/PiT_background_2005.rtf."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/10\/1\/22\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:27:13Z","timestamp":1760207233000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/10\/1\/22"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,1,29]]},"references-count":57,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,3]]}},"alternative-id":["a10010022"],"URL":"https:\/\/doi.org\/10.3390\/a10010022","relation":{"has-preprint":[{"id-type":"doi","id":"10.20944\/preprints201611.0116.v1","asserted-by":"object"}]},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,1,29]]}}}