{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:11:22Z","timestamp":1784178682491,"version":"3.55.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,11,19]],"date-time":"2018-11-19T00:00:00Z","timestamp":1542585600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1065250 and 1618695"],"award-info":[{"award-number":["1065250 and 1618695"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2019,1,31]]},"abstract":"<jats:p>\n            <jats:italic>Discovery<\/jats:italic>\n            is an important aspect of the civil litigation process in the United States of America, in which all parties to a lawsuit are permitted to request relevant evidence from other parties. With the rapid growth of digital content, the emerging need for \u201ce-discovery\u201d has created a strong demand for techniques that can be used to review massive collections both for \u201cresponsiveness\u201d (i.e., relevance) to the request and for \u201cprivilege\u201d (i.e., presence of legally protected content that the party performing the review may have a right to withhold). In this process, the party performing the review may incur costs of two types, namely,\n            <jats:italic>annotation costs<\/jats:italic>\n            (deriving from the fact that human reviewers need to be paid for their work) and\n            <jats:italic>misclassification costs<\/jats:italic>\n            (deriving from the fact that failing to correctly determine the responsiveness or privilege of a document may adversely affect the interests of the parties in various ways). Relying exclusively on automatic classification would minimize annotation costs but could result in substantial misclassification costs, while relying exclusively on manual classification could generate the opposite consequences. This article proposes a\n            <jats:italic>risk minimization<\/jats:italic>\n            framework (called MINECORE, for \u201c&lt;underline&gt;min&lt;\/underline&gt;imizing the &lt;underline&gt;e&lt;\/underline&gt;xpected &lt;underline&gt;co&lt;\/underline&gt;sts of &lt;underline&gt;re&lt;\/underline&gt;view\u201d) that seeks to strike an optimal balance between these two extreme stands. In MINECORE (a) the documents are first automatically classified for both responsiveness and privilege, and then (b) some of the automatically classified documents are annotated by human reviewers for responsiveness (typically by junior reviewers) and\/or, in cascade, for privilege (typically by senior reviewers), with the overall goal of minimizing the expected cost (i.e., the\n            <jats:italic>risk<\/jats:italic>\n            ) of the entire process. Risk minimization is achieved by optimizing, for both responsiveness and privilege, the choice of which documents to manually review. We present a simulation study in which classes from a standard text classification test collection (RCV1-v2) are used as surrogates for responsiveness and privilege. The results indicate that MINECORE can yield substantially lower total cost than any of a set of strong baselines.\n          <\/jats:p>","DOI":"10.1145\/3268928","type":"journal-article","created":{"date-parts":[[2018,11,20]],"date-time":"2018-11-20T14:19:23Z","timestamp":1542723563000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Jointly Minimizing the Expected Costs of Review for Responsiveness and Privilege in E-Discovery"],"prefix":"10.1145","volume":"37","author":[{"given":"Douglas W.","family":"Oard","sequence":"first","affiliation":[{"name":"University of Maryland, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4221-6427","authenticated-orcid":false,"given":"Fabrizio","family":"Sebastiani","sequence":"additional","affiliation":[{"name":"Consiglio Nazionale delle Ricerche, Pisa, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jyothi K.","family":"Vinjumur","sequence":"additional","affiliation":[{"name":"University of Maryland, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,11,19]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Foundations of Rational Choice Under Risk","author":"Anand Paul","unstructured":"Paul Anand . 1993. Foundations of Rational Choice Under Risk . Oxford University Press , Oxford, UK . Paul Anand. 1993. Foundations of Rational Choice Under Risk. Oxford University Press, Oxford, UK."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505708"},{"key":"e_1_2_1_3_1","unstructured":"Jason R. Baron Michael D. Berman and Ralph C. Losey (Eds.). 2016. Perspectives on Predictive Coding and Other Advanced Search and Review Technologies for the Legal Practitioner. ABA Book Publishing Washington.  Jason R. Baron Michael D. Berman and Ralph C. Losey (Eds.). 2016. Perspectives on Predictive Coding and Other Advanced Search and Review Technologies for the Legal Practitioner. ABA Book Publishing Washington."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806597"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2742548"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609601"},{"key":"e_1_2_1_7_1","volume-title":"Grossman","author":"Cormack Gordon V.","year":"2015","unstructured":"Gordon V. Cormack and Maura R . Grossman . 2015 . Autonomy and reliability of continuous active learning for technology-assisted review. CoRR abs\/1504.06868. Gordon V. Cormack and Maura R. Grossman. 2015. Autonomy and reliability of continuous active learning for technology-assisted review. CoRR abs\/1504.06868."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767771"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 18th Text Retrieval Conference (TREC\u201909)","author":"Gordon","unstructured":"Gordon V. Cormack and Mona Mojdeh. 2009. Machine learning for information retrieval: TREC 2009 web, relevance feedback and legal tracks . In Proceedings of the 18th Text Retrieval Conference (TREC\u201909) . Gordon V. Cormack and Mona Mojdeh. 2009. Machine learning for information retrieval: TREC 2009 web, relevance feedback and legal tracks. In Proceedings of the 18th Text Retrieval Conference (TREC\u201909)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.2307\/2987588"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007413511361"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148216"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the ICAIL 2013 Workshop on Standards for Using Predictive Coding (DESI\u201913)","author":"Gabriel Manfred","year":"2013","unstructured":"Manfred Gabriel , Chris Paskach , and David Sharpe . 2013 . The challenge and promise of predictive coding for privilege . In Proceedings of the ICAIL 2013 Workshop on Standards for Using Predictive Coding (DESI\u201913) . Manfred Gabriel, Chris Paskach, and David Sharpe. 2013. The challenge and promise of predictive coding for privilege. In Proceedings of the ICAIL 2013 Workshop on Standards for Using Predictive Coding (DESI\u201913)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2015.137"},{"key":"e_1_2_1_15_1","article-title":"Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review","volume":"17","author":"Grossman Maura R.","year":"2011","unstructured":"Maura R. Grossman and Gordon V. Cormack . 2011 . Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review . Richmond J. Law Technol. 17 , 3 (2011), Article 5. Maura R. Grossman and Gordon V. Cormack. 2011. Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review. Richmond J. Law Technol. 17, 3 (2011), Article 5.","journal-title":"Richmond J. Law Technol."},{"key":"e_1_2_1_16_1","first-page":"1","article-title":"The Grossman-Cormack glossary of technology-assisted review, with foreword by John M. Facciola, U.S. magistrate judge","volume":"7","author":"Grossman Maura R.","year":"2013","unstructured":"Maura R. Grossman and Gordon V. Cormack . 2013 . The Grossman-Cormack glossary of technology-assisted review, with foreword by John M. Facciola, U.S. magistrate judge . Fed. Courts Law Rev. 7 , 1 (2013), 1 -- 34 . Maura R. Grossman and Gordon V. Cormack. 2013. The Grossman-Cormack glossary of technology-assisted review, with foreword by John M. Facciola, U.S. magistrate judge. Fed. Courts Law Rev. 7, 1 (2013), 1--34.","journal-title":"Fed. Courts Law Rev."},{"key":"e_1_2_1_17_1","volume-title":"The Sedona Conference Glossary: E-Discovery and Digital Information Management (4th ed.).","author":"Sherry","unstructured":"Sherry B. Harris and Paul H. McVoy (Eds.). 2014 . The Sedona Conference Glossary: E-Discovery and Digital Information Management (4th ed.). Retrieved from http:\/\/bit.ly\/2Bhz0TB. Sherry B. Harris and Paul H. McVoy (Eds.). 2014. The Sedona Conference Glossary: E-Discovery and Digital Information Management (4th ed.). Retrieved from http:\/\/bit.ly\/2Bhz0TB."},{"key":"e_1_2_1_18_1","volume-title":"Advances in Kernel Methods\u2014Support Vector Learning","author":"Joachims Thorsten","unstructured":"Thorsten Joachims . 1999. Making large-scale SVM learning practical . In Advances in Kernel Methods\u2014Support Vector Learning , Bernhard Sch\u00f6lkopf , Christopher J. Burges, and Alexander J. Smola (Eds.). The MIT Press , Cambridge, MA, Chapter 11, 169--184. Thorsten Joachims. 1999. Making large-scale SVM learning practical. In Advances in Kernel Methods\u2014Support Vector Learning, Bernhard Sch\u00f6lkopf, Christopher J. Burges, and Alexander J. Smola (Eds.). The MIT Press, Cambridge, MA, Chapter 11, 169--184."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 17th International Conference on Machine Learning (ICML\u201900)","author":"Joachims Thorsten","year":"2000","unstructured":"Thorsten Joachims . 2000 . Estimating the generalization performance of a SVM efficiently . In Proceedings of the 17th International Conference on Machine Learning (ICML\u201900) . 431--438. Thorsten Joachims. 2000. Estimating the generalization performance of a SVM efficiently. In Proceedings of the 17th International Conference on Machine Learning (ICML\u201900). 431--438."},{"key":"e_1_2_1_20_1","volume-title":"The MIT Press","author":"Joachims Thorsten","unstructured":"Thorsten Joachims . 2006. Transductive support vector machines . In Semi-Supervised Learning, Olivier Chapelle, Bernhard Sch\u00f6lkopf, and Alexander Zien (Eds.). The MIT Press , Cambridge , MA , 105--117. Thorsten Joachims. 2006. Transductive support vector machines. In Semi-Supervised Learning, Olivier Chapelle, Bernhard Sch\u00f6lkopf, and Alexander Zien (Eds.). The MIT Press, Cambridge, MA, 105--117."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.2307\/1914185"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 20th International Joint Conference on Artifical Intelligence (IJCAI\u201907)","author":"Kapoor Ashish","year":"2007","unstructured":"Ashish Kapoor , Eric Horvitz , and Sumit Basu . 2007 . Selective supervision: Guiding supervised learning with decision-theoretic active learning . In Proceedings of the 20th International Joint Conference on Artifical Intelligence (IJCAI\u201907) . 877--882. Ashish Kapoor, Eric Horvitz, and Sumit Basu. 2007. Selective supervision: Guiding supervised learning with decision-theoretic active learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence (IJCAI\u201907). 877--882."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 14th International Conference on Machine Learning (ICML\u201997)","author":"Koller Daphne","year":"1997","unstructured":"Daphne Koller and Mehran Sahami . 1997 . Hierarchically classifying documents using very few words . In Proceedings of the 14th International Conference on Machine Learning (ICML\u201997) . 170--178. Daphne Koller and Mehran Sahami. 1997. Hierarchically classifying documents using very few words. In Proceedings of the 14th International Conference on Machine Learning (ICML\u201997). 170--178."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 17th ACM International Conference on Research and Development in Information Retrieval (SIGIR\u201994)","author":"David","unstructured":"David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers . In Proceedings of the 17th ACM International Conference on Research and Development in Information Retrieval (SIGIR\u201994) . 3--12. David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th ACM International Conference on Research and Development in Information Retrieval (SIGIR\u201994). 3--12."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/1005332.1005345"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the Workshop on Using Advanced Data Analysis in eDiscovery 8 Related Disciplines to Identify and Protect Sensitive Information in Large Collections (DESI\u201917)","author":"Oard Douglas","year":"2017","unstructured":"Douglas Oard , Jyothi Vinjumur , and Fabrizio Sebastiani . 2017 . When is it rational to review for privilege? In Proceedings of the Workshop on Using Advanced Data Analysis in eDiscovery 8 Related Disciplines to Identify and Protect Sensitive Information in Large Collections (DESI\u201917) . Douglas Oard, Jyothi Vinjumur, and Fabrizio Sebastiani. 2017. When is it rational to review for privilege? In Proceedings of the Workshop on Using Advanced Data Analysis in eDiscovery 8 Related Disciplines to Identify and Protect Sensitive Information in Large Collections (DESI\u201917)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000025"},{"key":"e_1_2_1_28_1","volume-title":"Advances in Large Margin Classifiers, Alexander Smola, Peter Bartlett","author":"Platt John C.","unstructured":"John C. Platt . 2000. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods . In Advances in Large Margin Classifiers, Alexander Smola, Peter Bartlett , Bernard Sch\u00f6lkopf , and Dale Schuurmans (Eds.). The MIT Press , Cambridge, MA, 61--74. John C. Platt. 2000. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Advances in Large Margin Classifiers, Alexander Smola, Peter Bartlett, Bernard Sch\u00f6lkopf, and Dale Schuurmans (Eds.). The MIT Press, Cambridge, MA, 61--74."},{"key":"e_1_2_1_29_1","volume-title":"The TREC 2002 filtering track report. In Proceedings of the 11th Text REtrieval Conference (TREC\u201902)","author":"Stephen","unstructured":"Stephen E. Robertson and Ian Soboroff. 2002 . The TREC 2002 filtering track report. In Proceedings of the 11th Text REtrieval Conference (TREC\u201902) . Stephen E. Robertson and Ian Soboroff. 2002. The TREC 2002 filtering track report. In Proceedings of the 11th Text REtrieval Conference (TREC\u201902)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/1672957.1672981"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2015.7363867"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2005.102"},{"key":"e_1_2_1_33_1","doi-asserted-by":"crossref","unstructured":"Burr Settles. 2012. Active Learning. Morgan 8 Claypool Publishers San Rafael.   Burr Settles. 2012. Active Learning. Morgan 8 Claypool Publishers San Rafael.","DOI":"10.1007\/978-3-031-01560-1"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the NIPS Workshop on Cost-Sensitive Learning.","author":"Settles Burr","year":"2008","unstructured":"Burr Settles , Mark Craven , and Lewis Friedland . 2008 . Active learning with real annotation costs . In Proceedings of the NIPS Workshop on Cost-Sensitive Learning. Burr Settles, Mark Craven, and Lewis Friedland. 2008. Active learning with real annotation costs. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 23rd International Conference on Computational Linguistics (COLING\u201910)","author":"Tomanek Katrin","year":"2010","unstructured":"Katrin Tomanek and Udo Hahn . 2010 . A comparison of models for cost-sensitive active learning . In Proceedings of the 23rd International Conference on Computational Linguistics (COLING\u201910) . 1247--1255. Katrin Tomanek and Udo Hahn. 2010. A comparison of models for cost-sensitive active learning. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING\u201910). 1247--1255."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2746090.2746101"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2854946.2854964"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609506"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval (SDAIR\u201995)","author":"Wiener Erik D.","unstructured":"Erik D. Wiener , Jan O. Pedersen , and Andreas S. Weigend . 1995. A neural network approach to topic spotting . In Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval (SDAIR\u201995) . 317--332. Erik D. Wiener, Jan O. Pedersen, and Andreas S. Weigend. 1995. A neural network approach to topic spotting. In Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval (SDAIR\u201995). 317--332."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775151"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3268928","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3268928","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3268928","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:07:19Z","timestamp":1750212439000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3268928"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11,19]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,1,31]]}},"alternative-id":["10.1145\/3268928"],"URL":"https:\/\/doi.org\/10.1145\/3268928","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11,19]]},"assertion":[{"value":"2017-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-11-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}