{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T21:51:19Z","timestamp":1776117079378,"version":"3.50.1"},"reference-count":143,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2022,3,30]],"date-time":"2022-03-30T00:00:00Z","timestamp":1648598400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000870","name":"John D. and Catherine T. MacArthur Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000870","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000935","name":"Russell Sage Foundation","doi-asserted-by":"publisher","award":["1908-17713"],"award-info":[{"award-number":["1908-17713"]}],"id":[{"id":"10.13039\/100000935","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,3,30]]},"abstract":"<jats:p>Despite growing calls for participation in AI design, there are to date few empirical studies of what these processes look like and how they can be structured for meaningful engagement with domain experts. In this paper, we examine a notable yet understudied AI design process in the legal domain that took place over a decade ago, the impact of which still informs legal automation efforts today. Specifically, we examine the design and evaluation activities that took place from 2006 to 2011 within the Text REtrieval Conference's (TREC) Legal Track, a computational research venue hosted by the National Institute of Standards and Technologies. The Legal Track of TREC is notable in the history of AI research and practice because it relied on a range of participatory approaches to facilitate the design and evaluation of new computational techniques-in this case, for automating attorney document review for civil litigation matters. Drawing on archival research and interviews with coordinators of the Legal Track of TREC, our analysis reveals how an interactive simulation methodology allowed computer scientists and lawyers to become co-designers and helped bridge the chasm between computational research and real-world, high-stakes litigation practice. In analyzing this case from the recent past, our aim is to empirically ground contemporary critiques of AI development and evaluation and the calls for greater participation as a means to address them.<\/jats:p>","DOI":"10.1145\/3512898","type":"journal-article","created":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T16:54:59Z","timestamp":1649350499000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["An Uncommon Task: Participatory Design in Legal AI"],"prefix":"10.1145","volume":"6","author":[{"given":"Fernando","family":"Delgado","sequence":"first","affiliation":[{"name":"Cornell University, Ithaca, NY, USA"}]},{"given":"Solon","family":"Barocas","sequence":"additional","affiliation":[{"name":"Microsoft Research &amp; Cornell University, New York, NY, USA"}]},{"given":"Karen","family":"Levy","sequence":"additional","affiliation":[{"name":"Cornell University, Ithaca, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,4,7]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"edited by Charles Ragin and H.S. Becker","author":"Abbott Andrew","unstructured":"Andrew Abbott . 1992. \" What do cases do? Some notes on activity in sociological analysis.\" In What is a case? , edited by Charles Ragin and H.S. Becker . New York : Cambridge University Press . Andrew Abbott. 1992. \"What do cases do? Some notes on activity in sociological analysis.\" In What is a case?, edited by Charles Ragin and H.S. Becker. New York: Cambridge University Press."},{"key":"e_1_2_1_2_1","volume-title":"Bridging the Great Divide: Social Science, Technical Systems, and Cooperative Work","author":"Agre Philip E.","unstructured":"Philip E. Agre . 1997. Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI . In Bridging the Great Divide: Social Science, Technical Systems, and Cooperative Work , edited by Geof Bowker, Les Gasser, Leigh Star, and Bill Turner. Erlbaum . Philip E. Agre. 1997. Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI. In Bridging the Great Divide: Social Science, Technical Systems, and Cooperative Work, edited by Geof Bowker, Les Gasser, Leigh Star, and Bill Turner. Erlbaum."},{"key":"e_1_2_1_3_1","unstructured":"Julia Angwin Jeff Larson Surya Mattu and Laura Kirchner. 2016. Machine Bias. ProPublica. https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal sentencing.  Julia Angwin Jeff Larson Surya Mattu and Laura Kirchner. 2016. Machine Bias. ProPublica. https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal sentencing."},{"key":"e_1_2_1_4_1","volume-title":"The Sedona Conference Journal 6(1).","author":"Baron Jason R.","year":"2005","unstructured":"Jason R. Baron . 2005 . Toward a Federal Benchmarking Standard for Evaluating Information Retrieval Products Used In E-Discovery . The Sedona Conference Journal 6(1). Jason R. Baron. 2005. Toward a Federal Benchmarking Standard for Evaluating Information Retrieval Products Used In E-Discovery. The Sedona Conference Journal 6(1)."},{"key":"e_1_2_1_5_1","volume-title":"The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery. The Sedona Conference Journal 8(2).","author":"Ed Jason R.","year":"2007","unstructured":"Jason R. Baron ( Ed .). 2007 . The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery. The Sedona Conference Journal 8(2). Jason R. Baron (Ed.). 2007. The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery. The Sedona Conference Journal 8(2)."},{"key":"e_1_2_1_6_1","volume-title":"The TREC Legal Track: Origins and Reflections on the First Year. The Sedona Conference Journal 8(1).","author":"Baron Jason R.","year":"2007","unstructured":"Jason R. Baron . 2007 . The TREC Legal Track: Origins and Reflections on the First Year. The Sedona Conference Journal 8(1). Jason R. Baron. 2007. The TREC Legal Track: Origins and Reflections on the First Year. The Sedona Conference Journal 8(1)."},{"key":"e_1_2_1_7_1","volume-title":"Legal Track. Presentation. Text REtrieval Conference 25th Anniversary.","author":"Baron Jason R.","year":"2016","unstructured":"Jason R. Baron . 2016 . Legal Track. Presentation. Text REtrieval Conference 25th Anniversary. Jason R. Baron. 2016. Legal Track. Presentation. Text REtrieval Conference 25th Anniversary."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the Fifteenth Text REtrieval Conference.","author":"Baron Jason R.","unstructured":"Jason R. Baron , David D. Lewis , and Douglas W. Oard . 2006. TREC Legal Track 2006 Overview . In Proceedings of the Fifteenth Text REtrieval Conference. Jason R. Baron, David D. Lewis, and Douglas W. Oard. 2006. TREC Legal Track 2006 Overview. In Proceedings of the Fifteenth Text REtrieval Conference."},{"key":"e_1_2_1_9_1","volume-title":"The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery. The Sedona Conference Journal 8(2).","author":"Ed Jason R.","year":"2007","unstructured":"Jason R. Baron ( Ed .). 2007 . The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery. The Sedona Conference Journal 8(2). Jason R. Baron (Ed.). 2007. The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery. The Sedona Conference Journal 8(2)."},{"key":"e_1_2_1_10_1","unstructured":"Jason R. Baron Bruce Hedin Douglas W. Oard and Stephen Tomlinson. 2008. TREC Legal Track 2008 Interactive Guidelines. http:\/\/trec-legal.umiacs.umd.edu\/guidelines\/2008InteractiveGuidelines.pdf  Jason R. Baron Bruce Hedin Douglas W. Oard and Stephen Tomlinson. 2008. TREC Legal Track 2008 Interactive Guidelines. http:\/\/trec-legal.umiacs.umd.edu\/guidelines\/2008InteractiveGuidelines.pdf"},{"key":"e_1_2_1_11_1","unstructured":"Jason R. Baron Bruce Hedin Douglas W. Oard and Stephen Tomlinson. 2009. TREC Legal Track 2009 Interactive Guidelines. http:\/\/trec-legal.umiacs.umd.edu\/LT09_ITG_final.pdf  Jason R. Baron Bruce Hedin Douglas W. Oard and Stephen Tomlinson. 2009. TREC Legal Track 2009 Interactive Guidelines. http:\/\/trec-legal.umiacs.umd.edu\/LT09_ITG_final.pdf"},{"key":"e_1_2_1_12_1","volume-title":"Perspectives on Predictive Coding: and Other Advanced Search Methods for the Legal Practitioner","author":"Jason","year":"2016","unstructured":"Baron, Jason R., Ralph C. Losey , and Michael D. Berman . Perspectives on Predictive Coding: and Other Advanced Search Methods for the Legal Practitioner . Chicago, Illinois : American Bar Association , Section of Litigation, 2016 . Baron, Jason R., Ralph C. Losey, and Michael D. Berman. Perspectives on Predictive Coding: and Other Advanced Search Methods for the Legal Practitioner. Chicago, Illinois: American Bar Association, Section of Litigation, 2016."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951717718854"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery","author":"Beede Emma","unstructured":"Emma Beede , Elizabeth Baylor , Fred Hersch , Anna Iurchenko , Lauren Wilcox , Paisan Ruamviboonsuk , and Laura M. Vardoulakis . 2020. A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy . In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery , New York, NY, USA, 1--12. Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, and Laura M. Vardoulakis. 2020. A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1--12."},{"key":"e_1_2_1_15_1","volume-title":"Howard J. C. Nicols, David Stanton, and K. Krasnow Waterman.","author":"Bieser Art","year":"2009","unstructured":"Art Bieser , Christopher H. Boehning , Michael Roman Gaske , Howard J. C. Nicols, David Stanton, and K. Krasnow Waterman. 2009 . Reflections of the Topic Authorities : About the 2009 TREC Legal Track Interactive Task . https:\/\/trec-legal.umiacs.umd.edu\/other\/2009TaReflections.pdf Art Bieser, Christopher H. Boehning, Michael Roman Gaske, Howard J. C. Nicols, David Stanton, and K. Krasnow Waterman. 2009. Reflections of the Topic Authorities: About the 2009 TREC Legal Track Interactive Task. https:\/\/trec-legal.umiacs.umd.edu\/other\/2009TaReflections.pdf"},{"key":"e_1_2_1_16_1","doi-asserted-by":"crossref","unstructured":"Jeanette Blomberg Lucy Suchman and Randall Trigg. 1996. Reflections on a Work-Oriented Design Project. Human-Computer Interaction 11.  Jeanette Blomberg Lucy Suchman and Randall Trigg. 1996. Reflections on a Work-Oriented Design Project. Human-Computer Interaction 11.","DOI":"10.1207\/s15327051hci1103_3"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning","author":"Blum Avrim","year":"2015","unstructured":"Avrim Blum and Mortiz Hardt . 2015 . The Ladder: A Reliable Leaderboard for Machine Learning Competitions . In Proceedings of the 32nd International Conference on Machine Learning . Lille, France. Avrim Blum and Mortiz Hardt. 2015. The Ladder: A Reliable Leaderboard for Machine Learning Competitions. In Proceedings of the 32nd International Conference on Machine Learning. Lille, France."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-7373(91)90030-B"},{"key":"e_1_2_1_19_1","volume-title":"Guidelines and Best Practices for Implementing the 2015 Discovery Amendments Concerning Proportionality","author":"Bolch Judicial Institute","year":"2021","unstructured":"Bolch Judicial Institute , Guidelines and Best Practices for Implementing the 2015 Discovery Amendments Concerning Proportionality ( Third Edition) ( 2021 ). Bolch Judicial Institute, Guidelines and Best Practices for Implementing the 2015 Discovery Amendments Concerning Proportionality (Third Edition) (2021)."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/1732323.1732339"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2347635.2347642"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-016-9259-4"},{"key":"e_1_2_1_23_1","volume-title":"To-wards a Synthesis of Judicial Perspectives on Technology-Assisted Review, 82 DEF. COUNS. J.309, 313--19","author":"Brickell Julia L.","year":"2015","unstructured":"Julia L. Brickell & Peter J. Pizzi , To-wards a Synthesis of Judicial Perspectives on Technology-Assisted Review, 82 DEF. COUNS. J.309, 313--19 ( 2015 ). Julia L. Brickell & Peter J. Pizzi, To-wards a Synthesis of Judicial Perspectives on Technology-Assisted Review, 82 DEF. COUNS. J.309, 313--19 (2015)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300271"},{"issue":"1","key":"e_1_2_1_25_1","first-page":"101","article-title":"). As We May Think","volume":"176","author":"Bush Vannevar","year":"1945","unstructured":"Vannevar Bush . (July 1945 ). As We May Think . The Atlantic Monthly. 176 ( 1 ): 101 -- 108 . Vannevar Bush. (July 1945). As We May Think. The Atlantic Monthly. 176 (1): 101--108.","journal-title":"The Atlantic Monthly."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359206"},{"key":"e_1_2_1_27_1","unstructured":"Calo Ryan and Danielle Keats Citron. \"The Automated Administrative State: A Crisis of Legitimacy.\" Emory Law Journal 70 no. 4 (2021): 797.  Calo Ryan and Danielle Keats Citron. \"The Automated Administrative State: A Crisis of Legitimacy.\" Emory Law Journal 70 no. 4 (2021): 797."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction, 3, CSCW.","author":"Chandrasekharan E.","unstructured":"E. Chandrasekharan , C. Gandhi , M. W. Mustelier , and E. Gilbert . 2019. CrossMod: A cross-community learning-based system to assist reddit moderators . In Proceedings of the ACM on Human-Computer Interaction, 3, CSCW. E. Chandrasekharan, C. Gandhi, M. W. Mustelier, and E. Gilbert. 2019. CrossMod: A cross-community learning-based system to assist reddit moderators. In Proceedings of the ACM on Human-Computer Interaction, 3, CSCW."},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Ang\u00e8le Christin. 2017. Algorithms in Practice: Comparing Web Journalism and Criminal Justice. Big Data & Society (December).  Ang\u00e8le Christin. 2017. Algorithms in Practice: Comparing Web Journalism and Criminal Justice. Big Data & Society (December).","DOI":"10.1177\/2053951717718855"},{"key":"e_1_2_1_30_1","volume-title":"Technological Due Process","author":"Citron Danielle K.","unstructured":"Danielle K. Citron . 2008. Technological Due Process . Washington University Law Review Vol 85 1249--1313. Danielle K. Citron. 2008. Technological Due Process. Washington University Law Review Vol 85 1249--1313."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/275537.275544"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/122860.122861"},{"key":"e_1_2_1_33_1","doi-asserted-by":"crossref","unstructured":"Cobbe Jennifer. \"Administrative law and the machines of government: judicial review of automated public-sector decision-making.\" Legal Studies 39 no. 4 (2019): 636--655.  Cobbe Jennifer. \"Administrative law and the machines of government: judicial review of automated public-sector decision-making.\" Legal Studies 39 no. 4 (2019): 636--655.","DOI":"10.1017\/lst.2019.9"},{"key":"e_1_2_1_34_1","unstructured":"Cary Coglianese and David Lehr. \"Regulating by Robot: Administrative Decision Making in the Machine-Learning Era.\" Georgetown Law Journal 105 (2017): 17--8.  Cary Coglianese and David Lehr. \"Regulating by Robot: Administrative Decision Making in the Machine-Learning Era.\" Georgetown Law Journal 105 (2017): 17--8."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1038\/538311a"},{"key":"e_1_2_1_36_1","unstructured":"Kade Crockford. 2018. Risk assessment tools in the criminal justice system: inaccurate unfair and unjust? ACLU of Massachusetts. https:\/\/privacysos.org\/blog\/risk-assessment-tools-criminal justice-system-inaccurate-unfair-unjust.  Kade Crockford. 2018. Risk assessment tools in the criminal justice system: inaccurate unfair and unjust? ACLU of Massachusetts. https:\/\/privacysos.org\/blog\/risk-assessment-tools-criminal justice-system-inaccurate-unfair-unjust."},{"key":"e_1_2_1_37_1","unstructured":"Da Silva Moore v. Publicis Groupe & MSL Groupe 287 F.R.D. 181 (S.D.N.Y. 2012)  Da Silva Moore v. Publicis Groupe & MSL Groupe 287 F.R.D. 181 (S.D.N.Y. 2012)"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2669617"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1080\/10618600.2017.1384734"},{"key":"e_1_2_1_40_1","unstructured":"David Donoho and H. Monajemi. 2018. Stats 385 Stanford Lecture 01: The Revolution is here! https:\/\/stats285.github.io\/2018\/assets\/lectures18\/Lecture-01--20180925.pdf  David Donoho and H. Monajemi. 2018. Stats 385 Stanford Lecture 01: The Revolution is here! https:\/\/stats285.github.io\/2018\/assets\/lectures18\/Lecture-01--20180925.pdf"},{"key":"e_1_2_1_41_1","first-page":"123","volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Dumais Susan","year":"2005","unstructured":"Susan Dumais and N. Belkin . 2005 . The TREC interactive tracks: putting the user into search. In Ellen M. Voorhees and Donna K. Harman , editors, TREC: Experiment and Evaluation in Information Retrieval , p. 123 -- 152 . Susan Dumais and N. Belkin. 2005. The TREC interactive tracks: putting the user into search. In Ellen M. Voorhees and Donna K. Harman, editors, TREC: Experiment and Evaluation in Information Retrieval, p. 123--52."},{"key":"e_1_2_1_42_1","unstructured":"Seth Katsuya Endo. 2018. Technological Opacity & Procedural Injustice. 59 British Columbia Law Review 821--876.  Seth Katsuya Endo. 2018. Technological Opacity & Procedural Injustice. 59 British Columbia Law Review 821--876."},{"key":"e_1_2_1_43_1","volume-title":"Hotchkiss, 2007. University of Washington (UW) at Legal TREC Interactive 2007. In Proceedings of the Sixteenth Text REtrieval Conference.","author":"Efthimis","unstructured":"Efthimis N. Efthimiadis and Mary A . Hotchkiss, 2007. University of Washington (UW) at Legal TREC Interactive 2007. In Proceedings of the Sixteenth Text REtrieval Conference. Efthimis N. Efthimiadis and Mary A. Hotchkiss, 2007. University of Washington (UW) at Legal TREC Interactive 2007. In Proceedings of the Sixteenth Text REtrieval Conference."},{"key":"e_1_2_1_44_1","volume-title":"Gelbach","author":"Engstrom David Freeman","year":"2020","unstructured":"David Freeman Engstrom and Jonah B . Gelbach . 2020 . Legal Tech, Civil Procedure , and the Future of Adversarialism. 169 University of Pennsylvania Law Review . David Freeman Engstrom and Jonah B. Gelbach. 2020. Legal Tech, Civil Procedure, and the Future of Adversarialism. 169 University of Pennsylvania Law Review."},{"key":"e_1_2_1_45_1","volume-title":"Dewey","author":"Flanagan Patrick","year":"2019","unstructured":"Patrick Flanagan and Michelle H . Dewey . 2019 . Where Do We Go from Here? Transformation and Acceleration of Legal Analytics in Practice 35 Georgia State Law Review . Patrick Flanagan and Michelle H. Dewey. 2019. Where Do We Go from Here? Transformation and Acceleration of Legal Analytics in Practice 35 Georgia State Law Review."},{"key":"e_1_2_1_46_1","volume-title":"Engineering Knowledge: The Construction of Knowledge in Artificial Intelligence. Social Studies of Science 23(3).","author":"Forsythe Diana E.","year":"1993","unstructured":"Diana E. Forsythe . 1993 . Engineering Knowledge: The Construction of Knowledge in Artificial Intelligence. Social Studies of Science 23(3). Diana E. Forsythe. 1993. Engineering Knowledge: The Construction of Knowledge in Artificial Intelligence. Social Studies of Science 23(3)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372840"},{"key":"e_1_2_1_48_1","unstructured":"Maura R. Grossman Conor R. Crowley and Joe Looby. 2008. Reflections of the Topic Authorities. https:\/\/trec-legal.umiacs.umd.edu\/other\/Tareflections2008.doc  Maura R. Grossman Conor R. Crowley and Joe Looby. 2008. Reflections of the Topic Authorities. https:\/\/trec-legal.umiacs.umd.edu\/other\/Tareflections2008.doc"},{"key":"e_1_2_1_49_1","volume-title":"Cormack","author":"Grossman Maura R.","year":"2011","unstructured":"Maura R. Grossman and Gordon V . Cormack . 2011 . Technology-Assisted Review in E-Discovery Can be More Effective and More Efficient Than Exhaustive Manual Review. Richmond Journal of Law and Technology 17(3). Maura R. Grossman and Gordon V. Cormack. 2011. Technology-Assisted Review in E-Discovery Can be More Effective and More Efficient Than Exhaustive Manual Review. Richmond Journal of Law and Technology 17(3)."},{"key":"e_1_2_1_50_1","volume-title":"ICAIL 2011 Workshop on Setting Standards for Searching Electronically Stored Information in Discovery Proceedings (DESI IV Workshop).","author":"Maura","unstructured":"Maura R. Grossman and Gordon V. Cormack. 2011. Inconsistent Assessment of Responsiveness in E-Discovery: Difference of Opinion or Human Error? ICAIL 2011 Workshop on Setting Standards for Searching Electronically Stored Information in Discovery Proceedings (DESI IV Workshop). Maura R. Grossman and Gordon V. Cormack. 2011. Inconsistent Assessment of Responsiveness in E-Discovery: Difference of Opinion or Human Error? ICAIL 2011 Workshop on Setting Standards for Searching Electronically Stored Information in Discovery Proceedings (DESI IV Workshop)."},{"key":"e_1_2_1_51_1","first-page":"32","article-title":"?The Grossman-Cormack Glossary of Technology-Assisted Review","volume":"1","author":"Grossman Maura R.","year":"2013","unstructured":"Maura R. Grossman and Gordon V. Cormack . 2013 . ?The Grossman-Cormack Glossary of Technology-Assisted Review .\" Federal Courts Law Review 1 , 32 . Maura R. Grossman and Gordon V. Cormack. 2013. ?The Grossman-Cormack Glossary of Technology-Assisted Review.\" Federal Courts Law Review 1, 32.","journal-title":"Federal Courts Law Review"},{"key":"e_1_2_1_52_1","volume-title":"Cormack","author":"Grossman Maura R.","year":"2019","unstructured":"Maura R. Grossman and Gordon V . Cormack . 2019 . Quantifying Success : Using Data Science to Measure the Accuracy of Technology-Assisted Review in Electronic Discovery IN Walters, E. (Ed.). (2019). Data-driven law: data analytics and the new legal services. Boca Raton, FL : CRC Press . Maura R. Grossman and Gordon V. Cormack. 2019. Quantifying Success: Using Data Science to Measure the Accuracy of Technology-Assisted Review in Electronic Discovery IN Walters, E. (Ed.). (2019). Data-driven law: data analytics and the new legal services. Boca Raton, FL: CRC Press."},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction, 4 CSCW, 1--37","author":"Halfaker Aaron","unstructured":"Aaron Halfaker and R. S. Geiger . 2020. Ores: Lowering barriers with participatory machine learning in wikipedia . In Proceedings of the ACM on Human-Computer Interaction, 4 CSCW, 1--37 . Aaron Halfaker and R. S. Geiger. 2020. Ores: Lowering barriers with participatory machine learning in wikipedia. In Proceedings of the ACM on Human-Computer Interaction, 4 CSCW, 1--37."},{"key":"#cr-split#-e_1_2_1_54_1.1","unstructured":"M.M. Hall and and E. Toms. 2021. Building a Common Framework for IIR Evaluation. Vol. 8138 LNCS. Lecture Notes in Computer Science. doi:10.1007\/978--3--642--40802--1_3. 10.1007\/978--3--642--40802--1_3"},{"key":"#cr-split#-e_1_2_1_54_1.2","unstructured":"M.M. Hall and and E. Toms. 2021. Building a Common Framework for IIR Evaluation. Vol. 8138 LNCS. Lecture Notes in Computer Science. doi:10.1007\/978--3--642--40802--1_3."},{"key":"e_1_2_1_55_1","first-page":"21","volume-title":"The TREC Test Collections","author":"Harman Donna K.","unstructured":"Donna K. Harman . 2005. The TREC Test Collections . In Ellen M. Voorhees and Donna K. Harman, editors, TREC : Experiment and Evaluation in Information Retrieval , pages 21 -- 52 . Donna K. Harman. 2005. The TREC Test Collections. In Ellen M. Voorhees and Donna K. Harman, editors, TREC: Experiment and Evaluation in Information Retrieval, pages 21--52."},{"key":"e_1_2_1_56_1","doi-asserted-by":"crossref","unstructured":"C. N. Harrington & A. M. Piper. 2019. Deconstructing Community-Based Collaborative Design: Towards More Equitable Participatory Design Engagements. PACM HCI 3(November).  C. N. Harrington & A. M. Piper. 2019. Deconstructing Community-Based Collaborative Design: Towards More Equitable Participatory Design Engagements. PACM HCI 3(November).","DOI":"10.1145\/3359318"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the Eighteenth Text REtrieval Conference.","author":"Hedin Bruce","year":"2009","unstructured":"Bruce Hedin , Stephen Tomlinson , Jason R. Baron , and Douglas Oard . 2009 . TREC Legal Track 2009 Overview . In Proceedings of the Eighteenth Text REtrieval Conference. Bruce Hedin, Stephen Tomlinson, Jason R. Baron, and Douglas Oard. 2009. TREC Legal Track 2009 Overview. In Proceedings of the Eighteenth Text REtrieval Conference."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.3138\/utlj.2017-0044"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10506-010-9100-1"},{"key":"e_1_2_1_60_1","unstructured":"International Association of Public Participation [IAP2]. (n.d.). IAP2 Spectrum of Public Participation. https:\/\/cdn.ymaws.com\/www.iap2.org\/resource\/resmgr\/pillars\/Spectrum_8.5x11_Print.pdf  International Association of Public Participation [IAP2]. (n.d.). IAP2 Spectrum of Public Participation. https:\/\/cdn.ymaws.com\/www.iap2.org\/resource\/resmgr\/pillars\/Spectrum_8.5x11_Print.pdf"},{"key":"e_1_2_1_61_1","unstructured":"In re: Enron Corporation Securities Litigation  In re: Enron Corporation Securities Litigation"},{"key":"e_1_2_1_62_1","volume-title":"Science at the Bar: Law, Science and Technology in America","author":"Jasanoff Sheila","unstructured":"Sheila Jasanoff . 1995. Science at the Bar: Law, Science and Technology in America . Harvard University Press : Cambridge : Massachusetts. Sheila Jasanoff. 1995. Science at the Bar: Law, Science and Technology in America. Harvard University Press: Cambridge: Massachusetts."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372874"},{"key":"e_1_2_1_64_1","volume-title":"Quantitative Legal Prediction--or--How I Learned to Stop Worrying and Start Preparing for the Data-Driven Future of the Legal Services Industry, 62 Emory L. J. 823","author":"Katz Daniel M.","year":"2013","unstructured":"Daniel M. Katz , Quantitative Legal Prediction--or--How I Learned to Stop Worrying and Start Preparing for the Data-Driven Future of the Legal Services Industry, 62 Emory L. J. 823 ( 2013 ). Daniel M. Katz, Quantitative Legal Prediction--or--How I Learned to Stop Worrying and Start Preparing for the Data-Driven Future of the Legal Services Industry, 62 Emory L. J. 823 (2013)."},{"key":"e_1_2_1_65_1","volume-title":"Mulligan","author":"Kluttz Daniel N.","year":"2019","unstructured":"Daniel N. Kluttz and Deirdre K . Mulligan . 2019 . Automated Decision Support Technologies and the Legal Profession. 34 Berkeley Technology Law Journal 853. Daniel N. Kluttz and Deirdre K. Mulligan. 2019. Automated Decision Support Technologies and the Legal Profession. 34 Berkeley Technology Law Journal 853."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445938"},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 3, CSCW","author":"Noothigattu Ritesh","year":"2019","unstructured":"Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu , Siheon Lee , Alexandros Psomas , 2019 . WeBuildAI: Participatory framework for algorithmic governance . In Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--35. Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, et al. 2019. WeBuildAI: Participatory framework for algorithmic governance. In Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--35."},{"key":"e_1_2_1_68_1","volume-title":"Harry Hochheiser.","author":"Lazar Jonathan","year":"2017","unstructured":"Jonathan Lazar , Jinjuan Heidi Feng , Harry Hochheiser. 2017 . \"Case Studies.\" In Research Methods in Human Computer Interaction (2nd Edition), edited by Jonathon Lazar, Jinjuan Heidi Feng, Harry Hocheiser, 153--185. Morgan Kaufmann . Jonathan Lazar, Jinjuan Heidi Feng, Harry Hochheiser. 2017. \"Case Studies.\" In Research Methods in Human Computer Interaction (2nd Edition), edited by Jonathon Lazar, Jinjuan Heidi Feng, Harry Hocheiser, 153--185. Morgan Kaufmann."},{"key":"e_1_2_1_69_1","volume-title":"Cornell University. Rule 26. Duty to Disclose","author":"Legal Information Institute","unstructured":"Legal Information Institute , Cornell University. Rule 26. Duty to Disclose ; General Provisions Governing Discovery . https:\/\/www.law.cornell.edu\/rules\/frcp\/rule_26. Legal Information Institute, Cornell University. Rule 26. Duty to Disclose; General Provisions Governing Discovery. https:\/\/www.law.cornell.edu\/rules\/frcp\/rule_26."},{"key":"e_1_2_1_70_1","volume-title":"Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Conference on Human Factors in Computing Systems - Proceedings, 1--15;","author":"Liao Q. V.","year":"2020","unstructured":"Q. V. Liao , D. Gruen , and S. Miller ( 2020 ). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Conference on Human Factors in Computing Systems - Proceedings, 1--15; Q. V. Liao, D. Gruen, and S. Miller (2020). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Conference on Human Factors in Computing Systems - Proceedings, 1--15;"},{"key":"e_1_2_1_71_1","unstructured":"Mark Liberman. 2015. Reproducible Research and the Common Task Method. April 1. https:\/\/www.simonsfoundation.org\/event\/reproducible-research-and-the-common-task-method\/  Mark Liberman. 2015. Reproducible Research and the Common Task Method. April 1. https:\/\/www.simonsfoundation.org\/event\/reproducible-research-and-the-common-task-method\/"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3210604.3210614"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301019.3320000"},{"key":"e_1_2_1_74_1","volume-title":"The Decade of Discovery. 10th Mountain Films","author":"Looby Joseph","unstructured":"Joseph Looby . 2014. The Decade of Discovery. 10th Mountain Films . Joseph Looby. 2014. The Decade of Discovery. 10th Mountain Films."},{"key":"#cr-split#-e_1_2_1_75_1.1","unstructured":"Kristian Lum and William Isaac. 2016. To Predict and Serve? The Royal Statistical Society. https:\/\/doi.org\/10.1111\/j.1740--9713.2016.00960.x. 10.1111\/j.1740--9713.2016.00960.x"},{"key":"#cr-split#-e_1_2_1_75_1.2","unstructured":"Kristian Lum and William Isaac. 2016. To Predict and Serve? The Royal Statistical Society. https:\/\/doi.org\/10.1111\/j.1740--9713.2016.00960.x."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1057\/palgrave.ejis.3000512"},{"key":"e_1_2_1_77_1","article-title":"Submerged Precedent","volume":"16","author":"McCuskey Elizabeth Y.","year":"2016","unstructured":"Elizabeth Y. McCuskey . 2016 . Submerged Precedent . Nevada Law Journal 16 : 515, 530--531 Elizabeth Y. McCuskey. 2016. Submerged Precedent. Nevada Law Journal 16: 515, 530--531","journal-title":"Nevada Law Journal"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419249.3420149"},{"key":"e_1_2_1_79_1","volume-title":"Conference on Human Factors in Computing Systems - Proceedings.","author":"Mucha H.","unstructured":"H. Mucha , D. Mevi\u00dfen , S. Robert , R. Jacobi , K. Meyer , W. Heusler , and D. Arztmann . 2020. Co-design futures for AI and space: A workbook sprint . Conference on Human Factors in Computing Systems - Proceedings. H. Mucha, D. Mevi\u00dfen, S. Robert, R. Jacobi, K. Meyer, W. Heusler, and D. Arztmann. 2020. Co-design futures for AI and space: A workbook sprint. Conference on Human Factors in Computing Systems - Proceedings."},{"key":"e_1_2_1_80_1","volume-title":"Towards Participatory Design Spaces for Explainable AI Interfaces in Expert Domains. CEUR Workshop Proceedings http:\/\/ceur-ws.org\/Vol-2796\/xi-ml-2020_mucha.pdf.","author":"Mucha H.","year":"2020","unstructured":"H. Mucha and S. Robert . ( 2020 ). Emerging Perspectives on Medical Decision Support : Co-Designing XAI; Henrik Mucha, Sebastian Robert et al . Towards Participatory Design Spaces for Explainable AI Interfaces in Expert Domains. CEUR Workshop Proceedings http:\/\/ceur-ws.org\/Vol-2796\/xi-ml-2020_mucha.pdf. H. Mucha and S. Robert. (2020). Emerging Perspectives on Medical Decision Support : Co-Designing XAI; Henrik Mucha, Sebastian Robert et al. Towards Participatory Design Spaces for Explainable AI Interfaces in Expert Domains. CEUR Workshop Proceedings http:\/\/ceur-ws.org\/Vol-2796\/xi-ml-2020_mucha.pdf."},{"key":"e_1_2_1_81_1","volume-title":"The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. 1125--1154.","author":"Muller Michael","year":"2012","unstructured":"Michael Muller and A. Druin . ( 2012 ). Participatory design: The third space in human-computer interaction . In The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. 1125--1154. Michael Muller and A. Druin. (2012). Participatory design: The third space in human-computer interaction. In The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications. 1125--1154."},{"key":"e_1_2_1_82_1","volume-title":"Text REtrieval Conference, Legal Track. https:\/\/trec.nist.gov\/data\/legal.html.","author":"National Institute of Standards and Technologies (NIST).","unstructured":"National Institute of Standards and Technologies (NIST). Text REtrieval Conference, Legal Track. https:\/\/trec.nist.gov\/data\/legal.html. National Institute of Standards and Technologies (NIST). Text REtrieval Conference, Legal Track. https:\/\/trec.nist.gov\/data\/legal.html."},{"key":"e_1_2_1_83_1","volume-title":"TREC 2007 Legal Track: Interactive Challenge Task. https:\/\/trec-legal.umiacs.umd.edu\/guidelines\/interactivetask.html.","author":"National Institute of Standards and Technologies (NIST).","unstructured":"National Institute of Standards and Technologies (NIST). TREC 2007 Legal Track: Interactive Challenge Task. https:\/\/trec-legal.umiacs.umd.edu\/guidelines\/interactivetask.html. National Institute of Standards and Technologies (NIST). TREC 2007 Legal Track: Interactive Challenge Task. https:\/\/trec-legal.umiacs.umd.edu\/guidelines\/interactivetask.html."},{"key":"e_1_2_1_84_1","volume-title":"The Future of Information Retrieval Evaluation","author":"Oard Douglas W.","year":"2020","unstructured":"Douglas W. Oard , The Future of Information Retrieval Evaluation . 2020 . In Evaluating Information Retrieval and Access Tasks: NTCIR's Legacy of Research Impact. Springer . Douglas W. Oard, The Future of Information Retrieval Evaluation. 2020. In Evaluating Information Retrieval and Access Tasks: NTCIR's Legacy of Research Impact. Springer."},{"key":"e_1_2_1_85_1","volume-title":"Lewis","author":"Oard Douglas W.","year":"2010","unstructured":"Douglas W. Oard , Jason R. Baron , and David D . Lewis . 2010 . Some Lessons Learned to Date from the TREC Legal Track ( 2006--2009). https:\/\/trec-legal.umiacs.umd.edu\/other\/LessonsLearned.pdf Douglas W. Oard, Jason R. Baron, and David D. Lewis. 2010. Some Lessons Learned to Date from the TREC Legal Track (2006--2009). https:\/\/trec-legal.umiacs.umd.edu\/other\/LessonsLearned.pdf"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10506-010-9093-9"},{"key":"e_1_2_1_87_1","volume-title":"Proceedings of the Seventeenth Text REtrieval Conference.","author":"Oard Douglas W.","year":"2008","unstructured":"Douglas W. Oard , Bruce Hedin , Stephen Tomlinson , Jason R. Baron , 2008 . TREC Legal Track 2008 Overview . In Proceedings of the Seventeenth Text REtrieval Conference. Douglas W. Oard, Bruce Hedin, Stephen Tomlinson, Jason R. Baron, 2008. TREC Legal Track 2008 Overview. In Proceedings of the Seventeenth Text REtrieval Conference."},{"key":"e_1_2_1_88_1","volume-title":"Pace and Laura Zakaras","author":"Nicholas","year":"2012","unstructured":"Nicholas M. Pace and Laura Zakaras . 2012 . Where the Money Goes : Understanding Litigant Expenditures for Producing Electronic Discovery . Santa Monica, CA : RAND Corporation Nicholas M. Pace and Laura Zakaras. 2012. Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery. Santa Monica, CA: RAND Corporation"},{"key":"e_1_2_1_89_1","volume-title":"A Rule of Persons","author":"Pasquale Frank A.","unstructured":"Frank A. Pasquale . 2018. A Rule of Persons , Not Machines : The Limits of Legal Automation. 87 George Washington Law Review 1. Frank A. Pasquale. 2018. A Rule of Persons, Not Machines: The Limits of Legal Automation. 87 George Washington Law Review 1."},{"key":"e_1_2_1_90_1","volume-title":"Baron","author":"Paul George L.","year":"2007","unstructured":"George L. Paul and Jason R . Baron . 2007 . Information Inflation : Can the Legal System Adapt? 13 Richmond Journal of Law and Technology . George L. Paul and Jason R. Baron. 2007. Information Inflation: Can the Legal System Adapt? 13 Richmond Journal of Law and Technology."},{"key":"e_1_2_1_91_1","unstructured":"Andrew Peck. 2011. Search Forward. Legal Tech News (October).  Andrew Peck. 2011. Search Forward. Legal Tech News (October)."},{"key":"e_1_2_1_92_1","volume-title":"Cases of \"What is a case?'.\" In What is a case?, edited by C. Ragin and H.S. Becker","author":"Ragin Charles","unstructured":"Charles Ragin . 1992. \"Introduction : Cases of \"What is a case?'.\" In What is a case?, edited by C. Ragin and H.S. Becker . New York : Cambridge University Press . Charles Ragin. 1992. \"Introduction: Cases of \"What is a case?'.\" In What is a case?, edited by C. Ragin and H.S. Becker. New York: Cambridge University Press."},{"key":"e_1_2_1_93_1","volume-title":"Proceedings of the Eighteenth Text REtrieval Conference.","author":"Rangan Venkat","year":"2009","unstructured":"Venkat Rangan and Maojin Jiang . 2009 . Clearwell Systems at TREC 2009 Legal Interactive . In Proceedings of the Eighteenth Text REtrieval Conference. Venkat Rangan and Maojin Jiang. 2009. Clearwell Systems at TREC 2009 Legal Interactive. In Proceedings of the Eighteenth Text REtrieval Conference."},{"key":"e_1_2_1_94_1","unstructured":"Dana A. Remus. 2014. The Uncertain Promise of Predictive Coding. 99 Iowa Law Review 1691 1692.  Dana A. Remus. 2014. The Uncertain Promise of Predictive Coding. 99 Iowa Law Review 1691 1692."},{"key":"e_1_2_1_95_1","volume-title":"Remus and Frank Levy","author":"Dana","year":"2017","unstructured":"Dana A. Remus and Frank Levy . 2017 . Can Robots Be Lawyers? Computers, Lawyers, and the Practice of Law. 30 Georgetown Journal of Legal Ethics . Dana A. Remus and Frank Levy. 2017. Can Robots Be Lawyers? Computers, Lawyers, and the Practice of Law. 30 Georgetown Journal of Legal Ethics."},{"key":"e_1_2_1_96_1","doi-asserted-by":"crossref","unstructured":"David Ribes Andrew S Hoffman Steven C Slota and Geoffrey C Bowker. 2019. The Logic of Domains. Social Studies of Science 49 no. 3.  David Ribes Andrew S Hoffman Steven C Slota and Geoffrey C Bowker. 2019. The Logic of Domains. Social Studies of Science 49 no. 3.","DOI":"10.1177\/0306312719849709"},{"key":"e_1_2_1_97_1","unstructured":"Rio Tinto PLC v. Vale S.A. 306 F.R.D. 125 (S.D.N.Y. 2015)  Rio Tinto PLC v. Vale S.A. 306 F.R.D. 125 (S.D.N.Y. 2015)"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1177\/0165551507086989"},{"key":"e_1_2_1_99_1","doi-asserted-by":"crossref","unstructured":"Herbert L. Roitblat Anne Kershaw and Patrick Oot. 2010. Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review. 61 American Society for Information Science and Technology 70.  Herbert L. Roitblat Anne Kershaw and Patrick Oot. 2010. Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review. 61 American Society for Information Science and Technology 70.","DOI":"10.1002\/asi.21233"},{"key":"e_1_2_1_100_1","unstructured":"L. Rosenfeld and P. Morville. 2002. Information architecture for the World Wide Web 2nd edn. O'Reilly Media Sebastopol.  L. Rosenfeld and P. Morville. 2002. Information architecture for the World Wide Web 2nd edn. O'Reilly Media Sebastopol."},{"key":"e_1_2_1_101_1","volume-title":"Simoni","author":"Rowe Brent R.","year":"2010","unstructured":"Brent R. Rowe , Dallas W. Wod , Albert N. Link , and Diglio A . Simoni . 2010 . Economic Impact Assessment of NIST'S Text REtrieval Conference (TREC) Program . https:\/\/trec.nist.gov\/pubs\/2010.economic.impact.pdf. Brent R. Rowe, Dallas W. Wod, Albert N. Link, and Diglio A. Simoni. 2010. Economic Impact Assessment of NIST'S Text REtrieval Conference (TREC) Program. https:\/\/trec.nist.gov\/pubs\/2010.economic.impact.pdf."},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.5555\/1315930.1315946"},{"key":"e_1_2_1_103_1","doi-asserted-by":"crossref","unstructured":"Tefko Saracevic. 2016. The Notion of Relevance in Information Science: Everybody knows what relevance is. But what is it really?. Synthesis Lectures on Information Concepts Retrieval and Services. 8. i-109. 10.2200\/S00723ED1V01Y201607ICR050.  Tefko Saracevic. 2016. The Notion of Relevance in Information Science: Everybody knows what relevance is. But what is it really?. Synthesis Lectures on Information Concepts Retrieval and Services. 8. i-109. 10.2200\/S00723ED1V01Y201607ICR050.","DOI":"10.2200\/S00723ED1V01Y201607ICR050"},{"key":"e_1_2_1_104_1","volume-title":"Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW, 383--388","author":"Saxena D.","unstructured":"D. Saxena and S. Guha . 2020. Conducting participatory design to improve algorithms in public services: Lessons and challenges . In Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW, 383--388 . D. Saxena and S. Guha. 2020. Conducting participatory design to improve algorithms in public services: Lessons and challenges. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW, 383--388."},{"key":"e_1_2_1_105_1","unstructured":"Erich Schweighofer. The Revolution in Legal Information Retrieval or: The Empire Strikes Back. Journal of Information Law and Technology (JILT) 1999 (1).  Erich Schweighofer. The Revolution in Legal Information Retrieval or: The Empire Strikes Back. Journal of Information Law and Technology (JILT) 1999 (1)."},{"key":"e_1_2_1_106_1","volume-title":"Sedona Conference. https:\/\/thesedonaconference.org.","author":"The","unstructured":"The Sedona Conference. https:\/\/thesedonaconference.org. The Sedona Conference. https:\/\/thesedonaconference.org."},{"key":"e_1_2_1_107_1","volume-title":"Sedona Conference. 2008","author":"The","year":"2008","unstructured":"The Sedona Conference. 2008 . Call for Volunteers to be Part of TREC Legal Track 2008 . The Sedona Conference. 2008. Call for Volunteers to be Part of TREC Legal Track 2008."},{"key":"e_1_2_1_108_1","volume-title":"Sedona Conference.","author":"The","year":"2009","unstructured":"The Sedona Conference. 2009 . An Open Letter to Law Firms and Companies in the Legal Tech Sector. The Sedona Conference. 2009. An Open Letter to Law Firms and Companies in the Legal Tech Sector."},{"key":"e_1_2_1_109_1","volume-title":"Sedona Conference.","author":"The","year":"2017","unstructured":"The Sedona Conference. 2017 b. TAR Case Law Primer. https:\/\/thesedonaconference.org\/node\/279. The Sedona Conference. 2017b. TAR Case Law Primer. https:\/\/thesedonaconference.org\/node\/279."},{"key":"e_1_2_1_110_1","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19)","author":"Andrew","unstructured":"Andrew D. Selbst et al. 2019. Fairness and Abstraction in Sociotechnical Systems . In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19) . ACM, New York, NY, USA, 59--68. Andrew D. Selbst et al. 2019. Fairness and Abstraction in Sociotechnical Systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19). ACM, New York, NY, USA, 59--68."},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372827"},{"key":"e_1_2_1_112_1","volume-title":"The Sedona Conference Journal.","author":"Shaffer Judge Craig B.","year":"2012","unstructured":"Judge Craig B. Shaffer . 2012 . ?Defensible' by What Standard? The Sedona Conference Journal. Judge Craig B. Shaffer. 2012. ?Defensible' by What Standard? The Sedona Conference Journal."},{"key":"e_1_2_1_113_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445971"},{"key":"e_1_2_1_114_1","volume-title":"David Ribes and Geoffrey C Bowker. Prospecting (in) the data sciences","author":"Slota Stephen C","year":"2020","unstructured":"Stephen C Slota , Andrew S. Hoffman , David Ribes and Geoffrey C Bowker. Prospecting (in) the data sciences . Big Data & Society. January-June: 1--12, 2020 . Stephen C Slota, Andrew S. Hoffman, David Ribes and Geoffrey C Bowker. Prospecting (in) the data sciences. Big Data & Society. January-June: 1--12, 2020."},{"key":"e_1_2_1_115_1","doi-asserted-by":"crossref","unstructured":"Phillip Smith Norman D. Geddes Rober Beatty and Caroline Hayes. 2012. Human-Centered Design of Decision-Support Systems. In The Human-Computer Interaction Handbook: Fundamentals Evolving Technologies and Emerging Applications: 590--615.  Phillip Smith Norman D. Geddes Rober Beatty and Caroline Hayes. 2012. Human-Centered Design of Decision-Support Systems. In The Human-Computer Interaction Handbook: Fundamentals Evolving Technologies and Emerging Applications: 590--615.","DOI":"10.1201\/b11963-ch-26"},{"key":"e_1_2_1_116_1","volume-title":"Proceedings of the Nineteenth Text REtrieval Conference.","author":"Smucker Mark D.","year":"2010","unstructured":"Mark D. Smucker , Charles L. A. Clarke , Gordon V. Cormack , and Olga Vechtomova . 2010 . University of Waterloo at TREC 2010: Legal Interactive . In Proceedings of the Nineteenth Text REtrieval Conference. Mark D. Smucker, Charles L. A. Clarke, Gordon V. Cormack, and Olga Vechtomova. 2010. University of Waterloo at TREC 2010: Legal Interactive. In Proceedings of the Nineteenth Text REtrieval Conference."},{"key":"e_1_2_1_117_1","volume-title":"Fostering Historical Research in CSCW & HCI. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (CSCW '19)","author":"Soden Robert","year":"2019","unstructured":"Robert Soden , David Ribes , Maggie Jack , Will Sutherland , Vera Khovanskaya , Seyram Avle , Phoebe Sengers , and Susanne B\u00f8dker . 2019 . Fostering Historical Research in CSCW & HCI. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (CSCW '19) . Association for Computing Machinery, New York, NY, USA, 517--522. Robert Soden, David Ribes, Maggie Jack, Will Sutherland, Vera Khovanskaya, Seyram Avle, Phoebe Sengers, and Susanne B\u00f8dker. 2019. Fostering Historical Research in CSCW & HCI. In Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing (CSCW '19). Association for Computing Machinery, New York, NY, USA, 517--522."},{"key":"e_1_2_1_118_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479603"},{"issue":"2","key":"e_1_2_1_119_1","first-page":"163","article-title":"The methodology of participatory design","volume":"52","author":"Spinuzzi C.","year":"2005","unstructured":"C. Spinuzzi . 2005 . The methodology of participatory design . Technical Communication 52 ( 2 ): 163 -- 174 . C. Spinuzzi. 2005. The methodology of participatory design. Technical Communication 52 (2): 163--174.","journal-title":"Technical Communication"},{"key":"e_1_2_1_120_1","volume-title":"The Art of Case Study Research","author":"Robert E.","unstructured":"Robert E. Stake 1995. The Art of Case Study Research . Sage Publications . Robert E. Stake 1995. The Art of Case Study Research. Sage Publications."},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1177\/030631289019003001"},{"key":"e_1_2_1_122_1","volume-title":"Cambridge, Massachusetts: The MIT Press.","author":"Star Susan Leigh","year":"1989","unstructured":"Susan Leigh Star . ( 1989) 2015 . The Structure of Ill-Structured Solutions: Boundary Objects and Heterogeneous Distributed Problem Solving. In Bowker, Geoffrey C., Stefan Timmermans, Adele E. Clarke, and Ellen Balka, eds. Boundary Objects and beyond: Working with Leigh Star . Cambridge, Massachusetts: The MIT Press. Susan Leigh Star. (1989) 2015. The Structure of Ill-Structured Solutions: Boundary Objects and Heterogeneous Distributed Problem Solving. In Bowker, Geoffrey C., Stefan Timmermans, Adele E. Clarke, and Ellen Balka, eds. Boundary Objects and beyond: Working with Leigh Star. Cambridge, Massachusetts: The MIT Press."},{"issue":"1","key":"e_1_2_1_123_1","first-page":"79","article-title":"Uniformity in Procedural Rules and the Attributes of a Sound Procedural System: The Case for Presumptive Limits","volume":"49","author":"Subrin Stephen","year":"1997","unstructured":"Stephen Subrin . 1997 . Uniformity in Procedural Rules and the Attributes of a Sound Procedural System: The Case for Presumptive Limits . Alabama Law Review 49 ( 1 ): 79 -- 102 . Stephen Subrin. 1997. Uniformity in Procedural Rules and the Attributes of a Sound Procedural System: The Case for Presumptive Limits. Alabama Law Review 49(1): 79--102.","journal-title":"Alabama Law Review"},{"key":"e_1_2_1_124_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00749282"},{"key":"e_1_2_1_125_1","doi-asserted-by":"publisher","DOI":"10.1145\/223248.223263"},{"key":"e_1_2_1_126_1","volume-title":"Proceedings of the Nineteenth Text REtrieval Conference.","author":"Sun Ying","year":"2010","unstructured":"Ying Sun . 2010 . UB at TREC 2010 Legal Interactive . In Proceedings of the Nineteenth Text REtrieval Conference. Ying Sun. 2010. UB at TREC 2010 Legal Interactive. In Proceedings of the Nineteenth Text REtrieval Conference."},{"key":"e_1_2_1_127_1","doi-asserted-by":"publisher","DOI":"10.1177\/0735275112457914"},{"issue":"1","key":"e_1_2_1_128_1","first-page":"27","article-title":"Distributed development to enable user participation: Multilevel design in the HISP network","volume":"21","author":"Titlestad O. H.","year":"2009","unstructured":"O. H. Titlestad , K. Staring , and J. Braa , 2009 . Distributed development to enable user participation: Multilevel design in the HISP network . Scandinavian Journal of Information Systems 21 ( 1 ), 27 -- 50 . O. H. Titlestad, K. Staring, and J. Braa, 2009. Distributed development to enable user participation: Multilevel design in the HISP network. Scandinavian Journal of Information Systems 21 (1), 27--50.","journal-title":"Scandinavian Journal of Information Systems"},{"key":"e_1_2_1_129_1","volume-title":"Proceedings of the Sixteenth Text REtrieval Conference.","author":"Tomlinson Stephen","unstructured":"Stephen Tomlinson , Douglas Oard , Jason Baron , and P. Thompson . 2007. TREC Legal Track 2007 Overview . In Proceedings of the Sixteenth Text REtrieval Conference. Stephen Tomlinson, Douglas Oard, Jason Baron, and P. Thompson. 2007. TREC Legal Track 2007 Overview. In Proceedings of the Sixteenth Text REtrieval Conference."},{"key":"e_1_2_1_130_1","first-page":"215","article-title":"Participatory design and design for values","volume":"11","author":"der Velden Maja Van","year":"2014","unstructured":"Maja Van der Velden , C M\u00f6rtberg , J Van den Hoven , PE Vermaas , and I Van de Poel . 2014 . Participatory design and design for values . Development 11 , 3 (2014), 215 -- 236 Maja Van der Velden, C M\u00f6rtberg, J Van den Hoven, PE Vermaas, and I Van de Poel. 2014. Participatory design and design for values. Development 11, 3 (2014), 215--236","journal-title":"Development"},{"key":"e_1_2_1_131_1","volume-title":"The heuristics of case analysis.\" In What is a case?, edited by Charles Ragin and H.S. Becker","author":"Vaughan Diane","unstructured":"Diane Vaughan . 1992. \" Theory Elaboration : The heuristics of case analysis.\" In What is a case?, edited by Charles Ragin and H.S. Becker . New York : Cambridge University Press . Diane Vaughan. 1992. \"Theory Elaboration: The heuristics of case analysis.\" In What is a case?, edited by Charles Ragin and H.S. Becker. New York: Cambridge University Press."},{"key":"e_1_2_1_132_1","unstructured":"Victor Stanley v. Creative Pipe Inc. 250 F.R.D. 251 (D. Md. 2008)  Victor Stanley v. Creative Pipe Inc. 250 F.R.D. 251 (D. Md. 2008)"},{"key":"#cr-split#-e_1_2_1_133_1.1","doi-asserted-by":"crossref","unstructured":"E. M. Voorhees. 2002. The Philosophy of Information Retrieval Evaluation. 355--370. https:\/\/doi.org\/10.1007\/3--540--45691-0_34. 10.1007\/3--540--45691-0_34","DOI":"10.1007\/3-540-45691-0_34"},{"key":"#cr-split#-e_1_2_1_133_1.2","doi-asserted-by":"crossref","unstructured":"E. M. Voorhees. 2002. The Philosophy of Information Retrieval Evaluation. 355--370. https:\/\/doi.org\/10.1007\/3--540--45691-0_34.","DOI":"10.1007\/3-540-45691-0_34"},{"key":"e_1_2_1_134_1","volume-title":"Text Retrieval Conference (TREC). Encyclopedia of Library and Information Sciences.","author":"Voorhees E. M.","year":"2009","unstructured":"E. M. Voorhees and Donna Harman . 2009 , Text Retrieval Conference (TREC). Encyclopedia of Library and Information Sciences. E. M. Voorhees and Donna Harman. 2009, Text Retrieval Conference (TREC). Encyclopedia of Library and Information Sciences."},{"key":"e_1_2_1_135_1","volume-title":"TREC 2008 at the University at Buffalo: Legal and Blog Track. In Proceedings of the Seventeenth Text REtrieval Conference.","author":"Wang Jianqiang","year":"2008","unstructured":"Jianqiang Wang , Ying Sun , Omar Mukhtar , and Rohini Srihari . 2008 . TREC 2008 at the University at Buffalo: Legal and Blog Track. In Proceedings of the Seventeenth Text REtrieval Conference. Jianqiang Wang, Ying Sun, Omar Mukhtar, and Rohini Srihari. 2008. TREC 2008 at the University at Buffalo: Legal and Blog Track. In Proceedings of the Seventeenth Text REtrieval Conference."},{"key":"e_1_2_1_136_1","volume-title":"Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '18)","author":"Wolf Christine T.","unstructured":"Christine T. Wolf , Haiyi Zhu , Julia Bullard , Min Kyung Lee , and Jed R. Brubaker . 2018. The Changing Contours of \"Participation\" in Data-driven, Algorithmic Ecosystems: Challenges, Tactics, and an Agenda . In Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '18) . Association for Computing Machinery, New York, NY, USA, 377--384. Christine T. Wolf, Haiyi Zhu, Julia Bullard, Min Kyung Lee, and Jed R. Brubaker. 2018. The Changing Contours of \"Participation\" in Data-driven, Algorithmic Ecosystems: Challenges, Tactics, and an Agenda. In Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '18). Association for Computing Machinery, New York, NY, USA, 377--384."},{"key":"e_1_2_1_137_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300468"},{"key":"e_1_2_1_138_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-019-09497-z"},{"key":"e_1_2_1_139_1","volume-title":"Bogdan Kylych, Smitha Mill and Richard Zemel.","author":"Zhou Angela","year":"2020","unstructured":"Angela Zhou , David Madra , Inioluwa Deborah Raji , Bogdan Kylych, Smitha Mill and Richard Zemel. 2020 . Call for Participation: Participatory Approaches to Machine Learning . https:\/\/participatoryml.github.io\/. Angela Zhou, David Madra, Inioluwa Deborah Raji, Bogdan Kylych, Smitha Mill and Richard Zemel. 2020. Call for Participation: Participatory Approaches to Machine Learning. https:\/\/participatoryml.github.io\/."},{"key":"e_1_2_1_140_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274463"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512898","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3512898","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:12:28Z","timestamp":1750191148000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512898"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,30]]},"references-count":143,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2022,3,30]]}},"alternative-id":["10.1145\/3512898"],"URL":"https:\/\/doi.org\/10.1145\/3512898","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,30]]},"assertion":[{"value":"2022-04-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}