{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T15:41:15Z","timestamp":1774539675601,"version":"3.50.1"},"reference-count":28,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2021,4,13]],"date-time":"2021-04-13T00:00:00Z","timestamp":1618272000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["IIS-1618695"],"award-info":[{"award-number":["IIS-1618695"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,4,13]]},"abstract":"<jats:p>This paper describes and assesses the value sensitive design (VSD) of a test collection: data used to train and evaluate a machine learning system for information retrieval. The project used the VSD framework and methods to design a test collection annotated for discretion. We conducted qualitative stakeholder interviews to develop values personas, which guided annotation of a collection of corporate emails for contextual notions of sensitivity. Both qualitative and quantitative evaluations of the method reveal that the values personas concretely shaped annotators' sensitivity judgments, and analysis of the test collection itself demonstrates that the sensitivity annotations have utility for identifying features that may correlate with email sensitivity. Values personas for training data annotation expand the toolkit of methods for value-sensitive machine learning.<\/jats:p>","DOI":"10.1145\/3449207","type":"journal-article","created":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T17:51:09Z","timestamp":1619113869000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Search with Discretion"],"prefix":"10.1145","volume":"5","author":[{"given":"Modassir","family":"Iqbal","sequence":"first","affiliation":[{"name":"University of Maryland - College Park, College Park, MD, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katie","family":"Shilton","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, College Park, MD, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mahmoud F.","family":"Sayed","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, College Park, MD, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Douglas","family":"Oard","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, College Park, MD, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonah Lynn","family":"Rivera","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, College Park, MD, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William","family":"Cox","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, College Park, MD, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,4,22]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Retrieved","author":"Adlin Tamara","year":"2010","unstructured":"Tamara Adlin and John Pruitt. 2010. The essential persona lifecycle: your guide to building and using personas. Morgan Kaufmann, Amsterdam; Boston. Retrieved May 26, 2020 from http:\/\/www.books24x7.com\/marc.asp?bookid=37227"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","unstructured":"M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Trans. Assoc. Comput. Linguist.6 (2018) 587--604. DOI:https:\/\/doi.org\/10.1162\/tacl_a_00041","DOI":"10.1162\/tacl_a_00041"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.2307\/2286841"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3303772.3303798"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/99332.99357"},{"key":"e_1_2_1_6_1","volume-title":"The Future of Email Archives A Report from the Task Force on Technical Approaches for Email Archives","author":"Council on Library and Information Resources. 2018.","year":"2018","unstructured":"Council on Library and Information Resources. 2018. The Future of Email Archives A Report from the Task Force on Technical Approaches for Email Archives. Council on Library and Information Resources. Retrieved from https:\/\/clir.wordpress.clir.org\/wp-content\/uploads\/sites\/6\/2018\/08\/CLIR-pub175.pdf"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287589"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 2008 International Conference on Digital Government Research (dg.o '08)","author":"Friedman Batya","year":"2008","unstructured":"Batya Friedman, Alan Borning, Janet L. Davis, Brian T. Gill, Peter H. Kahn, Travis Kriplean, and Peyina Lin. 2008. Laying the Foundations for Public Participation and Value Advocacy: Interaction Design for a Large Scale Urban Simulation. In Proceedings of the 2008 International Conference on Digital Government Research (dg.o '08), Digital Government Society of North America, 305--314."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208562"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000015"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","unstructured":"Batya Friedman Peter Kahn Alan Borning Ping Zhang and Dennis Galletta. 2006. Value Sensitive Design and Information Systems. In The Handbook of Information and Computer Ethics. DOI:https:\/\/doi.org\/10.1007\/978-94-007-7844-3_4","DOI":"10.1007\/978-94-007-7844-3_4"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_2_1_14_1","first-page":"1","article-title":"Think Before You Type: A Look at Email Privacy in the Work Place","volume":"11","author":"Hornug Meir","year":"2005","unstructured":"Meir Hornug. 2005. Think Before You Type: A Look at Email Privacy in the Work Place. Fordham J. Corp. Financ. Law 11, 1 (January 2005), 115.","journal-title":"Fordham J. Corp. Financ. Law"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_2_1_16_1","unstructured":"Justin Larner. Value-led Personas: A Methodology to Promote Sustainable User-centered Design? Retrieved from http:\/\/nordichi2014.appaholiclabs.com\/details_paper.php?id=6867104"},{"key":"e_1_2_1_17_1","article-title":"Measuring Privacy: An Empirical Test Using Context to Expose Confounding Variables. Columbia Sci","volume":"18","author":"Martin Kirsten","year":"2016","unstructured":"Kirsten Martin and Helen Nissenbaum. 2016. Measuring Privacy: An Empirical Test Using Context to Expose Confounding Variables. Columbia Sci. Technol. Law Rev.18, 1 (2017 2016), 176--218.","journal-title":"Technol. Law Rev."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/01972243.2016.1153012"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2957276.2957280"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.2016.0118"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531643"},{"key":"e_1_2_1_22_1","volume-title":"Privacy in context: Technology, policy, and the integrity of social life","author":"Nissenbaum Helen","unstructured":"Helen Nissenbaum. 2009. Privacy in context: Technology, policy, and the integrity of social life. Stanford University Press."},{"key":"e_1_2_1_23_1","volume-title":"Retrieved","author":"Oard Douglas W.","year":"2015","unstructured":"Douglas W. Oard, William Webber, David Kirsch, and Sergey Golitsynskiy. 2015. Avocado Research Email Collection - Linguistic Data Consortium. Retrieved October 15, 2020 from https:\/\/catalog.ldc.upenn.edu\/LDC2015T03"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Mahmoud","unstructured":"Mahmoud F. Sayed and Douglas W. Oard. 2019. Jointly Modeling Relevance and Sensitivity for Search Among Sensitive Content. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Paris, France."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.3390\/bdcc3010005"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","unstructured":"Qian Yang Aaron Steinfeld Carolyn Ros\u00e9 and John Zimmerman. 2020. Re-examining Whether Why and How Human-AI Interaction Is Uniquely Difficult to Design. DOI:https:\/\/doi.org\/10.1145\/3313831.3376301","DOI":"10.1145\/3313831.3376301"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274463"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401284"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449207","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449207","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3449207","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:22Z","timestamp":1750195702000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3449207"}},"subtitle":["Value Sensitive Design of Training Data for Information Retrieval"],"short-title":[],"issued":{"date-parts":[[2021,4,13]]},"references-count":28,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2021,4,13]]}},"alternative-id":["10.1145\/3449207"],"URL":"https:\/\/doi.org\/10.1145\/3449207","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,13]]},"assertion":[{"value":"2021-04-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}