{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T09:27:45Z","timestamp":1790069265889,"version":"4.0.1"},"reference-count":16,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2011,2,3]],"date-time":"2011-02-03T00:00:00Z","timestamp":1296691200000},"content-version":"vor","delay-in-days":94,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc of Assoc for Info"],"published-print":{"date-parts":[[2010,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper presents a comparative user study that investigates the relevance judgments made by assessors with a law background and assessors without. Four law students and four library and information studies (LIS) students were recruited to judge independently the relevance of 100 documents for each of four requests for production of responsive documents for litigation purposes, two requests for the tobacco Master Settlement Agreement collection and two for the Enron corporate emails collection. Both quantitative and qualitative methods are used to analyze data collected through the relevance judgment task, an entry questionnaire, and an exit interview. Being given the same task guidelines, the LIS student assessors judged <jats:italic>relevant<\/jats:italic> documents just as accurately as the law student assessors, while they judged <jats:italic>nonrelevant<\/jats:italic> documents slightly less accurately than the law student assessors. In addition, participants achieved moderate to substantial agreement on their relevance judgments. Relevance judgment speed varied significantly among participants, although on average it was about the same for the two groups. Factors influencing the accuracy and the speed of participants' relevance judgments are discussed based on preliminary analysis of qualitative data collected through the exit interviews.<\/jats:p>","DOI":"10.1002\/meet.14504701157","type":"journal-article","created":{"date-parts":[[2011,2,3]],"date-time":"2011-02-03T16:35:46Z","timestamp":1296750946000},"page":"1-10","source":"Crossref","is-referenced-by-count":2,"title":["A user study of relevance judgments for E\u2010Discovery"],"prefix":"10.1002","volume":"47","author":[{"given":"Jianqiang","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dagobert","family":"Soergel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2011,2,3]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"crossref","unstructured":"Bailey P. Craswell N. Soboroff I. Thomas P. de Vries A. P. Yilmaz E.(2008).Relevance assessment: are judges exchangeable and does it matter. In Proceedings of the 31st Annual international ACM SIGIR Conference on Research and Development in information Retrieval 667\u2013674.","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_2_8_3_1","doi-asserted-by":"crossref","unstructured":"Baron J. R. Lewis D. D. Oard D. W.(2006).TREC\u20102006 legal track overview. In Proceedings the Fifteenth Text REtrieval Conference (TREC 2006). Retrieved May 24 2010 fromhttp:\/\/trec.nist.gov.","DOI":"10.6028\/NIST.SP.500-272.legal-overview"},{"key":"e_1_2_8_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"e_1_2_8_5_1","volume-title":"Technical Report ASLIB Part","author":"Cleverdon C. W.","year":"1970"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(71)90024-6"},{"key":"e_1_2_8_8_1","doi-asserted-by":"crossref","unstructured":"Hedin B. Tomlinson S. Baron J. B. Oard D. W.(2009).Overview of the 2009 TREC legal track. In Proceedings the Eighteenth Text REtrieval Conference (TREC 2009). Retrieved May 24 2009 fromhttp:\/\/trec.nist.gov.","DOI":"10.6028\/NIST.SP.500-278.legal-overview"},{"issue":"1","key":"e_1_2_8_9_1","article-title":"An evidence perspective on topical relevance types and its implications for exploratory and task\u2010based retrieval","volume":"12","author":"Huang X.","year":"2006","journal-title":"Information Research"},{"key":"e_1_2_8_10_1","unstructured":"Huang X.(2009)Topical Relevance Rhetoric and Argumentation: A Cross\u2010Disciplinary Inquiry into Patterns of Thinking and Information StructuringPhD Dissertation University of Maryland. Retrieved May 26 2010 fromhttp:\/\/terpconnect.umd.edu\/\u223cxiaoli\/XH_DisserationAbstract.pdf"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_2_8_12_1","doi-asserted-by":"crossref","unstructured":"Oard D. W. Hedin B. TomlinsonS. BaronJ.B.(2008).Overview of the TREC 2008 legal track. In Proceedings of the Seventeenth Text REtrieval Conference. Retrieved May 24 2010 from:http:\/\/trec.nist.gov.","DOI":"10.6028\/NIST.SP.500-277.legal-overview"},{"key":"e_1_2_8_13_1","doi-asserted-by":"crossref","unstructured":"Voorhees E. M.(1998).Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 315\u2013323.","DOI":"10.1145\/290941.291017"},{"key":"e_1_2_8_14_1","doi-asserted-by":"crossref","unstructured":"Wang J. Sun Y. Mukhtar O Srihari R.(2008).TREC 2008 at the university at buffalo: legal and blog track. In Proceedings the Seventeenth Text REtrieval Conference (TREC 2008). Retrieved May 24 2009 fromhttp:\/\/trec.nist.gov.","DOI":"10.6028\/NIST.SP.500-277.legal-SUNY_Buffalo"},{"key":"e_1_2_8_15_1","doi-asserted-by":"crossref","unstructured":"Wang J. Sun Y. Thompson P.(2009).TREC 2009 at the university at buffalo: interactive legal e\u2010discovery with Enron emails. In Proceedings the Eighteenth Text REtrieval Conference (TREC 2009) Retrieved May 24 2009 fromhttp:\/\/trec.nist.gov.","DOI":"10.6028\/NIST.SP.500-278.legal-SUNY_Buffalo"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199802)49:2<115::AID-ASI3>3.0.CO;2-T"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(73)90096-X"}],"container-title":["Proceedings of the American Society for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fmeet.14504701157","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fmeet.14504701157","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/meet.14504701157","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,20]],"date-time":"2025-10-20T14:52:08Z","timestamp":1760971928000},"score":1,"resource":{"primary":{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/10.1002\/meet.14504701157"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,11]]},"references-count":16,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2010,11]]}},"alternative-id":["10.1002\/meet.14504701157"],"URL":"https:\/\/doi.org\/10.1002\/meet.14504701157","archive":["Portico"],"relation":{},"ISSN":["0044-7870","1550-8390"],"issn-type":[{"value":"0044-7870","type":"print"},{"value":"1550-8390","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,11]]}}}