{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T17:44:32Z","timestamp":1754156672964,"version":"3.41.2"},"reference-count":20,"publisher":"Emerald","issue":"1","license":[{"start":{"date-parts":[[2019,1,21]],"date-time":"2019-01-21T00:00:00Z","timestamp":1548028800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["AJIM"],"published-print":{"date-parts":[[2019,1,21]]},"abstract":"<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title>\n<jats:p>The effort in addition to relevance is a major factor for satisfaction and utility of the document to the actual user. The purpose of this paper is to propose a method in generating relevance judgments that incorporate effort without human judges\u2019 involvement. Then the study determines the variation in system rankings due to low effort relevance judgment in evaluating retrieval systems at different depth of evaluation.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title>\n<jats:p>Effort-based relevance judgments are generated using a proposed boxplot approach for simple document features, HTML features and readability features. The boxplot approach is a simple yet repeatable approach in classifying documents\u2019 effort while ensuring outlier scores do not skew the grading of the entire set of documents.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Findings<\/jats:title>\n<jats:p>The retrieval systems evaluation using low effort relevance judgments has a stronger influence on shallow depth of evaluation compared to deeper depth. It is proved that difference in the system rankings is due to low effort documents and not the number of relevant documents.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title>\n<jats:p>Hence, it is crucial to evaluate retrieval systems at shallow depth using low effort relevance judgments.<\/jats:p>\n<\/jats:sec>","DOI":"10.1108\/ajim-04-2018-0086","type":"journal-article","created":{"date-parts":[[2018,10,11]],"date-time":"2018-10-11T06:44:20Z","timestamp":1539240260000},"page":"2-17","source":"Crossref","is-referenced-by-count":1,"title":["Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment"],"prefix":"10.1108","volume":"71","author":[{"given":"Prabha","family":"Rajagopal","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sri Devi","family":"Ravana","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yun Sing","family":"Koh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vimala","family":"Balakrishnan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"issue":"5","key":"key2020092515354736900_ref001","first-page":"859","article-title":"A review of factors influencing user satisfaction in information retrieval","volume":"61","year":"2010","journal-title":"Journal of the Association for Information Science and Technology"},{"first-page":"539","article-title":"The effect of assessor errors on IR system evaluation","year":"2010","key":"key2020092515354736900_ref002"},{"first-page":"745","article-title":"Document features predicting assessor disagreement","year":"2013","key":"key2020092515354736900_ref003"},{"first-page":"1","article-title":"Enriching information retrieval with reading level prediction","year":"2011","key":"key2020092515354736900_ref004"},{"key":"key2020092515354736900_ref005","doi-asserted-by":"crossref","unstructured":"DuBay, W.H. (2004), \u201cThe principles of readability\u201d, Online Submission, available at: https:\/\/doi.org\/10.1080\/07060668509501661 (accessed September 2, 2018).","DOI":"10.1080\/07060668509501661"},{"issue":"4","key":"key2020092515354736900_ref006","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1629096.1629099","article-title":"A few good topics: experiments in topic set reduction for retrieval evaluation","volume":"27","year":"2009","journal-title":"ACM Transactions on Information Systems"},{"key":"key2020092515354736900_ref007","unstructured":"G\u00fastafsd\u00f3ttir, G.U. (2017), \u201cReadability tests \u2013 siteimprove help center\u201d, available at: https:\/\/support.siteimprove.com\/hc\/en-gb\/articles\/114094009592-Readability-tests (accessed August 1, 2017)."},{"first-page":"17","article-title":"Do batch and user evaluations give the same results?","year":"2000","key":"key2020092515354736900_ref008"},{"first-page":"1843","article-title":"Size and source matter: understanding inconsistencies in test collection-based evaluation","year":"2014","key":"key2020092515354736900_ref009"},{"first-page":"47","article-title":"Models and metrics: IR evaluation as a user process","year":"2012","key":"key2020092515354736900_ref010"},{"issue":"4","key":"key2020092515354736900_ref011","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1561\/1500000009","article-title":"Test collection based evaluation of information retrieval systems","volume":"4","year":"2010","journal-title":"Foundations and Trends\u00ae in Information Retrieval"},{"first-page":"1063","article-title":"Quantifying test collection quality based on the consistency of relevance judgements","year":"2011","key":"key2020092515354736900_ref012"},{"first-page":"1","article-title":"Automated readability index","year":"1967","key":"key2020092515354736900_ref013"},{"first-page":"225","article-title":"Why batch and user evaluations do not give the same results","year":"2001","key":"key2020092515354736900_ref014"},{"first-page":"277","article-title":"On obtaining effort based judgements for information retrieval","year":"2016","key":"key2020092515354736900_ref015"},{"first-page":"765","article-title":"Is relevance hard work? Evaluating the effort of making relevant assessments","year":"2013","key":"key2020092515354736900_ref016"},{"issue":"5","key":"key2020092515354736900_ref017","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1016\/S0306-4573(00)00010-8","article-title":"Variations in relevance judgments and the measurement of retrieval effectiveness","volume":"36","year":"2000","journal-title":"Information Processing and Management"},{"first-page":"74","article-title":"Evaluation by highly relevant documents","year":"2001","key":"key2020092515354736900_ref018"},{"first-page":"1053","article-title":"Effect of written instructions on assessor agreement","year":"2012","key":"key2020092515354736900_ref019"},{"first-page":"91","article-title":"Relevance and effort: an analysis of document utility","year":"2014","key":"key2020092515354736900_ref020"}],"container-title":["Aslib Journal of Information Management"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/AJIM-04-2018-0086\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/AJIM-04-2018-0086\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:00:35Z","timestamp":1753398035000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/ajim\/article\/71\/1\/2-17\/60615"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,1,21]]},"references-count":20,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,1,21]]}},"alternative-id":["10.1108\/AJIM-04-2018-0086"],"URL":"https:\/\/doi.org\/10.1108\/ajim-04-2018-0086","relation":{},"ISSN":["2050-3806"],"issn-type":[{"type":"print","value":"2050-3806"}],"subject":[],"published":{"date-parts":[[2019,1,21]]}}}