{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T04:54:30Z","timestamp":1755838470964,"version":"3.41.2"},"reference-count":23,"publisher":"Emerald","issue":"6","license":[{"start":{"date-parts":[[2015,11,16]],"date-time":"2015-11-16T00:00:00Z","timestamp":1447632000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2015,11,16]]},"abstract":"<jats:sec>\n               <jats:title content-type=\"abstract-heading\">Purpose<\/jats:title>\n               <jats:p> \u2013 In a system-based approach, replicating the web would require large test collections, and judging the relevancy of all documents per topic in creating relevance judgment through human assessors is infeasible. Due to the large amount of documents that requires judgment, there are possible errors introduced by human assessors because of disagreements. The paper aims to discuss these issues. <\/jats:p>\n            <\/jats:sec>\n            <jats:sec>\n               <jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title>\n               <jats:p> \u2013 This study explores exponential variation and document ranking methods that generate a reliable set of relevance judgments (pseudo relevance judgments) to reduce human efforts. These methods overcome problems with large amounts of documents for judgment while avoiding human disagreement errors during the judgment process. This study utilizes two key factors: number of occurrences of each document per topic from all the system runs; and document rankings to generate the alternate methods. <\/jats:p>\n            <\/jats:sec>\n            <jats:sec>\n               <jats:title content-type=\"abstract-heading\">Findings<\/jats:title>\n               <jats:p> \u2013 The effectiveness of the proposed method is evaluated using the correlation coefficient of ranked systems using mean average precision scores between the original Text REtrieval Conference (TREC) relevance judgments and pseudo relevance judgments. The results suggest that the proposed document ranking method with a pool depth of 100 could be a reliable alternative to reduce human effort and disagreement errors involved in generating TREC-like relevance judgments. <\/jats:p>\n            <\/jats:sec>\n            <jats:sec>\n               <jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title>\n               <jats:p> \u2013 Simple methods proposed in this study show improvement in the correlation coefficient in generating alternate relevance judgment without human assessors while contributing to information retrieval evaluation.<\/jats:p>\n            <\/jats:sec>","DOI":"10.1108\/ajim-03-2015-0046","type":"journal-article","created":{"date-parts":[[2015,11,5]],"date-time":"2015-11-05T03:59:16Z","timestamp":1446695956000},"page":"700-714","source":"Crossref","is-referenced-by-count":10,"title":["Ranking retrieval systems using pseudo relevance judgments"],"prefix":"10.1108","volume":"67","author":[{"given":"Sri Devi","family":"Ravana","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Prabha","family":"Rajagopal","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vimala","family":"Balakrishnan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2020121904432713400_b1","doi-asserted-by":"crossref","unstructured":"Bailey, P.\n               , \n                  Craswell, N.\n               , \n                  Soboroff, I.\n               , \n                  Thomas, P.\n               , \n                  de Vries, A.\n                and \n                  Yilmaz, E.\n                (2008), \u201cRelevance assessment: are judges exchangeable and does it matter?\u201d, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Singapore, pp. 667-674.","DOI":"10.1145\/1390334.1390447"},{"key":"key2020121904432713400_b2","doi-asserted-by":"crossref","unstructured":"Carpineto, C.\n                and \n                  Romano, G.\n                (2012), \u201cA survey of automatic query expansion in information retrieval\u201d, \n                  ACM Computing Surveys\n               , Vol. 44 No. 1, pp. 1-50.","DOI":"10.1145\/2071389.2071390"},{"key":"key2020121904432713400_b3","doi-asserted-by":"crossref","unstructured":"Carterette, B.\n                and \n                  Soboroff, I.\n                (2010), \u201cThe effect of assessor errors on IR system evaluation\u201d, Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Geneva, pp. 539-546.","DOI":"10.1145\/1835449.1835540"},{"key":"key2020121904432713400_b4","doi-asserted-by":"crossref","unstructured":"Cleverdon, C.W.\n                (1991), \u201cThe significance of the cranfield tests on index languages\u201d, Proceedings of the 14th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Chicago, IL, pp. 3-12.","DOI":"10.1145\/122860.122861"},{"key":"key2020121904432713400_b5","unstructured":"Greengrass, E.\n                (2000), \n                  Information Retrieval: A Survey\n               , University of Maryland, Baltimore County, MD."},{"key":"key2020121904432713400_b6","doi-asserted-by":"crossref","unstructured":"Hersh, W.\n               , \n                  Elliot, D.\n               , \n                  Hickam, D.\n               , \n                  Wolf, S.\n                and \n                  Molnar, A.\n                (1995), \u201cTowards new measures of information retrieval evaluation\u201d, Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Seattle, WA, pp. 164-170.","DOI":"10.1145\/215206.215355"},{"key":"key2020121904432713400_b7","doi-asserted-by":"crossref","unstructured":"Huffman, S.B.\n                and \n                  Hochster, M.\n                (2007), \u201cHow well does result relevance predict session satisfaction?\u201d, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Amsterdam, pp. 567-574.","DOI":"10.1145\/1277741.1277839"},{"key":"key2020121904432713400_b8","unstructured":"Mandl, T.\n                (2008), \u201cRecent developments in the evaluation of information retrieval systems: moving towards diversity and practical relevance\u201d, \n                  Informatica\n               , Vol. 32 No. 1, pp. 27-38."},{"key":"key2020121904432713400_b9","doi-asserted-by":"crossref","unstructured":"Nuray, R.\n                and \n                  Can, F.\n                (2003), \u201cAutomatic ranking of retrieval systems in imperfect environments\u201d, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Toronto, pp. 379-380.","DOI":"10.1145\/860435.860510"},{"key":"key2020121904432713400_b10","unstructured":"Rajagopal, P.\n               , \n                  Ravana, S.D.\n                and \n                  Ismail, M.A.\n                (2014), \u201cRelevance judgments exclusive of human assessors in large scale information retrieval evaluation experimentation\u201d, \n                  Malaysian Journal of Computer Science\n               , Vol. 27 No. 2, pp. 80-94."},{"key":"key2020121904432713400_b11","unstructured":"Rasmussen, E.\n                (2002), \u201cEvaluation in information retrieval\u201d, \n                  The MIR\/MDL Evaluation Project White Paper Collection\n               , 3rd ed., ACM, Toronto, pp. 45-49."},{"key":"key2020121904432713400_b12","unstructured":"Ravana, S.D.\n                (2011), \u201cExperimental evaluation of information retrieval systems\u201d, unpublished manuscript, Computing and Information Systems \u2013 Theses, The University of Melbourne, Melbourne."},{"key":"key2020121904432713400_b13","unstructured":"Saeid, M.\n               , \n                  Abd Ghani, A.A.\n                and \n                  Selamat, H.\n                (2011), \u201cRank-order weighting of web attributes for website evaluation\u201d, \n                  The International Arab Journal of Information Technology\n               , Vol. 8 No. 1, pp. 30-38."},{"key":"key2020121904432713400_b14","doi-asserted-by":"crossref","unstructured":"Scholer, F.\n               , \n                  Turpin, A.\n                and \n                  Sanderson, M.\n                (2011), \u201cQuantifying test collection quality based on the consistency of relevance judgements\u201d, Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information \u2013 SIGIR, ACM, Beijing, pp. 1063-1072.","DOI":"10.1145\/2009916.2010057"},{"key":"key2020121904432713400_b15","doi-asserted-by":"crossref","unstructured":"Smucker, M.\n                and \n                  Jethani, C.\n                (2012), \u201cTime to judge relevance as an indicator of assessor error\u201d, Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Portland, OR, pp. 1153-1154.","DOI":"10.1145\/2348283.2348515"},{"key":"key2020121904432713400_b16","doi-asserted-by":"crossref","unstructured":"Soboroff, I.\n               , \n                  Nicholas, C.\n                and \n                  Cahan, P.\n                (2001), \u201cRanking retrieval systems without relevance judgments\u201d, Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, New Orleans, LA, pp. 66-73.","DOI":"10.1145\/383952.383961"},{"key":"key2020121904432713400_b17","doi-asserted-by":"crossref","unstructured":"Turpin, A.\n               , \n                  Scholer, F.\n               , \n                  Jarvelin, K.\n               , \n                  Wu, M.\n                and \n                  Culpepper, J.\n                (2009), \u201cIncluding summaries in system evaluation\u201d, Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Boston, MA, pp. 508-515.","DOI":"10.1145\/1571941.1572029"},{"key":"key2020121904432713400_b18","unstructured":"Varathan, K.D.\n               , \n                  Sembok, T.M.T.\n               , \n                  Kadir, R.A.\n                and \n                  Omar, N.\n                (2014), \u201cSemantic indexing for question answering system\u201d, \n                  Malaysian Journal of Computer Science\n               , Vol. 27 No. 4, pp. 261-274."},{"key":"key2020121904432713400_b19","doi-asserted-by":"crossref","unstructured":"Voorhees, E.M.\n                (2000), \u201cVariations in relevance judgments and the measurement of retrieval effectiveness\u201d, \n                  Information processing & management\n               , Vol. 36 No. 5, pp. 697-716.","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"key2020121904432713400_b20","doi-asserted-by":"crossref","unstructured":"Vorhees, E.M.\n                (2005), \u201cOverview of TREC 2005\u201d, NIST Special Publication 500-266: The Fourteenth Text REtrieval Conference Proceedings (TREC), November 15-18, Gaithersburg, MD.","DOI":"10.6028\/NIST.SP.500-266"},{"key":"key2020121904432713400_b21","doi-asserted-by":"crossref","unstructured":"Webber, W.\n               , \n                  Chandar, P.\n                and \n                  Carterette, B.\n                (2012), \u201cAlternative assessor disagreement and retrieval depth\u201d, Proceedings of the 21st ACM International Conference on Information and Knowledge Management, ACM, Maui, HI, pp. 125-134.","DOI":"10.1145\/2396761.2396781"},{"key":"key2020121904432713400_b22","doi-asserted-by":"crossref","unstructured":"Yilmaz, E.\n                and \n                  Aslam, J.\n                (2006), \u201cEstimating average precision with incomplete and imperfect judgments\u201d, Proceedings of the 15th ACM International Conference on Information and Knowledge Management \u2013 CIKM, ACM, Arlington, VA, pp. 102-111.","DOI":"10.1145\/1183614.1183633"},{"key":"key2020121904432713400_b23","doi-asserted-by":"crossref","unstructured":"Zobel, J.\n                (1998), \u201cHow reliable are the results of large-scale information retrieval experiments?\u201d, Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval \u2013 SIGIR, ACM, Melbourne, pp. 307-314.","DOI":"10.1145\/290941.291014"}],"container-title":["Aslib Journal of Information Management"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/AJIM-03-2015-0046","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/AJIM-03-2015-0046\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/AJIM-03-2015-0046\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:00:26Z","timestamp":1753398026000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/ajim\/article\/67\/6\/700-714\/60559"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,11,16]]},"references-count":23,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2015,11,16]]}},"alternative-id":["10.1108\/AJIM-03-2015-0046"],"URL":"https:\/\/doi.org\/10.1108\/ajim-03-2015-0046","relation":{},"ISSN":["2050-3806"],"issn-type":[{"type":"print","value":"2050-3806"}],"subject":[],"published":{"date-parts":[[2015,11,16]]}}}