{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T07:48:29Z","timestamp":1769154509119,"version":"3.49.0"},"reference-count":97,"publisher":"Emerald","issue":"3","license":[{"start":{"date-parts":[[2020,6,10]],"date-time":"2020-06-10T00:00:00Z","timestamp":1591747200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["EL"],"published-print":{"date-parts":[[2020,6,10]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title><jats:p>The purpose of this paper is to present a review on the use of the recall metric for evaluating information retrieval systems, especially search engines.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title><jats:p>This paper investigates different researchers\u2019 views about recall metrics.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Findings<\/jats:title><jats:p>Five different definitions for recall were identified. For the first group, recall refers to completeness, but it does not specify where all the relevant documents are located. For the second group, recall refers to retrieving all the relevant documents from the collection. However, it seems that the term \u201ccollection\u201d is ambiguous. For the third group (first approach), collection means the index of search engines and, for the fourth group (second approach), collection refers to the Web. For the fifth group (third approach), ranking of the retrieved documents should also be accounted for in calculating recall.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Practical implications<\/jats:title><jats:p>It can be said that in the first, second and third approaches, the components of the retrieval algorithm, the retrieval algorithm and crawler, and the retrieval algorithm and crawler and ranker, respectively, are evaluated. To determine the effectiveness of search engines for the use of users, it is better to use the third approach in recall measurement.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title><jats:p>The value of this paper is to collect, identify and analyse literature that is used in recall. In addition, different views of researchers about recall are identified.<\/jats:p><\/jats:sec>","DOI":"10.1108\/el-12-2019-0287","type":"journal-article","created":{"date-parts":[[2020,6,10]],"date-time":"2020-06-10T05:55:35Z","timestamp":1591768535000},"page":"477-492","source":"Crossref","is-referenced-by-count":5,"title":["Three approaches to measuring recall on the Web: a systematic review"],"prefix":"10.1108","volume":"38","author":[{"given":"Mahdi","family":"Zeynali Tazehkandi","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohsen","family":"Nowkarizi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"issue":"6","key":"key2020090209440728000_ref001","doi-asserted-by":"crossref","first-page":"835","DOI":"10.1016\/j.compeleceng.2011.10.005","article-title":"An overview of web search evaluation methods","volume":"37","year":"2011","journal-title":"Computers and Electrical Engineering"},{"issue":"3","key":"key2020090209440728000_ref002","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1007\/s10115-011-0391-7","article-title":"How many performance measures to evaluate information retrieval systems?","volume":"30","year":"2012","journal-title":"Knowledge and Information Systems"},{"issue":"2","key":"key2020090209440728000_ref003","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/BF02458356","article-title":"On the overlap, the precision and estimated recall of search engines: a case study of the query \u2018Erdos\u2019","volume":"42","year":"1998","journal-title":"Scientometrics"},{"key":"key2020090209440728000_ref004","volume-title":"Text Information Retrieval Systems","year":"2007"},{"issue":"9","key":"key2020090209440728000_ref005","doi-asserted-by":"crossref","first-page":"1879","DOI":"10.1002\/asi.22675","article-title":"Ranking, relevance judgment, and precision of information retrieval on children\u2019s queries: evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and Ask Kids","volume":"63","year":"2012","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"key2020090209440728000_ref006","first-page":"93","article-title":"Information retrieval effectiveness of Turkish search engines","volume-title":"International Conference on Advances in Information Systems","year":"2002"},{"key":"key2020090209440728000_ref007","first-page":"122","article-title":"The normalized recall and related measures","volume-title":"ACM SIGIR Forum","year":"1983"},{"key":"key2020090209440728000_ref008","volume-title":"Information and Society","year":"2017"},{"issue":"1","key":"key2020090209440728000_ref009","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1002\/(SICI)1097-4571(199401)45:1<12::AID-ASI2>3.0.CO;2-L","article-title":"The relationship between recall and precision","volume":"45","year":"1994","journal-title":"Journal of the American Society for Information Science"},{"key":"key2020090209440728000_ref010","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1145\/1008992.1009000","article-title":"Retrieval evaluation with incomplete information","volume-title":"Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","year":"2004"},{"key":"key2020090209440728000_ref011","volume-title":"Knowledge and Knowing in Library and Information Science: A Philosophical Framework","year":"2001"},{"issue":"3","key":"key2020090209440728000_ref012","first-page":"447","article-title":"Relevance: language, semantics, philosophy","volume":"52","year":"2004","journal-title":"Library Trends"},{"issue":"1","key":"key2020090209440728000_ref013","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1002\/aris.1440370109","article-title":"The concept of information","volume":"37","year":"2003","journal-title":"Annual Review of Information Science and Technology"},{"key":"key2020090209440728000_ref014","first-page":"127","article-title":"Search engines for the world wide web: a comparative study and evaluation methodology","volume-title":"Proceedings of the Annual Meeting of the American Society for Information Science","year":"1996"},{"issue":"7","key":"key2020090209440728000_ref015","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1108\/eb051463","article-title":"Estimating the recall performance of web search engines","volume":"49","year":"1997","journal-title":"ASLIB Proceedings"},{"issue":"1","key":"key2020090209440728000_ref016","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1108\/eb026487","article-title":"Evaluation tests of information retrieval systems","volume":"26","year":"1970","journal-title":"Journal of Documentation"},{"issue":"2","key":"key2020090209440728000_ref017","article-title":"Evaluating the performance of information retrieval systems using test collections","volume":"18","year":"2013","journal-title":"Information Research"},{"issue":"1","key":"key2020090209440728000_ref018","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/0020-0271(71)90024-6","article-title":"A definition of relevance for information retrieval","volume":"7","year":"1971","journal-title":"Information Storage and Retrieval"},{"key":"key2020090209440728000_ref019","volume-title":"Search Engines: Information Retrieval in Practice","year":"2010"},{"issue":"5","key":"key2020090209440728000_ref020","doi-asserted-by":"crossref","first-page":"757","DOI":"10.1108\/14684521011084609","article-title":"Performance evaluation and comparison of the five most used search engines in retrieving web resources","volume":"34","year":"2010","journal-title":"Online Information Review"},{"key":"key2020090209440728000_ref021","first-page":"1","article-title":"An evaluation of popular search engines on finding turkish document","volume-title":"Second International Conference on Internet and Web Applications and Services (ICIW \u201807)","year":"2007"},{"issue":"2","key":"key2020090209440728000_ref022","doi-asserted-by":"crossref","first-page":"302","DOI":"10.1108\/EL-03-2015-0035","article-title":"Effectiveness of Google keyword suggestion on users\u2019 relevance judgment","volume":"34","year":"2016","journal-title":"The Electronic Library"},{"issue":"6","key":"key2020090209440728000_ref023","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1177\/016555159802400604","article-title":"Measuring recall","volume":"24","year":"1998","journal-title":"Journal of Information Science"},{"issue":"2","key":"key2020090209440728000_ref024","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/S0306-4573(98)00041-7","article-title":"Finding information on the world wide web: the retrieval effectiveness of search engines","volume":"35","year":"1999","journal-title":"Information Processing and Management"},{"key":"key2020090209440728000_ref025","article-title":"Union of sets","volume-title":"Encyclopedia of Mathematics","year":"2001","edition":"2nd ed."},{"issue":"1","key":"key2020090209440728000_ref026","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1108\/00220410510577970","article-title":"Library and information science and the philosophy of science","volume":"61","year":"2005","journal-title":"Journal of Documentation"},{"issue":"5","key":"key2020090209440728000_ref027","first-page":"217","article-title":"The foundation of the concept of relevance","volume":"61","year":"2010","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"1","key":"key2020090209440728000_ref028","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1108\/EL-07-2018-0142","article-title":"Retrieval effectiveness of image search engines","volume":"37","year":"2019","journal-title":"The Electronic Library"},{"issue":"2","key":"key2020090209440728000_ref029","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/S0306-4573(99)00056-4","article-title":"Real life, real users, and real needs: a study and analysis of user queries on the web","volume":"36","year":"2000","journal-title":"Information Processing and Management"},{"issue":"4","key":"key2020090209440728000_ref030","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1145\/582415.582418","article-title":"Cumulated gain-based evaluation of IR techniques","volume":"20","year":"2002","journal-title":"ACM Transactions on Information Systems (Tois)"},{"issue":"2","key":"key2020090209440728000_ref031","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1002\/asi.5090060209","article-title":"Operational criteria for designing information retrieval systems","volume":"6","year":"1955","journal-title":"American Documentation"},{"issue":"3","key":"key2020090209440728000_ref032","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1177\/014107680309600304","article-title":"Five steps to conducting a systematic review","volume":"96","year":"2003","journal-title":"Journal of the Royal Society of Medicine"},{"issue":"4","key":"key2020090209440728000_ref033","first-page":"39","article-title":"A comparative study of BYG search engines","volume":"2","year":"2013","journal-title":"American Journal of Engineering Research"},{"issue":"1","key":"key2020090209440728000_ref034","first-page":"124","article-title":"Precision and relative recall of search engines: a comparative study of Google and Yahoo","volume":"38","year":"2009","journal-title":"Singapore Journal of Library and Information Management"},{"issue":"2","key":"key2020090209440728000_ref035","first-page":"87","article-title":"Evaluating the searching capabilities of search engines and metasearch engines: a comparative study","volume":"57","year":"2010","journal-title":"Annals of Library and Information Studies"},{"key":"key2020090209440728000_ref036","article-title":"Information retrieval systems: Characteristics","volume-title":"Testing, and Evaluation","year":"1979","edition":"2nd ed."},{"issue":"9","key":"key2020090209440728000_ref037","doi-asserted-by":"crossref","first-page":"1763","DOI":"10.1002\/asi.23304","article-title":"Evaluating the retrieval effectiveness of web search engines using a representative query sample","volume":"66","year":"2015","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"key2020090209440728000_ref038","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1145\/1835449.1835551","article-title":"PRES: a score metric for evaluating recall-oriented information retrieval application","volume-title":"Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval","year":"2010"},{"issue":"6","key":"key2020090209440728000_ref039","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1002\/asi.10408","article-title":"Measuring retrieval effectiveness: a new proposal and a first experimental validation","volume":"55","year":"2004","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"1","key":"key2020090209440728000_ref040","first-page":"1","article-title":"Rethinking the recall measure in appraising information retrieval systems and providing a new measure by using Persian search engines","volume":"17","year":"2019","journal-title":"International Journal of Information Science and Management"},{"issue":"1","key":"key2020090209440728000_ref041","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1108\/00330330710724908","article-title":"Putting Google scholar to the test: a preliminary study","volume":"41","year":"2007","journal-title":"Program"},{"key":"key2020090209440728000_ref042","first-page":"26","article-title":"Why is relevance still the basic notion in information science","volume-title":"Proceedings of the 14th International Symposium on Information Science","year":"2015"},{"issue":"2","key":"key2020090209440728000_ref043","first-page":"42","article-title":"Precision and recall of five search engines for retrieval of scholarly information in the field of biotechnology","volume":"2","year":"2005","journal-title":"Webology"},{"key":"key2020090209440728000_ref044","unstructured":"Sirotkin, P. (2013), \u201cOn search engine evaluation metrics\u201d, Thesis, available at: https:\/\/docserv.uni-duesseldorf.de\/servlets\/DerivateServlet\/Derivate-25066\/On%20Search%20Engine%20Evaluation%20Metrics%20-%20final.pdf (accessed 12 November 2018)."},{"key":"key2020090209440728000_ref045","volume-title":"Web Search: Public Searching of the Web","year":"2006"},{"issue":"3","key":"key2020090209440728000_ref046","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<207::AID-ASI10>3.0.CO;2-1","article-title":"The relevance of recall and precision in user evaluation","volume":"45","year":"1994","journal-title":"Journal of the American Society for Information Science"},{"issue":"1","key":"key2020090209440728000_ref047","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<1::AID-ASI1>3.0.CO;2-3","article-title":"Some perspectives on the evaluation of information retrieval systems","volume":"47","year":"1996","journal-title":"Journal of the American Society for Information Science"},{"issue":"1","key":"key2020090209440728000_ref048","first-page":"21","article-title":"A comparative study of Google and Bing search engines in context of precision and relative recall parameter","volume":"4","year":"2012","journal-title":"International Journal on Computer Science and Engineering"},{"issue":"3","key":"key2020090209440728000_ref049","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1108\/EL-08-2014-0132","article-title":"Retrieval efficiency of select search engines vis-\u00e0-vis diverse open courseware formats","volume":"34","year":"2016","journal-title":"The Electronic Library"},{"issue":"12","key":"key2020090209440728000_ref050","doi-asserted-by":"crossref","first-page":"1073","DOI":"10.1002\/asi.1157","article-title":"Vox populi: the public searching of the web","volume":"52","year":"2001","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"2","key":"key2020090209440728000_ref051","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1002\/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z","article-title":"Measuring retrieval effectiveness based on user preference of documents","volume":"46","year":"1995","journal-title":"Journal of the American Society for Information Science"},{"key":"key2020090209440728000_ref052","doi-asserted-by":"publisher","DOI":"10.1108\/LHT-11-2019-0229","article-title":"Evaluating the effectiveness of Google, Parsijoo, Rismoon, and Yooz to retrieve Persian documents","year":"2020","journal-title":"Library Hi Tech"},{"issue":"3","key":"key2020090209440728000_ref053","doi-asserted-by":"crossref","first-page":"35","DOI":"10.5121\/ijcsit.2012.4304","article-title":"Evaluation of information retrieval systems","volume":"4","year":"2012","journal-title":"International Journal of Computer Science and Information Technology"},{"issue":"4","key":"key2020090209440728000_ref054","doi-asserted-by":"crossref","first-page":"515","DOI":"10.1108\/OIR-07-2015-0243","article-title":"Search engine effectiveness using query classification: a study","volume":"40","year":"2016","journal-title":"Online Information Review"},{"issue":"11","key":"key2020090209440728000_ref055","doi-asserted-by":"crossref","first-page":"2620","DOI":"10.1002\/asi.23958","article-title":"geNov: a new metric for measuring novelty and relevancy in biomedical information retrieval","volume":"68","year":"2017","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"2","key":"key2020090209440728000_ref056","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1504\/IJICBM.2016.074482","article-title":"Evaluation of information retrieval: precision and recall","volume":"12","year":"2016","journal-title":"International Journal of Indian Culture and Business Management"},{"issue":"2","key":"key2020090209440728000_ref057","first-page":"1015","article-title":"A survey on performance evaluation measures for information retrieval system","volume":"2","year":"2015","journal-title":"International Research Journal of Engineering and Technology"},{"key":"key2020090209440728000_ref058","first-page":"233","article-title":"Two axioms for evaluation measures in information retrieval","volume-title":"Proceedings of the 7th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","year":"1984"},{"key":"key2020090209440728000_ref059","first-page":"235","article-title":"Evaluating evaluation measure stability","volume-title":"ACM SIGIR Forum","year":"2017"},{"issue":"3","key":"key2020090209440728000_ref060","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1016\/S0306-4573(98)00065-X","article-title":"Discriminating meta-search: a framework for evaluation","volume":"35","year":"1999","journal-title":"Information Processing and Management"},{"issue":"3","key":"key2020090209440728000_ref061","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1108\/eb026538","article-title":"On the inverse relationship of recall and precision","volume":"28","year":"1972","journal-title":"Journal of Documentation"},{"key":"key2020090209440728000_ref062","volume-title":"Factors Determining the Performance of Indexing Systems","year":"1966"},{"issue":"1","key":"key2020090209440728000_ref063","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1002\/asi.5090190108","article-title":"Expected search length: a single measure of retrieval effectiveness based on the weak ordering action of retrieval systems","volume":"19","year":"1968","journal-title":"American Documentation"},{"issue":"2","key":"key2020090209440728000_ref064","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1002\/asi.4630240204","article-title":"On selecting a measure of retrieval effectiveness","volume":"24","year":"1973","journal-title":"Journal of the American Society for Information Science"},{"key":"key2020090209440728000_ref065","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1145\/1143844.1143874","article-title":"The relationship between precision-recall and ROC curves","volume-title":"Proceedings of the 23rd International Conference on Machine Learning","year":"2006"},{"issue":"2","key":"key2020090209440728000_ref066","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1108\/eb024608","article-title":"Search engines on the world wide web and information retrieval from the internet: a review and evaluation","volume":"21","year":"1997","journal-title":"Online and CD-ROM Review"},{"issue":"9","key":"key2020090209440728000_ref067","first-page":"1804","article-title":"Evaluation methods for web retrieval tasks considering hyperlink structure","volume":"86","year":"2003","journal-title":"IEICE Transactions on Information and Systems"},{"issue":"2-3","key":"key2020090209440728000_ref068","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/0306-4573(91)90046-O","article-title":"Determining the effectiveness of retrieval algorithms","volume":"27","year":"1991","journal-title":"Information Processing and Management"},{"issue":"3","key":"key2020090209440728000_ref069","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1002\/(SICI)1097-4571(198905)40:3<145::AID-ASI1>3.0.CO;2-I","article-title":"Recall\u2010precision trade\u2010off: a derivation","volume":"40","year":"1989","journal-title":"Journal of the American Society for Information Science"},{"key":"key2020090209440728000_ref070","unstructured":"Harper, D. (2000), \u201cOnline etymology dictionary\u201d, available at: www.etymonline.com (accessed 12 November 2018)."},{"issue":"1","key":"key2020090209440728000_ref071","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.0.CO;2-3","article-title":"Variations in relevance assessments and the measurement of retrieval effectiveness","volume":"47","year":"1996","journal-title":"Journal of the American Society for Information Science"},{"issue":"11-16","key":"key2020090209440728000_ref072","doi-asserted-by":"crossref","first-page":"1321","DOI":"10.1016\/S1389-1286(99)00024-9","article-title":"Results and challenges in web search evaluation","volume":"31","year":"1999","journal-title":"Computer Networks"},{"issue":"5837","key":"key2020090209440728000_ref073","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1126\/science.1126557","article-title":"Search technologies for the internet","volume":"317","year":"2007","journal-title":"Science"},{"issue":"5","key":"key2020090209440728000_ref074","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1108\/00242530610667567","article-title":"A comparative study on overlapping of search results in metasearch engines and their common underlying search engines","volume":"55","year":"2006","journal-title":"Library Review"},{"issue":"3","key":"key2020090209440728000_ref075","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1108\/EUM0000000007006","article-title":"Information retrieval techniques for evaluating search engines: a critical overview","volume":"52","year":"2000","journal-title":"ASLIB Proceedings"},{"key":"key2020090209440728000_ref076","volume-title":"An Introduction to Search Engines and Web Navigation","year":"2011"},{"key":"key2020090209440728000_ref077","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1108\/S1876-0562(2012)002012a009","article-title":"Chapter 7: personalised search engine evaluation: methodologies and metrics","volume-title":"Web Search Engine Research","year":"2012"},{"key":"key2020090209440728000_ref078","first-page":"578","article-title":"A comparative study of performance measures for information retrieval systems","volume-title":"Third International Conference on Information Technology: New Generations","year":"2006"},{"issue":"1","key":"key2020090209440728000_ref079","first-page":"2","article-title":"Rank-biased precision for measurement of retrieval effectiveness","volume":"27","year":"2008","journal-title":"ACM Transactions on Information Systems"},{"issue":"2","key":"key2020090209440728000_ref080","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1108\/00220410010803810","article-title":"The evaluation of WWW search engines","volume":"56","year":"2000","journal-title":"Journal of Documentation"},{"key":"key2020090209440728000_ref081","volume-title":"Concepts of Information Retrieval","year":"1989"},{"issue":"2","key":"key2020090209440728000_ref082","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1108\/EUM0000000006884","article-title":"The design of world wide web search engines: a critical review","volume":"31","year":"1997","journal-title":"Program"},{"issue":"3","key":"key2020090209440728000_ref083","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1145\/65943.65945","article-title":"A critical investigation of recall and precision as measures of retrieval system performance","volume":"7","year":"1989","journal-title":"ACM Transactions on Information Systems (Tois)"},{"issue":"2","key":"key2020090209440728000_ref084","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1108\/eb026466","article-title":"The parametric description of retrieval tests","volume":"25","year":"1969","journal-title":"Journal of Documentation"},{"issue":"4","key":"key2020090209440728000_ref085","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1177\/0165551507086989","article-title":"On the history of evaluation in IR","volume":"34","year":"2008","journal-title":"Journal of Information Science"},{"issue":"3","key":"key2020090209440728000_ref086","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1007\/s10791-007-9025-9","article-title":"On rank-based effectiveness measures and optimization","volume":"10","year":"2007","journal-title":"Information Retrieval"},{"key":"key2020090209440728000_ref087","volume-title":"Introduction to Modern Information Retrieval","year":"1983"},{"key":"key2020090209440728000_ref088","first-page":"138","article-title":"Evaluation of evaluation in information retrieval","volume-title":"Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","year":"1995"},{"issue":"2","key":"key2020090209440728000_ref089","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1108\/14684520310471716","article-title":"Think local, search global? Comparing search engines for searching geographically specific information","volume":"27","year":"2003","journal-title":"Online Information Review"},{"issue":"3","key":"key2020090209440728000_ref090","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1111\/j.1471-1842.2008.00822.x","article-title":"Beyond relevance and recall: testing new user\u2010centred measures of database performance","volume":"26","year":"2009","journal-title":"Health Information and Libraries Journal"},{"issue":"4","key":"key2020090209440728000_ref091","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1016\/0306-4573(92)90007-M","article-title":"Evaluation measures for interactive information retrieval","volume":"28","year":"1992","journal-title":"Information Processing and Management"},{"issue":"1\/2","key":"key2020090209440728000_ref092","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1108\/00012530310462661","article-title":"Information metrics and user studies","volume":"55","year":"2003","journal-title":"ASLIB Proceedings"},{"key":"key2020090209440728000_ref093","volume-title":"Automated Information Retrieval: Theory and Methods","year":"1997"},{"issue":"4","key":"key2020090209440728000_ref094","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1108\/eb026584","article-title":"Foundation of evaluation","volume":"30","year":"1974","journal-title":"Journal of Documentation"},{"issue":"4","key":"key2020090209440728000_ref095","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1016\/S0306-4573(03)00043-8","article-title":"New measurements for search engine evaluation proposed and tested","volume":"40","year":"2004","journal-title":"Information Processing and Management"},{"key":"key2020090209440728000_ref096","volume-title":"Techniques of Information Retrieval","year":"1970"},{"issue":"2","key":"key2020090209440728000_ref095a","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1002\/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z","article-title":"Measuring retrieval effectiveness based on user preference of documents","volume":"46","year":"1995","journal-title":"Journal of the American Society for Information Science"}],"container-title":["The Electronic Library"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/EL-12-2019-0287\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/EL-12-2019-0287\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T01:08:29Z","timestamp":1753405709000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/el\/article\/38\/3\/477-492\/90929"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,10]]},"references-count":97,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,6,10]]}},"alternative-id":["10.1108\/EL-12-2019-0287"],"URL":"https:\/\/doi.org\/10.1108\/el-12-2019-0287","relation":{},"ISSN":["0264-0473","0264-0473"],"issn-type":[{"value":"0264-0473","type":"print"},{"value":"0264-0473","type":"print"}],"subject":[],"published":{"date-parts":[[2020,6,10]]}}}