{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T09:48:49Z","timestamp":1791280129158,"version":"4.1.0"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2011,4,1]],"date-time":"2011-04-01T00:00:00Z","timestamp":1301616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000145","name":"Division of Information and Intelligent Systems","doi-asserted-by":"publisher","award":["IIS-0347933IIS-0713581"],"award-info":[{"award-number":["IIS-0347933IIS-0713581"]}],"id":[{"id":"10.13039\/100000145","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2011,4]]},"abstract":"<jats:p>\n            Developing effective retrieval models is a long-standing central challenge in information retrieval research. In order to develop more effective models, it is necessary to understand the deficiencies of the current retrieval models and the relative strengths of each of them. In this article, we propose a general methodology to\n            <jats:italic>analytically<\/jats:italic>\n            and\n            <jats:italic>experimentally<\/jats:italic>\n            diagnose the weaknesses of a retrieval function, which provides guidance on how to further improve its performance. Our methodology is motivated by the empirical observation that good retrieval performance is closely related to the use of various retrieval heuristics. We connect the weaknesses and strengths of a retrieval function with its implementations of these retrieval heuristics, and propose two strategies to check how well a retrieval function implements the desired retrieval heuristics. The first strategy is to formalize heuristics as constraints, and use constraint analysis to analytically check the implementation of retrieval heuristics. The second strategy is to define a set of relevance-preserving perturbations and perform diagnostic tests to empirically evaluate how well a retrieval function implements retrieval heuristics. Experiments show that both strategies are effective to identify the potential problems in implementations of the retrieval heuristics. The performance of retrieval functions can be improved after we fix these problems.\n          <\/jats:p>","DOI":"10.1145\/1961209.1961210","type":"journal-article","created":{"date-parts":[[2011,5,3]],"date-time":"2011-05-03T12:48:53Z","timestamp":1304426933000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":72,"title":["Diagnostic Evaluation of Information Retrieval Models"],"prefix":"10.1145","volume":"29","author":[{"given":"Hui","family":"Fang","sequence":"first","affiliation":[{"name":"University of Delaware"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tao","family":"Tao","sequence":"additional","affiliation":[{"name":"Microsoft Corporation"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chengxiang","family":"Zhai","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2011,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582416"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1099554.1099723"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291009"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the 46th Annual Meetings of the Association for Computational Linguistics.","author":"Fang H.","year":"2008","unstructured":"Fang , H. 2008 . A re-examination of query expansion using lexical resources . In Proceedings of the 46th Annual Meetings of the Association for Computational Linguistics. Fang, H. 2008. A re-examination of query expansion using lexical resources. In Proceedings of the 46th Annual Meetings of the Association for Computational Linguistics."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009004"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076116"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148193"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/35.3.243"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the Language Modeling and IR Workshop. 6--11","author":"Fuhr N.","year":"2001","unstructured":"Fuhr , N. 2001 . Language models and uncertain inference in information retrieval . In Proceedings of the Language Modeling and IR Workshop. 6--11 . Fuhr, N. 2001. Language models and uncertain inference in information retrieval. In Proceedings of the Language Modeling and IR Workshop. 6--11."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1041394.1041404"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076114"},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","unstructured":"Hiemstra D. 2000. A probabilistic justification for using tf-idf term wieghting in information retrieval. Int. J. Digital Libraries 131--139.  Hiemstra D. 2000. A probabilistic justification for using tf-idf term wieghting in information retrieval. Int. J. Digital Libraries 131--139.","DOI":"10.1007\/s007999900025"},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","unstructured":"Lafferty J. and Zhai C. 2003. Probabilistic relevance models based on document and query generation. In Language Modeling and Information Retrieval W. B. Croft and J. Lafferty Eds. Kluwer Academic Publishers.  Lafferty J. and Zhai C. 2003. Probabilistic relevance models based on document and query generation. In Language Modeling and Information Retrieval W. B. Croft and J. Lafferty Eds. Kluwer Academic Publishers.","DOI":"10.1007\/978-94-017-0171-6_1"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383972"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the Symposium on Document Analysis and Information Retrieval.","author":"Lopresti D.","unstructured":"Lopresti , D. and Zhou , J . 1996. Retrieval strategy for noisy text . In Proceedings of the Symposium on Document Analysis and Information Retrieval. Lopresti, D. and Zhou, J. 1996. Retrieval strategy for noisy text. In Proceedings of the Symposium on Document Analysis and Information Retrieval."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291008"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630270302"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 232--241","author":"Robertson S.","unstructured":"Robertson , S. and Walker , S . 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval . In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 232--241 . Robertson, S. and Walker, S. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval. 232--241."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/258525.258529"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 3rd Text REtrieval Conference (TREC-3). D. K. Harman Ed., 109--126","author":"Robertson S. E.","unstructured":"Robertson , S. E. , Walker , S. , Jones , S., M. Hancock-Beaulieu , M. , and Gatford , M . 1995. Okapi at TREC-3 . In Proceedings of the 3rd Text REtrieval Conference (TREC-3). D. K. Harman Ed., 109--126 . Robertson, S. E., Walker, S., Jones, S., M.Hancock-Beaulieu, M., and Gatford, M. 1995. Okapi at TREC-3. In Proceedings of the 3rd Text REtrieval Conference (TREC-3). D. K. Harman Ed., 109--126."},{"key":"e_1_2_1_22_1","volume-title":"Automatic Text Processing: The Transformation, Analysis and Retrieval of Information by Computer","author":"Salton G.","unstructured":"Salton , G. 1989. Automatic Text Processing: The Transformation, Analysis and Retrieval of Information by Computer . Addison-Wesley . Salton, G. 1989. Automatic Text Processing: The Transformation, Analysis and Retrieval of Information by Computer. Addison-Wesley."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(88)90021-0"},{"key":"e_1_2_1_24_1","volume-title":"Introduction to Modern Information Retrieval","author":"Salton G.","unstructured":"Salton , G. and McGill , M. 1983. Introduction to Modern Information Retrieval . McGraw-Hill . Salton, G. and McGill, M. 1983. Introduction to Modern Information Retrieval. McGraw-Hill."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630260106"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009001"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076117"},{"key":"e_1_2_1_28_1","first-page":"35","article-title":"Modern information retrieval: A brief overview","volume":"24","author":"Singhal A.","year":"2001","unstructured":"Singhal , A. 2001 . Modern information retrieval: A brief overview . Bull. IEEE Comput. Soc. Techn. Committee Data Engin. 24 , 4, 35 -- 43 . Singhal, A. 2001. Modern information retrieval: A brief overview. Bull. IEEE Comput. Soc. Techn. Committee Data Engin. 24, 4, 35--43.","journal-title":"Bull. IEEE Comput. Soc. Techn. Committee Data Engin."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/243199.243206"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the Symposium on Document Analysis and Information Retrieval. 149--162","author":"Singhal A.","unstructured":"Singhal , A. , Salton , G. , and Buckley , C . 1996b. Length normalization in degraded text collections . In Proceedings of the Symposium on Document Analysis and Information Retrieval. 149--162 . Singhal, A., Salton, G., and Buckley, C. 1996b. Length normalization in degraded text collections. In Proceedings of the Symposium on Document Analysis and Information Retrieval. 149--162."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the Text REtrieval Conference. 186--198","author":"Singhal A.","unstructured":"Singhal , A. , Choi , J. , Hindle , D. , Lewis , D. D. , and Pereira , F. C. N. 1998. ATT at TREC-7 . In Proceedings of the Text REtrieval Conference. 186--198 . Singhal, A., Choi, J., Hindle, D., Lewis, D. D., and Pereira, F. C. N. 1998. ATT at TREC-7. In Proceedings of the Text REtrieval Conference. 186--198."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383961"},{"key":"e_1_2_1_33_1","volume-title":"Eds","author":"Sparck Jones K.","year":"1997","unstructured":"Sparck Jones , K. and Willett , P. , Eds . 1997 . Readings in Information Retrieval. Morgan Kaufmann Publishers . Sparck Jones, K. and Willett, P., Eds. 1997. Readings in Information Retrieval. Morgan Kaufmann Publishers."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277794"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/125187.125188"},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","unstructured":"van Rijbergen C. J. 1977. A theoretical basis for theuse of co-occurrence data in information retrieval. J. Document. 106--119.  van Rijbergen C. J. 1977. A theoretical basis for theuse of co-occurrence data in information retrieval. J. Document. 106--119.","DOI":"10.1108\/eb026637"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/29.6.481"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297797.1297822"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/502795.502796"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/195705.195713"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/502585.502654"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.384019"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183696"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/281250.281256"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1961209.1961210","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1961209.1961210","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T10:59:50Z","timestamp":1750244390000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1961209.1961210"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,4]]},"references-count":45,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2011,4]]}},"alternative-id":["10.1145\/1961209.1961210"],"URL":"https:\/\/doi.org\/10.1145\/1961209.1961210","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2011,4]]},"assertion":[{"value":"2007-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2010-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-04-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}