{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:52:29Z","timestamp":1750308749608,"version":"3.41.0"},"reference-count":95,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2009,11,1]],"date-time":"2009-11-01T00:00:00Z","timestamp":1257033600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100002920","name":"Research Grants Council, University Grants Committee, Hong Kong","doi-asserted-by":"publisher","award":["PolyU 5226\/05E"],"award-info":[{"award-number":["PolyU 5226\/05E"]}],"id":[{"id":"10.13039\/501100002920","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2009,11]]},"abstract":"<jats:p>\n            A new principles framework is presented for retrieval evaluation of ranked outputs. It applies decision theory to model relevance decision preferences and shows that the Probability Ranking Principle (PRP) specifies optimal ranking. It has two new components, namely a probabilistic evaluation model and a general measure of retrieval effectiveness. Its probabilities may be interpreted as subjective or objective ones. Its performance measure is the expected weighted rank which is the weighted average rank of a retrieval list. Starting from this measure, the expected forward rank and some existing retrieval effectiveness measures (e.g., top\n            <jats:italic>n<\/jats:italic>\n            precision and discounted cumulative gain) are instantiated using suitable weighting schemes after making certain assumptions. The significance of these instantiations is that the ranking prescribed by PRP is shown to be optimal simultaneously for all these existing performance measures. In addition, the optimal expected weighted rank may be used to normalize the expected weighted rank of retrieval systems for (summary) performance comparison (across different topics) between systems. The framework also extends PRP and our evaluation model to handle graded relevance, thereby generalizing the discussed, existing measures (e.g., top\n            <jats:italic>n<\/jats:italic>\n            precision) and probabilistic retrieval models for graded relevance.\n          <\/jats:p>","DOI":"10.1145\/1629096.1629098","type":"journal-article","created":{"date-parts":[[2009,11,30]],"date-time":"2009-11-30T14:56:36Z","timestamp":1259592996000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Building a framework for the probability ranking principle by a family of expected weighted rank"],"prefix":"10.1145","volume":"27","author":[{"given":"Edward Kai Fung","family":"Dang","sequence":"first","affiliation":[{"name":"The Hong Kong Polytechnic University, Kowloon, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ho Chung","family":"Wu","sequence":"additional","affiliation":[{"name":"The Hong Kong Polytechnic University, Kowloon, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert Wing Pong","family":"Luk","sequence":"additional","affiliation":[{"name":"The Hong Kong Polytechnic University, Kowloon, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kam Fai","family":"Wong","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shatin, NT, Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2009,11,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582416"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2008.111"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1013230.511811"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/42005.42023"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026739"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630260107"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(92)90031-T"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277755"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009000"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390451"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(75)90001-1"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.5090190108"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133199"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148262"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/299917.299920"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148293"},{"volume-title":"Foresight: Its logical laws, its subjective sources. Translated from French in Studies in Subjective Probabilities, Kyburg","year":"1937","author":"De Finetti B.","key":"e_1_2_1_18_1"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1272516.1272529"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(88)90042-8"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0895480102412856"},{"key":"e_1_2_1_22_1","unstructured":"French S. 1982. Sequencing and Scheduling: An Introduction to the Mathematics of the Job-Shop. Ellis Horwood Ltd. Chichester UK.  French S. 1982. Sequencing and Scheduling: An Introduction to the Mathematics of the Job-Shop. Ellis Horwood Ltd. Chichester UK."},{"key":"e_1_2_1_23_1","unstructured":"French S. 1986. Decision Theory:An Introduction to the Mathematics of Rationality Ellis Horwood Chichester UK.   French S. 1986. Decision Theory:An Introduction to the Mathematics of Rationality Ellis Horwood Chichester UK."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/65943.65944"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/35.3.243"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9045-0"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(66)90006-4"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199112)42:10<703::AID-ASI3>3.0.CO;2-1"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199201)43:1<1::AID-ASI1>3.0.CO;2-5"},{"volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Harman D. K.","key":"e_1_2_1_30_1"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(73)90066-1"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1229179.1229181"},{"key":"e_1_2_1_34_1","unstructured":"Kando N. Kuriyams K. and Yoshioka M. 2001. Information retrieval system evaluation using multi-grade relevance judgments. Discussion on averageable single-numbered measures. Joho Shori Gakkai Kenkyu Hokoku 74 FI-63 105--112.  Kando N. Kuriyams K. and Yoshioka M. 2001. Information retrieval system evaluation using multi-grade relevance judgments. Discussion on averageable single-numbered measures. Joho Shori Gakkai Kenkyu Hokoku 74 FI-63 105--112."},{"key":"e_1_2_1_35_1","unstructured":"Kendall M. G. 1955. Rank Correlation Methods. Hafner Publishing New York.  Kendall M. G. 1955. Rank Correlation Methods. Hafner Publishing New York."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383970"},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Lafferty J. and Zhai C. 2003. Probabilistic relevance models based on document and query generation. In Language Modeling for Information Retrieval. W. B. Croft and J. Lafferty Eds. Kluwer Academic 1--10.  Lafferty J. and Zhai C. 2003. Probabilistic relevance models based on document and query generation. In Language Modeling for Information Retrieval. W. B. Croft and J. Lafferty Eds. Kluwer Academic 1--10.","DOI":"10.1007\/978-94-017-0171-6_1"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/258525.258546"},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Lavrenko V. and Croft W. B. 2003. Relevance models in information retrieval. In Language Modeling for Information Retrieval. W. B. Croft and J. Lafferty Eds. Kluwer Academic 11--56.  Lavrenko V. and Croft W. B. 2003. Relevance models in information retrieval. In Language Modeling for Information Retrieval. W. B. Croft and J. Lafferty Eds. Kluwer Academic 11--56.","DOI":"10.1007\/978-94-017-0171-6_2"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(94)90071-X"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9062-z"},{"volume-title":"Colloq. Math.","author":"Marczewski E.","key":"e_1_2_1_42_1"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/321033.321035"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076115"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277806"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183631"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v57:2"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.5090190406"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291008"},{"volume-title":"The Foundations of Mathematics and Other Logical Essays R","author":"Ramsey F. P.","key":"e_1_2_1_51_1","doi-asserted-by":"crossref","DOI":"10.4324\/9781315887814"},{"key":"e_1_2_1_52_1","unstructured":"Roberts F. S. 1979. Measurement Theory. Addison-Wesley Reading MA.  Roberts F. S. 1979. Measurement Theory. Addison-Wesley Reading MA."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026462"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026466"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026647"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-005-5665-9"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390453"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026654"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630270302"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-007-9025-9"},{"volume-title":"The SMART Retrieval System\u2014Experiments in Automatic Document Processing","author":"Rocchio J. J.","key":"e_1_2_1_61_1"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.2964\/jsik_KJ00002217135"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/860435.860529"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277756"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2006.07.020"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/361219.361220"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(90)90050-C"},{"key":"e_1_2_1_68_1","unstructured":"Shafer G. 1976. A Mathematical Theory of Evidence. Princeton University Press Princeton NJ.  Shafer G. 1976. A Mathematical Theory of Evidence. Princeton University Press Princeton NJ."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb01338.x"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/1328964.1328965"},{"volume-title":"Psychophysics: Introduction to Its Perceptual, Neural and Social Prospects","year":"1975","author":"Stevens S. S.","key":"e_1_2_1_71_1"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(77)90003-6"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630200110"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026584"},{"key":"e_1_2_1_75_1","unstructured":"van Rijsbergen C. J. 1979.Information Retrieval. Butterworths London.   van Rijsbergen C. J. 1979.Information Retrieval. Butterworths London."},{"volume-title":"Proceedings of the 8th Text Retrieval Conference. E. M. Voorhees and D. K. Harman, Eds. National Institute of Standards and Technology, 77--82","year":"1999","author":"Voorhees E.","key":"e_1_2_1_76_1"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383963"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(79)90030-X"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.iprm.2005.06.005"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133190"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9060-1"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390346"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(96)00028-3"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/279339.279340"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/253168.253205"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4379(89)90024-0"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2007.12.002"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277872"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/1361684.1361686"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199503)46:2%3C133::AID-ASI6%3E3.0.CO;2-Z"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/122860.122881"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183633"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0019-9958(65)90241-X"},{"volume":"4994","volume-title":"Proceedings of the 17th International Symposium on Foundations Intelligent Systems. J. G. Carbonell and J. Siekmann, Eds. Lecture Notes in Computer Science","author":"Zhou B.","key":"e_1_2_1_94_1"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03204312"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1629096.1629098","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1629096.1629098","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:22:19Z","timestamp":1750278139000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1629096.1629098"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,11]]},"references-count":95,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2009,11]]}},"alternative-id":["10.1145\/1629096.1629098"],"URL":"https:\/\/doi.org\/10.1145\/1629096.1629098","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2009,11]]},"assertion":[{"value":"2007-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2009-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2009-11-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}