{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,20]],"date-time":"2025-10-20T16:31:28Z","timestamp":1760977888312,"version":"3.41.0"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2001,10,1]],"date-time":"2001-10-01T00:00:00Z","timestamp":1001894400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2001,10]]},"abstract":"<jats:p>Experimental approaches are widely employed to benchmark the performance of an information retrieval (IR) system. Measurements in terms of recall and precision are computed as performance indicators. Although they are good at assessing the retrieval effectiveness of an IR system, they fail to explore deeper aspects such as its underlying functionality and explain why the system shows such performance. Recently, inductive (i.e., theoretical) evaluation of IR systems has been proposed to circumvent the controversies of the experimental methods. Several studies have adopted the inductive approach, but they mostly focus on theoretical modeling of IR properties by using some metalogic. In this article, we propose to use inductive evaluation for functional benchmarking of IR models as a complement of the traditional experiment-based performance benchmarking. We define a functional benchmark suite in two stages: the evaluation criteria based on the notion of \"aboutness,\" and the formal evaluation methodology using the criteria. The proposed benchmark has been successfully applied to evaluate various well-known classical and logic-based IR models. The functional benchmarking results allow us to compare and analyze the functionality of the different IR models.<\/jats:p>","DOI":"10.1145\/502795.502796","type":"journal-article","created":{"date-parts":[[2002,7,27]],"date-time":"2002-07-27T11:29:00Z","timestamp":1027769340000},"page":"337-370","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Application of aboutness to functional benchmarking in information retrieval"],"prefix":"10.1145","volume":"19","author":[{"given":"Kam-Fai","family":"Wong","sequence":"first","affiliation":[{"name":"The Chinese University of Hong Kong, Shatin, N.T., Hong Kong"}]},{"given":"Dawei","family":"Song","sequence":"additional","affiliation":[{"name":"University of Queensland, Australia"}]},{"given":"Peter","family":"Bruza","sequence":"additional","affiliation":[{"name":"University of Queensland, Australia"}]},{"given":"Chun-Hung","family":"Cheng","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Shatin, N.T., Hong Kong"}]}],"member":"320","published-online":{"date-parts":[[2001,10]]},"reference":[{"volume-title":"Information Retrieval and Hypertext","author":"AGOSTI M.","key":"e_1_2_1_1_1","unstructured":"AGOSTI , M. AND SMEATON , A. F. , EDS. 1996. Information Retrieval and Hypertext . Kluwer , Hingham, Mass .]] AGOSTI, M. AND SMEATON, A. F., EDS. 1996. Information Retrieval and Hypertext. Kluwer, Hingham, Mass.]]"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the Second Workshop on Information Retrieval, Uncertainty and Logic (Glasgow).]]","author":"AMATI G.","year":"1996","unstructured":"AMATI , G. AND GEORGATOS , K. 1996 . Relevance as deduction: A logical view of information retrieval . In Proceedings of the Second Workshop on Information Retrieval, Uncertainty and Logic (Glasgow).]] AMATI, G. AND GEORGATOS, K. 1996. Relevance as deduction: A logical view of information retrieval. In Proceedings of the Second Workshop on Information Retrieval, Uncertainty and Logic (Glasgow).]]"},{"volume-title":"Modern Information Retrieval","author":"BAEZA-YATES R.","key":"e_1_2_1_3_1","unstructured":"BAEZA-YATES , R. AND RIBEIRO-NETO , B. 1999. Modern Information Retrieval . ACM Press and Addison-Wesley , New York .]] BAEZA-YATES, R. AND RIBEIRO-NETO, B. 1999. Modern Information Retrieval. ACM Press and Addison-Wesley, New York.]]"},{"key":"e_1_2_1_4_1","unstructured":"BARWISE J. 1989. The Situation in Logic.InCLSI Lecture Notes 17 Stanford Calif.]]  BARWISE J. 1989. The Situation in Logic.InCLSI Lecture Notes 17 Stanford Calif.]]"},{"key":"e_1_2_1_5_1","first-page":"33","volume-title":"Situation Theory and its Applications, R. Cooper, et al. Eds., CLSI Lecture Notes 1","author":"BARWISE J.","unstructured":"BARWISE , J. AND ETCHEMENDY , J. 1990. Information, Infons and Inference . In Situation Theory and its Applications, R. Cooper, et al. Eds., CLSI Lecture Notes 1 , 33 - 78 .]] BARWISE, J. AND ETCHEMENDY, J. 1990. Information, Infons and Inference. In Situation Theory and its Applications, R. Cooper, et al. Eds., CLSI Lecture Notes 1, 33-78.]]"},{"volume-title":"Information Flow-The Logic of Distributed Systems","author":"BARWISE J.","key":"e_1_2_1_6_1","unstructured":"BARWISE , J. AND SELIGMAN , J. 1997. Information Flow-The Logic of Distributed Systems . Cambridge University Press , Cambridge, MA .]] BARWISE,J.AND SELIGMAN, J. 1997. Information Flow-The Logic of Distributed Systems. Cambridge University Press, Cambridge, MA.]]"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00130692"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/188490.188521"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of WIRUL 96, the Second Workshop on Information Retrieval, Uncertainty and Logic (Glasgow).]]","author":"BRUZA P.D.","year":"1996","unstructured":"BRUZA , P.D. AND LALMAS , M. 1996 . Logic based information retrieval: Is it really worth it? In Proceedings of WIRUL 96, the Second Workshop on Information Retrieval, Uncertainty and Logic (Glasgow).]] BRUZA,P.D.AND LALMAS, M. 1996. Logic based information retrieval: Is it really worth it? In Proceedings of WIRUL 96, the Second Workshop on Information Retrieval, Uncertainty and Logic (Glasgow).]]"},{"volume-title":"Preferential models of query by navigation","author":"BRUZA P.D.","key":"e_1_2_1_10_1","unstructured":"BRUZA , P.D. AND VAN LINDER , B. 1998. Preferential models of query by navigation . In Information Retrieval, Logic and Uncertainty, F. Crestani, M. Lalmas, and C. J. van Rijsbergen, Eds. Springer- Verlag , New York .]] BRUZA,P.D.AND VAN LINDER, B. 1998. Preferential models of query by navigation. In Information Retrieval, Logic and Uncertainty, F. Crestani, M. Lalmas, and C. J. van Rijsbergen, Eds. Springer- Verlag, New York.]]"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/646129.678851"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/359203.359208"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the International Conference on Advances in Intelligent Systems: Theory and Applications (AISTA 2000)","author":"BRUZA P.D.","year":"2000","unstructured":"BRUZA , P.D. , SONG , D. , WONG , K.F. , AND CHENG , C. H. 2000 b. Commonsense aboutness for information retrieval . In Proceedings of the International Conference on Advances in Intelligent Systems: Theory and Applications (AISTA 2000) (Canberra, Australia, February 2-4), 317- 324.]] BRUZA,P.D.,SONG, D., WONG,K.F.,AND CHENG, C. H. 2000b. Commonsense aboutness for information retrieval. In Proceedings of the International Conference on Advances in Intelligent Systems: Theory and Applications (AISTA 2000) (Canberra, Australia, February 2-4), 317- 324.]]"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/312624.312696"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Fourth Text Retrieval Conference (TREC-4), 509-526","author":"CRESTANI F.","year":"1995","unstructured":"CRESTANI , F. , RUTHVEN , I. , SANDERSON , M. , AND VAN RIJSBERGEN , C. J. 1995 . The troubles with using a logical model of IR on a large collection of documents . In Proceedings of the Fourth Text Retrieval Conference (TREC-4), 509-526 .]] CRESTANI, F., RUTHVEN, I., SANDERSON, M., AND VAN RIJSBERGEN, C. J. 1995. The troubles with using a logical model of IR on a large collection of documents. In Proceedings of the Fourth Text Retrieval Conference (TREC-4), 509-526.]]"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026939"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/215206.215373"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/290159.290161"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/290159.290161"},{"key":"e_1_2_1_20_1","first-page":"62","volume-title":"Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI-97)","author":"DUBOIS D.","year":"1997","unstructured":"DUBOIS , D. , FARINAS DEL CERRO , L. , HERZIG , A. , AND PRADE , H. 1997 . Qualitative relevance and independence: A roadmap . In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI-97) , 62 - 67 .]] DUBOIS, D., FARINAS DEL CERRO, L., HERZIG, A., AND PRADE, H. 1997. Qualitative relevance and independence: A roadmap. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI-97), 62-67.]]"},{"volume-title":"Data Structures &amp","author":"FRAKES W.B.","key":"e_1_2_1_21_1","unstructured":"FRAKES , W.B. AND BAEZA-YATES , R. , EDS. 1992. Information Retrieval , Data Structures &amp ; Algorithms. Prentice-Hall , Englewood, Cliffs, N.J.]] FRAKES,W.B.AND BAEZA-YATES, R., EDS. 1992. Information Retrieval, Data Structures &amp; Algorithms. Prentice-Hall, Englewood, Cliffs, N.J.]]"},{"key":"e_1_2_1_23_1","volume-title":"Information Retrieval: New Systems and Current Research","volume":"2","author":"HUIBERS T.W.","year":"1994","unstructured":"HUIBERS , T.W. C. AND BRUZA , P. D. 1994 . Situations, a general framework for studying information retrieval . In Information Retrieval: New Systems and Current Research , vol. 2 , Taylor Graham, Ed.]] HUIBERS,T.W.C.AND BRUZA, P. D. 1994. Situations, a general framework for studying information retrieval. In Information Retrieval: New Systems and Current Research, vol. 2, Taylor Graham, Ed.]]"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/381984.381986"},{"key":"e_1_2_1_25_1","series-title":"Lecture Notes in Computer Science","volume-title":"Symbolic and Quantitative Approaches to Uncertainty","author":"HUNTER A.","unstructured":"HUNTER , A. 1995. Using default logic in information retrieval . In Symbolic and Quantitative Approaches to Uncertainty , Lecture Notes in Computer Science , vol. 946 , Springer-Verlag , New York , 235-242.]] HUNTER, A. 1995. Using default logic in information retrieval. In Symbolic and Quantitative Approaches to Uncertainty, Lecture Notes in Computer Science, vol. 946, Springer-Verlag, New York, 235-242.]]"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/850949.853619"},{"issue":"1","key":"e_1_2_1_27_1","first-page":"17","article-title":"On the problem of 'aboutness' in document analysis","volume":"1","author":"HUTCHINS W. J.","year":"1977","unstructured":"HUTCHINS , W. J. 1977 . On the problem of 'aboutness' in document analysis . J. Inf. 1 , 1 , 17 - 35 .]] HUTCHINS, W. J. 1977. On the problem of 'aboutness' in document analysis. J. Inf. 1, 1, 17-35.]]","journal-title":"J. Inf."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(90)90101-5"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(97)00041-1"},{"volume-title":"Applications of Uncertainty Formalisms","author":"LALMAS M.","key":"e_1_2_1_30_1","unstructured":"LALMAS , M. 1997. Information retrieval and Dempster-Shafer's theory of evidence . In Applications of Uncertainty Formalisms , A. Hunter and S. Parson, Eds., Chapter 8, Lecture Notes in Computer Science, Springer-Verlag , New York.]] LALMAS, M. 1997. Information retrieval and Dempster-Shafer's theory of evidence. In Applications of Uncertainty Formalisms, A. Hunter and S. Parson, Eds., Chapter 8, Lecture Notes in Computer Science, Springer-Verlag, New York.]]"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888998002124"},{"volume-title":"Towards a Theory of Information. The Status of Partial Objects in Semantics","author":"LANDMAN F. W.","key":"e_1_2_1_33_1","unstructured":"LANDMAN , F. W. 1986. Towards a Theory of Information. The Status of Partial Objects in Semantics . Foris , Dordrecht .]] LANDMAN, F. W. 1986. Towards a Theory of Information. The Status of Partial Objects in Semantics. Foris, Dordrecht.]]"},{"volume-title":"Text Retrieval and Filtering: Analytic Models of Performance","author":"LOSEE R. M.","key":"e_1_2_1_34_1","unstructured":"LOSEE R. M. 1998. Text Retrieval and Filtering: Analytic Models of Performance . Kluwer , Hinghamn, Mass .]] LOSEE R. M. 1998. Text Retrieval and Filtering: Analytic Models of Performance. Kluwer, Hinghamn, Mass.]]"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/249622.249628"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630280107"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133188"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(89)90019-8"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/38.8.643"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/308915.308916"},{"key":"e_1_2_1_41_1","unstructured":"VAN RIJSBERGEN C. J. 1993. The state of information retrieval: Logic and information. Comput. Bull. February.]]  VAN RIJSBERGEN C. J. 1993. The state of information retrieval: Logic and information. Comput. Bull. February.]]"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/29.6.481"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/253168.253208"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/75334.75344"},{"volume-title":"Information Retrieval","author":"VAN RIJSBERGEN C. J.","key":"e_1_2_1_45_1","unstructured":"VAN RIJSBERGEN , C. J. 1979. Information Retrieval , Second edition. Butterworths , London .]] VAN RIJSBERGEN, C. J. 1979. Information Retrieval, Second edition. Butterworths, London.]]"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/225878.225884"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/243199.243268"},{"volume-title":"Automatic Text Processing","author":"SALTON G.","key":"e_1_2_1_48_1","unstructured":"SALTON , G. 1988. Automatic Text Processing . Addison-Wesley , Reading, Mass .]] SALTON, G. 1988. Automatic Text Processing. Addison-Wesley, Reading, Mass.]]"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(97)00055-1"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/243199.243206"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.384017"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.5555\/646813.707508"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the Fourth World Multiconference on Systemics, Cybernetics and Informatics (SCI 2000)","author":"SONG D.","year":"2000","unstructured":"SONG , D. , WONG , K.F. , BRUZA , P.D. , AND CHENG , C. H. 2000 b. Towards a commonsense aboutness theory for information retrieval modeling . In Proceedings of the Fourth World Multiconference on Systemics, Cybernetics and Informatics (SCI 2000) (Orlando, Fl., July) 23-26.]] SONG, D., WONG,K.F.,BRUZA,P.D.,AND CHENG, C. H. 2000b. Towards a commonsense aboutness theory for information retrieval modeling. In Proceedings of the Fourth World Multiconference on Systemics, Cybernetics and Informatics (SCI 2000) (Orlando, Fl., July) 23-26.]]"},{"key":"e_1_2_1_55_1","first-page":"389","volume-title":"Proceedings of the Twelfth International Florida Artificial Intelligence Society Conference (FLAIRS'99)","author":"SONG D.","year":"1999","unstructured":"SONG , D. , WONG , K.F. , BRUZA , P.D. , AND CHENG , C. H. 1999 . Towards functional benchmarking of information retrieval models . In Proceedings of the Twelfth International Florida Artificial Intelligence Society Conference (FLAIRS'99) (Orlando, Fl., May 3-5) , 389 - 393 .]] SONG, D., WONG,K.F.,BRUZA,P.D.,AND CHENG, C. H. 1999. Towards functional benchmarking of information retrieval models. In Proceedings of the Twelfth International Florida Artificial Intelligence Society Conference (FLAIRS'99) (Orlando, Fl., May 3-5), 389-393.]]"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/35.3.279"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/243199.243202"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/502795.502796","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/502795.502796","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T18:55:43Z","timestamp":1750272943000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/502795.502796"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2001,10]]},"references-count":54,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2001,10]]}},"alternative-id":["10.1145\/502795.502796"],"URL":"https:\/\/doi.org\/10.1145\/502795.502796","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2001,10]]},"assertion":[{"value":"2001-10-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}