{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T00:50:51Z","timestamp":1774399851710,"version":"3.50.1"},"reference-count":60,"publisher":"Elsevier BV","issue":"5","license":[{"start":{"date-parts":[[1998,9,1]],"date-time":"1998-09-01T00:00:00Z","timestamp":904608000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.elsevier.com\/tdm\/userlicense\/1.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information Processing &amp; Management"],"published-print":{"date-parts":[[1998,9]]},"DOI":"10.1016\/s0306-4573(98)00025-9","type":"journal-article","created":{"date-parts":[[2002,7,25]],"date-time":"2002-07-25T15:07:29Z","timestamp":1027609649000},"page":"599-621","source":"Crossref","is-referenced-by-count":165,"title":["From highly relevant to not relevant: examining different regions of relevance 1This paper is a revised version of the conference paper: Spink, A. and Greisdorf, H. (1997) Partial relevance judgments during interactive information retrieval: An exploratory study. Proceedings of the 60th Annual Meeting of the American Society for Information Science, November 1\u20136, Washington, DC, pp. 323\u2013334. 1"],"prefix":"10.1016","volume":"34","author":[{"given":"Amanda","family":"Spink","sequence":"first","affiliation":[]},{"given":"Howard","family":"Greisdorf","sequence":"additional","affiliation":[]},{"given":"Judy","family":"Bateman","sequence":"additional","affiliation":[]}],"member":"78","reference":[{"key":"10.1016\/S0306-4573(98)00025-9_BIB1","unstructured":"Barnydt, G. C. (1964). A comparison of relevance assessments by three types of evaluators. Proceedings of the American Documentation Institute, October 5\u20138, 1964, pp. 383\u2013385. American Documentation Institute, Washington, DC"},{"issue":"3","key":"10.1016\/S0306-4573(98)00025-9_BIB2","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J","article-title":"User-defined relevance criteria: An exploratory study","volume":"45","author":"Barry","year":"1994","journal-title":"Journal of the American Society for Information Science"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB3","unstructured":"Bateman, J. (1997). Changes in users' relevance criteria: An information seeking study. Unpublished dissertation proposal. School of Library and Information Sciences, University of North Texas"},{"issue":"6","key":"10.1016\/S0306-4573(98)00025-9_BIB4","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1016\/S0306-4573(96)00039-8","article-title":"Document familiarity, relevance and Bradford's Law: The Getty Online Searching Project Report No. 5","volume":"32","author":"Bates","year":"1996","journal-title":"Information Processing and Management"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB5","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1108\/eb026722","article-title":"ASK for information retrieval: Pt 1. Background and theory","volume":"38","author":"Belkin","year":"1982","journal-title":"Journal of Documentation"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB6","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1108\/eb026726","article-title":"ASK for information retrieval: Part II. Results of a design study","volume":"38","author":"Belkin","year":"1982","journal-title":"Journal of Documentation"},{"issue":"3","key":"10.1016\/S0306-4573(98)00025-9_BIB7","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1108\/EUM0000000007198","article-title":"The development of a method for the evaluation of interactive information retrieval systems","volume":"53","author":"Borlund","year":"1997","journal-title":"Journal of Documentation"},{"issue":"2","key":"10.1016\/S0306-4573(98)00025-9_BIB8","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1002\/asi.4630240204","article-title":"On selecting a measure of retrieval effectiveness (Part 1)","volume":"24","author":"Cooper","year":"1973","journal-title":"Journal of the American Society for Information Science"},{"issue":"6","key":"10.1016\/S0306-4573(98)00025-9_BIB9","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1002\/asi.4630240603","article-title":"On selecting a measure of retrieval effectiveness (Part 2)","volume":"24","author":"Cooper","year":"1973","journal-title":"Journal of the American Society for Information Science"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB10","first-page":"3","article-title":"Information needs and uses","volume":"21","author":"Dervin","year":"1986","journal-title":"Annual Review of Information Science and Technology"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB11","unstructured":"Doyle, L. B. (1963). Is relevance an adequate criterion for retrieval system evaluation? In Automation and scientific communication, short papers, ed. H. P. Luhn, Pt. 2, pp. 199\u2013200. American Documentation Institute, Washington, DC"},{"issue":"4","key":"10.1016\/S0306-4573(98)00025-9_BIB12","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1016\/0306-4573(88)90042-8","article-title":"Measuring relevance judgments","volume":"24","author":"Eisenberg","year":"1988","journal-title":"Information Processing and Management"},{"issue":"5","key":"10.1016\/S0306-4573(98)00025-9_BIB13","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1002\/(SICI)1097-4571(198809)39:5<293::AID-ASI1>3.0.CO;2-I","article-title":"Order effects: A study of the possible influence of presentation order on user judgments of document relevance","volume":"39","author":"Eisenberg","year":"1988","journal-title":"Journal of the American Society for Information Science"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB14","unstructured":"Eisenberg, M. B., and Hu, X. (1987). Dichotomous relevance judgments and the evaluation of information systems. Proceedings of the Annual Meeting of the American Society for Information Science, Vol. 24, pp. 66\u201370"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB15","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1177\/016555158400800105","article-title":"Theory and explanation in information retrieval research","volume":"8","author":"Ellis","year":"1984","journal-title":"Journal of Information Science"},{"issue":"3","key":"10.1016\/S0306-4573(98)00025-9_BIB16","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1108\/eb026843","article-title":"A behavioral approach to information retrieval system design","volume":"45","author":"Ellis","year":"1989","journal-title":"Journal of Documentation"},{"issue":"1","key":"10.1016\/S0306-4573(98)00025-9_BIB17","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<23::AID-ASI3>3.0.CO;2-4","article-title":"The dilemma of measurement in information retrieval research","volume":"47","author":"Ellis","year":"1996","journal-title":"Journal of the American Society for Information Science"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB18","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1002\/asi.5090070408","article-title":"Seven years work in the organization of materials in a special library","volume":"7","author":"Gull","year":"1956","journal-title":"American Documentation"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB19","doi-asserted-by":"crossref","unstructured":"Harman, D. (1992). Relevance feedback revisited. Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, June 21\u201324, Copenhagen, Denmark, pp. 1\u201310","DOI":"10.1145\/133160.133167"},{"issue":"4","key":"10.1016\/S0306-4573(98)00025-9_BIB20","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1016\/0306-4573(93)90037-E","article-title":"The first text retrieval conference (TREC-1), Rockville, MD, USA, 4\u20136 November, 1992","volume":"29","author":"Harman","year":"1993","journal-title":"Information Processing and Management"},{"issue":"9","key":"10.1016\/S0306-4573(98)00025-9_BIB21","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1002\/(SICI)1097-4571(199210)43:9<602::AID-ASI3>3.0.CO;2-Q","article-title":"Psychological relevance and information science","volume":"43","author":"Harter","year":"1992","journal-title":"Journal of the American Society for Information Science"},{"issue":"1","key":"10.1016\/S0306-4573(98)00025-9_BIB22","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.0.CO;2-3","article-title":"Variations in relevance assessments and the measurement of retrieval effectiveness","volume":"47","author":"Harter","year":"1996","journal-title":"Journal of the American Society for Information Science"},{"issue":"1","key":"10.1016\/S0306-4573(98)00025-9_BIB23","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1108\/eb026960","article-title":"Cognitive perspectives of information retrieval interaction: Elements of a cognitive IR theory","volume":"52","author":"Ingwersen","year":"1996","journal-title":"Journal of Documentation"},{"issue":"2","key":"10.1016\/S0306-4573(98)00025-9_BIB24","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1086\/602442","article-title":"Relevance judgments of actual users and secondary judges: A comparative study","volume":"62","author":"Janes","year":"1992","journal-title":"Library Quarterly"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB25","unstructured":"Kuhlthau, C. C. (1993). Seeking meaning: A process approach to library and information science. Ablex Publishing, Norwood, NJ"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB26","unstructured":"Kuhlthau, C. C, Spink, A., and Cool, C. (1992). Exploration of stages in the information search process in online information retrieval: Communication between users and intermediaries. Proceedings of the 55th Annual Meeting of the American Society for Information Science, Vol. 29. October 1992. Pittsburgh, PA, pp. 67\u201371"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB27","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1002\/asi.4630360516","article-title":"Relevance?","volume":"36","author":"Meadow","year":"1985","journal-title":"Journal of the American Society for Information Science"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB28","unstructured":"Newby, G. B. (1992). An investigation of the role of navigation for information retrieval. Proceedings of the 55th Annual Meeting of the American Society for Information Science, Vol. 29, pp. 20\u201325"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB29","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1002\/asi.4630200405","article-title":"Some independent agreements and resolved disagreements about answer-providing documents","volume":"20","author":"O'Conner","year":"1969","journal-title":"American Documentation"},{"issue":"1","key":"10.1016\/S0306-4573(98)00025-9_BIB30","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/0306-4573(93)90026-A","article-title":"Term and citation retrieval: A field study","volume":"29","author":"Pao","year":"1993","journal-title":"Information Processing and Management"},{"issue":"3","key":"10.1016\/S0306-4573(98)00025-9_BIB31","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1086\/602592","article-title":"The nature of relevance in information retrieval: An empirical study","volume":"63","author":"Park","year":"1993","journal-title":"Library Quarterly"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB32","first-page":"63","article-title":"Evaluation of information systems and services","volume":"2","author":"Rees","year":"1967","journal-title":"Annual Review of Information Science and Technology"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB33","unstructured":"Rees, A. M., and Schultz, D. G., (1967). A field experiment approach to the study of relevance assessments in relation to document searching, 2 vols. Center for Documentation and Communication Research, School of Library Science, Case Western Reserve University, Cleveland, OH"},{"issue":"4","key":"10.1016\/S0306-4573(98)00025-9_BIB34","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1002\/(SICI)1097-4571(198807)39:4<235::AID-ASI3>3.0.CO;2-H","article-title":"Performance measures for information retrieval systems: An experimental approach","volume":"39","author":"Regazzi","year":"1988","journal-title":"Journal of the American Society for Information Science"},{"issue":"4","key":"10.1016\/S0306-4573(98)00025-9_BIB35","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1016\/0306-4573(92)90004-J","article-title":"On the evaluation of IR systems","volume":"28","author":"Robertson","year":"1992","journal-title":"Information Processing and Management"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB36","unstructured":"Robins, D. (1997). Shifts in focus in information retrieval interaction. Proceedings of the 60th Annual Meeting of the American Society for Information Science, November 1997, Washington, DC (pp.)"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB37","first-page":"157","article-title":"Psychometric measurement and information retrieval","volume":"23","author":"Rorvig","year":"1988","journal-title":"Annual Review of Information Science and Technology"},{"issue":"1","key":"10.1016\/S0306-4573(98)00025-9_BIB38","first-page":"33","article-title":"Online searching: What measures satisfaction?","volume":"12","author":"Sandore","year":"1990","journal-title":"Library and Information Science Research"},{"issue":"6","key":"10.1016\/S0306-4573(98)00025-9_BIB39","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1002\/asi.4630260604","article-title":"Relevance: A review of and framework for the thinking on the notion in information science","volume":"26","author":"Saracevic","year":"1975","journal-title":"Journal of the American Society for Information Science"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB40","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (1995). Evaluation of evaluation in information retrieval. Proceedings of the 18th ACM SIGIR International Conference on Research and Development in Information Retrieval. Association of Computing Machinery, Seattle, WA, pp. 138\u2013146","DOI":"10.1145\/215206.215351"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB41","unstructured":"Saracevic, T. (1996a). Interactive models in information retrieval (IR): A review and proposal. Proceedings of the 59th Annual Meeting of the American Society for Information Science, Vol. 33, pp. 3\u20139"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB42","unstructured":"Saracevic, T. (1996b). Relevance reconsidered:'96. Proceedings of COLIS 2: Second International Conference on Conceptions of Library and Information Science: Integration in Perspective, October 13\u201316, 1996. The Royal School of Librarianship, Copenhagen, Denmark, pp. 201\u2013218"},{"issue":"3","key":"10.1016\/S0306-4573(98)00025-9_BIB43","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1002\/(SICI)1097-4571(198805)39:3<161::AID-ASI2>3.0.CO;2-0","article-title":"A study of information seeking and retrieving: Part I Background and methodology","volume":"39","author":"Saracevic","year":"1988","journal-title":"Journal of the American Society for Information Science"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB44","unstructured":"Saracevic, T., Spink, A., and Wu, M. M. (1997). Users and intermediaries in interactive information retrieval (IR): What are they talking about? Proceedings of UM97: The 6th International Conference on User Modeling, ed. A. Jameson, C. Paris and C. Tasso, June 2\u20135, 1997, Sardinia \u2014 Italy. International Center for Mechanical Sciences. Courses and Lectures, Vol. 383, pp. 43\u201354. Springer Wein, New York"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB45","unstructured":"Saracevic, T., and Su, L. (1990). Modeling and measuring the user-intermediary-computer interaction in online searching. Proceedings of the 52nd Annual Meeting of the American Society for Information Science, Vol. 26, pp. 75\u201380"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB46","unstructured":"Schamber, L. (1991). User's criteria for evaluation in multimedia information seeking and use situations. Unpublished doctoral dissertation, Syracuse University"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB47","first-page":"3","article-title":"Relevance and information behavior","volume":"29","author":"Schamber","year":"1994","journal-title":"Annual Review of Information Science and Technology"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB48","doi-asserted-by":"crossref","first-page":"755","DOI":"10.1016\/0306-4573(90)90050-C","article-title":"A re-examination of relevance: Toward a dynamic, situational definition","volume":"26","author":"Schamber","year":"1990","journal-title":"Information Processing and Management"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB49","unstructured":"Smithson, S. (1990). The evaluation of information retrieval systems: A case study approach. In Informetrics 10 \u2014 Prospects for Intelligent Retrieval, ed. K. F. Jones, pp. 75\u201389. Aslib, London"},{"issue":"3","key":"10.1016\/S0306-4573(98)00025-9_BIB50","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1016\/0306-4573(94)00048-8","article-title":"Reflections on TREC","volume":"31","author":"Sparck Jones","year":"1995","journal-title":"Information Processing and Management"},{"issue":"8","key":"10.1016\/S0306-4573(98)00025-9_BIB51","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1002\/(SICI)1097-4571(199608)47:8<603::AID-ASI4>3.0.CO;2-X","article-title":"A multiple search session model of end-user behavior: An exploratory study","volume":"47","author":"Spink","year":"1996","journal-title":"Journal of the American Society for Information Science"},{"issue":"5","key":"10.1016\/S0306-4573(98)00025-9_BIB52","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1002\/(SICI)1097-4571(199705)48:5<382::AID-ASI2>3.0.CO;2-R","article-title":"A study of interactive feedback during mediated online searching","volume":"48","author":"Spink","year":"1997","journal-title":"Journal of the American Society for Information Science"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB53","doi-asserted-by":"crossref","unstructured":"Spink, A., Goodrum, A., and Robins, D. (1998). Elicitation behavior during mediated information retrieval. Information Processing and Management, 34(1)","DOI":"10.1016\/S0306-4573(97)00059-9"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB54","first-page":"33","article-title":"Feedback in information retrieval","volume":"31","author":"Spink","year":"1996","journal-title":"Annual Review of Information Science and Technology"},{"issue":"4","key":"10.1016\/S0306-4573(98)00025-9_BIB55","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1016\/0306-4573(92)90007-M","article-title":"Evaluation measures for interactive information retrieval","volume":"28","author":"Su","year":"1994","journal-title":"Information Processing and Management"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB56","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1016\/0306-4573(88)90043-X","article-title":"A total relevance and document interaction effects model for the evaluation of information retrieval processes","volume":"24","author":"Tiamiyu","year":"1988","journal-title":"Information Processing and Management"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB57","unstructured":"Williams, F. W. (1992). Reasoning with statistics: How to read quantitative research. Harcourt Brace Jovanovich, Fort Worth"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB58","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1108\/eb026702","article-title":"On user studies and information needs","volume":"37","author":"Wilson","year":"1981","journal-title":"Journal of Documentation"},{"issue":"4","key":"10.1016\/S0306-4573(98)00025-9_BIB59","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1016\/S0306-4573(97)00028-9","article-title":"Information behavior: An interdisciplinary perspective","volume":"83","author":"Wilson","year":"1997","journal-title":"Information Processing and Management"},{"key":"10.1016\/S0306-4573(98)00025-9_BIB60","unstructured":"Xie, H. (1997). Planned and situated aspects in interactive IR: Patterns of user interactive intentions and information seeking. Proceedings of the 60th Annual Meeting of the American Society for Information Science, November 1997, pp. 101\u2013110"}],"container-title":["Information Processing &amp; Management"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.elsevier.com\/content\/article\/PII:S0306457398000259?httpAccept=text\/xml","content-type":"text\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/api.elsevier.com\/content\/article\/PII:S0306457398000259?httpAccept=text\/plain","content-type":"text\/plain","content-version":"vor","intended-application":"text-mining"}],"deposited":{"date-parts":[[2019,4,27]],"date-time":"2019-04-27T14:55:57Z","timestamp":1556376957000},"score":1,"resource":{"primary":{"URL":"https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0306457398000259"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[1998,9]]},"references-count":60,"journal-issue":{"issue":"5","published-print":{"date-parts":[[1998,9]]}},"alternative-id":["S0306457398000259"],"URL":"https:\/\/doi.org\/10.1016\/s0306-4573(98)00025-9","relation":{},"ISSN":["0306-4573"],"issn-type":[{"value":"0306-4573","type":"print"}],"subject":[],"published":{"date-parts":[[1998,9]]}}}