{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,22]],"date-time":"2025-10-22T20:18:01Z","timestamp":1761164281232,"version":"build-2065373602"},"reference-count":10,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2016,2,24]],"date-time":"2016-02-24T00:00:00Z","timestamp":1456272000000},"content-version":"vor","delay-in-days":419,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":["asistdl.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Proc. Assoc. Info. Sci. Tech."],"published-print":{"date-parts":[[2015,1]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>In this study we investigate whether and why users change their preferences when assessing search engine results over time. We conducted a study with 35 subjects who were asked to rank and assign relevance scores to the same set of search results for three times, with a few weeks period between each round. The subjects were then exposed to the differences in their judgements and were asked to explain them. A new coefficient to measure change was introduced to assess the results of the experiment. We found that all the subjects judge the vast majority of the results differently in every round. However, there was less change in relevance judgements than in rankings. Most of the subjects were satisfied with their changes, and did not perceive them as mistakes but rather as a legitimate phenomenon, since they believe that time influences the relevance assessment. Our analysis reveals that the main factors that caused these changes were due to categorical thinking, influence of the learnt information, and environmental and emotional changes.<\/jats:p>","DOI":"10.1002\/pra2.2015.145052010067","type":"journal-article","created":{"date-parts":[[2016,2,27]],"date-time":"2016-02-27T05:56:50Z","timestamp":1456552610000},"page":"1-4","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["How and why do users change their assessment of search results over time?"],"prefix":"10.1002","volume":"52","author":[{"given":"Maayan","family":"Zhitomirsky\u2010Geffet","sequence":"first","affiliation":[{"name":"Bar\u2010Ilan University  Ramat\u2010Gan Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Judit","family":"Bar\u2010Ilan","sequence":"additional","affiliation":[{"name":"Bar\u2010Ilan University  Ramat\u2010Gan Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Levene","sequence":"additional","affiliation":[{"name":"Birkbeck University  London UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2016,2,24]]},"reference":[{"key":"e_1_2_5_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20608"},{"key":"e_1_2_5_3_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20941"},{"key":"e_1_2_5_4_1","doi-asserted-by":"publisher","DOI":"10.1108\/14684521111193166"},{"key":"e_1_2_5_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2004.10.007"},{"issue":"4","key":"e_1_2_5_6_1","first-page":"598","article-title":"Relevance ranking on Google","volume":"35","author":"Hariri N.","year":"2011","journal-title":"Are top ranked results considered more relevant by the users? Online Information Review"},{"key":"e_1_2_5_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0953-5438(98)00012-5"},{"key":"e_1_2_5_8_1","unstructured":"MullainathanS.(2000).Thinking through categories. MIT working paper. Retrieved fromwww.haas.berkeley.edu\/groups\/finance\/cat3.pdf"},{"key":"e_1_2_5_9_1","first-page":"201","volume-title":"CoLIS 2: Proceedings of the Second Conference on Conception of Library and Information Science: Integration in Perspectives","author":"Saracevic T.","year":"1996"},{"key":"e_1_2_5_10_1","first-page":"1063","volume-title":"Quantifying test collection quality based on the consistency of relevance judgements","author":"Scholer F.","year":"2011"},{"key":"e_1_2_5_11_1","first-page":"757","volume-title":"Characterizing the value of personalizing search","author":"Teevan J.","year":"2007"}],"container-title":["Proceedings of the Association for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fpra2.2015.145052010067","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fpra2.2015.145052010067","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/pra2.2015.145052010067","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/pra2.2015.145052010067","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/pra2.2015.145052010067","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T18:06:01Z","timestamp":1761069961000},"score":1,"resource":{"primary":{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/10.1002\/pra2.2015.145052010067"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,1]]},"references-count":10,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,1]]}},"alternative-id":["10.1002\/pra2.2015.145052010067"],"URL":"https:\/\/doi.org\/10.1002\/pra2.2015.145052010067","archive":["Portico"],"relation":{},"ISSN":["2373-9231","2373-9231"],"issn-type":[{"type":"print","value":"2373-9231"},{"type":"electronic","value":"2373-9231"}],"subject":[],"published":{"date-parts":[[2015,1]]},"assertion":[{"value":"2016-02-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}