{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T03:14:29Z","timestamp":1761621269224,"version":"3.41.2"},"reference-count":64,"publisher":"Emerald","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,7,18]]},"abstract":"<jats:sec>\n                  <jats:title>Purpose<\/jats:title>\n                  <jats:p>\u2013 One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the \u201cwisdom of crowds\u201d. The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Design\/methodology\/approach<\/jats:title>\n                  <jats:p>\u2013 In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated \u201cwisdom of crowds\u201d judgements were used as a means to compare the differences between human assessments of search results and search engine\u2019s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Findings<\/jats:title>\n                  <jats:p>\u2013 It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Practical implications<\/jats:title>\n                  <jats:p>\u2013 The proposed \u201cwisdom of crowds\u201d-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine\u2019s ranking over time to changes in users preferences.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Originality\/value<\/jats:title>\n                  <jats:p>\u2013 This is a first study that applies the notion of \u201cwisdom of crowds\u201d to examine an under-explored in the literature phenomenon of \u201cchange in time\u201d in user evaluation of relevance.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1108\/ajim-10-2015-0165","type":"journal-article","created":{"date-parts":[[2016,6,30]],"date-time":"2016-06-30T04:38:33Z","timestamp":1467261513000},"page":"407-427","source":"Crossref","is-referenced-by-count":9,"title":["Testing the stability of \u201cwisdom of crowds\u201d judgments of search results over time and their similarity with the search engine rankings"],"prefix":"10.1108","volume":"68","author":[{"given":"Maayan","family":"Zhitomirsky-Geffet","sequence":"first","affiliation":[{"name":"Department of Information Science,Bar-Ilan University,Ramat Gan,Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Judit","family":"Bar-Ilan","sequence":"additional","affiliation":[{"name":"Department of Information Science,Bar-Ilan University,Ramat Gan,Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Levene","sequence":"additional","affiliation":[{"name":"Department of Computer Science,BirkBeck University of London,London,UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"2025072816492745700_b1","doi-asserted-by":"crossref","unstructured":"Agichtein, E.\n          , Brill, E. and Dumais, S. (2006), \u201cImproving web search ranking by incorporating user behavior information\u201d, Proceedings of SIGIR\u201906, ACM, New York, NY, pp. 19-26.","DOI":"10.1145\/1148170.1148177"},{"key":"2025072816492745700_b2","doi-asserted-by":"crossref","unstructured":"Bao, S.\n          , Xue, G., Wu, X., Yu, Y., Fei, B. and Su, Z. (2007), \u201cOptimizing web search using social annotations\u201d, in Patel-Schnider, P., Shenoy, P., Williamson, C. and Zurko, M. (Eds), WWW \u201807: Proceedings of the 16th International Conference on World Wide Web, ACM, New York, NY, pp. 501-510.","DOI":"10.1145\/1242572.1242640"},{"issue":"5","key":"2025072816492745700_b3","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1108\/eb024320","article-title":"The design of browsing and berrypicking techniques for the online search interface","volume":"13","author":"","year":"1989","journal-title":"Online Review"},{"issue":"6","key":"2025072816492745700_b6","doi-asserted-by":"crossref","first-page":"854","DOI":"10.1108\/14684521111193166","article-title":"A method to assess search engine results","volume":"35","author":"","year":"2011","journal-title":"Online Information Review"},{"issue":"9","key":"2025072816492745700_b4","first-page":"1254","article-title":"User rankings of search engine results","volume":"58","author":"","year":"2007","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"1","key":"2025072816492745700_b5","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1002\/asi.20941","article-title":"Presentation bias is significant in determining user preference for search results \u2013 a user study","volume":"60","author":"","year":"2009","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"1","key":"2025072816492745700_b7","first-page":"23","article-title":"Changes in relevance criteria: a longitudinal study","volume":"35","author":"","year":"1998","journal-title":"Journal of the American Society for Information Science"},{"issue":"9","key":"2025072816492745700_b8","first-page":"1879","article-title":"Ranking, relevance judgment, and precision of information retrieval on children\u2019s queries: evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids","volume":"63","author":"","year":"2012","journal-title":"Journal of Association for Information Science"},{"issue":"10","key":"2025072816492745700_b9","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1109\/MC.2011.323","article-title":"Twitter mood as a stock market predictor","volume":"44","author":"","year":"2011","journal-title":"IEEE Computer"},{"issue":"5","key":"2025072816492745700_b10","first-page":"142","article-title":"A cognitive view of the situational dynamism of user centered relevance estimation","volume":"45","author":"","year":"1994","journal-title":"Journal of the Association for Information Science"},{"key":"2025072816492745700_b11","unstructured":"Case, D.O.\n           (2012),                   Looking for Information: A Survey of Research on Information Seeking, Needs and Behavior                , Emerald Group Publishing Limited, Bingley."},{"key":"2025072816492745700_b12","unstructured":"Cen, R.\n          , Liu, Y., Zhang, M., Ru, L. and Ma, S. (2009), \u201cAutomatic search engine performance evaluation with the wisdom of crowds\u201d, Proceedings of AIRS, 5th Asia Information Retrieval Symposium, AIRS 2009, Sapporo, October 21-23."},{"key":"2025072816492745700_b13","unstructured":"Chitika\n           (2013), \u201cChitika insights: the value of Google positioning\u201d, available at: https:\/\/cdn2.hubspot.net\/hub\/239330\/file-61331237-pdf\/ChitikaInsights-ValueofGoogleResultsPositioning.pdf (accessed 1 October 2015)."},{"key":"2025072816492745700_b14","doi-asserted-by":"crossref","unstructured":"Choochaiwattana, W.\n           and Spring, M.B. (2009), \u201cApplying social annotations to retrieve and re-rank web resources\u201d, in Mahadevan, V. and Xie, Y. (Eds), ICIME \u201809: Proceeding of 2009 International Conference on Information Management and Engineering, Kuala Lumpur, IEEE, Los-Alamitos, CA, 3-5 April, pp. 215-219.","DOI":"10.1109\/ICIME.2009.41"},{"issue":"7307","key":"2025072816492745700_b15","doi-asserted-by":"crossref","first-page":"756","DOI":"10.1038\/nature09304","article-title":"Predicting protein structures with a multiplayer online game","volume":"466","author":"","year":"2010","journal-title":"Nature"},{"key":"2025072816492745700_b16","unstructured":"comScore\n           (2015), \u201ccomScore releases November 2015 US desktop search engine rankings\u201d, comScore qSearchTM, available at: www.comscore.com\/Insights\/Rankings\/comScore-Releases-November-2015-US-Desktop-Search-Engine-Rankings (accessed 1 October 2015)."},{"key":"2025072816492745700_b17","unstructured":"Dervin, B.\n           (1992),                   Qualitative Research in Information Management                , Libraries Unlimited, Englewood, CO, pp. 61-84."},{"key":"2025072816492745700_b18","doi-asserted-by":"crossref","unstructured":"Dou, Z.\n          , Song, R., Yuan, X. and Wen, J. (2008), \u201cAre click-through data adequate for learning web search rankings?\u201d, Proceedings of CIKM\u201908. ACM, New York, NY, pp. 73-82.","DOI":"10.1145\/1458082.1458095"},{"key":"2025072816492745700_b19","unstructured":"Du, J.T.\n           (2010), \u201cMultitasking, cognitive coordination and cognitive shifts during web searching\u201d, unpublished PhD, Queensland University of Technology, Queensland."},{"issue":"8","key":"2025072816492745700_b20","doi-asserted-by":"crossref","first-page":"1446","DOI":"10.1002\/asi.21551","article-title":"Towards a web search model: Integrating multitasking, cognitive coordination and cognitive shifts","volume":"62","author":"","year":"2011","journal-title":"Journal of the American Society for Information Science and Technology (JASIST)"},{"issue":"4","key":"2025072816492745700_b21","first-page":"171","article-title":"A behavioural approach to information retrieval design","volume":"49","author":"","year":"1989","journal-title":"Journal of Documentation"},{"key":"2025072816492745700_b22","unstructured":"Fisher, K.E.\n          , Erdelez, S. and McKechnie, L.E.F. (Eds) (2005),                   Theories of Information Behavior                , ASIS&T Monograph Series, Medford, NJ, Information Today."},{"issue":"7070","key":"2025072816492745700_b23","doi-asserted-by":"crossref","first-page":"900","DOI":"10.1038\/438900a","article-title":"Internet encyclopedia go head to head","volume":"438","author":"","year":"2005","journal-title":"Nature"},{"issue":"4","key":"2025072816492745700_b24","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1108\/14684521111161954","article-title":"Relevance ranking on google. Are top ranked results considered more relevant by the users?","volume":"35","author":"","year":"2011","journal-title":"Online Information Review"},{"key":"2025072816492745700_b25","doi-asserted-by":"crossref","unstructured":"Harris, C.G.\n           (2014), \u201cThe beauty contest revisited: measuring consensus rankings of relevance using a game\u201d, Proceedings of the First International Workshop on Gamification for Information Retrieval, ACM, pp. 17-21.","DOI":"10.1145\/2594776.2594780"},{"key":"2025072816492745700_b501","doi-asserted-by":"crossref","unstructured":"Harshavardhan, A.\n          , Gandhe, A., Ross, L., Ssu-Hsin, Y. and Benyuan, L. (2013), \u201cOnline social networks flu trend tracker: a novel sensory approach to predict flu trends\u201d, in Gabriel, J., Schier, J., Van Huffel, S., Conchon, E., Correia, C., Fred, A. and Gamboa, H. (Eds),                   Biomedical Engineering Systems and Technologies                , Communications in Computer and Information Science, Vol. 357, Springer Berlin Heidelberg, pp. 353-368.","DOI":"10.1007\/978-3-642-38256-7_24"},{"issue":"1","key":"2025072816492745700_b26","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1016\/j.ipm.2004.10.007","article-title":"How are we searching the web? A comparison of nine search engine transaction logs","volume":"42","author":"","year":"2006","journal-title":"Information Processing and Management"},{"issue":"2","key":"2025072816492745700_b28","first-page":"1","article-title":"Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search","volume":"25","author":"","year":"2007","journal-title":"ACM Transactions on Information Systems"},{"issue":"3","key":"2025072816492745700_b62","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1111\/j.1468-2958.1993.tb00305.x","article-title":"A comprehensive model of cancer-related information seeking applied to magazines","volume":"19","author":"","year":"1993","journal-title":"Human Communications Research"},{"key":"2025072816492745700_b29","doi-asserted-by":"crossref","unstructured":"Kawase, R.\n          , Siehndel, P., Pereira Nunes, B., Herder, E. and Nejdl, W. (2014), \u201cExploiting the wisdom of the crowds for characterizing and connecting heterogeneous resources\u201d, Proceedings of HT\u201914, Santiago, 1-4 September.","DOI":"10.1145\/2631775.2631797"},{"key":"2025072816492745700_b30","doi-asserted-by":"crossref","unstructured":"Keenoy, K.\n           and Levene, M. (2005), \u201cPersonalisation of web search\u201d, in Anand, S.S. and Mobasher, B. (Eds),                   Intelligent Techniques for Web Personalization (ITWP)                , Lecture Notes in Computer Science (LNCS), Springer-Verlag, Berlin, pp. 201-228.","DOI":"10.1007\/11577935_11"},{"key":"2025072816492745700_b31","doi-asserted-by":"crossref","unstructured":"Knight, S.A.\n           and Spink, A. (2008), \u201cToward a web search information behavior model\u201d, in Spink, A. and Zimmer, M. (Eds),                   Web Search: Multidisciplinary Perspectives                , Springer-Verlag, Berlin, pp. 209-234.","DOI":"10.1007\/978-3-540-75829-7_12"},{"issue":"5","key":"2025072816492745700_b59","first-page":"361","article-title":"Inside the search process: information seeking from the user 's perspective","volume":"42","author":"","year":"1991","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"6","key":"2025072816492745700_b32","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1108\/00220410810912451","article-title":"The retrieval effectiveness of web search engines: considering results descriptions","volume":"64","author":"","year":"2008","journal-title":"Journal of Documentation"},{"issue":"9","key":"2025072816492745700_b33","doi-asserted-by":"crossref","first-page":"1763","DOI":"10.1002\/asi.23304","article-title":"Evaluating the retrieval effectiveness of web search engines using a representative query sample","volume":"66","author":"","year":"2015","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2025072816492745700_b63","doi-asserted-by":"crossref","unstructured":"Liu, D.\n          , Hua, X., Yang, L., Wang, M. and Zhang, H. (2009), \u201cTag ranking\u201d, WWW\u201909: Proceedings of the 18th International Conference on World Wide Web, ACM, New York, NY, pp. 351-360.","DOI":"10.1145\/1526709.1526757"},{"key":"2025072816492745700_b34","doi-asserted-by":"crossref","unstructured":"Marchionini, G.\n           (1995),                   Information Seeking in Electronic Environments                , Cambridge University Press, Cambridge.","DOI":"10.1017\/CBO9780511626388"},{"key":"2025072816492745700_b35","doi-asserted-by":"crossref","unstructured":"Micarelli, A.\n          , Gasparetti, F., Sciarrone, F. and Gauch, S. (2007), \u201cPersonalized search on the world wide web\u201d, in Brusilovsky, P., Kobsa, A. and Nejdl, W. (Eds),                   The Adaptive Web                , LNCS 4321, Springer-Verlag, Berlin and Heidelberg, pp. 195-230.","DOI":"10.1007\/978-3-540-72079-9_6"},{"issue":"3","key":"2025072816492745700_b503","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1136\/amiajnl-2014-002901","article-title":"Using the wisdom of the crowds to find critical errors in biomedical ontologies: a study of SNOMED CT","volume":"22","author":"","year":"2015","journal-title":"Journal of American Medical Information Association"},{"issue":"1684","key":"2025072816492745700_b38","article-title":"Quantifying trading behavior in financial markets using Google trends","volume":"3","author":"","year":"2013","journal-title":"Scientific Reports"},{"key":"2025072816492745700_b39","unstructured":"Rees, A.M.\n           and Schultz, D.G. (1967),                   A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching                , Vols 1-2, Western Reserve University, School of Library Science, Center for Documentation and Communication Research, Cleveland, OH."},{"issue":"1","key":"2025072816492745700_b41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1500000040","article-title":"Search result diversification","volume":"9","author":"","year":"2015","journal-title":"Foundations and Trends in Information Retrieval"},{"issue":"13","key":"2025072816492745700_b42","doi-asserted-by":"crossref","first-page":"2126","DOI":"10.1002\/asi.20681","article-title":"Relevance: a review of the literature and a framework for thinking on the notion in information science, part III: behaviour and effects of relevance","volume":"58","author":"","year":"2007","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"2025072816492745700_b43","doi-asserted-by":"crossref","unstructured":"Scholer, F.\n          , Turpin, A. and Sanderson, M. (2011), \u201cQuantifying test collection quality based on the consistency of relevance judgments\u201d, SIGIR\u201911: Proceedings of the 34th international ACM SIGIR Conference, 24-28 July, Beijing and New York, NY, pp. 1063-1072.","DOI":"10.1145\/2009916.2010057"},{"key":"2025072816492745700_b44","doi-asserted-by":"crossref","unstructured":"Scholer, F.\n          , Kelly, D., Wu, W.C., Lee, H.S. and Webber, W. (2013), \u201cThe effect of threshold priming and need for cognition on relevance calibration and assessment\u201d, Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp. 623-632.","DOI":"10.1145\/2484028.2484090"},{"key":"2025072816492745700_b45","doi-asserted-by":"crossref","unstructured":"Singh, V.K.\n          , Piryani, R., Uddin, A. and Pinto, D. (2013), \u201cA content-based e-resource recommender system to augment ebook-based learning\u201d, inRamanna, S., Lingras, P., Sombattheera, C. and Krishna, A. (Eds).                   Multi-disciplinary Trends in Artificial Intelligence                , Springer, Berlin and Heidelberg, pp. 257-268.","DOI":"10.1007\/978-3-642-44949-9_24"},{"issue":"2","key":"2025072816492745700_b47","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/0306-4573(94)90065-5","article-title":"Information retrieval evaluation in practice: a case study approach","volume":"30","author":"","year":"1994","journal-title":"Information Processing and Management"},{"issue":"8","key":"2025072816492745700_b504","first-page":"382","article-title":"Study of interactive feedback during mediated information retrieval","volume":"45","author":"","year":"1997","journal-title":"Journal of the American Society for Information Science"},{"issue":"6","key":"2025072816492745700_b60","doi-asserted-by":"crossref","first-page":"845","DOI":"10.1108\/14684520710841801","article-title":"Cognitive shifts related to interactive information retrieval","volume":"31","author":"","year":"2007","journal-title":"Online Information Review"},{"key":"2025072816492745700_b48","unstructured":"Surowiecki, J.\n           (2005),                   The Wisdom of Crowds                , Doubleday, New York, NY."},{"issue":"8","key":"2025072816492745700_b64","first-page":"676","article-title":"Use of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies","volume":"52","author":"","year":"2001","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"3","key":"2025072816492745700_b50","first-page":"295","article-title":"Changes in search tactics and relevance judgments when preparing a research proposal: a summary of findings of a longitudinal study","volume":"4","author":"","year":"2001","journal-title":"Information Retrieval"},{"issue":"5","key":"2025072816492745700_b49","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1108\/EUM0000000007127","article-title":"Changes in relevance criteria and problem stages in task performance","volume":"56","author":"","year":"2000","journal-title":"Journal of Documentation"},{"issue":"4","key":"2025072816492745700_b51","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1016\/S0306-4573(03)00043-8","article-title":"New measurements for search engine evaluation proposed and tested","volume":"40","author":"","year":"2004","journal-title":"Information Processing & Management"},{"key":"2025072816492745700_b52","unstructured":"Veronis, J.\n           (2006), \u201cA comparative study of six search engines\u201d, available at: www.up.univ-mrs.fr\/veronis\/pdf\/2006-comparative-study.pdf (accessed 15 April 2006)."},{"key":"2025072816492745700_b53","unstructured":"Wang, P.\n           and White, M.D. (1995), \u201cDocument use during a research project: a longitudinal study\u201d, Proceedings of American Society for Information Science, Vol. 32, pp. 181-188."},{"issue":"3","key":"2025072816492745700_b61","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1108\/EUM0000000007145","article-title":"Models in information behaviour research","volume":"55","author":"","year":"1999","journal-title":"Journal of Documentation"},{"key":"2025072816492745700_b54","doi-asserted-by":"crossref","unstructured":"Yanbe, Y.\n          , Jatowt, A., Nakamura, S. and Tanaka, K. (2007), \u201cCan social bookmarking enhance search in the web?\u201d, in Larson, R., Rasmussen, E., Sugimoto, S. and Toms, E. (Eds), JCDL \u201807 Proceedings of the 2007 Conference on Digital Libraries, ACM, New York, NY, pp. 107-116.","DOI":"10.1145\/1255175.1255198"},{"issue":"2","key":"2025072816492745700_b56","first-page":"1","article-title":"Mining query subtopics from social tags","volume":"20","author":"","year":"2015","journal-title":"Information Research"},{"issue":"5","key":"2025072816492745700_b57","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1108\/OIR-03-2014-0073","article-title":"Maximizing agreement on diverse ontologies with \u2018wisdom of crowds\u2019 relation classification","volume":"38","author":"","year":"2014","journal-title":"Online Information Review"},{"key":"2025072816492745700_b58","doi-asserted-by":"crossref","unstructured":"Zhitomirsky-Geffet, M.\n          , Erez, E.S. and Bar-Ilan, J. (2016), \u201cTowards multi-viewpoint\u2008 ontology construction by collaboration of non-experts and crowdsourcing: the case of the effect of diet on health\u201d,                   Journal of the Association for Information Science and Technology (JASIST)                . doi: 10.1002\/asi.23686.","DOI":"10.1002\/asi.23686"},{"issue":"3","key":"2025072816492745700_frd1","first-page":"305","article-title":"How many relevances in information retrieval?","volume":"10","author":"","year":"1998","journal-title":"Interacting with Computers"},{"key":"2025072816492745700_frd2","unstructured":"Saracevic, T.\n           (1996), \u201cRelevance reconsidered\u201d, (1996), CoLIS 2: Proceedings of the Second Conference on Conception of Library and Information Science: Integration in Perspectives, The Royal School of Librarianship, 13-16 October, Copenhagen, pp. 201-218."},{"issue":"2","key":"2025072816492745700_frd3","first-page":"1","article-title":"Some observations on user search behavior","volume":"9","author":"","year":"2006","journal-title":"Australian Journal of Intelligent Information Processing Systems"}],"container-title":["Aslib Journal of Information Management"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/AJIM-10-2015-0165","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/AJIM-10-2015-0165\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/ajim\/article-pdf\/68\/4\/407\/360067\/ajim-10-2015-0165.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/www.emerald.com\/ajim\/article-pdf\/68\/4\/407\/360067\/ajim-10-2015-0165.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T20:49:46Z","timestamp":1753735786000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.emerald.com\/ajim\/article\/68\/4\/407\/60568\/Testing-the-stability-of-wisdom-of-crowds"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,7,18]]},"references-count":64,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,7,18]]}},"URL":"https:\/\/doi.org\/10.1108\/ajim-10-2015-0165","relation":{},"ISSN":["2050-3806","2050-3814"],"issn-type":[{"type":"print","value":"2050-3806"},{"type":"electronic","value":"2050-3814"}],"subject":[],"published":{"date-parts":[[2016,7,18]]}}}