{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,19]],"date-time":"2025-12-19T14:42:02Z","timestamp":1766155322754,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":77,"publisher":"ACM","license":[{"start":{"date-parts":[[2014,7,3]],"date-time":"2014-07-03T00:00:00Z","timestamp":1404345600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2014,7,3]]},"DOI":"10.1145\/2600428.2609577","type":"proceedings-article","created":{"date-parts":[[2014,7,8]],"date-time":"2014-07-08T12:20:11Z","timestamp":1404822011000},"page":"435-444","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":37,"title":["Multidimensional relevance modeling via psychometrics and crowdsourcing"],"prefix":"10.1145","author":[{"given":"Yinglong","family":"Zhang","sequence":"first","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}]},{"given":"Jin","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}]},{"given":"Jacek","family":"Gwizdka","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}]}],"member":"320","published-online":{"date-parts":[[2014,7,3]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9204-1"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.103.3.411"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.22707"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630300505"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3%3C149::AID-ASI5%3E3.0.CO;2-J"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"e_1_3_2_1_8_1","volume-title":"Proceedings of the ASIS Annual Meeting. 35","author":"Bateman J.","year":"1998","unstructured":"Bateman , J. 1998 . Changes in Relevance Criteria: A Longitudinal Study . Proceedings of the ASIS Annual Meeting. 35 , 23--32. Bateman, J. 1998. Changes in Relevance Criteria: A Longitudinal Study. Proceedings of the ASIS Annual Meeting. 35, 23--32."},{"key":"e_1_3_2_1_9_1","volume-title":"et al","author":"Behrend T.S.","year":"2011","unstructured":"Behrend , T.S. et al . 2011 . The viability of crowdsourcing for survey research. Behavior research methods. 43, 3, 800--813. Behrend, T.S. et al. 2011. The viability of crowdsourcing for survey research. Behavior research methods. 43, 3, 800--813."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010039"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10286"},{"key":"e_1_3_2_1_12_1","first-page":"105","article-title":"Beyond topicality: A two stage view of relevance and the retrieval process","volume":"18","author":"Boyce B.","year":"1982","unstructured":"Boyce , B. 1982 . Beyond topicality: A two stage view of relevance and the retrieval process . IP & M 18 , 3, 105 -- 109 . Boyce, B. 1982. Beyond topicality: A two stage view of relevance and the retrieval process. IP & M 18, 3, 105--109.","journal-title":"IP & M"},{"key":"e_1_3_2_1_13_1","first-page":"85","article-title":"Sources of information on specific subjects. Engineering","volume":"137","author":"Bradford S.C.","year":"1934","unstructured":"Bradford , S.C. 1934 . Sources of information on specific subjects. Engineering : An Illustrated Weekly Journal (London). 137 , 26, 85 -- 86 . Bradford, S.C. 1934. Sources of information on specific subjects. Engineering: An Illustrated Weekly Journal (London). 137, 26, 85--86.","journal-title":"An Illustrated Weekly Journal (London)."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2000.10474246"},{"key":"e_1_3_2_1_15_1","first-page":"673","article-title":"The Elaboration Likelihood Model of Persuasion","volume":"11","author":"Cacioppo J.T.","year":"1984","unstructured":"Cacioppo , J.T. and Petty , R.E. 1984 . The Elaboration Likelihood Model of Persuasion . Advances in Consumer Research. 11 , 1 673 -- 675 . Cacioppo, J.T. and Petty, R.E. 1984. The Elaboration Likelihood Model of Persuasion. Advances in Consumer Research. 11, 1 673--675.","journal-title":"Advances in Consumer Research."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433411"},{"key":"e_1_3_2_1_17_1","unstructured":"Cognitive Interviewing: http:\/\/www.uk.sagepub.com\/textbooks\/Book225856?prodId=Book225856. Accessed: 2014-01--24.  Cognitive Interviewing: http:\/\/www.uk.sagepub.com\/textbooks\/Book225856?prodId=Book225856. Accessed: 2014-01--24."},{"volume-title":"Statistical power analysis for the behavioral sciences","author":"Cohen J.","key":"e_1_3_2_1_18_1","unstructured":"Cohen , J. 1988. Statistical power analysis for the behavioral sciences . L. Erlbaum Associates . Cohen, J. 1988. Statistical power analysis for the behavioral sciences. L. Erlbaum Associates."},{"volume-title":"Proceedings of the 14th National Online Meeting, 77--84","author":"Cool C.","key":"e_1_3_2_1_19_1","unstructured":"Cool , C. et al. 1993. Characteristics of Texts affecting relevance judgements . Proceedings of the 14th National Online Meeting, 77--84 . Cool, C. et al. 1993. Characteristics of Texts affecting relevance judgements. Proceedings of the 14th National Online Meeting, 77--84."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2011.07.001"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026436"},{"volume-title":"Communication in Business: Strategies and Skills","author":"Dwyer J.","key":"e_1_3_2_1_22_1","unstructured":"Dwyer , J. 2002. Communication in Business: Strategies and Skills . Prentice Hall . Dwyer, J. 2002. Communication in Business: Strategies and Skills. Prentice Hall."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484066"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9181-9"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.2307\/3236261"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Furr M. 2011. Scale Construction and Psychometrics for Social and Personality Psychology. SAGE.  Furr M. 2011. Scale Construction and Psychometrics for Social and Personality Psychology. SAGE.","DOI":"10.4135\/9781446287866"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.48.1.82"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199510)46:9%3C646::AID-ASI2%3E3.0.CO;2-1"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(02)00032-8"},{"volume-title":"Studies in the way of words","author":"Grice H.P.","key":"e_1_3_2_1_30_1","unstructured":"Grice , H.P. 1989. Studies in the way of words . Harvard University Press . Grice, H.P. 1989. Studies in the way of words. Harvard University Press."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2578153.2578198"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199210)43:9<602::AID-ASI3>3.0.CO;2-Q"},{"key":"e_1_3_2_1_33_1","unstructured":"Hatcher L. 2013. Advanced statistics in research: reading understanding and writing up data analysis results. ShadowFinch Media LLC.  Hatcher L. 2013. Advanced statistics in research: reading understanding and writing up data analysis results. ShadowFinch Media LLC."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10132"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-28997-2_16"},{"key":"e_1_3_2_1_36_1","unstructured":"Hox J.J. and Bechger T.M. 2007. An introduction to structural equation modeling.  Hox J.J. and Bechger T.M. 2007. An introduction to structural equation modeling."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1080\/10705519909540118"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.22811"},{"key":"e_1_3_2_1_39_1","first-page":"270","article-title":"Characteristics of Errors in Accounts Receivable and Inventory Audits","volume":"56","author":"Johnson J.R.","year":"1981","unstructured":"Johnson , J.R. 1981 . Characteristics of Errors in Accounts Receivable and Inventory Audits . The Accounting Review. 56 , 2, 270 -- 293 . Johnson, J.R. et al. 1981. Characteristics of Errors in Accounts Receivable and Inventory Audits. The Accounting Review. 56, 2, 270--293.","journal-title":"The Accounting Review."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396779"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009947"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"volume-title":"Information retrieval systems: characteristics, testing, and evaluation","author":"Lancaster F.W.","key":"e_1_3_2_1_43_1","unstructured":"Lancaster , F.W. 1968. Information retrieval systems: characteristics, testing, and evaluation . Wiley . Lancaster, F.W. 1968. Information retrieval systems: characteristics, testing, and evaluation. Wiley."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-0271(68)90029-6"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(94)00030-7"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2009.5295247"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000016"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.88.3.588"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630280107"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2464464.2464485"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199709)48:9%3C810::AID-ASI6%3E3.0.CO;2-U"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36973-5_2"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"crossref","unstructured":"Mueller R.O. and Hancock G.R. 2008. Best practices in structural equation modeling. Best practices in quantitative methods. 488--508.  Mueller R.O. and Hancock G.R. 2008. Best practices in structural equation modeling. Best practices in quantitative methods. 488--508.","DOI":"10.4135\/9781412995627.d38"},{"key":"e_1_3_2_1_54_1","volume-title":"Machine Learning: A Probabilistic Perspective","author":"Murphy K.P.","year":"2012","unstructured":"Murphy , K.P. 2012 . Machine Learning: A Probabilistic Perspective . Mit Press . Murphy, K.P. 2012. Machine Learning: A Probabilistic Perspective. Mit Press."},{"key":"e_1_3_2_1_55_1","unstructured":"Pearson - Modern Measurement: Theory Principles and Applications of Mental Appraisal 2\/E - Steven J. Osterlind: http:\/\/www.pearsonhighered.com\/educator\/product\/Modern-Measurement-Theory-Principles-and-Applications-of-Mental-Appraisal\/9780137010257.page. Accessed: 2014-01--24.  Pearson - Modern Measurement: Theory Principles and Applications of Mental Appraisal 2\/E - Steven J. Osterlind: http:\/\/www.pearsonhighered.com\/educator\/product\/Modern-Measurement-Theory-Principles-and-Applications-of-Mental-Appraisal\/9780137010257.page. Accessed: 2014-01--24."},{"key":"e_1_3_2_1_56_1","unstructured":"Principles and Practice of Structural Equation Modeling: Third Edition: http:\/\/www.guilford.com\/cgi-bin\/cartscript.cgi?page=pr\/kline.htm & dir=research\/res_quant. Accessed: 2014-01--24.  Principles and Practice of Structural Equation Modeling: Third Edition: http:\/\/www.guilford.com\/cgi-bin\/cartscript.cgi?page=pr\/kline.htm & dir=research\/res_quant. Accessed: 2014-01--24."},{"key":"e_1_3_2_1_57_1","volume-title":"International Conference on Scientific Information -- Two Volumes: http:\/\/books.nap.edu\/openbook.php?record_id=10866 & page=687","author":"Proceedings","year":"2014","unstructured":"Proceedings of the International Conference on Scientific Information -- Two Volumes: http:\/\/books.nap.edu\/openbook.php?record_id=10866 & page=687 . Accessed : 2014 -01-26. Proceedings of the International Conference on Scientific Information -- Two Volumes: http:\/\/books.nap.edu\/openbook.php?record_id=10866 & page=687. Accessed: 2014-01-26."},{"key":"e_1_3_2_1_58_1","unstructured":"Rees A.M. and Schultz D.G. 1967. A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Final Report to the National Science Foundation. Volume I.  Rees A.M. and Schultz D.G. 1967. A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Final Report to the National Science Foundation. Volume I."},{"key":"e_1_3_2_1_59_1","unstructured":"Relevance as process: judgements in the context of scholarly research: http:\/\/www.informationr.net\/ir\/10--2\/paper226. Accessed: 2014-01--24.  Relevance as process: judgements in the context of scholarly research: http:\/\/www.informationr.net\/ir\/10--2\/paper226. Accessed: 2014-01--24."},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000009"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v58:13"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v58:13"},{"key":"e_1_3_2_1_63_1","first-page":"3","article-title":"Relevance and Information Behavior","volume":"29","author":"Schamber L.","year":"1994","unstructured":"Schamber , L. 1994 . Relevance and Information Behavior . Annual Review of Information Science and Technology (ARIST). 29 , 3 -- 48 . Schamber, L. 1994. Relevance and Information Behavior. Annual Review of Information Science and Technology (ARIST). 29, 3--48.","journal-title":"Annual Review of Information Science and Technology (ARIST)."},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02294318"},{"key":"e_1_3_2_1_65_1","unstructured":"Tabachnick B.G. and Fidell L.S. 2012. Using Multivariate Statistics. Pearson Education Limited.  Tabachnick B.G. and Fidell L.S. 2012. Using Multivariate Statistics. Pearson Education Limited."},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(97)00081-2"},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2006.09.008"},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.14.1.87.14765"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"crossref","unstructured":"Tsikrika T. and Lalmas M. 2007. Combining Evidence for Relevance Criteria: A Framework and Experiments in Web Retrieval. Advances in Information Retrieval. G. Amati et al. eds. Springer Berlin Heidelberg. 481--493.   Tsikrika T. and Lalmas M. 2007. Combining Evidence for Relevance Criteria: A Framework and Experiments in Web Retrieval. Advances in Information Retrieval. G. Amati et al. eds. Springer Berlin Heidelberg. 481--493.","DOI":"10.1007\/978-3-540-71496-5_43"},{"key":"e_1_3_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007127"},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291017"},{"key":"e_1_3_2_1_72_1","unstructured":"Wilson D. and Sperber D. 2002. Relevance Theory. Handbook of Pragmatics. G. Ward and L. Horn eds. Blackwell.  Wilson D. and Sperber D. 2002. Relevance Theory. Handbook of Pragmatics. G. Ward and L. Horn eds. Blackwell."},{"key":"e_1_3_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1080\/02664763.2011.610445"},{"key":"e_1_3_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1177\/0011000006288127"},{"key":"e_1_3_2_1_75_1","unstructured":"Wright S. Correlation and causation.  Wright S. Correlation and causation."},{"key":"e_1_3_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v57:7"},{"key":"e_1_3_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9206-z"}],"event":{"name":"SIGIR '14: The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Gold Coast Queensland Australia","acronym":"SIGIR '14"},"container-title":["Proceedings of the 37th international ACM SIGIR conference on Research &amp; development in information retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2600428.2609577","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2600428.2609577","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:01:26Z","timestamp":1750230086000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2600428.2609577"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,7,3]]},"references-count":77,"alternative-id":["10.1145\/2600428.2609577","10.1145\/2600428"],"URL":"https:\/\/doi.org\/10.1145\/2600428.2609577","relation":{},"subject":[],"published":{"date-parts":[[2014,7,3]]},"assertion":[{"value":"2014-07-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}