{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,7]],"date-time":"2026-02-07T15:35:58Z","timestamp":1770478558464,"version":"3.49.0"},"reference-count":9,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2008,11,30]],"date-time":"2008-11-30T00:00:00Z","timestamp":1228003200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2008,11,30]]},"abstract":"<jats:p>Relevance evaluation is an essential part of the development and maintenance of information retrieval systems. Yet traditional evaluation approaches have several limitations; in particular, conducting new editorial evaluations of a search system can be very expensive. We describe a new approach to evaluation called TERC, based on the crowdsourcing paradigm, in which many online users, drawn from a large community, each performs a small evaluation task.<\/jats:p>","DOI":"10.1145\/1480506.1480508","type":"journal-article","created":{"date-parts":[[2008,12,30]],"date-time":"2008-12-30T17:45:31Z","timestamp":1230659131000},"page":"9-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":208,"title":["Crowdsourcing for relevance evaluation"],"prefix":"10.1145","volume":"42","author":[{"given":"Omar","family":"Alonso","sequence":"first","affiliation":[{"name":"A9.com, Palo Alto, CA"}]},{"given":"Daniel E.","family":"Rose","sequence":"additional","affiliation":[{"name":"A9.com, Palo Alto, CA"}]},{"given":"Benjamin","family":"Stewart","sequence":"additional","affiliation":[{"name":"A9.com, Palo Alto, CA"}]}],"member":"320","published-online":{"date-parts":[[2008,11,30]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Amazon Mechanical Turk http:\/\/www.mturk.com  Amazon Mechanical Turk http:\/\/www.mturk.com"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1142055.1142067"},{"key":"e_1_2_1_3_1","unstructured":"Brendan O'Connor \"Search Engine Relevance: An Empirical Test\" http:\/\/blog.doloreslabs.com\/2008\/04\/search-engine-relevance-an-empirical-test\/#more-35 accessed April 13 2008.  Brendan O'Connor \"Search Engine Relevance: An Empirical Test\" http:\/\/blog.doloreslabs.com\/2008\/04\/search-engine-relevance-an-empirical-test\/#more-35 accessed April 13 2008."},{"key":"e_1_2_1_4_1","unstructured":"Jeff Howe. \"The Rise of Crowdsourcing\". Wired June 2006. http:\/\/www.wired.com\/wired\/archive\/14.06\/crowds.html  Jeff Howe. \"The Rise of Crowdsourcing\". Wired June 2006. http:\/\/www.wired.com\/wired\/archive\/14.06\/crowds.html"},{"key":"e_1_2_1_5_1","volume-title":"Springer","author":"Ingwersen Peter","year":"2005"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2007.289"},{"key":"e_1_2_1_7_1","first-page":"4","article-title":"Why Is Web Search So Hard.. to Evaluate?","volume":"3","author":"Rose Daniel E.","year":"2004","journal-title":"Journal of Web Engineering"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v58:13"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297797.1297822"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1480506.1480508","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1480506.1480508","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T13:29:59Z","timestamp":1750253399000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1480506.1480508"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2008,11,30]]},"references-count":9,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2008,11,30]]}},"alternative-id":["10.1145\/1480506.1480508"],"URL":"https:\/\/doi.org\/10.1145\/1480506.1480508","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2008,11,30]]},"assertion":[{"value":"2008-11-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}