{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,31]],"date-time":"2025-12-31T05:24:18Z","timestamp":1767158658919,"version":"build-2238731810"},"reference-count":27,"publisher":"Proceedings of the National Academy of Sciences","issue":"17","content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2012,4,24]]},"abstract":"<jats:p>\n                    Searching for relevant content in a massive amount of multimedia information is facilitated by accurately annotating each image, video, or song with a large number of relevant semantic keywords, or\n                    <jats:italic>tags<\/jats:italic>\n                    . We introduce game-powered machine learning, an integrated approach to annotating multimedia content that combines the effectiveness of\n                    <jats:italic>human computation<\/jats:italic>\n                    , through online games, with the scalability of\n                    <jats:italic>machine learning<\/jats:italic>\n                    . We investigate this framework for labeling music. First, a socially-oriented music annotation game called\n                    <jats:italic>Herd It<\/jats:italic>\n                    collects reliable music annotations based on the \u201cwisdom of the crowds.\u201d Second, these annotated examples are used to train a supervised machine learning system. Third, the machine learning system\n                    <jats:italic>actively<\/jats:italic>\n                    directs the annotation games to collect new data that will most benefit future model iterations. Once trained, the system can automatically annotate a corpus of music much larger than what could be labeled using human computation alone. Automatically annotated songs can be retrieved based on their semantic relevance to text-based queries (e.g., \u201cfunky jazz with saxophone,\u201d \u201cspooky electronica,\u201d etc.). Based on the results presented in this paper, we find that actively coupling annotation games with machine learning provides a reliable and scalable approach to making searchable massive amounts of multimedia data.\n                  <\/jats:p>","DOI":"10.1073\/pnas.1014748109","type":"journal-article","created":{"date-parts":[[2012,3,29]],"date-time":"2012-03-29T20:20:15Z","timestamp":1333052415000},"page":"6411-6416","update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":23,"title":["Game-powered machine learning"],"prefix":"10.1073","volume":"109","author":[{"given":"Luke","family":"Barrington","sequence":"first","affiliation":[{"name":"Electrical and Computer Engineering Department, University of California at San Diego, La Jolla, CA 92093; and"}]},{"given":"Douglas","family":"Turnbull","sequence":"additional","affiliation":[{"name":"Computer Science Department, Ithaca College, Ithaca, NY 14850"}]},{"given":"Gert","family":"Lanckriet","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering Department, University of California at San Diego, La Jolla, CA 92093; and"}]}],"member":"341","published-online":{"date-parts":[[2012,3,28]]},"reference":[{"key":"e_1_3_5_1_2","unstructured":"http:\/\/www.facebook.com\/press\/info.php?statistics February. (2012)."},{"key":"e_1_3_5_2_2","unstructured":"http:\/\/www.youtube.com\/t\/fact_sheet February. (2012)."},{"key":"e_1_3_5_3_2","unstructured":"http:\/\/www.apple.com\/itunes\/features\/ February. (2012)."},{"key":"e_1_3_5_4_2","first-page":"107","article-title":"The anatomy of a large-scale hypertextual web search engine","volume":"30","author":"Brin S","year":"1998","unstructured":"S Brin, L Page, The anatomy of a large-scale hypertextual web search engine. Comput Netw 30, 107\u2013117 (1998).","journal-title":"Comput Netw"},{"key":"e_1_3_5_5_2","unstructured":"Securities and Exchange Commission Form S-1. http:\/\/www.sec.gov\/Archives\/edgar\/data\/1230276\/000119312511032963\/ds1.htm February. (2011)."},{"key":"e_1_3_5_6_2","unstructured":"http:\/\/mturk.amazon.com 2012)."},{"key":"e_1_3_5_7_2","unstructured":"http:\/\/www.galaxyzoo.org 2012)."},{"key":"e_1_3_5_8_2","doi-asserted-by":"crossref","first-page":"756","DOI":"10.1038\/nature09304","article-title":"Predicting protein structures with a multiplayer online game","volume":"466","author":"Cooper S","year":"2010","unstructured":"S Cooper, et al., Predicting protein structures with a multiplayer online game. Nature 466, 756\u2013760 (2010).","journal-title":"Nature"},{"key":"e_1_3_5_9_2","doi-asserted-by":"crossref","first-page":"1465","DOI":"10.1126\/science.1160379","article-title":"reCAPTCHA: human-based character recognition via web security measures","volume":"321","author":"von Ahn L","year":"2008","unstructured":"L von Ahn, B Maurer, C McMillen, D Abraham, M Blum, reCAPTCHA: human-based character recognition via web security measures. Science 321, 1465\u20131468 (2008).","journal-title":"Science"},{"key":"e_1_3_5_10_2","volume-title":"In 23rd Conference on Neural Information Processing Systems (NIPS)","author":"Whitehill J","year":"2009","unstructured":"J Whitehill, P Ruvolo, J Bergsma, T Wu, J Movellan, Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In 23rd Conference on Neural Information Processing Systems (NIPS) (MIT Press, MA, 2009)."},{"key":"e_1_3_5_11_2","volume-title":"In 22nd International Conference on Human Factors in Computing Systems","author":"von Ahn L","year":"2004","unstructured":"L von Ahn, L Dabbish, Labeling images with a computer game. In 22nd International Conference on Human Factors in Computing Systems (ACM SIGCHI, 2004)."},{"key":"e_1_3_5_12_2","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1109\/MC.2006.196","article-title":"Games with a purpose","volume":"39","author":"von Ahn L","year":"2006","unstructured":"L von Ahn, Games with a purpose. IEEE Computer Magazine 39, 92\u201394 (2006).","journal-title":"IEEE Computer Magazine"},{"key":"e_1_3_5_13_2","volume-title":"In 27th International Conference on Human Factors in Computing Systems","author":"Law E","year":"2009","unstructured":"E Law, L von Ahn, Input-agreement: a new mechanism for collecting data using human computation games. In 27th International Conference on Human Factors in Computing Systems (ACM SIGCHI, 2009)."},{"key":"e_1_3_5_14_2","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1023\/B:VISI.0000013087.49260.fb","article-title":"Robust real-time face detection","volume":"57","author":"Viola P","year":"2004","unstructured":"P Viola, M Jones, Robust real-time face detection. Int J Comput Vision 57, 137\u2013154 (2004).","journal-title":"Int J Comput Vision"},{"key":"e_1_3_5_15_2","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1080\/09298210802479300","article-title":"A web-based game for collecting music metadata","volume":"37","author":"Mandel M","year":"2008","unstructured":"M Mandel, D Ellis, A web-based game for collecting music metadata. J New Music Res 37, 151\u2013165 (2008).","journal-title":"J New Music Res"},{"key":"e_1_3_5_16_2","doi-asserted-by":"crossref","first-page":"300","DOI":"10.1145\/3166.3170","article-title":"Designing for usability: key principles and what designers think","volume":"28","author":"Gould J","year":"1985","unstructured":"J Gould, C Lewis, Designing for usability: key principles and what designers think. Commun ACM 28, 300\u2013311 (1985).","journal-title":"Commun ACM"},{"key":"e_1_3_5_17_2","unstructured":"ed D Rohrl (International Game Developers Association Casual Games White Paper. 2008)."},{"key":"e_1_3_5_18_2","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1111\/j.1467-9280.2006.01691.x","article-title":"Message in a ballad: the role of music preferences in interpersonal perception","volume":"17","author":"Rentfrow P","year":"2006","unstructured":"P Rentfrow, S Gosling, Message in a ballad: the role of music preferences in interpersonal perception. Psychol Sci 17, 236\u2013242 (2006).","journal-title":"Psychol Sci"},{"key":"e_1_3_5_19_2","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1109\/TASL.2007.913750","article-title":"Semantic annotation and retrieval of music and sound effects","volume":"16","author":"Turnbull D","year":"2008","unstructured":"D Turnbull, L Barrington, D Torres, G Lanckriet, Semantic annotation and retrieval of music and sound effects. IEEE T Acoust Speech 16, 467\u2013476 (2008).","journal-title":"IEEE T Acoust Speech"},{"key":"e_1_3_5_20_2","volume-title":"In 11th International Society for Music Information Retrieval (ISMIR) Conference","author":"Coviello E","year":"2010","unstructured":"E Coviello, L Barrington, GRG Lanckriet, AB Chan, Automatic music tagging with time series models. In 11th International Society for Music Information Retrieval (ISMIR) Conference (International Society for Music Information Retrieval , Netherlands, 2010)."},{"key":"e_1_3_5_21_2","volume-title":"In 21st Conference on Neural Information Processing Systems (NIPS)","author":"Eck D","year":"2007","unstructured":"D Eck, P Lamere, T Bertin-Mahieux, S Green, Automatic generation of social tags for music recommendation. In 21st Conference on Neural Information Processing Systems (NIPS) (MIT Press, MA, 2007)."},{"key":"e_1_3_5_22_2","volume-title":"In 9th International Society for Music Information Retrieval (ISMIR) Conference","author":"Mandel M","year":"2008","unstructured":"M Mandel, D Ellis, Multiple-instance learning for music information retrieval. In 9th International Society for Music Information Retrieval (ISMIR) Conference (International Society for Music Information Retrieval , PA, 2008)."},{"key":"e_1_3_5_23_2","volume-title":"University of Wisconsin\u2014Madison: Computer Sciences Technical Report","author":"Settles B","year":"2010","unstructured":"B Settles, Active learning literature survey. University of Wisconsin\u2014Madison: Computer Sciences Technical Report, 2010)."},{"key":"e_1_3_5_24_2","article-title":"Audio tag classification","author":"Downie JS","year":"2008","unstructured":"JS Downie, Audio tag classification. Music Information Retrieval Evaluation eXchange (MIREX), http:\/\/music-ir.org\/mirex\/wiki\/2008:Audio_Tag_Classification_Results. (2008).","journal-title":"Music Information Retrieval Evaluation eXchange (MIREX)"},{"key":"e_1_3_5_25_2","volume-title":"In 11th International Conference on Machine Learning (ICML)","author":"Lewis D","year":"1994","unstructured":"D Lewis, J Catlett, Heterogeneous uncertainty sampling for supervised learning. In 11th International Conference on Machine Learning (ICML) (Morgan Kaufmann, CA, 1994)."},{"key":"e_1_3_5_26_2","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1613\/jair.295","article-title":"Active learning with statistical models","volume":"4","author":"Cohn DA","year":"1996","unstructured":"DA Cohn, Z Ghahramani, MI Jordan, Active learning with statistical models. Journal of Artificial Intelligence Results 4, 129\u2013145 (1996).","journal-title":"Journal of Artificial Intelligence Results"},{"key":"e_1_3_5_27_2","first-page":"203","article-title":"A probabilistic analysis of EM for mixtures of separated, spherical Gaussians","volume":"8","author":"Dasgupta S","year":"2007","unstructured":"S Dasgupta, LJ Schulman, A probabilistic analysis of EM for mixtures of separated, spherical Gaussians. J Mach Learn Res 8, 203\u2013226 (2007).","journal-title":"J Mach Learn Res"}],"updated-by":[{"DOI":"10.1073\/pnas.1205806109","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2012,5,7]],"date-time":"2012-05-07T00:00:00Z","timestamp":1336348800000}}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.1014748109","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,7]],"date-time":"2022-06-07T04:25:04Z","timestamp":1654575904000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/full\/10.1073\/pnas.1014748109"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,3,28]]},"references-count":27,"journal-issue":{"issue":"17","published-print":{"date-parts":[[2012,4,24]]}},"alternative-id":["10.1073\/pnas.1014748109"],"URL":"https:\/\/doi.org\/10.1073\/pnas.1014748109","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,3,28]]},"assertion":[{"value":"2012-03-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}