{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:23:29Z","timestamp":1750307009769,"version":"3.41.0"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2012,9,1]],"date-time":"2012-09-01T00:00:00Z","timestamp":1346457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2012,9]]},"abstract":"<jats:p>Text mining has been very successful in extracting huge amounts of commonsense knowledge from data, but the extracted knowledge tends to be extremely noisy. Manual construction of knowledge repositories, on the other hand, tends to produce high-quality data in very small amounts. We propose an architecture to combine the best of both worlds: A game with a purpose that induces humans to clean up data automatically extracted by text mining. First, a text miner trained on a set of known commonsense facts harvests many more candidate facts from corpora. Then, a simple slot-machine-with-a-purpose game presents these candidate facts to the players for verification by playing. As a result, a new dataset of high precision commonsense knowledge is created. This combined architecture is able to produce significantly better commonsense facts than the state-of-the-art text miner alone. Furthermore, we report that bootstrapping (i.e., training the text miner on the output of the game) improves the subsequent performance of the text miner.<\/jats:p>","DOI":"10.1145\/2337542.2337544","type":"journal-article","created":{"date-parts":[[2012,10,12]],"date-time":"2012-10-12T20:56:02Z","timestamp":1350075362000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Bootstrapping a Game with a Purpose for Commonsense Collection"],"prefix":"10.1145","volume":"3","author":[{"given":"Ama\u00e7","family":"Herda\u011fdelen","sequence":"first","affiliation":[{"name":"CIMeC, University of Trento"}]},{"given":"Marco","family":"Baroni","sequence":"additional","affiliation":[{"name":"CIMeC, University of Trento"}]}],"member":"320","published-online":{"date-parts":[[2012,9]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csda.2010.11.018"},{"volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence. 2670--2676","author":"Banko M.","key":"e_1_2_1_2_1","unstructured":"Banko , M. , Cafarella , M. , Soderland , S. , Broadhead , M. , and Etzioni , O . 2007. Open information extraction from the web . In Proceedings of the International Joint Conference on Artificial Intelligence. 2670--2676 . Banko, M., Cafarella, M., Soderland, S., Broadhead, M., and Etzioni, O. 2007. Open information extraction from the web. In Proceedings of the International Joint Conference on Artificial Intelligence. 2670--2676."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli_a_00016"},{"volume-title":"Proceedings of the 15th Turkish Symposium on Artificial Intelligence and Neural Networks. 277--284","author":"Bicic E.","key":"e_1_2_1_4_1","unstructured":"Bicic , E. and Yuret , D . 2006. Clustering word pairs to answer analogy questions . In Proceedings of the 15th Turkish Symposium on Artificial Intelligence and Neural Networks. 277--284 . Bicic, E. and Yuret, D. 2006. Clustering word pairs to answer analogy questions. In Proceedings of the 15th Turkish Symposium on Artificial Intelligence and Neural Networks. 277--284."},{"key":"e_1_2_1_5_1","unstructured":"Buitelaar P. and Cimiano P. 2008. Bridging the Gap between Text and Knowledge. IOS Amsterdam. Buitelaar P. and Cimiano P. 2008. Bridging the Gap between Text and Knowledge . IOS Amsterdam."},{"key":"e_1_2_1_6_1","unstructured":"Canu S. Grandvalet Y. Guigue V. and Rakotomamonjy A. 2005. VM and kernel methods. Matlab Toolbox. Canu S. Grandvalet Y. Guigue V. and Rakotomamonjy A. 2005. VM and kernel methods. Matlab Toolbox."},{"volume-title":"Proceedings of the 24th Conference on Artificial Intelligence.","author":"Carlson A.","key":"e_1_2_1_7_1","unstructured":"Carlson , A. , Betteridge , J. , Kisiel , B. , Settles , B. , Hruschka Jr ., E., and Mitchell , T . 2010. Toward an architecture for never-ending language learning . In Proceedings of the 24th Conference on Artificial Intelligence. Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka Jr., E., and Mitchell, T. 2010. Toward an architecture for never-ending language learning. In Proceedings of the 24th Conference on Artificial Intelligence."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/89086.89095"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2005.10.010"},{"volume-title":"Proceeedings of the Workshop on Creating Speech and Language Data With Amazon\u2019s Mechanical Turk. NAACL.","author":"Gordon J.","key":"e_1_2_1_11_1","unstructured":"Gordon , J. , Durme , B. V. , and Schubert , L . 2010. Evaluation of commonsense knowledge with mechanical turk . In Proceeedings of the Workshop on Creating Speech and Language Data With Amazon\u2019s Mechanical Turk. NAACL. Gordon, J., Durme, B. V., and Schubert, L. 2010. Evaluation of commonsense knowledge with mechanical turk. In Proceeedings of the Workshop on Creating Speech and Language Data With Amazon\u2019s Mechanical Turk. NAACL."},{"volume-title":"Proceedings of the AAAI Workshop on Collaboratively-Built Knowledge Sources and Artificial Intelligence. ACM","author":"Gordon J.","key":"e_1_2_1_12_1","unstructured":"Gordon , J. , Van Durme , B. , and Schubert , L . 2010. Learning from the web: Extracting general world knowledge from noisy text . In Proceedings of the AAAI Workshop on Collaboratively-Built Knowledge Sources and Artificial Intelligence. ACM , New York. Gordon, J., Van Durme, B., and Schubert, L. 2010. Learning from the web: Extracting general world knowledge from noisy text. In Proceedings of the AAAI Workshop on Collaboratively-Built Knowledge Sources and Artificial Intelligence. ACM, New York."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2009.72"},{"volume-title":"Proceedings of the EACL GEMS Workshop. 33--40","author":"Herda\u01e7delen A.","key":"e_1_2_1_14_1","unstructured":"Herda\u01e7delen , A. and Baroni , M . 2009. BagPack: A general framework to represent semantic relations . In Proceedings of the EACL GEMS Workshop. 33--40 . Herda\u01e7delen, A. and Baroni, M. 2009. BagPack: A general framework to represent semantic relations. In Proceedings of the EACL GEMS Workshop. 33--40."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219745"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the ACM Conference on Computers and Human Interaction (CHI-08)","author":"Lieberman H.","year":"2008","unstructured":"Lieberman , H. 2008 . Usable AI requires commonsense knowledge . In Proceedings of the ACM Conference on Computers and Human Interaction (CHI-08) . ACM, New York. Lieberman, H. 2008. Usable AI requires commonsense knowledge. In Proceedings of the ACM Conference on Computers and Human Interaction (CHI-08). ACM, New York."},{"volume-title":"Proceedings of the IUI\u201907","author":"Lieberman H.","key":"e_1_2_1_17_1","unstructured":"Lieberman , H. , Smith , D. , and Teeters , A . 2007. Common consensus: A web-based game for collecting commonsense goals . In Proceedings of the IUI\u201907 . Lieberman, H., Smith, D., and Teeters, A. 2007. Common consensus: A web-based game for collecting commonsense goals. In Proceedings of the IUI\u201907."},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Liu H. and Singh P. 2004. Commonsense reasoning in and over natural language. In Knowledge-Based Intelligent Information and Engineering Systems Springer Berlin 293--306. Liu H. and Singh P. 2004. Commonsense reasoning in and over natural language. In Knowledge-Based Intelligent Information and Engineering Systems Springer Berlin 293--306.","DOI":"10.1007\/978-3-540-30134-9_40"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the National Conference on Artificial Intelligence.","volume":"20","author":"Matuszek C.","unstructured":"Matuszek , C. , Witbrock , M. , Kahlert , R. , Cabral , J. , Schneider , D. , Shah , P. , and Lenat , D . 2005. Searching for common sense: Populating CYC from the web . In Proceedings of the National Conference on Artificial Intelligence. Vol. 20 , 1430. Matuszek, C., Witbrock, M., Kahlert, R., Cabral, J., Schneider, D., Shah, P., and Lenat, D. 2005. Searching for common sense: Populating CYC from the web. In Proceedings of the National Conference on Artificial Intelligence. Vol. 20, 1430."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Teddington Conference on the Mechanization of Thought Processes. 75--91","author":"McCarthy J.","year":"1959","unstructured":"McCarthy , J. 1959 . Programs with common sense . In Proceedings of the Teddington Conference on the Mechanization of Thought Processes. 75--91 . McCarthy, J. 1959. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes. 75--91."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/345124.345145"},{"volume-title":"Proceedings of CogSci. 1623","author":"Quesada J.","key":"e_1_2_1_22_1","unstructured":"Quesada , J. , Mangalath , P. , and Kintsch , W . 2004. Analogy-making as predication using relational information and LSA vectors . In Proceedings of CogSci. 1623 . Quesada, J., Mangalath, P., and Kintsch, W. 2004. Analogy-making as predication using relational information and LSA vectors. In Proceedings of CogSci. 1623."},{"key":"e_1_2_1_23_1","unstructured":"Quinn A. J. and Bederson B. B. 2009. A taxonomy of distributed human computation. Tech. rep. HCIL-2009-23 University of Maryland College Park. Quinn A. J. and Bederson B. B. 2009. A taxonomy of distributed human computation. Tech. rep. HCIL-2009-23 University of Maryland College Park."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557914.1557948"},{"volume-title":"Proceedings of the AAAI Workshop on Contexts and Ontologies: Theory, Practice and Applications.","author":"Ramachandran D.","key":"e_1_2_1_25_1","unstructured":"Ramachandran , D. , Reagan , P. , and Goolsbey , K . 2005. First-orderized research CYC: Expressivity and efficiency in a common-sense ontology . In Proceedings of the AAAI Workshop on Contexts and Ontologies: Theory, Practice and Applications. Ramachandran, D., Reagan, P., and Goolsbey, K. 2005. First-orderized research CYC: Expressivity and efficiency in a common-sense ontology. In Proceedings of the AAAI Workshop on Contexts and Ontologies: Theory, Practice and Applications."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.3115\/1119239.1119241"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.1995.1067"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the Workshop on Common Sense and Intelligent User Interfaces.","author":"Speer R.","year":"2007","unstructured":"Speer , R. 2007 . Open mind commons: An inquisitive approach to learning common sense . In Proceedings of the Workshop on Common Sense and Intelligent User Interfaces. Speer, R. 2007. Open mind commons: An inquisitive approach to learning common sense. In Proceedings of the Workshop on Common Sense and Intelligent User Interfaces."},{"volume-title":"Proceedings of the 23rd International Florida Artificial Intelligence Research Society Conference (FLAIRS). 104--109","author":"Speer R.","key":"e_1_2_1_29_1","unstructured":"Speer , R. , Havasi , C. , and Surana , H . 2010. Using verbosity: Common sense data from games with a purpose . In Proceedings of the 23rd International Florida Artificial Intelligence Research Society Conference (FLAIRS). 104--109 . Speer, R., Havasi, C., and Surana, H. 2010. Using verbosity: Common sense data from games with a purpose. In Proceedings of the 23rd International Florida Artificial Intelligence Research Society Conference (FLAIRS). 104--109."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1597735.1597751"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.3115\/1220175.1220215"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2006.32.3.379"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/1599081.1599195"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-005-0913-1"},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Turney P. Littman M. Bigham J. and Shnayder V. 2003. Combining independent modules in lexical multiple-choice problems. In Recent Advances in Natural Language Processing III: Selected Papers from RANLP. 101--110. Turney P. Littman M. Bigham J. and Shnayder V. 2003. Combining independent modules in lexical multiple-choice problems. In Recent Advances in Natural Language Processing III: Selected Papers from RANLP . 101--110.","DOI":"10.1075\/cilt.260.11tur"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the AAAI Spring Symposium on Knowledge Collection from Volunteer Contributors (KCVC\u201905)","author":"Vanderwende L.","year":"2005","unstructured":"Vanderwende , L. 2005 . Volunteers created the web . In Proceedings of the AAAI Spring Symposium on Knowledge Collection from Volunteer Contributors (KCVC\u201905) . Vanderwende, L. 2005. Volunteers created the web. In Proceedings of the AAAI Spring Symposium on Knowledge Collection from Volunteer Contributors (KCVC\u201905)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124784"},{"key":"e_1_2_1_39_1","unstructured":"Wright M. Chodzko J. and Luk D. 1996. HAL\u2019s Legacy: 2001\u2019s Computer as Dream and Reality. MIT Press 193--209. Wright M. Chodzko J. and Luk D. 1996. HAL\u2019s Legacy: 2001\u2019s Computer as Dream and Reality . MIT Press 193--209."},{"key":"e_1_2_1_40_1","volume-title":"-H","author":"Yu C.-H.","year":"2010","unstructured":"Yu , C.-H. and Chen , H . -H . 2010 . Commonsense knowledge mining from the web. In Proceedings of AAAI. Yu, C.-H. and Chen, H.-H. 2010. Commonsense knowledge mining from the web. In Proceedings of AAAI."}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2337542.2337544","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2337542.2337544","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T08:49:00Z","timestamp":1750236540000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2337542.2337544"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,9]]},"references-count":39,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2012,9]]}},"alternative-id":["10.1145\/2337542.2337544"],"URL":"https:\/\/doi.org\/10.1145\/2337542.2337544","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2012,9]]},"assertion":[{"value":"2011-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2011-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2012-09-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}