{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T05:10:07Z","timestamp":1752469807718,"version":"3.41.2"},"reference-count":84,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T00:00:00Z","timestamp":1634083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,13]]},"abstract":"<jats:p>The life cycle of a peer-produced dataset follows the phases of growth, maturity, and decline. Paying crowdworkers is a proven method to collect and organize information into structured tables. However, these tabular representations may contain inaccuracies due to errors or data changing over time. Thus, the maturation phase of a dataset can benefit from the additional human examination. One method to improve accuracy is to recruit additional paid crowdworkers to verify and correct errors. An alternative method relies on unpaid contributors, collectively editing the dataset during regular use. We describe two case studies to examine different strategies for human verification and maintenance of in-flux tabular datasets. The first case study examines traditional micro-task verification strategies with paid crowdworkers, while the second examines long-term maintenance strategies with unpaid contributions from non-crowdworkers. Two paid verification strategies that produced more accurate corrections at a lower cost per accurate correction were redundant data collection followed by final verification from a trusted crowdworker and allowing crowdworkers to review any data freely. In the unpaid maintenance strategies, contributors provided more accurate corrections when asked to review data matching their interests. This research identifies considerations and future approaches to collectively improving information accuracy and longevity of tabular information.<\/jats:p>","DOI":"10.1145\/3479592","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T02:30:11Z","timestamp":1634610611000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Case Studies on the Motivation and Performance of Contributors Who Verify and Maintain In-Flux Tabular Datasets"],"prefix":"10.1145","volume":"5","author":[{"given":"Shaun","family":"Wallace","sequence":"first","affiliation":[{"name":"Brown University, Providence, RI, USA"}]},{"given":"Alexandra","family":"Papoutsaki","sequence":"additional","affiliation":[{"name":"Pomona College, Claremont, CA, USA"}]},{"given":"Neilly H.","family":"Tan","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"Hua","family":"Guo","sequence":"additional","affiliation":[{"name":"Twitter Inc., San Francisco, CA, USA"}]},{"given":"Jeff","family":"Huang","sequence":"additional","affiliation":[{"name":"Brown University, Providence, RI, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Bryan Alexander. 2020. The little spreadsheet that could and did: crowdsourcing COVID-19 higher education data and stories. https:\/\/bryanalexander.org\/research-topics\/the-little-spreadsheet-that-could-and-did-crowdsourcing-covid-19-higher-education-data-and-stories\/. (Accessed on 04\/12\/2021)."},{"key":"e_1_2_1_2_1","volume-title":"Intrinsic and extrinsic motivation. The review of economic studies","author":"Benabou Roland","year":"2003","unstructured":"Roland Benabou and Jean Tirole. 2003. Intrinsic and extrinsic motivation. The review of economic studies, Vol. 70, 3 (2003), 489--520."},{"key":"e_1_2_1_3_1","volume-title":"http:\/\/csrankings.org\/ Retrieved","author":"Berger Emery D.","year":"2020","unstructured":"Emery D. Berger. 2020. CSRankings. http:\/\/csrankings.org\/ Retrieved April 27, 2020 from"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441955.2441979"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380039"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702329"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2016.2715"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300733"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359164"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053359"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1216295.1216309"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3148148"},{"key":"e_1_2_1_14_1","volume-title":"Nebraska Symposium on Motivation: Perspectives on Motivation","volume":"38","author":"Deci Edward L","year":"1991","unstructured":"Edward L Deci and Richard M Ryan. 1991. A motivational approach to self: Integration in personality. In Nebraska Symposium on Motivation: Perspectives on Motivation, Vol. 38 (1991), 237--288."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924421.1924442"},{"volume-title":"Proc","author":"Dumitrache Anca","key":"e_1_2_1_16_1","unstructured":"Anca Dumitrache. 2015. Crowdsourcing disagreement for collecting semantic annotation. In Proc. ESWC. Springer, New York, NY, USA, 701--710."},{"key":"e_1_2_1_17_1","volume-title":"Chi, and Steven Dow","author":"Egelman Serge","year":"2014","unstructured":"Serge Egelman, Ed H. Chi, and Steven Dow. 2014. Ways of Knowing in HCI .Springer New York, New York, NY, Chapter Crowdsourcing in HCI Research, 267--289."},{"volume-title":"Proceedings of the 11th AAAI Conference on Human Computation (AAAIWS'11--11)","author":"Faridani Siamak","key":"e_1_2_1_18_1","unstructured":"Siamak Faridani, Bj\u00f6rn Hartmann, and Panagiotis G. Ipeirotis. 2011. What's the Right Price? Pricing Tasks for Finishing on Time. In Proceedings of the 11th AAAI Conference on Human Computation (AAAIWS'11--11). AAAI, Menlo Park, CA, USA, 26--31. http:\/\/dl.acm.org\/citation.cfm?id=2908698.2908703"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313629"},{"key":"e_1_2_1_20_1","volume-title":"Understanding the complementary nature of paid and volunteer crowds for content creation. Avances en Interacci\u00f3n Humano-Computadora","author":"Flores-Saviaga Claudia","year":"2020","unstructured":"Claudia Flores-Saviaga, Ricardo Granados, Liliana Savage, Lizbeth Escobedo, and Saiph Savage. 2020. Understanding the complementary nature of paid and volunteer crowds for content creation. Avances en Interacci\u00f3n Humano-Computadora, Vol. 1, 1 (2020), 37--44."},{"key":"e_1_2_1_21_1","volume-title":"About that 1 billion Microsoft Office figure... https:\/\/www.zdnet.com\/article\/about-that-1-billion-microsoft-office-figure\/. [Online","author":"Foley Mary Jo","year":"2020","unstructured":"Mary Jo Foley. 2010. About that 1 billion Microsoft Office figure... https:\/\/www.zdnet.com\/article\/about-that-1-billion-microsoft-office-figure\/. [Online; accessed 2020-04--20]."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702443"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824062"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824122"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983740"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1093\/cje\/bet075"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3295750.3298921"},{"key":"e_1_2_1_28_1","volume-title":"Wikipedia","volume":"20","author":"Hill Benjamin Mako","year":"2020","unstructured":"Benjamin Mako Hill and Aaron Shaw. 2020. Wikipedia and the End of Open Collaboration. Wikipedia, Vol. 20 (2020)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.mcm.2012.01.006"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741102"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3020165.3020178"},{"volume-title":"HCOMP","author":"Kenneth Huang Ting-Hao","key":"e_1_2_1_32_1","unstructured":"Ting-Hao Kenneth Huang and Jeffrey P Bigham. 2017. A 10-Month-Long Deployment Study of On-Demand Recruiting for Low-Latency Crowdsourcing.. In HCOMP. AAAI, Menlo Park, CA, USA, 61--70."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2017.0750"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v11i1.14900"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1525\/cmr.2015.57.4.63"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314043"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518927"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783367"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.3163\/1536-5050.99.4.010"},{"key":"e_1_2_1_44_1","volume-title":"Alfred Benedikt Brendel, and Simon Trang","author":"Lichtenberg Sascha","year":"2020","unstructured":"Sascha Lichtenberg, Tim-Benjamin Lembcke, Mattheus Brening, Alfred Benedikt Brendel, and Simon Trang. 2020. Can Gamification lead to Increase Paid Crowdworkers Output?. In Wirtschaftsinformatik (Zentrale Tracks). 1188--1202."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.25300\/MISQ\/2019\/14439"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v1i1.13075"},{"key":"e_1_2_1_47_1","volume-title":"How I Learned to Stop Worrying and Love the Crowd. https:\/\/www.youtube.com\/watch?v=FL9Q43zO1BQ (Online","author":"Marcus Adam","year":"2020","unstructured":"Adam Marcus. 2013. How I Learned to Stop Worrying and Love the Crowd. https:\/\/www.youtube.com\/watch?v=FL9Q43zO1BQ (Online; accessed on 2020-04--20)."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11115-020-00478-1"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1037\/per0000191"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025687"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753587"},{"volume-title":"Motivation of workers on microtask crowdsourcing platforms","author":"Naderi Babak","key":"e_1_2_1_53_1","unstructured":"Babak Naderi. 2018. Motivation of workers on microtask crowdsourcing platforms .Springer, New York, NY, USA."},{"volume-title":"SemTab@ ISWC","author":"Nguyen Phuc","key":"e_1_2_1_54_1","unstructured":"Phuc Nguyen, Ikuya Yamada, Natthawut Kertkeidkachorn, Ryutaro Ichise, and Hideaki Takeda. 2020. MTab4Wikidata at SemTab 2020: Tabular Data Annotation with Wikidata.. In SemTab@ ISWC. ACM, New York, NY, USA, 86--95."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2615569.2615644"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/287893.287899"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13230"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3020165.3020170"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610503"},{"key":"e_1_2_1_60_1","unstructured":"Wikimedia Project. 2021. Wikipedia and Wikidata Tools - Meta. https:\/\/meta.wikimedia.org\/wiki\/Wikipedia_and_Wikidata_Tools. (Accessed on 07\/10\/2021)."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376403"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998267"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.5555\/2208923.2208929"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675246"},{"volume-title":"Gamification in education and business","author":"Richter Ganit","key":"e_1_2_1_66_1","unstructured":"Ganit Richter, Daphne R Raban, and Sheizaf Rafaeli. 2015. Studying gamification: the effect of rewards and incentives on motivation. In Gamification in education and business. Springer, New York, NY, USA, 21--46."},{"key":"e_1_2_1_67_1","volume-title":"Fifth International AAAI Conference on Weblogs and Social Media. AAAI","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Fifth International AAAI Conference on Weblogs and Social Media. AAAI, Menlo Park, CA, USA, 321--328."},{"key":"e_1_2_1_68_1","volume-title":"Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist","author":"Ryan Richard M","year":"2000","unstructured":"Richard M Ryan and Edward L Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, Vol. 55, 1 (2000), 68."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v5i1.13301"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819985"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0069958"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376783"},{"volume-title":"ISWC Satellites","author":"Szekely Pedro A","key":"e_1_2_1_73_1","unstructured":"Pedro A Szekely, Daniel Garijo, Jay Pujara, Divij Bhatia, and Jiasheng Wu. 2019. T2WML: A Cell-Based Language to Map Tables into Wikidata Records.. In ISWC Satellites. ACM, New York, NY, USA, 45--48."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2629489"},{"key":"e_1_2_1_75_1","volume-title":"Model for Tabular Data and Metadata on the Web. https:\/\/www.w3.org\/TR\/tabular-data-model\/. [Online","author":"C.","year":"2020","unstructured":"W3C. 2015. Model for Tabular Data and Metadata on the Web. https:\/\/www.w3.org\/TR\/tabular-data-model\/. [Online; accessed 2020-04--20]."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415243"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v5i1.13300"},{"key":"e_1_2_1_78_1","volume-title":"Do Monetary Incentives Create a Spillover Effect on Free Knowledge Contribution? Evidence from a Natural Experiment. Evidence from a Natural Experiment (June 25","author":"Wang Jing","year":"2018","unstructured":"Jing Wang, Gen Li, and Kai-Lung Hui. 2018. Do Monetary Incentives Create a Spillover Effect on Free Knowledge Contribution? Evidence from a Natural Experiment. Evidence from a Natural Experiment (June 25, 2018) (2018), 1--24."},{"key":"e_1_2_1_79_1","volume-title":"2011 IEEE Seventh International Conference on. IEEE","author":"Wiggins Andrea","year":"2011","unstructured":"Andrea Wiggins, Greg Newman, Robert D Stevenson, and Kevin Crowston. 2011. Mechanisms for data quality and validation in citizen science. In e-Science Workshops (eScienceW), 2011 IEEE Seventh International Conference on. IEEE, Piscataway, NJ, USA, 14--19."},{"key":"e_1_2_1_80_1","volume-title":"Assessing the value of coooperation in wikipedia. arXiv preprint cs\/0702140","author":"Wilkinson Dennis M","year":"2007","unstructured":"Dennis M Wilkinson and Bernardo A Huberman. 2007. Assessing the value of coooperation in wikipedia. arXiv preprint cs\/0702140 (2007), 1--14."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287098.3287104"},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics","author":"Zaidan Omar F","year":"2011","unstructured":"Omar F Zaidan and Chris Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, Stroudsburg, PA, USA, 1220--1229."},{"key":"e_1_2_1_83_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3372117","article-title":"Web Table Extraction, Retrieval, and Augmentation: A Survey","volume":"11","author":"Zhang Shuo","year":"2020","unstructured":"Shuo Zhang and Krisztian Balog. 2020. Web Table Extraction, Retrieval, and Augmentation: A Survey. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 11, 2 (2020), 1--35.","journal-title":"ACM Transactions on Intelligent Systems and Technology (TIST)"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741656"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479592","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3479592","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:56:48Z","timestamp":1752469008000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479592"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":84,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2021,10,13]]}},"alternative-id":["10.1145\/3479592"],"URL":"https:\/\/doi.org\/10.1145\/3479592","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2021,10,13]]},"assertion":[{"value":"2021-10-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}