{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T22:33:43Z","timestamp":1767652423962,"version":"3.41.0"},"reference-count":19,"publisher":"Walter de Gruyter GmbH","issue":"1","funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/F00575X\/1"],"award-info":[{"award-number":["EP\/F00575X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["Doctoral Training Allowance"],"award-info":[{"award-number":["Doctoral Training Allowance"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["ES\/L011859\/1"],"award-info":[{"award-number":["ES\/L011859\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Crowdsourcing has revolutionised the way tasks can be completed but the process is frequently inefficient, costing practitioners time and money. This research investigates whether crowdsourcing can be optimised with a validation process, as measured by four criteria: quality; cost; noise; and speed. A validation model is described, simulated and tested on real data from an online crowdsourcing game to collect data about human language. Results show that by adding an agreement validation (or a like\/upvote) step fewer annotations are required, noise and collection time are reduced and quality may be improved.<\/jats:p>","DOI":"10.1515\/itit-2017-0020","type":"journal-article","created":{"date-parts":[[2018,2,28]],"date-time":"2018-02-28T22:25:59Z","timestamp":1519856759000},"page":"41-49","source":"Crossref","is-referenced-by-count":3,"title":["Optimising crowdsourcing efficiency: Amplifying human computation with validation"],"prefix":"10.1515","volume":"60","author":[{"given":"Jon","family":"Chamberlain","sequence":"first","affiliation":[{"name":"University of Essex , Wivenhoe Park , Colchester CO4 3SQ , UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Udo","family":"Kruschwitz","sequence":"additional","affiliation":[{"name":"University of Essex , Wivenhoe Park , Colchester CO4 3SQ , UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Massimo","family":"Poesio","sequence":"additional","affiliation":[{"name":"Queen Mary University , Mile End Rd , London E1 4NS , UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"374","published-online":{"date-parts":[[2018,2,28]]},"reference":[{"key":"2023033119475994629_j_itit-2017-0020_ref_001_w2aab3b7b2b1b6b1ab1b7b1Aa","doi-asserted-by":"crossref","unstructured":"R. Artstein and M. Poesio, Inter-coder agreement for computational linguistics, Computational Linguistics 34 (2008), 555\u2013596.10.1162\/coli.07-034-R2","DOI":"10.1162\/coli.07-034-R2"},{"key":"2023033119475994629_j_itit-2017-0020_ref_002_w2aab3b7b2b1b6b1ab1b7b2Aa","doi-asserted-by":"crossref","unstructured":"Yochai Benkler and Helen Nissenbaum, Commons-based Peer Production and Virtue, Journal of Political Philosophy 14 (2006), 394\u2013419.10.1111\/j.1467-9760.2006.00235.x","DOI":"10.1111\/j.1467-9760.2006.00235.x"},{"key":"2023033119475994629_j_itit-2017-0020_ref_003_w2aab3b7b2b1b6b1ab1b7b3Aa","doi-asserted-by":"crossref","unstructured":"Michael S. Bernstein, Greg Little, Robert C. Miller, Bj\u00f6rn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell and Katrina Panovich, Soylent: A Word Processor with a Crowd Inside, in: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (UIST\u201910), pp.\u2009313\u2013322, 2010.","DOI":"10.1145\/1866029.1866078"},{"key":"2023033119475994629_j_itit-2017-0020_ref_004_w2aab3b7b2b1b6b1ab1b7b4Aa","doi-asserted-by":"crossref","unstructured":"Amiangshu Bosu, Christopher S. Corley, Dustin Heaton, Debarshi Chatterji, Jeffrey C. Carver and Nicholas A. Kraft, Building Reputation in StackOverflow: An Empirical Investigation, in: Proceedings of the 10th Working Conference on Mining Software Repositories (MSR\u201913), pp.\u200989\u201392, 2013.","DOI":"10.1109\/MSR.2013.6624013"},{"key":"2023033119475994629_j_itit-2017-0020_ref_005_w2aab3b7b2b1b6b1ab1b7b5Aa","doi-asserted-by":"crossref","unstructured":"Daren C. Brabham, Crowdsourcing, The MIT Press, 2013.","DOI":"10.7551\/mitpress\/9693.001.0001"},{"key":"2023033119475994629_j_itit-2017-0020_ref_006_w2aab3b7b2b1b6b1ab1b7b6Aa","doi-asserted-by":"crossref","unstructured":"J. Chamberlain, Groupsourcing: Distributed Problem Solving Using Social Networks, in: Proceedings of 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201914), 2014.","DOI":"10.1609\/hcomp.v2i1.13162"},{"key":"2023033119475994629_j_itit-2017-0020_ref_007_w2aab3b7b2b1b6b1ab1b7b7Aa","unstructured":"J. Chamberlain, M. Poesio and U. Kruschwitz, Phrase Detectives Corpus 1.0 Crowdsourced Anaphoric Coreference, in: Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC\u201916), may 2016."},{"key":"2023033119475994629_j_itit-2017-0020_ref_008_w2aab3b7b2b1b6b1ab1b7b8Aa","doi-asserted-by":"crossref","unstructured":"Ido Guy, Inbal Ronen, Naama Zwerdling, Irena Zuyev-Grabovitch and Michal Jacovi, What is Your Organization \u2018Like\u2019?: A Study of Liking Activity in the Enterprise, in: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI \u201916, pp.\u20093025\u20133037, ACM, New York, NY, USA, 2016.","DOI":"10.1145\/2858036.2858540"},{"key":"2023033119475994629_j_itit-2017-0020_ref_009_w2aab3b7b2b1b6b1ab1b7b9Aa","doi-asserted-by":"crossref","unstructured":"Matthias Hirth, Tobias Ho\u00dffeld and Phuoc Tran-Gia, Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms, Mathematical and Computer Modelling 57 (2013), 2918\u20132932.10.1016\/j.mcm.2012.01.006","DOI":"10.1016\/j.mcm.2012.01.006"},{"key":"2023033119475994629_j_itit-2017-0020_ref_010_w2aab3b7b2b1b6b1ab1b7c10Aa","unstructured":"J. Howe, Crowdsourcing: Why the power of the crowd is driving the future of business, Crown Publishing Group, 2008."},{"key":"2023033119475994629_j_itit-2017-0020_ref_011_w2aab3b7b2b1b6b1ab1b7c11Aa","unstructured":"Faiza Khattak and Ansaf Salleb-Aouissi, Quality control of crowd labeling through expert evaluation, in: Proceedings of the 2nd Workshop on Computational Social Science and the Wisdom of Crowds (NIPS\u201911), 2011."},{"key":"2023033119475994629_j_itit-2017-0020_ref_012_w2aab3b7b2b1b6b1ab1b7c12Aa","doi-asserted-by":"crossref","unstructured":"Anand P. Kulkarni, Matthew Can and Bjoern Hartmann, Turkomatic: Automatic Recursive Task and Workflow Design for Mechanical Turk, in: CHI \u201911 Extended Abstracts on Human Factors in Computing Systems, pp.\u20092053\u20132058, ACM, New York, NY, USA, 2011.","DOI":"10.1145\/1979742.1979865"},{"key":"2023033119475994629_j_itit-2017-0020_ref_013_w2aab3b7b2b1b6b1ab1b7c13Aa","doi-asserted-by":"crossref","unstructured":"M. Lafourcade, A. Joubert and N. Le Brun, Games with a Purpose (GWAPS), John Wiley & Sons, 2015.","DOI":"10.1002\/9781119136309"},{"key":"2023033119475994629_j_itit-2017-0020_ref_014_w2aab3b7b2b1b6b1ab1b7c14Aa","doi-asserted-by":"crossref","unstructured":"Greg Little, Lydia B. Chilton, Max Goldman and Robert C. Miller, TurKit: Human Computation Algorithms on Mechanical Turk, in: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, UIST \u201910, pp.\u200957\u201366, ACM, New York, NY, USA, 2010.","DOI":"10.1145\/1866029.1866040"},{"key":"2023033119475994629_j_itit-2017-0020_ref_015_w2aab3b7b2b1b6b1ab1b7c15Aa","doi-asserted-by":"crossref","unstructured":"M. Poesio, J. Chamberlain, U. Kruschwitz, L. Robaldo and L. Ducceschi, Phrase Detectives: Utilizing Collective Intelligence for Internet-Scale Language Resource Creation, ACM Transactions on Interactive Intelligent Systems 3 (2013), 1\u201344.","DOI":"10.1145\/2448116.2448119"},{"key":"2023033119475994629_j_itit-2017-0020_ref_016_w2aab3b7b2b1b6b1ab1b7c16Aa","doi-asserted-by":"crossref","unstructured":"W. Rafelsberger and A. Scharl, Games with a purpose for social networking platforms, in: Proceedings of the 20th ACM Conference on Hypertext and Hypermedia, 2009.","DOI":"10.1145\/1557914.1557948"},{"key":"2023033119475994629_j_itit-2017-0020_ref_017_w2aab3b7b2b1b6b1ab1b7c17Aa","doi-asserted-by":"crossref","unstructured":"Victor S. Sheng, Foster Provost and Panagiotis G. Ipeirotis, Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers, in: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201908), pp.\u2009614\u2013622, 2008.","DOI":"10.1145\/1401890.1401965"},{"key":"2023033119475994629_j_itit-2017-0020_ref_018_w2aab3b7b2b1b6b1ab1b7c18Aa","doi-asserted-by":"crossref","unstructured":"Brian Sidlauskas, Calvin Bernard, Devin Bloom, Whitcomb Bronaugh, Michael Clementson and Richard P. Vari, Ichthyologists Hooked on Facebook, Science 332 (2011), 537.10.1126\/science.332.6029.537-c","DOI":"10.1126\/science.332.6029.537-c"},{"key":"2023033119475994629_j_itit-2017-0020_ref_019_w2aab3b7b2b1b6b1ab1b7c19Aa","doi-asserted-by":"crossref","unstructured":"R. Snow, B. O\u2019Connor, D. Jurafsky and A.\u2009Y. Ng, Cheap and fast - but is it good?: Evaluating non-expert annotations for natural language tasks, in: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP\u201908), 2008.","DOI":"10.3115\/1613715.1613751"}],"container-title":["it - Information Technology"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.degruyter.com\/view\/j\/itit.2018.60.issue-1\/itit-2017-0020\/itit-2017-0020.xml","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/itit-2017-0020\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/itit-2017-0020\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,2]],"date-time":"2025-07-02T04:20:40Z","timestamp":1751430040000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/itit-2017-0020\/html"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,1]]},"references-count":19,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2018,2,28]]},"published-print":{"date-parts":[[2018,3,1]]}},"alternative-id":["10.1515\/itit-2017-0020"],"URL":"https:\/\/doi.org\/10.1515\/itit-2017-0020","relation":{},"ISSN":["2196-7032","1611-2776"],"issn-type":[{"type":"electronic","value":"2196-7032"},{"type":"print","value":"1611-2776"}],"subject":[],"published":{"date-parts":[[2018,2,1]]}}}