{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T04:20:11Z","timestamp":1692764411353},"reference-count":24,"publisher":"Institute of Electronics, Information and Communications Engineers (IEICE)","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IEICE Trans. Inf. & Syst."],"published-print":{"date-parts":[[2017]]},"DOI":"10.1587\/transinf.2016dap0024","type":"journal-article","created":{"date-parts":[[2017,3,31]],"date-time":"2017-03-31T22:23:57Z","timestamp":1490999037000},"page":"750-757","source":"Crossref","is-referenced-by-count":0,"title":["A Novel Label Aggregation with Attenuated Scores for Ground-Truth Identification of Dataset Annotation with Crowdsourcing"],"prefix":"10.1587","volume":"E100.D","author":[{"given":"Ratchainant","family":"THAMMASUDJARIT","sequence":"first","affiliation":[{"name":"Faculty of Information and Communication Technology, Mahidol University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anon","family":"PLANGPRASOPCHOK","sequence":"additional","affiliation":[{"name":"National Electronics and Computer Technology Center"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Charnyote","family":"PLUEMPITIWIRIYAWEJ","sequence":"additional","affiliation":[{"name":"Faculty of Information and Communication Technology, Mahidol University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"532","reference":[{"key":"1","doi-asserted-by":"crossref","unstructured":"[1] R. Snow, B. O'Connor, D. Jurafsky, and A.Y. Ng, \u201cCheap and fast-but is it good? Evaluation non-expert annotiations for natural language tasks,\u201d Proc. Conference on Empirical Methods in Natural Language Processing, pp.254-263, 2008.","DOI":"10.3115\/1613715.1613751"},{"key":"2","doi-asserted-by":"crossref","unstructured":"[2] N. Littlestone and M. Warmuth, \u201cThe Weighted Majority Algorithm,\u201d Information and Computation, vol.108, pp.212-261, 1994.","DOI":"10.1006\/inco.1994.1009"},{"key":"3","doi-asserted-by":"crossref","unstructured":"[3] P.-Y. Hsueh, P. Melville, and V. Sindhwani, \u201cData quality from crowdsourcing: a study of annotation selection criteria,\u201d Proc. NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pp.27-35, 2009.","DOI":"10.3115\/1564131.1564137"},{"key":"4","unstructured":"[4] V.C. Raykar, S. Yu, L.H. Zhao, G.H. Valadez, C. Florin, L. Bogoni, and L. Moy, \u201cLearning From Crowds,\u201d J. Mach. Learn. Res., vol.11, pp.1297-1322, 2010."},{"key":"5","unstructured":"[5] X. Wu, W. Fan, and Y. Yu, \u201cSembler: Ensembling Crowd Sequential Labeling for Improved Quality,\u201d Proc. 26th Association for the Advancement of Artificial Intelligence, pp.1713-1719, 2012."},{"key":"6","doi-asserted-by":"crossref","unstructured":"[6] F. Rodrigues, F. Pereira, and B. Ribeiro, \u201cLearning from multiple annotators: Distinguishing good from random labelers,\u201d Pattern Recognit. Lett., vol.34, no.12, pp.1428-1436, 2013.","DOI":"10.1016\/j.patrec.2013.05.012"},{"key":"7","unstructured":"[7] D. Hovy, T. Berg-Kirkpatrick, A. Vaswani, and E. Hovy, \u201cLearning Whom to Trust with MACE,\u201d Naacl-Hlt '13, vol.3, no.June, pp.1120-1130, 2013."},{"key":"8","doi-asserted-by":"crossref","unstructured":"[8] D. Hovy, B. Plank, and A. S\u00f8gaard, \u201cExperiments with crowdsourced re-annotation of a POS tagging data set,\u201d Proc. 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp.377-382, 2014.","DOI":"10.3115\/v1\/P14-2062"},{"key":"9","doi-asserted-by":"crossref","unstructured":"[9] M. Georgescu, \u201cAggregation of Crowdsourced Labels Based on Worker History Categories and Subject Descriptors,\u201d Proc. 4th International Conference on Web Intelligence, Mining, and Semantics, 2014.","DOI":"10.1145\/2611040.2611074"},{"key":"10","doi-asserted-by":"crossref","unstructured":"[10] M.J.y. Chung, M. Forbes, M. Cakmak, and R.P.N. Rao, \u201cAccelerating Imitation Learning through Crowdsourcing,\u201d Proc. IEEE International Conference on Robotics and Automation, pp.4777-4784, 2014.","DOI":"10.1109\/ICRA.2014.6907558"},{"key":"11","unstructured":"[11] D. Zhou, J. Platt, S. Basu, and Y. Mao, \u201cLearning from the wisdom of crowds by minimax entropy,\u201d Advances in Neural Information Processing Systems 25, pp.2204-2212, 2012."},{"key":"12","unstructured":"[12] J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan, \u201cWhose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise,\u201d Advances in Neural Information Processing Systems, pp.1-9, 2009."},{"key":"13","doi-asserted-by":"crossref","unstructured":"[13] G. Demartini, D.E. Difallah, and P. Cudr\u00e9-Mauroux, \u201cZencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking,\u201d Proc. World Wide Web Conference Committee (IW3C2), pp.469-478, 2012.","DOI":"10.1145\/2187836.2187900"},{"key":"14","unstructured":"[14] R. Thammasudjarit, A. Plangprasopchok, and C. Pluempitiwiriyawej, \u201cExploring Consensus Sequential Pattern For Ground Truth Identification in Corpus Annotation With Crowdsourcing,\u201d Symposium on Natural Language Processing, 2016."},{"key":"15","doi-asserted-by":"crossref","unstructured":"[15] A.P. Dempster, N.M. Laird, and D.B. Rubin, \u201cMaximum likelihood from incomplete data via the EM algorithm,\u201d J. Royal Statistical Society Series B Methodological, vol.39, no.1, pp.1-38, 1977.","DOI":"10.1111\/j.2517-6161.1977.tb01600.x"},{"key":"16","doi-asserted-by":"crossref","unstructured":"[16] A.P. Dawid and A.M. Skene, \u201cMaximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm,\u201d Applied Statistics, vol.28, no.1, p.20, 1979.","DOI":"10.2307\/2346806"},{"key":"17","doi-asserted-by":"crossref","unstructured":"[17] Y. Tian and J. Zhu, \u201cLearning from crowds in the presence of schools of thought,\u201d Proc. 18th ACM SIGKDD international conference on Knowledge discovery and data mining-KDD '12, New York, New York, USA, p.226, ACM Press, 2012.","DOI":"10.1145\/2339530.2339571"},{"key":"18","unstructured":"[18] Q. Liu, J. Peng, and A. Ihler, \u201cVariational Inference for Crowdsourcing,\u201d Proc. Advances in Neural Information Processing Systems, pp.701-709, 2012."},{"key":"19","unstructured":"[19] X. Chen, Q. Lin, and D. Zhou, \u201cOptimistic knowledge gradient policy for optimal budget allocation in crowdsourcing,\u201d Proc. 30th International Conference on Machine Learning (ICML-13), pp.64-72, 2013."},{"key":"20","unstructured":"[20] D. Zhou, Q. Liu, J.C. Platt, and C. Meek, \u201cAggregating Ordinal Labels from Crowds by Minimax Conditional Entropy,\u201d Proc. 31st International Conference on Machine Learning, pp.262-270, 2014."},{"key":"21","unstructured":"[21] G. Leech, \u201cAdding Linguistic Annotation,\u201d in Developing Linguistic Corpora: a Guide to Good Practice, ed. M. Wynne, pp.17-29, 2005."},{"key":"22","unstructured":"[22] A.J. Viera and J.M. Garrett, \u201cUnderstanding interobserver agreement: The kappa statistic,\u201d Family Medicine, vol.37, no.5, pp.360-363, 2005."},{"key":"23","doi-asserted-by":"crossref","unstructured":"[23] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, \u201cImageNet: A large-scale hierarchical image database,\u201d 2009 IEEE Conference on Comput. Vis. Pattern Recognit., pp.2-9, 2009.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"24","doi-asserted-by":"crossref","unstructured":"[24] A. Sheshadri and M. Lease, \u201cSQUARE: A Benchmark for Research on Computing Crowd Consensus,\u201d Proc. AAAI Conference on Human Computation and Crowdsourcing, pp.156-164, 2013.","DOI":"10.1609\/hcomp.v1i1.13088"}],"container-title":["IEICE Transactions on Information and Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jstage.jst.go.jp\/article\/transinf\/E100.D\/4\/E100.D_2016DAP0024\/_pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,22]],"date-time":"2023-08-22T23:36:08Z","timestamp":1692747368000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jstage.jst.go.jp\/article\/transinf\/E100.D\/4\/E100.D_2016DAP0024\/_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017]]},"references-count":24,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2017]]}},"URL":"https:\/\/doi.org\/10.1587\/transinf.2016dap0024","relation":{},"ISSN":["0916-8532","1745-1361"],"issn-type":[{"value":"0916-8532","type":"print"},{"value":"1745-1361","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017]]}}}