{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,28]],"date-time":"2025-09-28T20:45:20Z","timestamp":1759092320731,"version":"3.37.3"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,5,12]],"date-time":"2021-05-12T00:00:00Z","timestamp":1620777600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,5,12]],"date-time":"2021-05-12T00:00:00Z","timestamp":1620777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"National Key R&D Program","award":["No.2017YFB1400100"],"award-info":[{"award-number":["No.2017YFB1400100"]}]},{"name":"SDNFSC","award":["No.ZR2018MF014"],"award-info":[{"award-number":["No.ZR2018MF014"]}]},{"name":"SPKR&DP","award":["No.ZR2019LZH008"],"award-info":[{"award-number":["No.ZR2019LZH008"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Sci. Eng."],"published-print":{"date-parts":[[2021,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Microtask crowdsourcing is a form of crowdsourcing in which work is decomposed into a set of small, self-contained tasks, which each can typically be completed in a matter of minutes. Due to the various capabilities and knowledge background of the voluntary participants on the Internet, the answers collected from the crowd are ambiguous and the final answer aggregation is challenging. In this process, the choice of quality control strategies is important for ensuring the quality of the crowdsourcing results. Previous work on answer estimation mainly used expectation\u2013maximization (EM) approach. Unfortunately, EM provides local optimal solutions and the estimated results will be affected by the initial value. In this paper, we extend the local optimal result of EM and propose an approximate global optimal algorithm for answer aggregation of crowdsourcing microtasks with binary answers. Our algorithm is expected to improve the accuracy of real answer estimation through further likelihood maximization. First, three worker quality evaluation models are presented based on static and dynamic methods, respectively, and the local optimal results are obtained based on the maximum likelihood estimation method. Then, a dominance ordering model (DOM) is proposed according to the known worker responses and worker categories for the specified crowdsourcing task to reduce the space of potential task-response sequence while retaining the dominant sequence. Subsequently, a Cut-point neighbor detection algorithm is designed to iteratively search for the approximate global optimal estimation in a reduced space, which works on the proposed dominance ordering model (DOM). We conduct extensive experiments on both simulated and real-world datasets, and the experimental results illustrate that the proposed approach can obtain better estimation results and has higher performance than regular EM-based algorithms.<\/jats:p>","DOI":"10.1007\/s41019-021-00164-2","type":"journal-article","created":{"date-parts":[[2021,5,12]],"date-time":"2021-05-12T16:03:19Z","timestamp":1620835399000},"page":"294-309","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Achieving Approximate Global Optimization of Truth Inference for Crowdsourcing Microtasks"],"prefix":"10.1007","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8262-8883","authenticated-orcid":false,"given":"Lizhen","family":"Cui","sequence":"first","affiliation":[]},{"given":"Jing","family":"Chen","sequence":"additional","affiliation":[]},{"given":"Wei","family":"He","sequence":"additional","affiliation":[]},{"given":"Hui","family":"Li","sequence":"additional","affiliation":[]},{"given":"Wei","family":"Guo","sequence":"additional","affiliation":[]},{"given":"Zhiyuan","family":"Su","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,5,12]]},"reference":[{"issue":"1","key":"164_CR1","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1007\/s00778-019-00568-7","volume":"29","author":"Y Tong","year":"2020","unstructured":"Tong Y, Zhou Z, Zeng Y, Chen L, Shahabi C (2020) Spatial crowdsourcing: a survey. VLDB J 29(1):217\u2013250. https:\/\/doi.org\/10.1007\/s00778-019-00568-7","journal-title":"VLDB J"},{"doi-asserted-by":"publisher","unstructured":"Lu J, Li W, Wang Q, Zhang Y (2020) Research on data quality control of crowdsourcing annotation: a survey. In: IEEE International conference on dependable, autonomic and secure computing, International conference on pervasive intelligence and computing, International conference on cloud and big data computing, International conference on cyber science and technology congress, DASC\/PiCom\/CBDCom\/CyberSciTech 2020, Calgary, AB, Canada, 17\u201322 August 2020, pp 201\u2013208. https:\/\/doi.org\/10.1109\/DASC-PICom-CBDCom-CyberSciTech49142.2020.00044","key":"164_CR2","DOI":"10.1109\/DASC-PICom-CBDCom-CyberSciTech49142.2020.00044"},{"key":"164_CR3","doi-asserted-by":"publisher","first-page":"110611","DOI":"10.1016\/j.jss.2020.110611","volume":"167","author":"SS Bhatti","year":"2020","unstructured":"Bhatti SS, Gao X, Chen G (2020) General framework, opportunities and challenges for crowdsourcing techniques: a comprehensive survey. J Syst Softw 167:110611. https:\/\/doi.org\/10.1016\/j.jss.2020.110611","journal-title":"J Syst Softw"},{"doi-asserted-by":"publisher","unstructured":"Das Sarma A, Parameswaran A, Widom J (2016) Towards globally optimal crowdsourcing quality management: the uniform worker setting. In: Proceedings of the 2016 international conference on management of data, pp 47\u201362. https:\/\/doi.org\/10.1145\/2882903.2882953","key":"164_CR4","DOI":"10.1145\/2882903.2882953"},{"doi-asserted-by":"publisher","unstructured":"Cui L, Chen J, He W, Li H, Guo W (2020) A pruned DOM-based iterative strategy for approximate global optimization in crowdsourcing microtasks. In: Wang X, Zhang R, Lee Y, Sun L, Moon Y (eds.), Web and Big Data\u20144th international joint conference, APWeb-WAIM 2020, Tianjin, China, 18\u201320 September 2020, Proceedings, Part I. Lecture Notes in Computer Science, vol 12317, pp 779\u2013793. https:\/\/doi.org\/10.1007\/978-3-030-60259-8_57","key":"164_CR5","DOI":"10.1007\/978-3-030-60259-8_57"},{"issue":"5","key":"164_CR6","doi-asserted-by":"publisher","first-page":"541","DOI":"10.14778\/3055540.3055547","volume":"10","author":"Y Zheng","year":"2017","unstructured":"Zheng Y, Li G, Li Y, Shan C, Cheng R (2017) Truth inference in crowdsourcing: is the problem solved? Proc VLDB Endow 10(5):541\u2013552. https:\/\/doi.org\/10.14778\/3055540.3055547","journal-title":"Proc VLDB Endow"},{"issue":"11","key":"164_CR7","doi-asserted-by":"publisher","first-page":"1495","DOI":"10.14778\/2350229.2350264","volume":"5","author":"CC Cao","year":"2012","unstructured":"Cao CC, She J, Tong Y, Chen L (2012) Whom to ask?: jury selection for decision making tasks on micro-blog services. Proc VLDB Endow 5(11):1495\u20131506","journal-title":"Proc VLDB Endow"},{"doi-asserted-by":"publisher","unstructured":"Franklin MJ, Kossmann D, Kraska T, Ramesh S, Xin R (2011) Crowddb: answering queries with crowdsourcing. In: Proceedings of the ACM SIGMOD international conference on management of data, SIGMOD 2011, Athens, Greece, 12\u201316 June 2011, pp 61\u201372. https:\/\/doi.org\/10.1145\/1989323.1989331","key":"164_CR8","DOI":"10.1145\/1989323.1989331"},{"issue":"1","key":"164_CR9","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1007\/s10044-002-0173-7","volume":"6","author":"LI Kuncheva","year":"2003","unstructured":"Kuncheva LI, Whitaker CJ, Shipp CA, Duin RP (2003) Limits on the majority vote accuracy in classifier fusion. Pattern Anal Appl 6(1):22\u201331. https:\/\/doi.org\/10.1007\/s10044-002-0173-7","journal-title":"Pattern Anal Appl"},{"issue":"2","key":"164_CR10","doi-asserted-by":"publisher","first-page":"109","DOI":"10.14778\/2535568.2448944","volume":"6","author":"A Marcus","year":"2012","unstructured":"Marcus A, Karger RD, Madden S, Miller R, Oh S (2012) Counting with the crowd. Proc VLDB Endow 6(2):109\u2013120. https:\/\/doi.org\/10.14778\/2535568.2448944","journal-title":"Proc VLDB Endow"},{"issue":"12","key":"164_CR11","doi-asserted-by":"publisher","first-page":"1990","DOI":"10.14778\/2367502.2367555","volume":"5","author":"H Park","year":"2012","unstructured":"Park H, Pang R, Parameswaran AG, Garcia-Molina H, Polyzotis N, Widom J (2012) Deco: a system for declarative crowdsourcing. Proc VLDB Endow 5(12):1990\u20131993. https:\/\/doi.org\/10.14778\/2367502.2367555","journal-title":"Proc VLDB Endow"},{"doi-asserted-by":"publisher","unstructured":"Yan T, Kumar V, Ganesan D (2010) Crowdsearch: exploiting crowds for accurate real-time image search on mobile phones. In: Proceedings of the 8th international conference on mobile systems, applications, and services (MobiSys 2010), San Francisco, California, USA, 15\u201318 June 2010, pp. 77\u201390. https:\/\/doi.org\/10.1145\/1814433.1814443","key":"164_CR12","DOI":"10.1145\/1814433.1814443"},{"unstructured":"Khattak FK, Salleb-Aouissi A (2011) Quality control of crowd labeling through expert evaluation. In: Proceedings of the NIPS 2nd workshop on computational social science and the wisdom of crowds, vol 2, p 5","key":"164_CR13"},{"doi-asserted-by":"crossref","unstructured":"Demartini G, Difallah DE, Cudr\u00e9-Mauroux P (2012) Zencrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In: Proceedings of the 21st international conference on World Wide Web, pp 469\u2013478","key":"164_CR14","DOI":"10.1145\/2187836.2187900"},{"issue":"10","key":"164_CR15","doi-asserted-by":"publisher","first-page":"1040","DOI":"10.14778\/2336664.2336676","volume":"5","author":"X Liu","year":"2012","unstructured":"Liu X, Lu M, Ooi BC, Shen Y, Wu S, Zhang M (2012) CDAS: a crowdsourcing data analytics system. Proc VLDB Endow 5(10):1040\u20131051. https:\/\/doi.org\/10.14778\/2336664.2336676","journal-title":"Proc VLDB Endow"},{"issue":"10","key":"164_CR16","doi-asserted-by":"publisher","first-page":"1040","DOI":"10.14778\/2336664.2336676","volume":"5","author":"X Liu","year":"2012","unstructured":"Liu X, Lu M, Ooi BC, Shen Y, Wu S, Zhang M (2012) Cdas: a crowdsourcing data analytics system. Proc VLDB Endow 5(10):1040\u20131051","journal-title":"Proc VLDB Endow"},{"unstructured":"Imamura H, Sato I, Sugiyama M (2018) Analysis of minimax error rate for crowdsourcing and its application to worker clustering model. In: Dy, JG, Krause A (eds.), Proceedings of the 35th international conference on machine learning, ICML 2018, Proceedings of machine learning research, Stockholmsm\u00e4ssan, Stockholm, Sweden, 10\u201315 July 2018, vol 80, pp 2152\u20132161","key":"164_CR17"},{"doi-asserted-by":"publisher","unstructured":"Li Q, Li Y, Gao J, Zhao B, Fan W, Han J (2014) Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. In: Dyreson CE, Li F, \u00d6zsu MT (eds.), International conference on management of data, SIGMOD 2014, Snowbird, UT, USA, 22\u201327 June 2014, pp. 1187\u20131198. ACM. https:\/\/doi.org\/10.1145\/2588555.2610509","key":"164_CR18","DOI":"10.1145\/2588555.2610509"},{"doi-asserted-by":"crossref","unstructured":"Ipeirotis PG, Provost F, Wang J (2010) Quality management on amazon mechanical turk. In: Proceedings of the ACM SIGKDD workshop on human computation, pp 64\u201367","key":"164_CR19","DOI":"10.1145\/1837885.1837906"},{"doi-asserted-by":"crossref","unstructured":"Raykar VC, Yu S, Zhao LH, Jerebko A, Florin C, Valadez GH, Bogoni L, Moy L (2009) Supervised learning from multiple experts: whom to trust when everyone lies a bit. In: Proceedings of the 26th annual international conference on machine learning, pp 889\u2013896","key":"164_CR20","DOI":"10.1145\/1553374.1553488"},{"doi-asserted-by":"publisher","unstructured":"Venanzi M, Guiver J, Kazai G, Kohli P, Shokouhi M (2014) Community-based Bayesian aggregation models for crowdsourcing. In: Chung C, Broder AZ, Shim K, Suel T (eds.), 23rd international World Wide Web conference, WWW\u201914, Seoul, Republic of Korea, 7\u201311 April, pp 155\u2013164. https:\/\/doi.org\/10.1145\/2566486.2567989","key":"164_CR21","DOI":"10.1145\/2566486.2567989"},{"issue":"1","key":"164_CR22","first-page":"20","volume":"28","author":"AP Dawid","year":"1979","unstructured":"Dawid AP, Skene AM (1979) Maximum likelihood estimation of observer error-rates using the EM algorithm. J R Stat Soc Ser C (Applied Statistics) 28(1):20\u201328","journal-title":"J R Stat Soc Ser C (Applied Statistics)"},{"doi-asserted-by":"publisher","unstructured":"Li Y, Rubinstein BIP, Cohn T (2019) Truth inference at scale: a Bayesian model for adjudicating highly redundant crowd annotations. In: Liu L, White RW, Mantrach A, Silvestri F, McAuley JJ, Baeza-Yates R, Zia L (eds.), The World Wide Web conference, WWW 2019, San Francisco, CA, USA, 13\u201317 May, pp 1028\u20131038. https:\/\/doi.org\/10.1145\/3308558.3313459","key":"164_CR23","DOI":"10.1145\/3308558.3313459"},{"doi-asserted-by":"publisher","unstructured":"Kurup AR, Sajeev GP (2019) Aggregating unstructured submissions for reliable answers in crowdsourcing systems. In: 9th international symposium on embedded computing and system design, ISED 2019, Kollam, India, 13\u201314 December, pp 1\u20137. https:\/\/doi.org\/10.1109\/ISED48680.2019.9096224","key":"164_CR24","DOI":"10.1109\/ISED48680.2019.9096224"},{"issue":"10","key":"164_CR25","doi-asserted-by":"publisher","first-page":"3091","DOI":"10.1007\/s00453-020-00715-5","volume":"82","author":"S Li","year":"2020","unstructured":"Li S, Xu J, Ye M (2020) Approximating global optimum for probabilistic truth discovery. Algorithmica 82(10):3091\u20133116. https:\/\/doi.org\/10.1007\/s00453-020-00715-5","journal-title":"Algorithmica"},{"doi-asserted-by":"publisher","unstructured":"Wu M, Li Q, Wang S, Hou J (2019) A subjectivity-aware algorithm for label aggregation in crowdsourcing. In: Qiu M (ed.), 2019 IEEE international conference on computational science and engineering, CSE 2019, and IEEE international conference on embedded and ubiquitous computing, EUC 2019, New York, NY, USA, 1\u20133 August, pp 373\u2013378. https:\/\/doi.org\/10.1109\/CSE\/EUC.2019.00077","key":"164_CR26","DOI":"10.1109\/CSE\/EUC.2019.00077"},{"doi-asserted-by":"publisher","unstructured":"Patwardhan M, Sainani A, Sharma R, Karande S, Ghaisas S (2018) Towards automating disambiguation of regulations: using the wisdom of crowds. ACM\/IEEE international conference, pp 850\u2013855. https:\/\/doi.org\/10.1145\/3238147.3240727","key":"164_CR27","DOI":"10.1145\/3238147.3240727"}],"container-title":["Data Science and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41019-021-00164-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41019-021-00164-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41019-021-00164-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,30]],"date-time":"2021-07-30T07:58:49Z","timestamp":1627631929000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41019-021-00164-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,12]]},"references-count":27,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,9]]}},"alternative-id":["164"],"URL":"https:\/\/doi.org\/10.1007\/s41019-021-00164-2","relation":{},"ISSN":["2364-1185","2364-1541"],"issn-type":[{"type":"print","value":"2364-1185"},{"type":"electronic","value":"2364-1541"}],"subject":[],"published":{"date-parts":[[2021,5,12]]},"assertion":[{"value":"20 January 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 April 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 April 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 May 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}