{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T07:32:35Z","timestamp":1767339155505,"version":"build-2065373602"},"reference-count":32,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2024,8,15]],"date-time":"2024-08-15T00:00:00Z","timestamp":1723680000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Data labeling systems are designed to facilitate the training and validation of machine learning algorithms under the umbrella of crowdsourcing practices. The current paper presents a novel approach for designing a customized data labeling system, emphasizing two key aspects: an innovative payment mechanism for users and an efficient configuration of output results. The main problem addressed is the labeling of datasets where golden items are utilized to verify user performance and assure the quality of the annotated outputs. Our proposed payment mechanism is enhanced through a modified skip-based golden-oriented function that balances user penalties and prevents spam activities. Additionally, we introduce a comprehensive reporting framework to measure aggregated results and accuracy levels, ensuring the reliability of the labeling output. Our findings indicate that the proposed solutions are pivotal in incentivizing user participation, thereby reinforcing the applicability and profitability of newly launched labeling systems.<\/jats:p>","DOI":"10.3390\/a17080357","type":"journal-article","created":{"date-parts":[[2024,8,15]],"date-time":"2024-08-15T09:29:41Z","timestamp":1723714181000},"page":"357","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A System Design Perspective for Business Growth in a Crowdsourced Data Labeling Practice"],"prefix":"10.3390","volume":"17","author":[{"given":"Vahid","family":"Hajipour","sequence":"first","affiliation":[{"name":"Department of Industrial Engineering, West Tehran Branch, Islamic Azad University, Tehran 1468763785, Iran"},{"name":"Research Center, FANAP Co., Tehran 1657245030, Iran"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8023-0337","authenticated-orcid":false,"given":"Sajjad","family":"Jalali","sequence":"additional","affiliation":[{"name":"Research Center, FANAP Co., Tehran 1657245030, Iran"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2385-4781","authenticated-orcid":false,"given":"Francisco Javier","family":"Santos-Arteaga","sequence":"additional","affiliation":[{"name":"Department of Financial and Actuarial Economics & Statistics, Universidad Complutense de Madrid, 28223 Madrid, Spain"}]},{"given":"Samira","family":"Vazifeh Noshafagh","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering, Doctoral Programme in Materials, Mechatronics and Systems Engineering, University of Trento, 38123 Trento, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6900-3977","authenticated-orcid":false,"given":"Debora","family":"Di Caprio","sequence":"additional","affiliation":[{"name":"Department of Economics and Management, University of Trento, 38122 Trento, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1007\/s10796-015-9578-x","article-title":"Factors influencing the decision to crowdsource: A systematic literature review","volume":"18","author":"Thuan","year":"2016","journal-title":"Inf. Syst. Front."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"110611","DOI":"10.1016\/j.jss.2020.110611","article-title":"General framework, opportunities and challenges for crowdsourcing techniques: A Comprehensive survey","volume":"167","author":"Bhatti","year":"2020","journal-title":"J. Syst. Softw."},{"key":"ref_3","first-page":"200235","article-title":"AI-driven ensemble three machine learning to enhance digital marketing strategies in the food delivery business","volume":"18","author":"Yaiprasert","year":"2023","journal-title":"Intell. Syst. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"101494","DOI":"10.1016\/j.aei.2021.101494","article-title":"Achieving Knowledge-as-a-Service in IIoT-driven smart manufacturing: A crowdsourcing-based continuous enrichment method for Industrial Knowledge Graph","volume":"51","author":"Lyu","year":"2022","journal-title":"Adv. Eng. Inform."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1504\/IJVCM.2022.122161","article-title":"Crowdsourcing-based business model for online customer service: A case study","volume":"13","author":"Majava","year":"2022","journal-title":"Int. J. Value Chain Manag."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"105199","DOI":"10.1016\/j.knosys.2019.105199","article-title":"Effective Bayesian-network-based missing value imputation enhanced by crowdsourcing","volume":"190","author":"Ye","year":"2020","journal-title":"Knowl. -Based Syst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"100336","DOI":"10.1016\/j.patter.2021.100336","article-title":"Data and its (dis)contents: A survey of dataset development and use in machine learning research","volume":"2","author":"Paullada","year":"2021","journal-title":"Patterns"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/j.jss.2019.04.027","article-title":"A systematic literature review on crowdsourcing in software engineering","volume":"153","author":"Tosun","year":"2019","journal-title":"J. Syst. Softw."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1145\/3396863","article-title":"Approval Voting and Incentives in Crowdsourcing","volume":"8","author":"Shah","year":"2020","journal-title":"ACM Trans. Econ. Comput."},{"key":"ref_10","first-page":"1","article-title":"Double or nothing: Multiplicative incentive mechanisms for Crowdsourcing","volume":"17","author":"Shah","year":"2016","journal-title":"J. Mach. Learn. Res."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1111\/ijmr.12135","article-title":"Crowdsourcing: A Review and Suggestions for Future Research","volume":"20","author":"Ghezzi","year":"2018","journal-title":"Int. J. Manag. Rev."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"101001","DOI":"10.1016\/j.aei.2019.101001","article-title":"Crowdsourced reliable labeling of safety-rule violations on images of complex construction scenes for advanced vision-based workplace safety","volume":"42","author":"Wang","year":"2019","journal-title":"Adv. Eng. Inform."},{"key":"ref_13","first-page":"100132","article-title":"A web crowdsourcing framework for transfer learning and personalized Speech Emotion Recognition","volume":"6","author":"Vryzas","year":"2021","journal-title":"Mach. Learn. Appl."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"102110","DOI":"10.1016\/j.ijdrr.2021.102110","article-title":"An uncertainty-aware framework for reliable disaster damage assessment via crowdsourcing","volume":"55","author":"Khajwal","year":"2021","journal-title":"Int. J. Disaster Risk Reduct."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"He, X., Zhang, H., Yang, X., Guo, Y., and Bian, J. (2019, January 10\u201313). STAT: A Web-based Semantic Text Annotation Tool to Assist Building Mental Health Knowledge Base. Proceedings of the 2019 IEEE International Conference on Healthcare Informatics (ICHI), Xi\u2019an, China.","DOI":"10.1109\/ICHI.2019.8904503"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"He, X., Zhang, H., and Bian, J. (2020). User-centered design of a web-based crowdsourcing-integrated semantic text annotation tool for building a mental health knowledge base. J. Biomed. Inform., 110.","DOI":"10.1016\/j.jbi.2020.103571"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"103166","DOI":"10.1016\/j.ijhcs.2023.103166","article-title":"Engagement by Design Cards: A tool to involve designers and non-experts in the design of crowdsourcing initiatives","volume":"182","author":"Vasconcelos","year":"2024","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1016\/j.ijhcs.2017.04.005","article-title":"Gamified crowdsourcing: Conceptualization, literature review, and future agenda","volume":"106","author":"Morschheuser","year":"2017","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Hirth, M., Ho\u00dffeld, T., and Tran-Gia, P. (July, January 30). Cost-Optimal Validation Mechanisms and Cheat-Detection for Crowdsourcing Platforms. Proceedings of the 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, Seoul, Republic of Korea.","DOI":"10.1109\/IMIS.2011.91"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1109\/JSAC.2017.2659278","article-title":"Incentive Mechanism Design to Meet Task Criteria in Crowdsourcing: How to Determine Your Budget","volume":"35","author":"Wu","year":"2017","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Khan, A.R., and Garcia-Molina, H. (2017, January 14\u201319). CrowdDQS: Dynamic Question Selection in Crowdsourcing Systems. Proceedings of the 2017 ACM International Conference on Management of Data, Chicago, IL, USA.","DOI":"10.1145\/3035918.3064055"},{"key":"ref_22","first-page":"7","article-title":"Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions","volume":"51","author":"Daniel","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gadiraju, U., Kawase, R., Dietze, S., and Demartini, G. (2015, January 18\u201323). Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea.","DOI":"10.1145\/2702123.2702443"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1007\/s10462-016-9491-9","article-title":"Learning from crowdsourced labeled data: A survey","volume":"46","author":"Zhang","year":"2016","journal-title":"Artif. Intell. Rev."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B. (2012, January 11\u201315). Shepherding the crowd yields better work. Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, Seattle, DC, USA.","DOI":"10.1145\/2145204.2145355"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3172","DOI":"10.1109\/TNNLS.2018.2890148","article-title":"Crowdsourced Label Aggregation Using Bilayer Collaborative Clustering","volume":"30","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Baslyman, M., Amyot, D., and Mylopoulos, J. (2022). Reasoning about Confidence in Goal Satisfaction. Algorithms, 15.","DOI":"10.3390\/a15100343"},{"key":"ref_28","first-page":"200359","article-title":"IoT data sharing technology based on blockchain and federated learning algorithms","volume":"22","author":"Feng","year":"2024","journal-title":"Intell. Syst. Appl."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"100262","DOI":"10.1016\/j.iot.2020.100262","article-title":"IoT-based enterprise resource planning: Challenges, open issues, applications, architecture, and future research directions","volume":"11","author":"Tavana","year":"2020","journal-title":"Internet Things"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Stoykova, S., and Shakev, N. (2023). Artificial Intelligence for Management Information Systems: Opportunities, Challenges, and Future Directions. Algorithms, 16.","DOI":"10.3390\/a16080357"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"20827","DOI":"10.1007\/s11042-023-16325-6","article-title":"Crowdsourcing of labeling image objects: An online gamification application for data collection","volume":"83","author":"Bastanfard","year":"2024","journal-title":"Multimed. Tools Appl."},{"key":"ref_32","first-page":"1","article-title":"Crowdsourcing truth inference based on label confidence clustering","volume":"17","author":"Wu","year":"2023","journal-title":"ACM Trans. Knowl. Discov. Data"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/8\/357\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:37:16Z","timestamp":1760110636000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/8\/357"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,15]]},"references-count":32,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["a17080357"],"URL":"https:\/\/doi.org\/10.3390\/a17080357","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2024,8,15]]}}}