{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T17:49:35Z","timestamp":1777571375727,"version":"3.51.4"},"reference-count":43,"publisher":"National Library of Serbia","issue":"3","license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["ComSIS","COMPUT SCI INF SYST","COMPUT SCI INFORM SY","COMPUTER SCI INFORM","COMSIS J"],"published-print":{"date-parts":[[2022]]},"abstract":"<jats:p>This paper presents a novel web-based crowdsourcing platform for the assessment of the subjective and objective quality of experience (QoE) of the video service in the cloud-server environment. The user has the option to enter subjective QoE data for video service by filling out a web questionnaire. The objective QoE data of the cloud-server, network condition, and the user device is automatically captured by the crowdsourcing platform. Our proposed system collects both objective and subjective QoE simultaneously in real-time. The paper presents the key technologies used in the development of the platform and describes the functional requirements and design ideas of the system in detail. The system collects real-time comprehensive data to enhance the quality of the user experience to provide a valuable reference. The system is tested in a real-time environment and the test results are given in terms of the system performance. The crowdsourcing platform has new features of real-time network monitoring, the client device, and cloud monitoring, which currently has not been provided by existing web platforms and crowdsourcing frameworks. The results show that 1MB buffer is filled 100% very soon after starting watching videos from the crowdsourcing platform.<\/jats:p>","DOI":"10.2298\/csis220322038l","type":"journal-article","created":{"date-parts":[[2022,9,19]],"date-time":"2022-09-19T14:37:16Z","timestamp":1663598236000},"page":"1305-1328","source":"Crossref","is-referenced-by-count":26,"title":["Crowdsourcing platform for QoE evaluation for cloud multimedia services"],"prefix":"10.2298","volume":"19","author":[{"suffix":"Ali","given":"Asif","family":"Laghari","sequence":"first","affiliation":[{"name":"Department of Computer Science, Sindh Madressatul Islam University Karachi, Pakistan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hui","family":"He","sequence":"additional","affiliation":[{"name":"School of Computer Sciencee & Technology, Harbin Institute of Technology Harbin, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Asiya","family":"Khan","sequence":"additional","affiliation":[{"name":"School of Engineering, University of Plymouth United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"suffix":"Ali","given":"Rashid","family":"Laghari","sequence":"additional","affiliation":[{"name":"Interdisciplinary Research Center for Intelligent Manufacturing and Robotics (IRC-IMR), King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shoulin","family":"Yin","sequence":"additional","affiliation":[{"name":"School of Information and Communication Engineering, Harbin Engineering University Harbin China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiachi","family":"Wang","sequence":"additional","affiliation":[{"name":"Software College, Shenyang Normal University Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1078","reference":[{"key":"ref1","doi-asserted-by":"crossref","unstructured":"Laghari, Asif Ali, Hui He, Shahid Karim, Himat Ali Shah, and Nabin Kumar Karn. \u201dQuality of Experience Assessment of Video Quality in Social Clouds,\u201d Wireless Communications and Mobile Computing, pp. 1-10, 2017.","DOI":"10.1155\/2017\/8313942"},{"key":"ref2","doi-asserted-by":"crossref","unstructured":"Vaishnavi, Ishan, Pablo Cesar, Dick Bulterman, and Oliver Friedrich. \u201dFrom IPTV services to shared experiences: Challenges in architecture design,\u201d In Proc of Multimedia and Expo (ICME), 2010 IEEE International Conference on, pp. 1511-1516. IEEE, 2010. Suntec City, Singapore.","DOI":"10.1109\/ICME.2010.5583263"},{"key":"ref3","doi-asserted-by":"crossref","unstructured":"Aldhaibani O A, Al-Jumaili M H, Raschella A, et al. \u201dA centralized architecture for autonomic quality of experience oriented handover in dense networks,\u201d Computers & Electrical Engineering, vol. 94, no. 2, pp. 107352, 2021.","DOI":"10.1016\/j.compeleceng.2021.107352"},{"key":"ref4","doi-asserted-by":"crossref","unstructured":"Hsu, Wu-Hsiao, and Chi-Hsiang Lo. \u201dQoS\/QoE mapping and adjustment model in the cloudbased multimedia infrastructure,\u201d IEEE Systems Journal, vol. 8, no. 1, pp. 247-255. (2014)","DOI":"10.1109\/JSYST.2013.2253035"},{"key":"ref5","doi-asserted-by":"crossref","unstructured":"C. G. Bampis, Z. Li, I. Katsavounidis, T. -Y. Huang, C. Ekanadham and A. C. Bovik. \u201dTowards Perceptually Optimized Adaptive Video Streaming-A Realistic Quality of Experience Database,\u201d IEEE Transactions on Image Processing, vol. 30, pp. 5182-5197, 2021.","DOI":"10.1109\/TIP.2021.3073294"},{"key":"ref6","unstructured":"Laghari, Asif Ali, Hui He, Shehnila Zardari, and Muhammad Shafiq. \u201dSystematic Analysis of Quality of Experience (QoE) Frameworks for Multimedia Services,\u201d IJCSNS, vol. 17, no. 5. (2017)"},{"key":"ref7","doi-asserted-by":"crossref","unstructured":"Feldman, Michael, and Abraham Bernstein. \u201dBehavior-Based Quality Assurance in Crowdsourcing Markets,\u201d In Second AAAI Conference on Human Computation and Crowdsourcing. (2014)","DOI":"10.1609\/hcomp.v2i1.13171"},{"key":"ref8","doi-asserted-by":"crossref","unstructured":"Liguo Wang, Yin Shoulin, Hashem Alyami, et al. \u201dA novel deep learning-based single shot multibox detector model for object detection in optical remote sensing images,\u201d Geoscience Data Journal, 2022. https:\/\/doi.org\/10.1002\/gdj3.162","DOI":"10.1002\/gdj3.162"},{"key":"ref9","doi-asserted-by":"crossref","unstructured":"Samet, Nouha, Asma Ben Leta fa, Mohamed Hamdi, and Sami Tabbane. \u201dReal-Time User Experience Evaluation for Cloud-Based Mobile Video,\u201d In Advanced Information Networking and Applications Workshops (WAINA), 2016 30th International Conference on, pp. 204-208. IEEE. (2016)","DOI":"10.1109\/WAINA.2016.120"},{"key":"ref10","doi-asserted-by":"crossref","unstructured":"Wang, Yumei, Xiaojiang Zhou, Mengyao Sun, Lin Zhang, and XiaofeiWu. \u201dA new QoE-driven video cache management scheme with wireless cloud computing in cellular networks,\u201d Mobile Networks and Applications, vol. 22, no. 1, pp. 72-82. (2017)","DOI":"10.1007\/s11036-016-0689-5"},{"key":"ref11","doi-asserted-by":"crossref","unstructured":"Laghari, Asif Ali, Hui He, Muhammad Shafiq, and Asiya Khan. \u201dAssessing effect of Cloud distance on end user\u2019s Quality of Experience (QoE),\u201d In Computer and Communications (ICCC), 2016 2nd IEEE International Conference on, pp. 500-505. IEEE. (2016)","DOI":"10.1109\/CompComm.2016.7924751"},{"key":"ref12","doi-asserted-by":"crossref","unstructured":"De Pessemier, Toon, Isabelle Stevens, Lieven De Marez, Luc Martens, andWout Joseph. \u201dQuality assessment and usage behavior of a mobile voice-over-IP service,\u201d Telecommunication Systems, vol. 61, no. 3, pp. 417-432. (2016)","DOI":"10.1007\/s11235-014-9961-9"},{"key":"ref13","doi-asserted-by":"crossref","unstructured":"Noor, Tawfeeg S., Niemah I. Osman, and Is-Haka M. Mkwawa. \u201dThe impact of gender on the Quality of Experience for video services,\u201d In Automation and Computing (ICAC), 2016 22nd International Conference on, pp. 488-492. IEEE. (2016)","DOI":"10.1109\/IConAC.2016.7604967"},{"key":"ref14","doi-asserted-by":"crossref","unstructured":"Xia, Huichuan, and Brian McKernan. \u201dPrivacy in Crowdsourcing: a Review of the Threats and Challenges,\u201d Computer Supported Cooperative Work (CSCW), pp. 1-39. (2020)","DOI":"10.1007\/s10606-020-09374-0"},{"key":"ref15","doi-asserted-by":"crossref","unstructured":"Hansson, Karin, and Thomas Ludwig. \u201dCrowd dynamics: Conflicts, contradictions, and community in crowdsourcing,\u201d Computer Supported Cooperative Work (CSCW), vol. 28, no. 5, pp. 791-794. (2019)","DOI":"10.1007\/s10606-018-9343-z"},{"key":"ref16","doi-asserted-by":"crossref","unstructured":"Laghari, Khalil Ur Rehman, and Kay Connelly. \u201dToward total quality of experience: A QoE model in a communication ecosystem,\u201d IEEE Communications Magazine, vol. 50, no. 4. (2012)","DOI":"10.1109\/MCOM.2012.6178834"},{"key":"ref17","doi-asserted-by":"crossref","unstructured":"Agostino D, Brambilla M, Pavanetto S, et al. \u201dThe Contribution of Online Reviews for Quality Evaluation of Cultural Tourism Offers: The Experience of Italian Museums,\u201d Sustainability, vol. 13, 2021.","DOI":"10.3390\/su132313340"},{"key":"ref18","doi-asserted-by":"crossref","unstructured":"GuBo, AlazabMamoun, LinZiqi, et al. \u201dAI-Enabled Task Offloading for Improving Quality of Computational Experience in Ultra Dense Networks,\u201d ACM Transactions on Internet Technology (TOIT), vol. 22. no. 3, pp. 1-17, 2022.","DOI":"10.1145\/3491217"},{"key":"ref19","doi-asserted-by":"crossref","unstructured":"Chen, Yanjiao, Kaishun Wu, and Qian Zhang. \u201dFrom QoS to QoE: A tutorial on video quality assessment,\u201d IEEE Communications Surveys & Tutorials, vol. 17, no. 2, pp. 1126-1165. (2015)","DOI":"10.1109\/COMST.2014.2363139"},{"key":"ref20","doi-asserted-by":"crossref","unstructured":"Jain, Dhruv, Swapnil Agrawal, Satadal Sengupta, Pradipta De, Bivas Mitra, and Sandip Chakraborty. \u201dPrediction of quality degradation for mobile video streaming apps: A case study using YouTube,\u201d In Communication Systems and Networks (COMSNETS), 2016 8th International Conference on, pp. 1-2. IEEE. (2016)","DOI":"10.1109\/COMSNETS.2016.7440005"},{"key":"ref21","doi-asserted-by":"crossref","unstructured":"Kimura, Takuto, Masahiro Yokota, Arifumi Matsumoto, Kei Takeshita, Taichi Kawano, Kazumichi Sato, Hiroshi Yamamoto, Takanori Hayashi, Kohei Shiomoto, and Kenichi Miyazaki. \u201dQUVE: QoE Maximizing Framework for Video-Streaming,\u201d IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 1, pp. 138-153. (2017)","DOI":"10.1109\/JSTSP.2016.2632060"},{"key":"ref22","doi-asserted-by":"crossref","unstructured":"Ahammad, Parvez, R. Gaunker, Brian Kennedy, Mehrdad Reshadi, K. Kumar, A. K. Pathan, and Hariharan Kolam. \u201dA flexible platform for QoE-driven delivery of image-rich web applications,\u201d In Multimedia and Expo (ICME), 2015 IEEE International Conference on, pp. 1-6. IEEE. (2015)","DOI":"10.1109\/ICME.2015.7177516"},{"key":"ref23","doi-asserted-by":"crossref","unstructured":"Zhou, Chao, Lifeng Sun,Wenming Shi, and Shiqiang Yang. \u201dQOEYE: A Data Driven Platform for QoE Visualization and System Performance Monitoring,\u201d In Proceedings of the 23rd ACM international conference on Multimedia, pp. 741-742. ACM. (2015)","DOI":"10.1145\/2733373.2807968"},{"key":"ref24","doi-asserted-by":"crossref","unstructured":"Rainer, Benjamin, Markus Waltl, and Christian Timmerer. \u201dA web based subjective evaluation platform,\u201d In Quality of Multimedia Experience (QoMEX), 2013 Fifth International Workshop on, pp. 24-25. IEEE. (2013)","DOI":"10.1109\/QoMEX.2013.6603196"},{"key":"ref25","doi-asserted-by":"crossref","unstructured":"Ding, Yifan, Yang Geng, Ruiyi Wang, Yang Yang, and Wenjing Li. \u201dQoE-oriented resource management strategy by considering user preference for video content,\u201d In Network Operations and Management Symposium (APNOMS), 2014 16th Asia-Pacific, pp. 1-4. IEEE. (2014)","DOI":"10.1109\/APNOMS.2014.6996606"},{"key":"ref26","doi-asserted-by":"crossref","unstructured":"Laghari, Asif Ali, Hui He, Muhammad Ibrahim, and Salahuddin Shaikh. \u201dAutomatic Network Policy Change on the Basis of Quality of Experience (QoE),\u201d ?Procedia Computer Science, vol. 107, pp. 657-659. (2017)","DOI":"10.1016\/j.procs.2017.03.144"},{"key":"ref27","doi-asserted-by":"crossref","unstructured":"Garella, Juan Pablo, Eduardo Gramp\u00e4n, Rafael Sotelo, Javier Baliosian, Jose Joskowicz, Gustavo Guimerans, and Maria Simon. \u201dMonitoring QoE on digital terrestrial TV: a comprehensive approach,\u201d In Broadband Multimedia Systems and Broadcasting (BMSB), 2016 IEEE International Symposium on, pp. 1-6. IEEE, 2016.","DOI":"10.1109\/BMSB.2016.7522008"},{"key":"ref28","doi-asserted-by":"crossref","unstructured":"Ghadiyaram, Deepti, and Alan C. Bovik. \u201dMassive online crowdsourced study of subjective and objective picture quality,\u201d IEEE Transactions on Image Processing, vol. 25, no. 1, pp. 372-387. (2016)","DOI":"10.1109\/TIP.2015.2500021"},{"key":"ref29","doi-asserted-by":"crossref","unstructured":"Li, Xiaofei, and Yue You. \u201dKano Model Analysis Required in APP Interactive Design based on Mobile User Experience,\u201d International Journal of Multimedia and Ubiquitous Engineering, vol. 11, no. 11, pp. 247-258. (2016)","DOI":"10.14257\/ijmue.2016.11.11.21"},{"key":"ref30","doi-asserted-by":"crossref","unstructured":"Ho feld, Tobias, Matthias Hirth, Pavel Korshunov, Philippe Hanhart, Bruno Gardlo, Christian Keimel, and Christian Timmerer. \u201dSurvey of web-based crowdsourcing frameworks for subjective quality assessment,\u201d In Multimedia Signal Processing (MMSP), 2014 IEEE 16th International Workshop on, pp. 1-6. IEEE. (2014)","DOI":"10.1109\/MMSP.2014.6958831"},{"key":"ref31","unstructured":"Belda, Jordi, Mario Montagud, Fernando Boronat, Marc Martinez, and Javier Pastor. \u201dWersync: A web-based platform for distributed media synchronization and social interaction,\u201d ACM TVX. (2015)"},{"key":"ref32","unstructured":"Sensory Experience Lab, http:\/\/selab.itec.aau.at. Accessed 12. 24. 2016."},{"key":"ref33","doi-asserted-by":"crossref","unstructured":"Alhamazani, Khalid, Rajiv Ranjan, Prem Prakash Jayaraman, Karan Mitra, Meisong Wang, Zhiqiang George Huang, Lizhe Wang, and Fethi Rabhi. \u201dReal-time qos monitoring for cloudbased big data analytics applications in mobile environments,\u201d In?Mobile Data Management (MDM), 2014 IEEE 15th International Conference on, vol. 1, pp. 337-340. IEEE. (2014)","DOI":"10.1109\/MDM.2014.74"},{"key":"ref34","doi-asserted-by":"crossref","unstructured":"Ribeiro, Flovio, Dinei Florencio, Cha Zhang, and Michael Seltzer. \u201dCrowdmos: An approach for crowdsourcing mean opinion score studies,\u201d In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pp. 2416-2419. IEEE. (2011)","DOI":"10.1109\/ICASSP.2011.5946971"},{"key":"ref35","doi-asserted-by":"crossref","unstructured":"Keimel, Christian, Julian Habigt, Clemens Horch, and Klaus Diepold. \u201dQualitycrowd: a framework for crowd-based quality evaluation,\u201d In Picture Coding Symposium (PCS), pp. 245-248. IEEE. (2012)","DOI":"10.1109\/PCS.2012.6213338"},{"key":"ref36","unstructured":"Kraft, Sebastian, and Udo Z lzer. \u201dBeaqleJS: HTML5 and JavaScript based framework for the subjective evaluation of audio quality,\u201d In Linux Audio Conference, Karlsruhe, DE. (2014)"},{"key":"ref37","doi-asserted-by":"crossref","unstructured":"Gardlo, Bruno, Sebastian Egger, Michael Seufert, and Raimund Schatz. \u201dCrowdsourcing 2.0: Enhancing execution speed and reliability of web-based QoE testing,\u201d In Communications (ICC), 2014 IEEE International Conference on, pp. 1070-1075. IEEE. (2014)","DOI":"10.1109\/ICC.2014.6883463"},{"key":"ref38","unstructured":"https:\/\/support.hyperic.com\/display\/SIGAR\/Home."},{"key":"ref39","doi-asserted-by":"crossref","unstructured":"Schopf, Jennifer M., Laura Pearlman, Neill Miller, Carl Kesselman, Ian Foster, Mike D\u2019Arcy, and Ann Chervenak. \u201dMonitoring the grid with the Globus Toolkit MDS4,\u201d In Journal of Physics: Conference Series, vol. 46, no. 1, p. 521. IOP Publishing. (2006)","DOI":"10.1088\/1742-6596\/46\/1\/072"},{"key":"ref40","unstructured":"ITU- T Recommendation P. 910: \u201dSubjective video quality assessment methods for multimedia applications,\u201d International telecommunication Union, Geneva, Switzerland. (1996)"},{"key":"ref41","unstructured":"Jisi A and Shoulin Yin. \u201dA New Feature Fusion Network for Student Behavior Recognition in Education ,\u201d Journal of Applied Science and Engineering, vol. 24, no. 2, pp. 133-140, 2021."},{"key":"ref42","doi-asserted-by":"crossref","unstructured":"Laghari, Asif Ali, and Mureed Ali Laghari. \u201dQuality of experience assessment of calling services in social network,\u201d ICT Express, vol. 7, no. 2, pp. 158-161. (2021)","DOI":"10.1016\/j.icte.2021.04.011"},{"key":"ref43","doi-asserted-by":"crossref","unstructured":"Baloch, Jawwad Ali, Awais Khan Jumani, Asif Ali Laghari, Vania V. Estrela, and Ricardo T. Lopes. \u201dA Preliminary Study on Quality of Experience Assessment of Compressed Audio File Format,\u201d In 2021 IEEE URUCON, pp. 161-165. IEEE. (2021)","DOI":"10.1109\/URUCON53396.2021.9647114"}],"container-title":["Computer Science and Information Systems"],"original-title":[],"language":"en","deposited":{"date-parts":[[2023,8,11]],"date-time":"2023-08-11T08:20:57Z","timestamp":1691742057000},"score":1,"resource":{"primary":{"URL":"https:\/\/doiserbia.nb.rs\/Article.aspx?ID=1820-02142200038L"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":43,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022]]}},"URL":"https:\/\/doi.org\/10.2298\/csis220322038l","relation":{},"ISSN":["1820-0214","2406-1018"],"issn-type":[{"value":"1820-0214","type":"print"},{"value":"2406-1018","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022]]}}}