{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,2]],"date-time":"2025-11-02T02:44:35Z","timestamp":1762051475713,"version":"build-2065373602"},"reference-count":40,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,5,13]],"date-time":"2022-05-13T00:00:00Z","timestamp":1652400000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JSPS KAKENHI","award":["No. 22K12299"],"award-info":[{"award-number":["No. 22K12299"]}]},{"name":"the University of Aizu","award":["No. 22K12299"],"award-info":[{"award-number":["No. 22K12299"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Adaptive streaming has become a key technology for various multimedia services, such as online learning, mobile streaming, Internet TV, etc. However, because of throughput fluctuations, video quality may be dramatically varying during a streaming session. In addition, stalling events may occur when segments do not reach the user device before their playback deadlines. It is well-known that quality variations and stalling events cause negative impacts on Quality of Experience (QoE). Therefore, a main challenge in adaptive streaming is how to evaluate the QoE of streaming sessions taking into account the influences of these factors. Thus far, many models have been proposed to tackle this issue. In addition, a lot of QoE databases have been publicly available. However, there have been no extensive evaluations of existing models using various databases. To fill this gap, in this study, we conduct an extensive evaluation of thirteen models on twelve databases with different characteristics of viewing devices, codecs, and session durations. Through experiment results, important findings are provided with regard to QoE prediction of streaming sessions. In addition, some suggestions on the effective employment of QoE models are presented. The findings and suggestions are expected to be useful for researchers and service providers to make QoE assessments and improvements of streaming solutions in adaptive streaming.<\/jats:p>","DOI":"10.3390\/fi14050151","type":"journal-article","created":{"date-parts":[[2022,5,14]],"date-time":"2022-05-14T09:27:56Z","timestamp":1652520476000},"page":"151","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["QoE Models for Adaptive Streaming: A Comprehensive Evaluation"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1122-0650","authenticated-orcid":false,"given":"Duc","family":"Nguyen","sequence":"first","affiliation":[{"name":"Department of Information and Communication Engineering, Tohoku Institute of Technology, Sendai 982-8577, Japan"}]},{"given":"Nam","family":"Pham Ngoc","sequence":"additional","affiliation":[{"name":"College of Engineering and Computer Science, VinUniversity, Gia Lam District, Hanoi 100000, Vietnam"}]},{"given":"Truong Cong","family":"Thang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, The University of Aizu, Aizuwakamatsu 965-8580, Japan"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,13]]},"reference":[{"key":"ref_1","unstructured":"Conviva (2020, January 15). Conviva State of Streaming. Available online: https:\/\/www.conviva.com\/state-of-streaming\/."},{"key":"ref_2","unstructured":"Sandvine (2022, April 12). The Global Internet Phenomena Report. Available online: https:\/\/www.sandvine.com\/phenomena."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1109\/COMST.2014.2360940","article-title":"A survey on quality of experience of HTTP adaptive streaming","volume":"17","author":"Seufert","year":"2015","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ho\u00dffeld, T., Egger, S., Schatz, R., Fiedler, M., Masuch, K., and Lorentzen, C. (2012, January 5\u20137). Initial delay vs. interruptions: Between the devil and the deep blue sea. Proceedings of the 4th International Workshop on Quality of Multimedia Experience, Melbourne, VIC, Australia.","DOI":"10.1109\/QoMEX.2012.6263849"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2141","DOI":"10.1109\/JSAC.2016.2577361","article-title":"Perceptual quality of HTTP adaptive streaming strategies: Cross-experimental analysis of multi-laboratory and crowdsourced subjective studies","volume":"34","author":"Tavakoli","year":"2016","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Rehman, A., and Wang, Z. (2013, January 3\u20135). Perceptual experience of time-varying video quality. Proceedings of the 5th International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt am W\u00f6rthersee, Austria.","DOI":"10.1109\/QoMEX.2013.6603240"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Guo, Z., Wang, Y., and Zhu, X. (2015, January 27\u201330). Assessing the visual effect of non-periodic temporal variation of quantization stepsize in compressed video. Proceedings of the IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada.","DOI":"10.1109\/ICIP.2015.7351378"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1109\/TBC.2015.2460611","article-title":"Deriving and validating user experience model for DASH video streaming","volume":"61","author":"Liu","year":"2015","journal-title":"IEEE Trans. Broadcast."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Singh, K.D., Hadjadj-Aoul, Y., and Rubino, G. (2012, January 14\u201317). Quality of experience estimation for adaptive HTTP\/TCP video streaming using H. 264\/AVC. Proceedings of the IEEE Consumer Communications and Networking Conference (CCNC), Las Vegas, NV, USA.","DOI":"10.1109\/CCNC.2012.6181070"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3212","DOI":"10.1109\/TCSVT.2020.3035824","article-title":"Overall Quality Prediction for HTTP Adaptive Streaming using LSTM Network","volume":"31","author":"Tran","year":"2021","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_11","unstructured":"Vriendt, J.D., Vleeschauwer, D.D., and Robinson, D. (2013, January 27\u201331). Model for estimating QoE of video delivered using HTTP adaptive streaming. Proceedings of the IFIP\/IEEE International Symposium on Integrated Network Management (IM 2013), Ghent, Belgium."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1145\/2829988.2787486","article-title":"A control-theoretic approach for dynamic adaptive video streaming over HTTP","volume":"45","author":"Yin","year":"2015","journal-title":"Acm Sigcomm Comput. Commun. Rev."},{"key":"ref_13","unstructured":"(2017). Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport-Quality integration module. Standard No. Recommendation ITU-T P.1203.3."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"560","DOI":"10.1109\/TCSVT.2003.815165","article-title":"Overview of the H. 264\/AVC video coding standard","volume":"13","author":"Wiegand","year":"2003","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1649","DOI":"10.1109\/TCSVT.2012.2221191","article-title":"Overview of the high efficiency video coding (HEVC) standard","volume":"22","author":"Sullivan","year":"2012","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1109\/JSTSP.2016.2608329","article-title":"A Quality-of-Experience Index for Streaming Video","volume":"11","author":"Duanmu","year":"2017","journal-title":"IEEE J. Sel. Top. Signal Process."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"6135","DOI":"10.1109\/TIP.2018.2855403","article-title":"Quality-of-Experience for Adaptive Streaming Videos: An Expectation Confirmation Theory Motivated Approach","volume":"27","author":"Duanmu","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"474","DOI":"10.1109\/TBC.2018.2822870","article-title":"A Quality-of-Experience Database for Adaptive Video Streaming","volume":"64","author":"Duanmu","year":"2018","journal-title":"IEEE Trans. Broadcast."},{"key":"ref_19","unstructured":"Duanmu, Z., Liu, W., Li, Z., Chen, D., Wang, Z., Wang, Y., and Gao, W. (2020). Assessing the Quality-of-Experience of Adaptive Bitrate Video Streaming. arXiv."},{"key":"ref_20","unstructured":"Bampis, C.G., Li, Z., Katsavounidis, I., Huang, T., Ekanadham, C., and Bovik, A.C. (2018). Towards Perceptually Optimized End-to-end Adaptive Video Streaming. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Robitza, W., G\u00f6ring, S., Raake, A., Lindegren, D., Heikkil\u00e4, G., Gustafsson, J., List, P., Feiten, B., W\u00fcstenhagen, U., and Garcia, M.N. (2018, January 12\u201315). HTTP Adaptive Streaming QoE Estimation with ITU-T Rec. P.1203\u2014Open Databases and Software. Proceedings of the 9th ACM Multimedia Systems Conference, Amsterdam, The Netherlands.","DOI":"10.1145\/3204949.3208124"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3423421","article-title":"Cumulative Quality Modeling for HTTP Adaptive Streaming","volume":"Volume 17","author":"Tran","year":"2020","journal-title":"ACM Transactions on Multimedia Computing Communications and Applications"},{"key":"ref_23","unstructured":"Le Callet, P., M\u00f6ller, S., and Perkis, A. (2013). Qualinet White Paper on Definitions of Quality of Experience, Technical Report. Version 1.2."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1109\/TCSVT.2017.2768542","article-title":"A Subjective and Objective Study of Stalling Events in Mobile Streaming Videos","volume":"29","author":"Ghadiyaram","year":"2019","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"19507","DOI":"10.1109\/ACCESS.2022.3149592","article-title":"A Survey on Multimedia Services QoE Assessment and Machine Learning-Based Prediction","volume":"10","author":"Kougioumtzidis","year":"2022","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1016\/j.image.2018.05.017","article-title":"Feature-based prediction of streaming video QoE: Distortions, stalling and memory","volume":"68","author":"Bampis","year":"2018","journal-title":"Signal Process. Image Commun."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Tran, H.T.T., Nguyen, D.V., and Thang, T.C. (2020, January 8\u201311). Open Software for Bitstream-based Quality Prediction in Adaptive Video Streaming. Proceedings of the ACM Multimedia Systems Conference (MMSys\u201920), Istanbul, Turkey.","DOI":"10.1145\/3339825.3394925"},{"key":"ref_28","unstructured":"Duanmu, Z., Liu, W., Chen, D., Li, Z., Wang, Z., Wang, Y., and Gao, W. (2019). A Knowledge-Driven Quality-of-Experience Model for Adaptive Streaming Videos. arXiv."},{"key":"ref_29","unstructured":"(2017). Parametric Bitstream-Based Quality Assessment of Progressive Dowload and Adaptive Audiovisual Streaming Services over Reliable Transport-Video Quality Estimation Module. Standard No. Recommendation ITU-T P.1203.1."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Raake, A., Garcia, M.N., Robitza, W., List, P., G\u00f6ring, S., and Feiten, B. (June, January 31). A bitstream-based, scalable video-quality model for HTTP adaptive streaming: ITU-T P.1203.1. Proceedings of the Ninth International Conference on Quality of Multimedia Experience (QoMEX), Erfurt, Germany.","DOI":"10.1109\/QoMEX.2017.7965631"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Tran, H.T.T., Nguyen, D.V., Nguyen, D.D., Ngoc, N.P., and Thang, T.C. (May, January 29). An LSTM-based Approach for Overall Quality. Proceedings of the IEEE Conference on Computer Communications Conference (INFOCOM 2019), Paris, France.","DOI":"10.1109\/INFCOMW.2019.8845041"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"5217","DOI":"10.1109\/TIP.2017.2729891","article-title":"Study of Temporal Effects on Subjective Video Quality of Experience","volume":"26","author":"Bampis","year":"2017","journal-title":"IEEE Trans. Image Process."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2206","DOI":"10.1109\/TIP.2014.2312613","article-title":"Modeling the time-Varying subjective quality of HTTP video streams with rate adaptations","volume":"23","author":"Chen","year":"2014","journal-title":"IEEE Trans. Image Process."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Duanmu, Z., Ma, K., and Wang, Z. (2017, January 23\u201327). Quality-of-Experience of Adaptive Video Streaming: Exploring the Space of Adaptations. Proceedings of the 25th ACM international conference on Multimedia, Mountain View, CA, USA.","DOI":"10.1145\/3123266.3123418"},{"key":"ref_35","unstructured":"(2018, July 01). Recommendation ITU-T P.1203. ITU-T Rec. P.1203 Standalone Implementation. Available online: https:\/\/github.com\/itu-p1203."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Tran, H.T.T., Ngoc, N.P., Pham, A.T., and Thang, T.C. (2016, January 4\u20138). A Multi-Factor QoE Model for Adaptive Streaming over Mobile Networks. Proceedings of the IEEE Globecom Workshops (GC Wkshps), Washington, DC, USA.","DOI":"10.1109\/GLOCOMW.2016.7848818"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1109\/TBC.2016.2570012","article-title":"Video quality metric for streaming service using DASH standard","volume":"62","author":"Rosa","year":"2016","journal-title":"IEEE Trans. Broadcast."},{"key":"ref_38","unstructured":"(2012). Methods, Metrics and Procedures for Statistical Evaluation, Qualification and Comparison of Objective Quality Prediction Models. Standard No. Recommendation ITU-T P.1401."},{"key":"ref_39","unstructured":"(2015). Opinion Model for Network Planning of Video and Audio Streaming Applications. Standard No. Recommendation ITU-T G.1071."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1587\/transfun.E100.A.555","article-title":"A Histogram-Based Quality Model for HTTP Adaptive Streaming","volume":"100","author":"Tran","year":"2017","journal-title":"Ieice Trans. Fundam. Electron. Commun. Comput. Sci."}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/14\/5\/151\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:10:34Z","timestamp":1760137834000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/14\/5\/151"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,13]]},"references-count":40,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["fi14050151"],"URL":"https:\/\/doi.org\/10.3390\/fi14050151","relation":{},"ISSN":["1999-5903"],"issn-type":[{"type":"electronic","value":"1999-5903"}],"subject":[],"published":{"date-parts":[[2022,5,13]]}}}