{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,19]],"date-time":"2025-12-19T15:45:41Z","timestamp":1766159141000,"version":"3.37.3"},"reference-count":116,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,4,13]],"date-time":"2023-04-13T00:00:00Z","timestamp":1681344000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,13]],"date-time":"2023-04-13T00:00:00Z","timestamp":1681344000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007826","name":"Technische Universit\u00e4t Ilmenau","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007826","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Qual User Exp"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In many research fields, human-annotated data plays an important role as it is used to accomplish a multitude of tasks. One such example is in the field of multimedia quality assessment where subjective annotations can be used to train or evaluate quality prediction models. Lab-based tests could be one approach to get such quality annotations. They are usually performed in well-defined and controlled environments to ensure high reliability. However, this high reliability comes at a cost of higher time consumption and costs incurred. To mitigate this, crowd or online tests could be used. Usually, online tests cover a wider range of end devices, environmental conditions, or participants, which may have an impact on the ratings. To verify whether such online tests can be used for visual quality assessment, we designed three online tests. These online tests are based on previously conducted lab tests as this enables comparison of the results of both test paradigms. Our focus is on the quality assessment of high-resolution images and videos. The online tests use AVrate Voyager, which is a publicly accessible framework for online tests. To transform the lab tests into online tests, dedicated adaptations in the test methodologies are required. The considered modifications are, for example, a patch-based or centre cropping of the images and videos, or a randomly sub-sampling of the to-be-rated stimuli. Based on the analysis of the test results in terms of correlation and SOS analysis it is shown that online tests can be used as a reliable replacement for lab tests albeit with some limitations. These limitations relate to, e.g., lack of appropriate display devices, limitation of web technologies, and modern browsers considering support for different video codecs and formats.<\/jats:p>","DOI":"10.1007\/s41233-023-00055-6","type":"journal-article","created":{"date-parts":[[2023,4,13]],"date-time":"2023-04-13T09:03:21Z","timestamp":1681376601000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Quality assessment of higher resolution images and videos with remote testing"],"prefix":"10.1007","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6810-6969","authenticated-orcid":false,"given":"Steve","family":"G\u00f6ring","sequence":"first","affiliation":[]},{"given":"Rakesh Rao Ramachandra","family":"Rao","sequence":"additional","affiliation":[]},{"given":"Alexander","family":"Raake","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,4,13]]},"reference":[{"doi-asserted-by":"crossref","unstructured":"Albalawi U, Mohanty SP, Kougianos E (2015) A hardware architecture for better portable graphics (BPG) compression encoder. In: IEEE international symposium on nanoelectronic and information systems. IEEE. 2015:291\u2013296","key":"55_CR1","DOI":"10.1109\/iNIS.2015.12"},{"doi-asserted-by":"crossref","unstructured":"Ashimov D, Martini MG, Barman N (2020) Quality assessment of gaming videos compressed via AV1. In: QoMEX. IEEE. 1\u20134","key":"55_CR2","DOI":"10.1109\/QoMEX48832.2020.9123112"},{"unstructured":"AVRateNG. AVRateNG \u2013 github project. https:\/\/github.com\/Telecommunication-Telemedia-Assessment\/avrateNG (visited on 03\/07\/2020)","key":"55_CR3"},{"key":"55_CR4","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1109\/TBC.2013.2247171","volume":"59.2","author":"S Bae","year":"2013","unstructured":"Bae S, Kim J, Kim M, Cho S, Choi JS (2013) Assessments of subjective video quality on HEVC-encoded 4KUHD video for beyond-HDTV broadcasting services. IEEE Trans Broadcasting 59.2:209\u2013222","journal-title":"IEEE Trans Broadcasting"},{"doi-asserted-by":"crossref","unstructured":"Barman N, Martini M (2020) An evaluation of the next-generation image coding standard AVIF. In: Twelfth international conference on quality of multimedia experience (QoMEX). Athlone, Ireland, 1\u20134","key":"55_CR5","DOI":"10.1109\/QoMEX48832.2020.9123131"},{"unstructured":"Bellard F (2018) BPG Image format. https:\/\/bellard.org\/bpg\/ (visited on 03\/07\/2020)","key":"55_CR6"},{"doi-asserted-by":"crossref","unstructured":"Berger K, Koudota Y, Barkowsky M, Le Callet P (2015) Subjective quality assessment comparing UHD and HD resolution in HEVC transmission chains. In: 7th international on workshop on quality of multimedia experience (QoMEX). 1\u20136","key":"55_CR7","DOI":"10.1109\/QoMEX.2015.7148114"},{"doi-asserted-by":"crossref","unstructured":"Bosse S, Siekmann M, Rasch J, Wiegand T, Samek W (2016) Quality assessment of image patches distorted by image compression using crowdsourcing. In: 2016 IEEE ICME. 1\u20136","key":"55_CR8","DOI":"10.1109\/ICME.2016.7552958"},{"unstructured":"Brunnstr\u00f6m K, Cousseau R, Jonsson J, Koudota Y, Bagazov V, Barkowsky M (2014) VQEGPlayer: open source software for subjective video quality experiments in windows","key":"55_CR9"},{"unstructured":"Chen L-H, Bampis CG, Li Z, Norkin A, Bovik AC (2020) Perceptually optimizing deep image compression. arXiv: 2007.02711 [eess.IV]","key":"55_CR10"},{"doi-asserted-by":"crossref","unstructured":"Cheon M, Lee J (2014) Objective quality comparison of 4K UHD and up-scaled 4K UHD videos. In: IEEE international symposium on multimedia, 78\u201381","key":"55_CR11","DOI":"10.1109\/ISM.2014.33"},{"doi-asserted-by":"crossref","unstructured":"Cheon M, Lee J (2018) Subjective and objective quality assessment of compressed 4K UHD videos for immersive experience. In: IEEE transaction on circuits and systems for video technology 28.7 (2018), 1467\u20131480","key":"55_CR12","DOI":"10.1109\/TCSVT.2017.2683504"},{"unstructured":"Diaz AG (2018) Picture this: the best image format for the web in 2019. https:\/\/www.freecodecamp.org\/news\/best-image-format-for-web-in-2019-jpeg-webp-heic-avif-41ba0c1b2789\/ (visited on 03\/07\/2020)","key":"55_CR13"},{"unstructured":"Egge NE, Valin J-M, Terriberry TB, Daede T, Montgomery C (2015) Using Daala intra frames for still picture coding. In: Proceedings of picture coding symposium","key":"55_CR14"},{"doi-asserted-by":"crossref","unstructured":"Egger-Lampl S, Redi J, Hosfeld T, Hirth M, M\u00f6ller S, Naderi B, Keimel C, Saupe D (2017) Crowdsourcing quality of experience experiments. In: Evaluation in the crowd. Crowdsourcing and human-centered experiments. Springer, 154\u2013190","key":"55_CR15","DOI":"10.1007\/978-3-319-66435-4_7"},{"doi-asserted-by":"crossref","unstructured":"Figuerola S, Adzic V, Shah A, Kalva H (2013) Assessing internet video quality using crowdsourcing. In: Proceedings of the 2nd ACM international workshop on crowdsourcing for multimedia, 23\u201328","key":"55_CR16","DOI":"10.1145\/2506364.2506366"},{"doi-asserted-by":"crossref","unstructured":"Fremerey S, Singla A, Meseberg K, Raake A (2018) AVTrack360: an open dataset and software recording people\u2019s Head Rotations watching 360$$^{\\circ }$$ contents on an HMD. In: ACM multimedia systems conference 2018. 1\u20136","key":"55_CR17","DOI":"10.1145\/3204949.3208134"},{"key":"55_CR18","first-page":"372","volume":"25.1","author":"D Ghadiyaram","year":"2015","unstructured":"Ghadiyaram D, Bovik AC (2015) Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans Image Process 25.1:372\u2013387","journal-title":"IEEE Trans Image Process"},{"key":"55_CR19","doi-asserted-by":"publisher","first-page":"372","DOI":"10.1109\/TIP.2015.2500021","volume":"25.1","author":"D Ghadiyaram","year":"2016","unstructured":"Ghadiyaram D, Bovik AC (2016) Massive online crowdsourced study of subjective and objective picture quality. IEEE Trans Image Process 25.1:372\u2013387","journal-title":"IEEE Trans Image Process"},{"unstructured":"(2020) Google. A new image format for the Web. https:\/\/developers.google.com\/speed\/webp\/ (visited on 03\/07\/2020)","key":"55_CR20"},{"doi-asserted-by":"crossref","unstructured":"G\u00f6ring S, Kr\u00e4mmer C, Raake A (2019) cencro \u2013 speedup of video quality calculation using center cropping. In: 21st IEEE international symposium on multimedia (IEEE ISM). 1\u2013 8","key":"55_CR21","DOI":"10.1109\/ISM46123.2019.00010"},{"doi-asserted-by":"crossref","unstructured":"G\u00f6ring S, Raake A (2018) deimeq\u2013a deep neural network based hybrid noreference image quality model. In: 7th European workshop on visual information processing (EUVIP). IEEE. 1\u20136","key":"55_CR22","DOI":"10.1109\/EUVIP.2018.8611703"},{"doi-asserted-by":"crossref","unstructured":"G\u00f6ring S, Raake A (2019) Evaluation of intra-coding based image compression. In: 8th European workshop on visual information processing (EUVIP). IEEE. 1\u20136","key":"55_CR23","DOI":"10.1109\/EUVIP.2018.8611703"},{"doi-asserted-by":"crossref","unstructured":"G\u00f6ring S, Rao R, Fremerey S, Raake A (2021) AVRate voyager: an open source online testing platform. In: 2021 IEEE 23st international workshop on multimedia signal processing (MMSP). IEEE. 2021:1\u20136","key":"55_CR24","DOI":"10.1109\/MMSP53017.2021.9733561"},{"doi-asserted-by":"crossref","unstructured":"G\u00f6ring S, Rao Ramachandra Rao R, Feiten B, Raake A (2021) Modular framework and instances of pixel-based video quality models for UHD-1\/4K. In: IEEE Access 9, 31842\u201331864. https:\/\/ieeexplore.ieee.org\/document\/9355144","key":"55_CR25","DOI":"10.1109\/ACCESS.2021.3059932"},{"key":"55_CR26","first-page":"1","volume":"2018.14","author":"S G\u00f6ring","year":"2018","unstructured":"G\u00f6ring S, Skowronek J, Raake A (2018) DeViQ - A deep no reference video quality model. Electron Imaging Hum Vis Electron Imaging 2018.14:1\u20136","journal-title":"Electron Imaging Hum Vis Electron Imaging"},{"unstructured":"G\u00f6ring S, Zebelein J, Wedel S, Keller D, Raake A (2019) Analyze and predict the perceptibility of UHD video contents. In: Electronic imaging, HVEI. https:\/\/www.ingentaconnect.com\/content\/ist\/ei\/pre-prints\/content-ei2019-hvei-215;jsessionid$=$3hkwbc6tro23a.x-iclive-01#Cits","key":"55_CR27"},{"key":"55_CR28","first-page":"215","volume":"12","author":"S G\u00f6ring","year":"2019","unstructured":"G\u00f6ring S, Zebelein J, Wedel S, Keller D, Raake A (2019) Analyze and predict the perceptibility of UHD video contents. Electron Imaging 12:215","journal-title":"Electron Imaging"},{"doi-asserted-by":"crossref","unstructured":"G\u00f6tz-Hahn F, Hosu V, Lin H, Saupe D (2021) KonVid-150k: a dataset for no- reference video quality assessment of videos in-the-wild. In: IEEE Access 9. IEEE. 72139\u201372160","key":"55_CR29","DOI":"10.1109\/ACCESS.2021.3077642"},{"unstructured":"G\u00f6tz-Hahn F, Hosu V, Lin H, Saupe D (2021) The Konstanz 150k in-the-wild video database (KonVid-150k). http:\/\/database.mmsp-kn.de","key":"55_CR30"},{"doi-asserted-by":"crossref","unstructured":"Hanhart P, Korshunov P, Ebrahimi T (2013) Benchmarking of quality metrics on ultra-high definition video sequences. In: 2013 18th international conference on digital signal processing (DSP). 1\u20138","key":"55_CR31","DOI":"10.1109\/ICDSP.2013.6622760"},{"doi-asserted-by":"crossref","unstructured":"Hosfeld T, Keimel C, Hirth M, Gardlo B, Habigt J, Diepold K, Tran- Gia P (2014) Best practices for QoE Crowdtesting: QoE assessment with crowdsourcing. In: IEEE transactions on multimedia 16(2), 541\u2013558","key":"55_CR32","DOI":"10.1109\/TMM.2013.2291663"},{"doi-asserted-by":"crossref","unstructured":"Hosfeld T, Seufert M, Hirth M, Zinner T, Tran-Gia P, Schatz R (2011) Quantification of YouTube QoE via crowdsourcing. In: IEEE ISM, 494\u2013499","key":"55_CR33","DOI":"10.1109\/ISM.2011.87"},{"doi-asserted-by":"crossref","unstructured":"Hosfeld T, Schatz R, Egger S (2011) SOS: the MOS is not enough! In: 2012 third international workshop on quality of multimedia experience. IEEE, 131\u2013136","key":"55_CR34","DOI":"10.1109\/QoMEX.2011.6065690"},{"doi-asserted-by":"crossref","unstructured":"Hosfeld T, Seufert M, Hirth M, Zinner T, Tran-Gia P, Schatz R (2011) Quantification of youtube QoE via crowdsourcing. In: 2011 IEEE international symposium on multimedia. IEEE, 494\u2013499","key":"55_CR35","DOI":"10.1109\/ISM.2011.87"},{"issue":"2","key":"55_CR36","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1109\/TMM.2013.2291663","volume":"16","author":"T Hossfeld","year":"2013","unstructured":"Hossfeld T, Keimel C, Hirth M, Gardlo B, Habigt J, Diepold K, Tran- Gia P (2013) Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Trans Multimedia 16(2):541\u2013558","journal-title":"IEEE Trans Multimedia"},{"key":"55_CR37","doi-asserted-by":"publisher","first-page":"4041","DOI":"10.1109\/TIP.2020.2967829","volume":"29","author":"V Hosu","year":"2020","unstructured":"Hosu V, Lin H, Sziranyi T, Saupe D (2020) KonIQ-10k: an ecologically valid database for deep learning of blind image quality assessment. IEEE Trans Image Process 29:4041\u20134056","journal-title":"IEEE Trans Image Process"},{"doi-asserted-by":"crossref","unstructured":"Hosu V, Hahn F, Jenadeleh M, Lin H, Men H, Szir\u00e1nyi T, Li S, Saupe D (2017) The Konstanz natural video database. http:\/\/database.mmsp-kn.de","key":"55_CR38","DOI":"10.1109\/QoMEX.2017.7965673"},{"doi-asserted-by":"crossref","unstructured":"Hosu V, Hahn F, Jenadeleh M, Lin H, Men H, Szir\u00e1nyi T, Li S, Saupe D (2017) The Konstanz natural video database (KoNViD-1k). In: 2017 ninth international conference on quality of multimedia experience (QoMEX). IEEE. 1\u20136","key":"55_CR39","DOI":"10.1109\/QoMEX.2017.7965673"},{"unstructured":"(2017) ITU-T. ITU-T Rec. P.1203 (10\/17). Tech. rep. Int. Telecommunication Union","key":"55_CR40"},{"unstructured":"(2016) ITU-T. ITU-T Rec. P.913 (16\/03). Tech. rep. Int. Telecommunication Union","key":"55_CR41"},{"unstructured":"(2020) ITU-T. Recommendation H.266 (08\/20) - Versatile video coding. Tech. rep. International Telecommunication Union","key":"55_CR42"},{"unstructured":"(2014) ITU-T. Recommendation ITU-R BT.500- 13 \u2013 Methodology for the subjective assessment of the quality of television pictures. Tech. rep. International Telecommunication Union","key":"55_CR43"},{"unstructured":"(2019) ITU-T. Recommendation P.1204 - Video quality assessment of streaming services over reliable transport for resolutions up to 4K. Tech. rep. International Telecommunication Union","key":"55_CR44"},{"unstructured":"(2019) ITU-T. Recommendation P.1204 - Video quality assessment of streaming services over reliable transport for resolutions up to 4K. Tech. rep. Int. Telecommunication Union","key":"55_CR45"},{"key":"55_CR46","first-page":"1","volume":"4.1","author":"PA Kara","year":"2019","unstructured":"Kara PA, Robitza W, Pinter N, Martini MG, Raake A, Simon A (2019) Comparison of HD and UHD video quality with and without the influence of the labeling effect. Qual User Exp 4.1:1\u201329","journal-title":"Qual User Exp"},{"doi-asserted-by":"crossref","unstructured":"Kara PA, Robitza W, Raake A, Martini MG (2017) The label knows better: the impact of labeling effects on perceived quality of HD and UHD video streaming. In: 2017 ninth international conference on quality of multimedia experience (QoMEX). IEEE. 1\u20136","key":"55_CR47","DOI":"10.1109\/QoMEX.2017.7965674"},{"doi-asserted-by":"crossref","unstructured":"Keimel C, Habigt J, Diepold K (2012) Challenges in crowd-based video quality assessment. In: 2012 fourth international workshop on quality of multimedia experience","key":"55_CR48","DOI":"10.1109\/QoMEX.2012.6263866"},{"doi-asserted-by":"crossref","unstructured":"Keimel C, Oelbaum T, Diepold K (2009) No-reference video quality evaluation for high-definition video. In: 2009 IEEE ICASSP","key":"55_CR49","DOI":"10.1109\/ICASSP.2009.4959791"},{"doi-asserted-by":"crossref","unstructured":"Keimel C, Habigt J, Horch C, Diepold K (2012) Qualitycrowd\u2013a framework for crowd-based quality evaluation. In: PCS. IEEE. 245\u2013248","key":"55_CR50","DOI":"10.1109\/PCS.2012.6213338"},{"doi-asserted-by":"crossref","unstructured":"Keller D, Raake A, Vaalgamaa M, Paajanen E (2020) Let the music play: an automated test setup for blind subjective evaluation of music playback on mobile devices. In: QoMEX. IEEE, 1\u2013 4","key":"55_CR51","DOI":"10.1109\/QoMEX48832.2020.9123092"},{"doi-asserted-by":"crossref","unstructured":"Keller D, Seybold T, Skowronek J, Raake A (2019) Assessing texture dimensions and video quality in motion pictures using sensory evaluation techniques. In: QoMEX. IEEE","key":"55_CR52","DOI":"10.1109\/QoMEX.2019.8743189"},{"doi-asserted-by":"crossref","unstructured":"Lainema J, Hannuksela MM, Vadakital VKM, Aksu EB (2016) HEVC still image coding and high efficiency image file format. In: 2016 IEEE international conference on image processing (ICIP). IEEE, 71\u201375","key":"55_CR53","DOI":"10.1109\/ICIP.2016.7532321"},{"unstructured":"Lebreton P, Garcia M-N, Raake A (2016) AVRate: an open source modular audio\/visual subjective evaluation test interface. In: QoMEX","key":"55_CR54"},{"doi-asserted-by":"crossref","unstructured":"Lebreton P, Skodras E, M\u00e4ki T, Hupont I, Hirth M (2015) Bridging the gap between eye tracking and crowdsourcing. In: vol. 9394","key":"55_CR55","DOI":"10.1117\/12.2076745"},{"doi-asserted-by":"crossref","unstructured":"Lee W-C, Chang C-P, Peng W-H, Hang H-M (2020) A hybrid layered image compressor with deep-learning technique. In: 1\u20136","key":"55_CR56","DOI":"10.1109\/MMSP48831.2020.9287130"},{"key":"55_CR57","doi-asserted-by":"publisher","first-page":"935","DOI":"10.1109\/TMM.2011.2152382","volume":"13.5","author":"S Li","year":"2011","unstructured":"Li S, Zhang F, Ma L, Ngan KN (2011) Image quality assessment by separately evaluating detail losses and additive impairments. IEEE Trans Multimedia 13.5:935\u2013949","journal-title":"IEEE Trans Multimedia"},{"doi-asserted-by":"crossref","unstructured":"Song L, Tang X, Zhang W, Yang X, Xia P (2013) The SJTU 4K video sequence dataset. In: 5th international workshop on quality of multimedia experience (QoMEX), 34\u201335","key":"55_CR58","DOI":"10.1109\/QoMEX.2013.6603201"},{"unstructured":"Lin H, Hosu V, Saupe D (2020) DeepFLIQA: weak supervision for deep IQA feature learning. arXiv preprint arXiv:2001.08113","key":"55_CR59"},{"doi-asserted-by":"crossref","unstructured":"Lin H, Hosu V, Saupe D (2019) KADID- 10k: a large-scale artificially distorted IQA database. In: 2019 tenth international conference on quality of multimedia experience (QoMEX). IEEE, 1\u2013 3","key":"55_CR60","DOI":"10.1109\/QoMEX.2019.8743252"},{"doi-asserted-by":"crossref","unstructured":"Lin JY, Liu TJ, Wu ECH, Kuo CCJ (2014) A fusion-based video quality assessment (fvqa) index. In: APSIPA, 2014 Asia-Pacific, 1\u20135","key":"55_CR61","DOI":"10.1109\/APSIPA.2014.7041705"},{"doi-asserted-by":"crossref","unstructured":"Lin J, Akbari M, Fu H, Zhang Q, Wang S, Liang J, Liu D, Liang F, Zhang G, Tu C (2020) Learned variable-rate multi-frequency image compression using modulated generalized octave convolution. In: 2020 IEEE 22st international workshop on multimedia signal processing (MMSP). IEEE, 1\u20136","key":"55_CR62","DOI":"10.1109\/MMSP48831.2020.9287082"},{"doi-asserted-by":"crossref","unstructured":"Menon VV, Feldmann C, Amirpour H, Ghanbari M, Timmerer C (2022) VCA: video complexity analyzer. In: Proceedings of the 13th ACM multimedia systems conference, pp 259\u2013264","key":"55_CR63","DOI":"10.1145\/3524273.3532896"},{"unstructured":"Naderi B, Cutler R (2022) A crowdsourced implementation of ITU-T P. 910. In: arXiv preprint arXiv:2204.06784","key":"55_CR64"},{"key":"55_CR65","first-page":"1","volume":"6.1","author":"B Naderi","year":"2020","unstructured":"Naderi B, Jim\u00e9nez RZ, Hirth M, M\u00f6ller S, Metzger F, Hosfeld T (2020) Towards speech quality assessment using a crowdsourcing approach: evaluation of standardized methods. Qual User Exp 6.1:1\u201321","journal-title":"Qual User Exp"},{"doi-asserted-by":"crossref","unstructured":"Naderi B, Polzehl T, Wechsung I, K\u00f6ster F, M\u00f6ller S (2015) Effect of trapping questions on the reliability of speech quality judgments in a crowdsourcing paradigm. In: 16th annual conference of the international speech communication association","key":"55_CR66","DOI":"10.21437\/Interspeech.2015-589"},{"doi-asserted-by":"crossref","unstructured":"Nehm\u00e9 Y, Le Callet P, Dupont F, Farrugia J-P, Lavou\u00e9 G (2021) Exploring crowdsourcing for subjective quality assessment of 3D graphics. In: IEEE international workshop on multimedia signal processing (MMSP)","key":"55_CR67","DOI":"10.1109\/MMSP53017.2021.9733634"},{"unstructured":"(2018) Netflix. 4K Support. https:\/\/help.netflix.com\/en\/node\/13444 (visited on 03\/07\/2020)","key":"55_CR68"},{"unstructured":"Netflix. Netflix VMAF, https : \/ \/ github . com\/ Netflix\/ vmaf ; 2019-07-03","key":"55_CR69"},{"unstructured":"(2018) Netflix. VMAF 4K included. https:\/\/github.com\/Netflix\/vmaf (visited on 03\/07\/2020)","key":"55_CR70"},{"key":"55_CR71","first-page":"790","volume":"25.5","author":"T Nguyen","year":"2014","unstructured":"Nguyen T, Marpe D (2014) Objective performance evaluation of the HEVC main still picture profile. IEEE Trans Circuits Syst Video Technol 25.5:790\u2013797","journal-title":"IEEE Trans Circuits Syst Video Technol"},{"unstructured":"(2020) Nokiatech. About HEIF and MIAF. https:\/\/nokiatech.github.io\/heif\/ (visited on 03\/07\/2020)","key":"55_CR72"},{"doi-asserted-by":"crossref","unstructured":"Nowak S, R\u00fcger S (2010) How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. In: Proceedings of the international conference on multimedia information retrieval. MIR \u201910. Philadelphia, Pennsylvania, USA, ACM","key":"55_CR73","DOI":"10.1145\/1743384.1743478"},{"unstructured":"T. A. for Open Media. AV1 Image File Format (AVIF). 2019. https:\/\/aomediacodec.github.io\/av1-avif\/ (visited on 03\/07\/2020)","key":"55_CR74"},{"doi-asserted-by":"crossref","unstructured":"Perry S, Cruz L, Dumic E, Nguyen N, Pinheiro A, Alexiou E (2021) Comparison of remote subjective assessment strategies in the context of the JPEG Pleno point cloud activity. In: IEEE MMSP, 1\u20136","key":"55_CR75","DOI":"10.1109\/MMSP53017.2021.9733709"},{"unstructured":"Pinson M (2014) A new method for immersive audiovisual subjective testing.","key":"55_CR76"},{"unstructured":"Pinson MH (2019) ITS4S3: a video quality dataset with unrepeated videos, camera impairments, and public safety scenarios. Tech. rep. NTIA Technical Memorandum 19-538","key":"55_CR77"},{"doi-asserted-by":"crossref","unstructured":"Pinson MH, Wolf S (2003) Comparing subjective video quality testing methodologies. In: Visual communications and image processing 2003. Vol. 5150. International Society for Optics and Photonics. 573\u2013582","key":"55_CR78","DOI":"10.1117\/12.509908"},{"key":"55_CR79","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1016\/j.image.2014.10.009","volume":"30","author":"N Ponomarenko","year":"2015","unstructured":"Ponomarenko N et al (2015) Image database TID2013: Peculiarities, results and perspectives. Signal Process Image Commun 30:57\u201377","journal-title":"Signal Process Image Commun"},{"doi-asserted-by":"crossref","unstructured":"Raake A, Egger S (2014) Quality and quality of experience. In: M\u00f6ller S, Raake A (eds) Quality of experience: advanced concepts, applications and methods. Springer International Publishing, Cham, 11\u201333. https:\/\/doi.org\/10.1007\/978-3-319-02681-72","key":"55_CR80","DOI":"10.1007\/978-3-319-02681-7_2"},{"doi-asserted-by":"crossref","unstructured":"Raake A, et al (2020) Multi-model standard for bitstream-, pixel-based and hybrid video quality assessment of UHD\/4K: ITUT P.1204. In: IEEE Access 8, 193020\u2013193049. https:\/\/ieeexplore.ieee.org\/document\/9234526?source$=$authoralert","key":"55_CR81","DOI":"10.1109\/ACCESS.2020.3032080"},{"doi-asserted-by":"crossref","unstructured":"Rainer B, Timmerer C (2014) Quality of experience of web-based adaptive HTTP streaming clients in real-world environments using crowdsourcing. In: Proceedings of the workshop on design, quality and deployment of adaptive video streaming. VideoNext \u201914. Australia: ACM","key":"55_CR82","DOI":"10.1145\/2676652.2676656"},{"doi-asserted-by":"crossref","unstructured":"Ramachandra Rao RR, G\u00f6ring S, Vogel P, Pachatz N, Villarreal JJV, Robitza W, List P, Feiten B, Raake A (2019) Adaptive video streaming with current codecs and formats: extensions to parametric video quality model ITU-T P. 1203. In: Electronic imaging","key":"55_CR83","DOI":"10.2352\/ISSN.2470-1173.2019.10.IQSP-314"},{"doi-asserted-by":"crossref","unstructured":"Rao R, G\u00f6ring S, Steger R, Zadtootaghaj S, Barman N, Fremerey S, M\u00f6ller S, Raake A (2020) A large-scale evaluation of the bitstream-based video-quality model ITU-T P.1204.3 on gaming content. In: 2020 IEEE 22st international workshop on multimedia signal processing (MMSP). IEEE, 1\u20136","key":"55_CR84","DOI":"10.1109\/MMSP48831.2020.9287055"},{"doi-asserted-by":"crossref","unstructured":"Rao R, G\u00f6ring S, List P, Robitza W, Feiten B, W\u00fcstenhagen U, Raake A (2020) Bitstream-based model standard for 4K\/UHD: ITU-T P.1204.3 \u2013 model details, evaluation, analysis and open source implementation. In: Twelfth international conference on quality of multimedia experience (QoMEX). Athlone, Ireland","key":"55_CR85","DOI":"10.1109\/QoMEX48832.2020.9123110"},{"unstructured":"Rao R, G\u00f6ring S, Robitza W, Feiten B, Raake A (2019) AVT-VQDB-UHD-1: a large scale video quality database for UHD-1. In: 21st IEEE international symposium on multimedia (IEEE ISM). 1\u20138","key":"55_CR86"},{"unstructured":"Rao Ramachandra Rao R, G\u00f6ring S, Raake A. Towards high resolution video quality assessment in the crowd. In: QoMEX","key":"55_CR87"},{"unstructured":"Rashtchian C, Young P, Hodosh M, Hockenmaier J (2010) Collecting image annotations using Amazon\u2019s mechanical turk. In: Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with amazon\u2019s mechanical turk. Association for Computational Linguistics, Los Angeles","key":"55_CR88"},{"unstructured":"Recommendation I (2008) P. 910, Subjective video quality assessment methods for multimedia applications. In: International telecommunication union, Tech. Rep","key":"55_CR89"},{"doi-asserted-by":"crossref","unstructured":"Redi JA, Hosfeld T, Korshunov P, Mazza F, Povoa I, Keimel C (2013) Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal. In: Proceedings of the 2nd ACM international workshop on crowdsourcing for multimedia, 29\u201334","key":"55_CR90","DOI":"10.1145\/2506364.2506368"},{"doi-asserted-by":"crossref","unstructured":"Regal G, Schatz R, Schrammel J, Suette S (2018) VRate: a unity3D asset for integrating subjective assessment questionnaires in virtual environments. In: QoMEX. IEEE","key":"55_CR91","DOI":"10.1109\/QoMEX.2018.8463296"},{"doi-asserted-by":"crossref","unstructured":"Robitza W, Dethof A, G\u00f6ring S, Raake A, Polzehl T, Beyer A (2020) Are you still watching? Streaming video quality and engagement assessment in the crowd. In: Twelfth international conference on quality of multimedia experience (QoMEX). Athlone, Ireland","key":"55_CR92","DOI":"10.1109\/QoMEX48832.2020.9123148"},{"doi-asserted-by":"crossref","unstructured":"Robitza W, Kittur DG, Dethof AM, G\u00f6ring S, Feiten B, Raake A (2018) Measuring YouTube QoE with ITU-T P. 1203 under constrained bandwidth conditions. In: Tenth international conference on quality of multimedia experience (QoMEX). IEEE, 1\u20136","key":"55_CR93","DOI":"10.1109\/QoMEX.2018.8463363"},{"doi-asserted-by":"crossref","unstructured":"Robitza W, Rao RRR, G\u00f6ring S, Raake A (2021) Impact of spatial and temporal information on video quality and compressibility. In: 2021 13th international conference on quality of multimedia experience (QoMEX). IEEE, 65\u2013 68","key":"55_CR94","DOI":"10.1109\/QoMEX51781.2021.9465452"},{"doi-asserted-by":"crossref","unstructured":"Robitza W, et al (2018) HTTP adaptive streaming QoE estimation with ITU-T Rec. P.1203 \u2013 open databases and software. In: 9th ACM multimedia systems conference. Amsterdam","key":"55_CR95","DOI":"10.1145\/3204949.3208124"},{"doi-asserted-by":"crossref","unstructured":"Salas OF, Adzic V, Kalva H (2013) Subjective quality evaluations using crowdsourcing. In: 2013 picture coding symposium (PCS). IEEE, 418\u2013421","key":"55_CR96","DOI":"10.1109\/PCS.2013.6737772"},{"unstructured":"Saupe D, Hahn F, Hosu V, Zingman I, Rana M, Li S (2016) Crowd workers proven useful: a comparative study of subjective video quality assessment. In: QoMEX 2016: 8th international conference on quality of multimedia experience","key":"55_CR97"},{"doi-asserted-by":"crossref","unstructured":"Schoeffler M, Bartoschek S, St\u00f6ter F-R, Roess M, Westphal S, Edler B, Herre J (2018) webMUSHRA\u2013a comprehensive framework for web-based listening tests. J Open Res Softw 6.1","key":"55_CR98","DOI":"10.5334\/jors.187"},{"doi-asserted-by":"crossref","unstructured":"Seufert M, Hossfeld T (2016) One shot crowdtesting: approaching the extremes of crowdsourced subjective quality testing. In: Aug., 122\u2013126","key":"55_CR99","DOI":"10.21437\/PQS.2016-26"},{"doi-asserted-by":"crossref","unstructured":"Shahid M, Sogaard J, Pokhrel J, Brunnstr\u00f6m K, Wang K, Tavakoli S, Gracia N (2014) Crowdsourcing based subjective quality assessment of adaptive video streaming. In: QoMEX","key":"55_CR100","DOI":"10.1109\/QoMEX.2014.6982289"},{"key":"55_CR101","doi-asserted-by":"publisher","first-page":"430","DOI":"10.1109\/TIP.2005.859378","volume":"15.2","author":"HR Sheikh","year":"2006","unstructured":"Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15.2:430\u2013444","journal-title":"IEEE Trans Image Process"},{"key":"55_CR102","doi-asserted-by":"publisher","first-page":"3440","DOI":"10.1109\/TIP.2006.881959","volume":"15.11","author":"HR Sheikh","year":"2006","unstructured":"Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15.11:3440\u20133451","journal-title":"IEEE Trans Image Process"},{"doi-asserted-by":"crossref","unstructured":"Singla A, G\u00f6ring S, Raake A, Meixner B, Koenen R, Buchholz T (2019) Subjective quality evaluation of tile-based streaming for omnidirectional videos. In: Proceedings of the 10th ACM multimedia systems conference, pp 232\u2013242","key":"55_CR103","DOI":"10.1145\/3304109.3306218"},{"doi-asserted-by":"crossref","unstructured":"Singla A, Rao RRR, G\u00f6ring S, Raake A (2019) Assessing media qoe, simulator sickness and presence for omnidirectional videos with different test protocols. In: Conference on VR and 3D user interfaces (VR). IEEE, pp 1163\u20131164","key":"55_CR104","DOI":"10.1109\/VR.2019.8798291"},{"doi-asserted-by":"crossref","unstructured":"Sinno Z, Bovik AC (2019) Large-scale study of perceptual video quality. In: IEEE Trans Image Process 28(2)","key":"55_CR105","DOI":"10.1109\/TIP.2018.2869673"},{"unstructured":"Su H, Deng J, Fei-Fei L (2012) Crowdsourcing annotations for visual object detection.","key":"55_CR106"},{"unstructured":"Su S, Hosu V, Lin H, Zhang Y, Saupe D (2021) KonIQ$$++$$: boosting no-reference image quality assessment in the wild by jointly predicting image quality and defects. In: The 32nd British machine vision conference (BMVC)","key":"55_CR107"},{"doi-asserted-by":"crossref","unstructured":"Tang A, Boring S (2012) #EpicPlay: crowd-sourcing sports video highlights. In: Proceedings of the SIGCHI conference on human factors in computing systems. CHI \u201912. Austin, Texas, USA: Association for Computing Machinery","key":"55_CR108","DOI":"10.1145\/2207676.2208622"},{"key":"55_CR109","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1145\/2812802","volume":"59.2","author":"B Thomee","year":"2016","unstructured":"Thomee B, Shamma DA, Friedland G, Elizalde B, Ni K, Poland D, Borth D, Li L-J (2016) YFCC100M: the new data in multimedia research. Commun ACM 59.2:64\u201373","journal-title":"Commun ACM"},{"doi-asserted-by":"crossref","unstructured":"Uhrina M, Bienik J, Mizdos T (2020) QoE on H.264 and H.265: crowdsourcing versus laboratory testing. In: 2020 30th international conference radioelektronika (RADIOELEKTRONIKA), 1\u20136","key":"55_CR110","DOI":"10.1109\/RADIOELEKTRONIKA49387.2020.9092424"},{"doi-asserted-by":"crossref","unstructured":"Van Wallendael G, Coppens P, Paridaens T, Van Kets N, Van den Broeck W, Lambert P (2016) Perceptual quality of 4Kresolution video content compared to HD. In: 8th international conference on quality of multimedia experience (QoMEX), pp. 1\u20136","key":"55_CR111","DOI":"10.1109\/QoMEX.2016.7498935"},{"key":"55_CR112","doi-asserted-by":"publisher","first-page":"600","DOI":"10.1109\/TIP.2003.819861","volume":"13.4","author":"Z Wang","year":"2004","unstructured":"Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13.4:600\u2013612","journal-title":"IEEE Trans Image Process"},{"doi-asserted-by":"crossref","unstructured":"Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: Conference record of the thirty-seventh asilomar conference on signals, systems and computers, 2004. Vol. 2. IEEE. 1398\u20131402","key":"55_CR113","DOI":"10.1109\/ACSSC.2003.1292216"},{"doi-asserted-by":"crossref","unstructured":"Wiedemann O, Hosu V, Lin H, Saupe D (2018) \u201cDisregarding the big picture: Towards local image quality assessment\u201d. In: 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX). IEEE. 1\u20136","key":"55_CR114","DOI":"10.1109\/QoMEX.2018.8463384"},{"doi-asserted-by":"crossref","unstructured":"Xu J, Jiang X (2013) Research on subjective assessment method of ultra high definition video quality. In: 4th world congress on software engineering, 326\u2013330","key":"55_CR115","DOI":"10.1109\/WCSE.2013.61"},{"doi-asserted-by":"crossref","unstructured":"Zou N, Zhang H, Cricri F, Tavakoli H, Lainema J, Hannuksela M, Aksu E, Rahtu E (2020) L2C - learning to learn to compress. In: 2020 IEEE 22st international workshop on multimedia signal processing (MMSP). IEEE. 1\u20136","key":"55_CR116","DOI":"10.1109\/MMSP48831.2020.9287069"}],"container-title":["Quality and User Experience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41233-023-00055-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41233-023-00055-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41233-023-00055-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,11]],"date-time":"2024-12-11T21:59:54Z","timestamp":1733954394000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41233-023-00055-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,13]]},"references-count":116,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["55"],"URL":"https:\/\/doi.org\/10.1007\/s41233-023-00055-6","relation":{},"ISSN":["2366-0139","2366-0147"],"issn-type":[{"type":"print","value":"2366-0139"},{"type":"electronic","value":"2366-0147"}],"subject":[],"published":{"date-parts":[[2023,4,13]]},"assertion":[{"value":"30 June 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 April 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Conflict of interest of potential conflicts of interest: The authors have no potential conflicts of interest to disclose. Research involving Human Participants: Participants are asked in online or lab studies to rate specific aspects of image and video quality. All guidelines pertaining to such online\/lab studies conducted in the literature are followed. Informed consent: All participants in the lab\/online studies were made aware of the relevant data privacy aspects and also potential health affecting aspects if any.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"2"}}