{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T04:28:53Z","timestamp":1772166533371,"version":"3.50.1"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T00:00:00Z","timestamp":1718582400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T00:00:00Z","timestamp":1718582400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007210","name":"RWTH Aachen University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100007210","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Image Video Proc."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We present a study on the validity of quality assessment in the context of the development of visual media coding schemes. The work is motivated by the need for reliable means for decision-taking in standardization efforts of MPEG and JVET, i.e., the adoption or rejection of coding tools during the development process of the coding standard. The study includes results considering three means: objective quality metrics, remote expert viewing, which is a method designed in the context of MPEG standardization, and formal laboratory visual evaluation. The focus of this work is on the comparison of pairs of coded video sequences, e.g., a proposed change and an anchor scheme at a given rate point. An aggregation of performance measurements across multiple rate points, such as the Bj\u00f8ntegaard Delta rate, is out of the scope of this paper. The paper details the test setup for the subjective assessment methods and the objective quality metrics under consideration. The results of the three approaches are reviewed, analyzed, and compared with respect to their suitability for the decision-taking task. The study indicates that, subject to the chosen test content and test protocols, the results of remote expert viewing using a forced-choice scale can be considered more discriminatory than the results of na\u00efve viewers in the laboratory tests. The results further that, in general, the well-established quality metrics, such as PSNR, SSIM, or MS-SSIM, exhibit a high rate of correct decision-making when their results are compared with both types of viewing tests. Among the learning-based metrics, VMAF and AVQT appear to be most robust. For the development process of a coding standard, the selection of the most suitable means must be guided by the context, where a small number of carefully selected objective metrics, in combination with viewing tests for unclear cases, appears recommendable.<\/jats:p>","DOI":"10.1186\/s13640-024-00630-7","type":"journal-article","created":{"date-parts":[[2024,6,17]],"date-time":"2024-06-17T08:02:24Z","timestamp":1718611344000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Remote expert viewing, laboratory tests or objective metrics: which one(s) to trust?"],"prefix":"10.1186","volume":"2024","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8724-2752","authenticated-orcid":false,"given":"Mathias","family":"Wien","sequence":"first","affiliation":[]},{"given":"Joel","family":"Jung","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,6,17]]},"reference":[{"key":"630_CR1","unstructured":"F. Bossen, X. Li, V. Seregin, K. Sharman, K. Suehring, VTM and HM common test conditions and software reference configurations for SDR 4:2:0 10 bit video. Doc. JVET-AB2010, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, Mainz, DE, 28th meeting (2022),\u00a0https:\/\/jvet-experts.org\/doc_end_user\/documents\/28_Mainz\/wg11\/JVET-AB2010-v1.zip. Accessed 12 May 2023"},{"key":"630_CR2","unstructured":"J. Str\u00f6m, K. Andersson, R. Sj\u00f6berg, A. Segall, F. Bossen, G. Sullivan, J.-R. Ohm, A. Tourapis, Working practices using objective metrics forevaluation of video coding efficiency experiments. Doc. HSTP-VID-WPOM, International Telecommunication Union (2020),\u00a0http:\/\/handle.itu.int\/11.1002\/pub\/8160e8da-en . Accessed 12 May 2023"},{"key":"630_CR3","unstructured":"M. Wien, V. Baroncini, VVC verification test report for Ultra High Definition (UHD) Standard Dynamic Range (SDR) Video content. Doc. JVET-T2020, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, Teleconference 20nd meeting (2020),\u00a0https:\/\/jvet-experts.org\/doc_end_user\/documents\/20_Teleconference\/wg11\/JVET-T2020-v1.zip. Accessed 12 May 2023"},{"key":"630_CR4","unstructured":"M. Wien, V. Baroncini, VC verification test report for hd sdr and 360$$^\\circ$$ video content. Doc. JVET-V2020, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, Teleconference 22nd meeting (2021),\u00a0https:\/\/jvet-experts.org\/doc_end_user\/documents\/22_Teleconference\/wg11\/JVET-V2020-v1.zip. Accessed 12 May 2023"},{"key":"630_CR5","unstructured":"M. Wien, V. Baroncini, VVC verification test report for high dynamic range video content. Doc. JVET-W2020, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, Teleconference 23rd meeting (2021),\u00a0https:\/\/jvet-experts.org\/doc_end_user\/documents\/23_Teleconference\/wg11\/JVET-W2020-v1.zip. Accessed 12 May 2023"},{"key":"630_CR6","unstructured":"M. Wien, L. Yu, V. Baroncini, Guidelines for verification testing of visual media specifications. Doc. AG5N39, ISO\/IEC JTC 1\/SC 29\/AG 5 MPEG Visual Quality Evaluation, Teleconference 5th meeting (2021),\u00a0https:\/\/www.mpeg.org\/wp-content\/uploads\/mpeg_meetings\/136_OnLine\/w20975.zip. Accessed 12 May 2023"},{"issue":"9","key":"630_CR7","doi-asserted-by":"publisher","first-page":"1521","DOI":"10.1109\/jproc.2021.3062590","volume":"109","author":"JM Boyce","year":"2021","unstructured":"J.M. Boyce, R. Dore, A. Dziembowski, J. Fleureau, J. Jung, B. Kroon, B. Salahieh, V.K.M. Vadakital, L. Yu, MPEG immersive video coding standard. Proc. IEEE 109(9), 1521\u20131536 (2021). https:\/\/doi.org\/10.1109\/jproc.2021.3062590","journal-title":"Proc. IEEE"},{"key":"630_CR8","unstructured":"J. Jung, B. Kroon, Common test conditions for mpeg immersive video. Doc. WG4N203, ISO\/IEC JTC 1\/SC 29\/WG 4 MPEG Video Coding, online, 7th meeting (2022)"},{"issue":"1","key":"630_CR9","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1109\/jetcas.2018.2885981","volume":"9","author":"S Schwarz","year":"2019","unstructured":"S. Schwarz, M. Preda, V. Baroncini, M. Budagavi, P. Cesar, P.A. Chou, R.A. Cohen, M. Krivokuca, S. Lasserre, Z. Li, J. Llach, K. Mammou, R. Mekuria, O. Nakagami, E. Siahaan, A. Tabatabai, A.M. Tourapis, V. Zakharchenko, Emerging MPEG standards for point cloud compression. IEEE J. Emerg. Select. Top. Circ. Syst. 9(1), 133\u2013148 (2019). https:\/\/doi.org\/10.1109\/jetcas.2018.2885981","journal-title":"IEEE J. Emerg. Select. Top. Circ. Syst."},{"key":"630_CR10","unstructured":"MPEG 3D Graphics: CfP for dynamic mesh coding. Doc. WG7N231, ISO\/IEC JTC 1\/SC 29\/WG 7 MPEG 3D Graphics, online, 5th meeting (2021),\u00a0https:\/\/www.mpeg.org\/wp-content\/uploads\/mpeg_meetings\/136_OnLine\/w21000.zip. Accessed 12 May 2023"},{"key":"630_CR11","unstructured":"ISO\/IEC JTC 1\/SC 29: MPEG visual quality assessment: terms of reference. Doc. SC29N19020, ISO\/IEC JTC 1\/SC 29 (2020)"},{"key":"630_CR12","unstructured":"J. Jung, M. Wien, V. Baroncini, Guidelines for remote experts viewing sessions. Doc. N40, ISO\/IEC JTC 1\/SC 29\/AG 5 MPEG Visual Quality Evaluation, Teleconference 5th meeting (2021),\u00a0https:\/\/www.mpeg.org\/wp-content\/uploads\/mpeg_meetings\/136_OnLine\/w20976.zip . Accessed 12 May 2023"},{"key":"630_CR13","unstructured":"J. Jung, M. Wien, Analysis of the MIV verification test dry run and recommendations. Doc. m59791, ISO\/IEC JTC 1\/SC 29\/AG 5 MPEG Visual Quality Evaluation, online, 7th meeting (2022)"},{"key":"630_CR14","doi-asserted-by":"publisher","unstructured":"T. Tominaga, T. Hayashi, J. Okamoto, A. Takahashi, Performance comparisons of subjective quality assessment methods for mobile video. In: Second International Workshop on Quality of Multimedia Experience (QoMEX). IEEE (2010). https:\/\/doi.org\/10.1109\/qomex.2010.5517948","DOI":"10.1109\/qomex.2010.5517948"},{"key":"630_CR15","doi-asserted-by":"publisher","unstructured":"T. Kawano, K. Yamagishi, T. Hayashi, Performance comparison of subjective assessment methods for 3d video quality. In, Fourth International Workshop on Quality of Multimedia Experience. IEEE (2012). https:\/\/doi.org\/10.1109\/qomex.2012.6263833","DOI":"10.1109\/qomex.2012.6263833"},{"issue":"1","key":"630_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3427931","volume":"18","author":"Y Nehm\u00e9","year":"2020","unstructured":"Y. Nehm\u00e9, J.-P. Farrugia, F. Dupont, P.L. Callet, G. Lavou\u00e9, Comparison of subjective methods for quality assessment of 3d graphics in virtual reality. ACM Trans Appl Percept 18(1), 1\u201323 (2020). https:\/\/doi.org\/10.1145\/3427931","journal-title":"ACM Trans Appl Percept"},{"key":"630_CR17","doi-asserted-by":"publisher","first-page":"3442","DOI":"10.1109\/tmm.2021.3098450","volume":"24","author":"P Perez","year":"2021","unstructured":"P. Perez, L. Janowski, N. Garcia, M. Pinson, Subjective assessment experiments that recruit few observers with repetitions (fowr). IEEE Trans. Multimedia 24, 3442\u20133454 (2021). https:\/\/doi.org\/10.1109\/tmm.2021.3098450","journal-title":"IEEE Trans. Multimedia"},{"key":"630_CR18","doi-asserted-by":"crossref","unstructured":"A. Pastor, P. David, I. Katsavounidis, . Krasula, H. Tmar, P. Le\u00a0Callet, \u2019Discriminability-Experimental Cost\u2019 Tradeoff in Subjective Video Quality Assessment of Codec: DCR with EVP Rating Scale Versus ACR-HR. https:\/\/hal.science\/hal-04363990. Accessed 17 May 2024","DOI":"10.1109\/PCS60826.2024.10566279"},{"key":"630_CR19","doi-asserted-by":"publisher","unstructured":"A. Pastor, P. Lebreton, T. Vigier, P.L. Callet, Comparison of conditions for omnidirectional video with spatial audio in terms of subjective quality and impacts on objective metrics resolving power. In: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. https:\/\/doi.org\/10.1109\/icassp48485.2024.10448123","DOI":"10.1109\/icassp48485.2024.10448123"},{"key":"630_CR20","unstructured":"M. Wien, DNN viewing report. Doc. JVET-U0142, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, online, 21st meeting (2021)"},{"key":"630_CR21","unstructured":"M. Wien, EE1-related: Report on results of remote viewing session. Doc. JVET-V0173, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, online, 22nd meeting (2021)"},{"key":"630_CR22","unstructured":"M. Wien, V. Baroncini, A. Segall, EE1-related: Report on results of remote viewing session. Doc. JVET-W0186, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, online, 23rd meeting (2021)"},{"key":"630_CR23","unstructured":"M. Wien, EE1-related: Report on results of JVET-X remote viewing session. Doc. JVET-X0209, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, online, 24th meeting (2021)"},{"key":"630_CR24","unstructured":"M. Wien, AHG4: REV result for AHG11\/EE1 and AHG10\/Deblocking. Doc. JVET-Y0212, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, online, 25th meeting (2022)"},{"key":"630_CR25","unstructured":"M. Wien, [AHG4] REV result for AHG11\/EE1 and AHG4 new test sequences. Doc. JVET-Z0053, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, online, 26th meeting (2022)"},{"key":"630_CR26","unstructured":"ITU-R BT.500-14: Methodologies for the Subjective Assessment of the Quality of Television Images. ITU-R. https:\/\/www.itu.int\/rec\/R-REC-BT.500-14-201910-I\/en Accessed 2023-05-12"},{"key":"630_CR27","unstructured":"ITU-T P.808: Subjective Evaluation of Speech Quality with a Crowdsourcing Approach. ITU-T. https:\/\/www.itu.int\/rec\/T-REC-P.808\/en. Accessed 12 May 2023"},{"key":"630_CR28","unstructured":"ITU-T P.910: Subjective Video Quality Assessment Methods for Multimedia Applications. ITU-T. http:\/\/www.itu.int\/rec\/T-REC-P.910-200804-I\/en. Accessed 12 May 2023"},{"key":"630_CR29","unstructured":"M. Wien, V. Baroncini, Results of subjective testing of responses to the cfp for dynamic mesh coding. Doc. N57, ISO\/IEC JTC 1\/SC 29\/AG 5 MPEG Visual Quality Evaluation, Teleconference 7th meeting (2022)"},{"key":"630_CR30","unstructured":"FFmpeg. https:\/\/ffmpeg.org\/. Accessed 12 May 2023"},{"key":"630_CR31","unstructured":"VideoLan: VLC Media Player. https:\/\/www.videolan.org\/vlc\/index.html. Accessed 12 May 2023"},{"key":"630_CR32","unstructured":"V. Baroncini, A. Norkin, A.M. Kotra, K. Andersson, K. Misra, H. Jang, C.M. Tsai, D. Rusanovskyy, Subjective assessment of CE11 (deblocking filter) proposals. Doc. JVET-M0906, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, Marrakech, 13th meeting (2019)"},{"key":"630_CR33","unstructured":"M. Wien, P. Hanhart, Core experiment viewing test procedure and results. Doc. JVET-N0835, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, Geneva, 14th meeting (2019)"},{"key":"630_CR34","unstructured":"M. Wien, Core experiment viewing test procedure and results. Doc. JVET-O1118, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO\/IEC JTC 1\/SC 29, Gotenburg, 15th meeting (2019)"},{"key":"630_CR35","doi-asserted-by":"publisher","unstructured":"P. Hanhart, L. Krasula, P.L. Callet, T. Ebrahimi, How to benchmark objective quality metrics from paired comparison data? In: 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX). IEEE. https:\/\/doi.org\/10.1109\/qomex.2016.7498960","DOI":"10.1109\/qomex.2016.7498960"},{"key":"630_CR36","doi-asserted-by":"publisher","unstructured":"L. Krasula, K. Fliegel, P. Le\u00a0Callet, M. Klima, On the accuracy of objective image and video quality models: New methodology for performance evaluation. In: 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX). IEEE. https:\/\/doi.org\/10.1109\/qomex.2016.7498936","DOI":"10.1109\/qomex.2016.7498936"},{"issue":"1","key":"630_CR37","first-page":">011003","volume":"19","author":"C Li","year":"2010","unstructured":"C. Li, A.C. Bovik, Content-weighted video quality assessment using a three-component image model. J. Electron. Imag. 19(1), 011003 (2010)","journal-title":"J. Electron. Imag."},{"key":"630_CR38","unstructured":"F. Xiao, DCT-based video quality evaluation. Doc. Vol. 769, Final Project for EE392J"},{"key":"630_CR39","doi-asserted-by":"publisher","unstructured":"H.R. Sheikh, A.C. Bovik, Image information and visual quality 15(2), 430\u2013444. https:\/\/doi.org\/10.1109\/tip.2005.859378","DOI":"10.1109\/tip.2005.859378"},{"key":"630_CR40","unstructured":"AVQT. https:\/\/developer.apple.com\/videos\/play\/wwdc2021\/10145\/. Accessed 12 May 2023"},{"key":"630_CR41","unstructured":"VMAF. https:\/\/netflixtechblog.com\/toward-a-practical-perceptual-video-quality-metric-653f208b9652. Accessed 12 May 2023"},{"key":"630_CR42","doi-asserted-by":"publisher","unstructured":"S. Bosse, D. Maniry, K.R. Muller, T. Wiegand, W. Samek, Deep neural networks for no-reference and full-reference image quality assessment.\u00a0 IEEE Trans. Image Process. 2017, 27(1), 206\u2013219.\u00a0https:\/\/doi.org\/10.1109\/tip.2017.2760518","DOI":"10.1109\/tip.2017.2760518"},{"key":"630_CR43","doi-asserted-by":"publisher","unstructured":"R. Zhang, P. Isola, A.A. Efros, E. Shechtman, O. Wang, The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition. IEEE. https:\/\/doi.org\/10.1109\/cvpr.2018.00068","DOI":"10.1109\/cvpr.2018.00068"},{"key":"630_CR44","doi-asserted-by":"publisher","unstructured":"K. Ding, K. Ma, S. Wang, E.P. Simoncelli, Image quality assessment: Unifying structure and texture similarity, 1\u20131. https:\/\/doi.org\/10.1109\/tpami.2020.3045810","DOI":"10.1109\/tpami.2020.3045810"},{"key":"630_CR45","unstructured":"NIST\/SEMATECH: e-Handbook of Statistical Methods. http:\/\/www.itl.nist.gov\/div898\/handbook\/. Accessed 12 May 2023"}],"container-title":["EURASIP Journal on Image and Video Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13640-024-00630-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13640-024-00630-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13640-024-00630-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,4]],"date-time":"2024-07-04T11:06:18Z","timestamp":1720091178000},"score":1,"resource":{"primary":{"URL":"https:\/\/jivp-eurasipjournals.springeropen.com\/articles\/10.1186\/s13640-024-00630-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,17]]},"references-count":45,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["630"],"URL":"https:\/\/doi.org\/10.1186\/s13640-024-00630-7","relation":{"is-basis-for":[{"id-type":"doi","id":"10.52843\/cassyni.n1l6n7","asserted-by":"object"}]},"ISSN":["1687-5281"],"issn-type":[{"value":"1687-5281","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,17]]},"assertion":[{"value":"22 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 May 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 June 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All viewers were volunteers consenting to participation. Volunteers participating as na\u00efve viewers in the laboratory test sessions were recruited from students at RWTH Aachen University and were paid for participating in the tests. Viewers participating in expert viewing sessions were volunteers of organizations and companies participating in JVET and\/or MPEG.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"16"}}