{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T15:11:30Z","timestamp":1781622690516,"version":"3.54.5"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,12,31]],"date-time":"2020-12-31T00:00:00Z","timestamp":1609372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"rench National Research Agency as part of ANR-PISCo","award":["ANR-17-CE33-0005"],"award-info":[{"award-number":["ANR-17-CE33-0005"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Appl. Percept."],"published-print":{"date-parts":[[2021,1,31]]},"abstract":"<jats:p>Numerous methodologies for subjective quality assessment exist in the field of image processing. In particular, the Absolute Category Rating with Hidden Reference (ACR-HR), the Double Stimulus Impairment Scale (DSIS), and the Subjective Assessment Methodology for Video Quality (SAMVIQ) are considered three of the most prominent methods for assessing the visual quality of 2D images and videos. Are these methods valid\/accurate to evaluate the perceived quality of 3D graphics data? Is the presence of an explicit reference necessary, due to the lack of human prior knowledge on 3D graphics data compared to natural images\/videos? To answer these questions, we compare these three subjective methods (ACR-HR, DSIS, and SAMVIQ) on a dataset of high-quality colored 3D models, impaired with various distortions. These subjective experiments were conducted in a virtual reality environment. Our results show differences in the performance of the methods depending on the 3D contents and the types of distortions. We show that DSIS and SAMVIQ outperform ACR-HR in terms of accuracy and point out a stable performance. In regard to the time-effort, DSIS achieves the highest accuracy in the shortest assessment time. Results also yield interesting conclusions on the importance of a reference for judging the quality of 3D graphics. We finally provide recommendations regarding the influence of the number of observers on the accuracy.<\/jats:p>","DOI":"10.1145\/3427931","type":"journal-article","created":{"date-parts":[[2021,1,1]],"date-time":"2021-01-01T05:08:46Z","timestamp":1609477726000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["Comparison of Subjective Methods for Quality Assessment of 3D Graphics in Virtual Reality"],"prefix":"10.1145","volume":"18","author":[{"given":"Yana","family":"Nehm\u00e9","sequence":"first","affiliation":[{"name":"Univ Lyon, LIRIS CNRS, Villeurbanne, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jean-Philippe","family":"Farrugia","sequence":"additional","affiliation":[{"name":"Univ Lyon, LIRIS CNRS, Villeurbanne, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Florent","family":"Dupont","sequence":"additional","affiliation":[{"name":"Univ Lyon, LIRIS CNRS, Villeurbanne, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick Le","family":"Callet","sequence":"additional","affiliation":[{"name":"Univ Nantes, LS2N CNRS, Nantes, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillaume","family":"Lavou\u00e9","sequence":"additional","affiliation":[{"name":"Univ Lyon, LIRIS CNRS, Villeurbanne, France"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,12,31]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX\u201917)","author":"Alexiou E.","unstructured":"E. Alexiou and T. Ebrahimi . 2017. On subjective and objective quality evaluation of point cloud geometry . In Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX\u201917) . 1--3. E. Alexiou and T. Ebrahimi. 2017. On subjective and objective quality evaluation of point cloud geometry. In Proceedings of the 9th International Conference on Quality of Multimedia Experience (QoMEX\u201917). 1--3."},{"key":"e_1_2_2_2_1","volume-title":"Andrew G. Tescher (Ed.)","volume":"10396","author":"Alexiou Evangelos","year":"2017","unstructured":"Evangelos Alexiou and Touradj Ebrahimi . 2017 . On the performance of metrics to predict quality in point cloud representations. In Applications of Digital Image Processing XL , Andrew G. Tescher (Ed.) , Vol. 10396 . International Society for Optics and Photonics, 282--297. DOI:https:\/\/doi.org\/10.1117\/12.2275142 10.1117\/12.2275142 Evangelos Alexiou and Touradj Ebrahimi. 2017. On the performance of metrics to predict quality in point cloud representations. In Applications of Digital Image Processing XL, Andrew G. Tescher (Ed.), Vol. 10396. International Society for Optics and Photonics, 282--297. DOI:https:\/\/doi.org\/10.1117\/12.2275142"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1017\/ATSIP.2019.20"},{"key":"e_1_2_2_4_1","volume-title":"Proceedings of the 12fth International Conference on Quality of Multimedia Experience (QoMEX\u201920)","author":"Alexiou E.","unstructured":"E. Alexiou , N. Yang , and T. Ebrahimi . 2020. PointXR: A toolbox for visualization and subjective evaluation of point clouds in virtual reality . In Proceedings of the 12fth International Conference on Quality of Multimedia Experience (QoMEX\u201920) . 1--6. E. Alexiou, N. Yang, and T. Ebrahimi. 2020. PointXR: A toolbox for visualization and subjective evaluation of point clouds in virtual reality. In Proceedings of the 12fth International Conference on Quality of Multimedia Experience (QoMEX\u201920). 1--6."},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1117\/1.JEI.27.5.053013"},{"key":"e_1_2_2_6_1","volume-title":"Proceedings of the International Telecommunication Union.","author":"ITU-R","year":"2007","unstructured":"ITU-R BT.1788. 2007 . Methodology for the subjective assessment of video quality in multimedia applications . In Proceedings of the International Telecommunication Union. ITU-R BT.1788. 2007. Methodology for the subjective assessment of video quality in multimedia applications. In Proceedings of the International Telecommunication Union."},{"key":"e_1_2_2_7_1","volume-title":"Proceedings of the International Telecommunication Union.","author":"ITU-R","year":"2012","unstructured":"ITU-R BT.500-13. 2012 . Methodology for the subjective assessment of the quality of television pictures BT series broadcasting service . In Proceedings of the International Telecommunication Union. ITU-R BT.500-13. 2012. Methodology for the subjective assessment of the quality of television pictures BT series broadcasting service. In Proceedings of the International Telecommunication Union."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0165-1684(99)00018-3"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2006.886261"},{"key":"e_1_2_2_10_1","doi-asserted-by":"crossref","unstructured":"Luis A. da Silva Cruz Emil Dumic Evangelos Alexiou Joao Prazeres Rafael Duarte Manuela Pereira Antonio Pinheiro and Touradj Ebrahimi. 2019. Point cloud quality evaluation: Towards a definition for test conditions. In Proceedings of the International Conference on Quality of Multimedia Experience. 6. Retrieved from http:\/\/infoscience.epfl.ch\/record\/264995.  Luis A. da Silva Cruz Emil Dumic Evangelos Alexiou Joao Prazeres Rafael Duarte Manuela Pereira Antonio Pinheiro and Touradj Ebrahimi. 2019. Point cloud quality evaluation: Towards a definition for test conditions. In Proceedings of the International Conference on Quality of Multimedia Experience. 6. Retrieved from http:\/\/infoscience.epfl.ch\/record\/264995.","DOI":"10.1109\/QoMEX.2019.8743258"},{"key":"e_1_2_2_12_1","volume-title":"Heckbert","author":"Garland Michael","year":"1997","unstructured":"Michael Garland and Paul S . Heckbert . 1997 . Surface simplification using quadric error metrics. In Proceedings of the ACM Siggraph . 209--216. Michael Garland and Paul S. Heckbert. 1997. Surface simplification using quadric error metrics. In Proceedings of the ACM Siggraph. 209--216."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2996296"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBC.2010.2086750"},{"key":"e_1_2_2_15_1","volume-title":"Proceedings of the 3rd International Workshop on Video Processing and Quality Metrics for Consumer Electronics.","author":"Huynh-Thua Quan","year":"2007","unstructured":"Quan Huynh-Thua and Martlesham Heath . 2007 . Examination of the SAMVIQ methodology for the subjective assessment of multimedia quality . In Proceedings of the 3rd International Workshop on Video Processing and Quality Metrics for Consumer Electronics. Quan Huynh-Thua and Martlesham Heath. 2007. Examination of the SAMVIQ methodology for the subjective assessment of multimedia quality. In Proceedings of the 3rd International Workshop on Video Processing and Quality Metrics for Consumer Electronics."},{"key":"e_1_2_2_16_1","unstructured":"Alireza Javaheri Catarina Brites Fernando Pereira and Joao Ascenso. 2019. Point cloud rendering after coding: Impacts on subjective and objective quality. Retrieved from https:\/\/arxiv:eess.IV\/1912.09137.  Alireza Javaheri Catarina Brites Fernando Pereira and Joao Ascenso. 2019. Point cloud rendering after coding: Impacts on subjective and objective quality. Retrieved from https:\/\/arxiv:eess.IV\/1912.09137."},{"key":"e_1_2_2_17_1","first-page":"738","article-title":"Performance comparison of subjective assessment methods for stereoscopic 3D video quality. IEICE","volume":"4","author":"Kawano Taichi","year":"2014","unstructured":"Taichi Kawano , Kazuhisa Yamagishi , and Takanori Hayashi . 2014 . Performance comparison of subjective assessment methods for stereoscopic 3D video quality. IEICE Trans. Commun. E97.B , 4 (2014), 738 -- 745 . DOI:https:\/\/doi.org\/10.1587\/transcom.E97.B.738 10.1587\/transcom.E97.B.738 Taichi Kawano, Kazuhisa Yamagishi, and Takanori Hayashi. 2014. Performance comparison of subjective assessment methods for stereoscopic 3D video quality. IEICE Trans. Commun. E97.B, 4 (2014), 738--745. DOI:https:\/\/doi.org\/10.1587\/transcom.E97.B.738","journal-title":"Trans. Commun. E97.B"},{"key":"e_1_2_2_18_1","volume-title":"Performance comparison of subjective quality assessment methods for 4k video. IEICE Trans. 101-B","author":"Kawashima Kimiko","year":"2018","unstructured":"Kimiko Kawashima , Kazuhisa Yamagishi , and Takanori Hayashi . 2018. Performance comparison of subjective quality assessment methods for 4k video. IEICE Trans. 101-B ( 2018 ), 933--945. Kimiko Kawashima, Kazuhisa Yamagishi, and Takanori Hayashi. 2018. Performance comparison of subjective quality assessment methods for 4k video. IEICE Trans. 101-B (2018), 933--945."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/333329.333344"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1462048.1462052"},{"key":"e_1_2_2_21_1","volume-title":"Florent Dupont, Atilla Baskurt, and Touradj Ebrahimi.","author":"Lavoue Guillaume","year":"2006","unstructured":"Guillaume Lavoue , Elisa Drelie Gelasca , Florent Dupont, Atilla Baskurt, and Touradj Ebrahimi. 2006 . Perceptually driven 3D distance metrics with application to watermarking. Proceedings of the International Society for Optical Engineering (SPIE\u2019 06). 6312. DOI:https:\/\/doi.org\/10.1117\/12.686964 10.1117\/12.686964 Guillaume Lavoue, Elisa Drelie Gelasca, Florent Dupont, Atilla Baskurt, and Touradj Ebrahimi. 2006. Perceptually driven 3D distance metrics with application to watermarking. Proceedings of the International Society for Optical Engineering (SPIE\u201906). 6312. DOI:https:\/\/doi.org\/10.1117\/12.686964"},{"key":"#cr-split#-e_1_2_2_22_1.1","doi-asserted-by":"crossref","unstructured":"G. Lavou\u00e9 and R. K. Mantiuk. 2015. Quality assessment in computer graphics. In Visual Signal Quality Assessment: Quality of Experience. Springer 243--286. DOI:https:\/\/doi.org\/10.1007\/978-3-319-10368-6_9 10.1007\/978-3-319-10368-6_9","DOI":"10.1007\/978-3-319-10368-6_9"},{"key":"#cr-split#-e_1_2_2_22_1.2","doi-asserted-by":"crossref","unstructured":"G. Lavou\u00e9 and R. K. Mantiuk. 2015. Quality assessment in computer graphics. In Visual Signal Quality Assessment: Quality of Experience. Springer 243--286. DOI:https:\/\/doi.org\/10.1007\/978-3-319-10368-6_9","DOI":"10.1007\/978-3-319-10368-6_9"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00371-011-0602-y"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2012.03188.x"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.1.1.30"},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the International Telecommunication Union.","author":"ITU-T","year":"2009","unstructured":"ITU-T P.910. 2009 . Subjective video quality assessment methods for multimedia applications . In Proceedings of the International Telecommunication Union. ITU-T P.910. 2009. Subjective video quality assessment methods for multimedia applications. In Proceedings of the International Telecommunication Union."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2005.843364"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3047407"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/QoMEX.2018.8463296"},{"key":"e_1_2_2_30_1","volume-title":"Rushmeier","author":"Rogowitz Bernice E.","year":"2001","unstructured":"Bernice E. Rogowitz and Holly E . Rushmeier . 2001 . Are image quality metrics adequate to evaluate the quality of geometric objects? In Photonics West 2001-Electronic Imaging. In Proceedings of International Society for Optics and Photonics (SPIE\u2019 01). DOI:https:\/\/doi.org\/10.1117\/12.429504 10.1117\/12.429504 Bernice E. Rogowitz and Holly E. Rushmeier. 2001. Are image quality metrics adequate to evaluate the quality of geometric objects? In Photonics West 2001-Electronic Imaging. In Proceedings of International Society for Optics and Photonics (SPIE\u201901). DOI:https:\/\/doi.org\/10.1117\/12.429504"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2010.2042111"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.2352\/ISSN.2470-1173.2018.14.HVEI-525"},{"key":"e_1_2_2_33_1","volume-title":"Proceedings of the International Workshop on Image Media Quality and its Applications (IMQA\u201908)","author":"Pechard Stephane","year":"2008","unstructured":"Stephane Pechard , Romuald Pepion , Patrick Le Callet . 2008 . Suitable methodology in subjective video quality assessment: A resolution dependent paradigm . In Proceedings of the International Workshop on Image Media Quality and its Applications (IMQA\u201908) . Stephane Pechard, Romuald Pepion, Patrick Le Callet. 2008. Suitable methodology in subjective video quality assessment: A resolution dependent paradigm. In Proceedings of the International Workshop on Image Media Quality and its Applications (IMQA\u201908)."},{"key":"e_1_2_2_34_1","volume-title":"Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR\u201920)","author":"Subramanyam S.","unstructured":"S. Subramanyam , J. Li , I. Viola , and P. Cesar . 2020. Comparing the quality of highly realistic digital humans in 3DoF and 6DoF: A volumetric video case study . In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR\u201920) . 127--136. S. Subramanyam, J. Li, I. Viola, and P. Cesar. 2020. Comparing the quality of highly realistic digital humans in 3DoF and 6DoF: A volumetric video case study. In Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces (VR\u201920). 127--136."},{"key":"e_1_2_2_35_1","volume-title":"Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX\u201910)","author":"Tominaga T.","unstructured":"T. Tominaga , T. Hayashi , J. Okamoto , and A. Takahashi . 2010. Performance comparisons of subjective quality assessment methods for mobile video . In Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX\u201910) . 82--87. T. Tominaga, T. Hayashi, J. Okamoto, and A. Takahashi. 2010. Performance comparisons of subjective quality assessment methods for mobile video. In Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX\u201910). 82--87."},{"key":"e_1_2_2_36_1","volume-title":"Virtual reality (VR) media services over 3GPP","author":"TR","unstructured":"TR 26.918 V15.2.0. 2018. Virtual reality (VR) media services over 3GPP . Technical Specification Group Services and System Aspects (TSG-SA) in the 3rd Generation Partnership Project (3GPP). TR 26.918 V15.2.0. 2018. Virtual reality (VR) media services over 3GPP. Technical Specification Group Services and System Aspects (TSG-SA) in the 3rd Generation Partnership Project (3GPP)."},{"key":"e_1_2_2_37_1","volume-title":"Watson","author":"van Dijk Andre M.","year":"1995","unstructured":"Andre M. van Dijk , Jean-Bernard Martens , and Andrew B . Watson . 1995 . Quality asessment of coded images using numerical category scaling, Vol. 2451 . In Proceedings of the International Society for Optics and Photonics (SPIE\u2019 95). 90--101. DOI:https:\/\/doi.org\/10.1117\/12.201231 10.1117\/12.201231 Andre M. van Dijk, Jean-Bernard Martens, and Andrew B. Watson. 1995. Quality asessment of coded images using numerical category scaling, Vol. 2451. In Proceedings of the International Society for Optics and Photonics (SPIE\u201995). 90--101. DOI:https:\/\/doi.org\/10.1117\/12.201231"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3129505"},{"key":"e_1_2_2_39_1","volume-title":"Multimedia Group Test Plan. Report version 1.19","author":"Hands David","unstructured":"David Hands and Kjell Brunnstr\u00f6m . 2007. Multimedia Group Test Plan. Report version 1.19 . Video Quality Experts Group (VQEG) . David Hands and Kjell Brunnstr\u00f6m. 2007. Multimedia Group Test Plan. Report version 1.19. Video Quality Experts Group (VQEG)."},{"key":"e_1_2_2_40_1","volume-title":"Proceedings of the ACM SIGGRAPH Conference. ACM, 213--220","author":"Watson B.","unstructured":"B. Watson , A. Friedman , and A. McGaffey . 2001. Measuring and predicting visual fidelity . In Proceedings of the ACM SIGGRAPH Conference. ACM, 213--220 . DOI:https:\/\/doi.org\/10.1145\/383259.383283 10.1145\/383259.383283 B. Watson, A. Friedman, and A. McGaffey. 2001. Measuring and predicting visual fidelity. In Proceedings of the ACM SIGGRAPH Conference. ACM, 213--220. DOI:https:\/\/doi.org\/10.1145\/383259.383283"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196493"},{"key":"e_1_2_2_42_1","first-page":"1","article-title":"Subjective and objective quality assessment for volumetric video compression","volume":"323","author":"Zerman Emin","year":"2019","unstructured":"Emin Zerman , Pan Gao , Cagri Ozcinar , and Aljosa Smolic . 2019 . Subjective and objective quality assessment for volumetric video compression . Electron. Imag. 323 (2019), 1 -- 7 . Emin Zerman, Pan Gao, Cagri Ozcinar, and Aljosa Smolic. 2019. Subjective and objective quality assessment for volumetric video compression. Electron. Imag. 323 (2019), 1--7.","journal-title":"Electron. Imag."},{"key":"e_1_2_2_43_1","volume-title":"Proceedings of the 12th International Conference on Quality of Multimedia Experience (QoMEX\u201920)","author":"Zerman E.","unstructured":"E. Zerman , C. Ozcinar , P. Gao , and A. Smolic . 2020. Textured mesh vs coloured point cloud: A subjective study for volumetric video compression . In Proceedings of the 12th International Conference on Quality of Multimedia Experience (QoMEX\u201920) . 1--6. E. Zerman, C. Ozcinar, P. Gao, and A. Smolic. 2020. Textured mesh vs coloured point cloud: A subjective study for volumetric video compression. In Proceedings of the 12th International Conference on Quality of Multimedia Experience (QoMEX\u201920). 1--6."}],"container-title":["ACM Transactions on Applied Perception"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3427931","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3427931","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:23Z","timestamp":1750195463000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3427931"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,31]]},"references-count":43,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1,31]]}},"alternative-id":["10.1145\/3427931"],"URL":"https:\/\/doi.org\/10.1145\/3427931","relation":{},"ISSN":["1544-3558","1544-3965"],"issn-type":[{"value":"1544-3558","type":"print"},{"value":"1544-3965","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,31]]},"assertion":[{"value":"2019-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-12-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}