{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:28:28Z","timestamp":1740122908697,"version":"3.37.3"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2020,9,22]],"date-time":"2020-09-22T00:00:00Z","timestamp":1600732800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,9,22]],"date-time":"2020-09-22T00:00:00Z","timestamp":1600732800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"published-print":{"date-parts":[[2021,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Subjective experiments are considered the most reliable way to assess the perceived visual quality. However, observers\u2019 opinions are characterized by large diversity: in fact, even the same observer is often not able to exactly repeat his first opinion when rating again a given stimulus. This makes the Mean Opinion Score (MOS) alone, in many cases, not sufficient to get accurate information about the perceived visual quality. To this aim, it is important to have a measure characterizing to what extent the observed or predicted MOS value is reliable and stable. For instance, the Standard deviation of the Opinions of the Subjects (SOS) could be considered as a measure of reliability when evaluating the quality subjectively. However, we are not aware of the existence of models or algorithms that allow to objectively predict how much diversity would be observed in subjects\u2019 opinions in terms of SOS. In this work we observe, on the basis of a statistical analysis made on several subjective experiments, that the disagreement between the quality as measured by means of different objective video quality metrics (VQMs) can provide information on the diversity of the observers\u2019 ratings on a given processed video sequence (PVS). In light of this observation we: i) propose and validate a model for the SOS observed in a subjective experiment; ii) design and train Neural Networks (NNs) that predict the average diversity that would be observed among the subjects\u2019 ratings for a PVS starting from a set of VQMs values computed on such a PVS; iii) give insights into how the same NN based approach can be used to identify potential anomalies in the data collected in subjective experiments.<\/jats:p>","DOI":"10.1007\/s11042-020-09704-w","type":"journal-article","created":{"date-parts":[[2020,9,22]],"date-time":"2020-09-22T05:02:51Z","timestamp":1600750971000},"page":"3469-3487","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Modeling and estimating the subjects\u2019 diversity of opinions in video quality assessment: a neural network based approach"],"prefix":"10.1007","volume":"80","author":[{"given":"Lohic","family":"Fotio Tiotsop","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tomas","family":"Mizdos","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miroslav","family":"Uhrina","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marcus","family":"Barkowsky","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter","family":"Pocta","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8906-354X","authenticated-orcid":false,"given":"Enrico","family":"Masala","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,9,22]]},"reference":[{"key":"9704_CR1","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1016\/j.image.2019.01.004","volume":"74","author":"A Aldahdooh","year":"2019","unstructured":"Aldahdooh A, Masala E, Van Wallendael G, Lambert P, Barkowsky M (2019) Improving relevant subjective testing for validation: Comparing machine learning algorithms for finding similarities in VQA datasets using objective measures. Signal Process Image Commun 74:32\u201341","journal-title":"Signal Process Image Commun"},{"key":"9704_CR2","unstructured":"Aroussi S, Mellouk A (2014) Survey on machine learning-based QoE-QoS correlation models. In: 2014 International conference on computing, management and telecommunications (commantel). IEEE, pp 200\u2013204"},{"issue":"9","key":"9704_CR3","doi-asserted-by":"publisher","first-page":"5545","DOI":"10.1007\/s11042-019-08444-w","volume":"79","author":"A Bhattacharya","year":"2020","unstructured":"Bhattacharya A, Palit S (2020) Measurement of image degradation: a no-reference approach. Multimed Tools Appl 79(9):5545\u20135572","journal-title":"Multimed Tools Appl"},{"issue":"1","key":"9704_CR4","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1109\/TIP.2017.2760518","volume":"27","author":"S Bosse","year":"2018","unstructured":"Bosse S, Maniry D, M\u00fcller K, Wiegand T, Samek W (2018) Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans Image Process 27(1):206\u2013219. https:\/\/doi.org\/10.1109\/TIP.2017.2760518","journal-title":"IEEE Trans Image Process"},{"key":"9704_CR5","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1016\/j.image.2017.03.001","volume":"54","author":"Y Ding","year":"2017","unstructured":"Ding Y, Zhao Y, Zhao X (2017) Image quality assessment based on multi-feature extraction and synthesis with support vector regression. Signal Process Image Commun 54:81\u201392","journal-title":"Signal Process Image Commun"},{"key":"9704_CR6","doi-asserted-by":"publisher","unstructured":"Fotio Tiotsop L, Masala E, Aldahdooh A, Van Wallendael G, Barkowsky M (2019) Computing quality-of-experience ranges for video quality estimation. In: 2019 Eleventh international conference on quality of multimedia experience (qoMEX), pp 1\u20133. https:\/\/doi.org\/10.1109\/QoMEX.2019.8743303","DOI":"10.1109\/QoMEX.2019.8743303"},{"key":"9704_CR7","doi-asserted-by":"crossref","unstructured":"Fotio Tiotsop L, Servetti A, Masala E (2020) Full reference video quality measure improvement using neural networks. In: Proc. Intl. Conf. acoustics, Speech, and Signal Processing (ICASSP)","DOI":"10.1109\/ICASSP40776.2020.9053739"},{"key":"9704_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.image.2018.02.010","volume":"64","author":"PG Freitas","year":"2018","unstructured":"Freitas PG, Akamine WY, Farias MC (2018) Using multiple spatio-temporal features to estimate video quality. Signal Process Image Commun 64:1\u201310","journal-title":"Signal Process Image Commun"},{"key":"9704_CR9","doi-asserted-by":"publisher","unstructured":"Ho\u00dffeld T, Schatz R, Egger-Lampl S (2011) SOS: The MOS is not enough!. In: 2011 Third international workshop on quality of multimedia experience, pp 131\u2013136. https:\/\/doi.org\/10.1109\/QoMEX.2011.6065690","DOI":"10.1109\/QoMEX.2011.6065690"},{"key":"9704_CR10","unstructured":"ITU-T Rec. BT.500: Methodology for the subjective assessment of the quality of television pictures (2012)"},{"key":"9704_CR11","unstructured":"ITU-T Rec. P.1401: methods, metrics and procedures for statistical evaluation qualification and comparison of objective quality prediction models (2012)"},{"key":"9704_CR12","unstructured":"ITU-T Rec. P.910: Subjective video quality assessment methods for multimedia applications (1996)"},{"issue":"12","key":"9704_CR13","doi-asserted-by":"publisher","first-page":"2210","DOI":"10.1109\/TMM.2015.2484963","volume":"17","author":"L Janowski","year":"2015","unstructured":"Janowski L, Pinson M (2015) The accuracy of subjects in a quality experiment: a theoretical subject model. IEEE Trans Multimed 17(12):2210\u20132224","journal-title":"IEEE Trans Multimed"},{"issue":"6245","key":"9704_CR14","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1126\/science.aaa8415","volume":"349","author":"MI Jordan","year":"2015","unstructured":"Jordan MI, Mitchell TM (2015) Machine learning: trends, perspectives, and prospects. Science 349(6245):255\u2013260","journal-title":"Science"},{"key":"9704_CR15","doi-asserted-by":"publisher","unstructured":"Li Z, Bampis CG (2017) Recover subjective quality scores from noisy measurements. In: 2017 Data compression conference (DCC), pp 52\u201361. https:\/\/doi.org\/10.1109\/DCC.2017.26","DOI":"10.1109\/DCC.2017.26"},{"issue":"6","key":"9704_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1117\/1.JEI.24.6.061208","volume":"24","author":"DC Mocanu","year":"2015","unstructured":"Mocanu DC, Pokhrel J, Garella JP, Sepp\u00e4nen J, Liotou E, Narwaria M (2015) No-reference video quality measurement: added value of machine learning. J Electron Imaging 24(6):1\u201315. https:\/\/doi.org\/10.1117\/1.JEI.24.6.061208","journal-title":"J Electron Imaging"},{"issue":"6","key":"9704_CR17","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1109\/79.543975","volume":"13","author":"TK Moon","year":"1996","unstructured":"Moon TK (1996) The expectation-maximization algorithm. IEEE Signal Process Mag 13(6):47\u201360","journal-title":"IEEE Signal Process Mag"},{"issue":"3","key":"9704_CR18","doi-asserted-by":"publisher","first-page":"787","DOI":"10.1007\/s11042-011-0946-3","volume":"61","author":"M Mu","year":"2012","unstructured":"Mu M, Romaniak P, Mauthe A, Leszczuk M, Janowski L, Cerqueira E (2012) Framework for the integrated video quality assessment. Multimed Tools Appl 61(3):787\u2013817","journal-title":"Multimed Tools Appl"},{"key":"9704_CR19","doi-asserted-by":"crossref","unstructured":"Mushtaq MS, Augustin B, Mellouk A (2012) Empirical study based on machine learning approach to assess the QoS\/QoE correlation. In: 2012 17th european conference on networks and optical communications. IEEE, pp 1\u20137","DOI":"10.1109\/NOC.2012.6249939"},{"key":"9704_CR20","unstructured":"Netflix: VMAF - video multi-method assessment fusion v.0.6.2. https:\/\/github.com\/Netflix\/vmaf (2018)"},{"issue":"23","key":"9704_CR21","doi-asserted-by":"publisher","first-page":"16461","DOI":"10.1007\/s11042-015-3214-0","volume":"75","author":"P Paudyal","year":"2016","unstructured":"Paudyal P, Battisti F, Carli M (2016) Impact of video content and transmission impairments on quality of experience. Multimed Tools Appl 75(23):16461\u201316485","journal-title":"Multimed Tools Appl"},{"key":"9704_CR22","unstructured":"Pinson MH (2018) A video quality dataset with four-second unrepeated scenes. NTIA Technical Memo TM-18-532"},{"key":"9704_CR23","doi-asserted-by":"crossref","unstructured":"Reiter U, Brunnstr\u00f6m K, De Moor K, Larabi MC, Pereira M, Pinheiro A, You J, Zgank A (2014) Factors Influencing Quality of Experience. Springer International Publishing, Cham, pp 55\u201372","DOI":"10.1007\/978-3-319-02681-7_4"},{"key":"9704_CR24","doi-asserted-by":"publisher","unstructured":"Seufert M (2019) Fundamental advantages of considering quality of experience distributions over mean opinion scores. In: 2019 Eleventh international conference on quality of multimedia experience (QoMEX), pp 1\u20136. https:\/\/doi.org\/10.1109\/QoMEX.2019.8743296","DOI":"10.1109\/QoMEX.2019.8743296"},{"issue":"2","key":"9704_CR25","doi-asserted-by":"publisher","first-page":"430","DOI":"10.1109\/TIP.2005.859378","volume":"15","author":"HR Sheikh","year":"2006","unstructured":"Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15(2):430\u2013444","journal-title":"IEEE Trans Image Process"},{"issue":"8","key":"9704_CR26","doi-asserted-by":"publisher","first-page":"1525","DOI":"10.1016\/j.techfore.2012.04.013","volume":"79","author":"HA von der Gracht","year":"2012","unstructured":"von der Gracht HA (2012) Consensus measurement in delphi studies: Review and implications for future quality assurance. Technol Forecast Soc Chang 79 (8):1525\u20131536","journal-title":"Technol Forecast Soc Chang"},{"key":"9704_CR27","unstructured":"VQEG: Report on the validation of video quality models for high definition video content (v. 2.0). http:\/\/bit.ly\/2Z7GWDI (2010)"},{"key":"9704_CR28","doi-asserted-by":"publisher","unstructured":"Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The thrity-seventh asilomar conference on signals, systems computers, vol 2, pp 1398\u20131402. https:\/\/doi.org\/10.1109\/ACSSC.2003.1292216","DOI":"10.1109\/ACSSC.2003.1292216"},{"issue":"4","key":"9704_CR29","doi-asserted-by":"publisher","first-page":"600","DOI":"10.1109\/TIP.2003.819861","volume":"13","author":"Z Wang","year":"2004","unstructured":"Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600\u2013612","journal-title":"IEEE Trans Image Process"},{"key":"9704_CR30","unstructured":"Wierman MJ, Tastle WJ (2005) Consensus and dissention: theory and properties. In: NAFIPS 2005 - 2005 Annual meeting of the north american fuzzy information processing society, pp 75\u201379"},{"key":"9704_CR31","doi-asserted-by":"crossref","unstructured":"Xu L, Lin W, Kuo CCJ (2015) Visual quality assessment by machine learning. Springer, Berlin","DOI":"10.1007\/978-981-287-468-9"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-020-09704-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-020-09704-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-020-09704-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T00:07:35Z","timestamp":1632269255000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-020-09704-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,22]]},"references-count":31,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,1]]}},"alternative-id":["9704"],"URL":"https:\/\/doi.org\/10.1007\/s11042-020-09704-w","relation":{},"ISSN":["1380-7501","1573-7721"],"issn-type":[{"type":"print","value":"1380-7501"},{"type":"electronic","value":"1573-7721"}],"subject":[],"published":{"date-parts":[[2020,9,22]]},"assertion":[{"value":"7 April 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 July 2020","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 August 2020","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 September 2020","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with Ethical Standards"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of interests"}}]}}