{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:44:42Z","timestamp":1760237082910,"version":"build-2065373602"},"reference-count":34,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2020,2,25]],"date-time":"2020-02-25T00:00:00Z","timestamp":1582588800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>We need data sets of images and subjective scores to develop robust no reference (or blind) visual quality metrics for consumer applications. These applications have many uncontrolled variables because the camera creates the original media and the impairment simultaneously. We do not fully understand how this impacts the integrity of our subjective data. We put forward two new data sets of images from consumer cameras. The first data set, CCRIQ2, uses a strict experiment design, more suitable for camera performance evaluation. The second data set, VIME1, uses a loose experiment design that resembles the behavior of consumer photographers. We gather subjective scores through a subjective experiment with 24 participants using the Absolute Category Rating method. We make these two new data sets available royalty-free on the Consumer Digital Video Library. We also present their integrity analysis (proposing one new approach) and explore the possibility of combining CCRIQ2 with its legacy counterpart. We conclude that the loose experiment design yields unreliable data, despite adhering to international recommendations. This suggests that the classical subjective study design may not be suitable for studies using consumer content. Finally, we show that Ho\u00dffeld\u2013Schatz\u2013Egger    \u03b1    failed to detect important differences between the two data sets.<\/jats:p>","DOI":"10.3390\/jimaging6030007","type":"journal-article","created":{"date-parts":[[2020,2,25]],"date-time":"2020-02-25T08:12:22Z","timestamp":1582618342000},"page":"7","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Study of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5671-3726","authenticated-orcid":false,"given":"Jakub","family":"Nawa\u0142a","sequence":"first","affiliation":[{"name":"AGH University of Science and Technology, MP 30059 Krak\u00f3w, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3369-4726","authenticated-orcid":false,"given":"Margaret H.","family":"Pinson","sequence":"additional","affiliation":[{"name":"Institute for Telecommunication Sciences, National Telecommunications and Information Administration, Boulder, CO 80305, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9123-1039","authenticated-orcid":false,"given":"Miko\u0142aj","family":"Leszczuk","sequence":"additional","affiliation":[{"name":"AGH University of Science and Technology, MP 30059 Krak\u00f3w, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3151-2944","authenticated-orcid":false,"given":"Lucjan","family":"Janowski","sequence":"additional","affiliation":[{"name":"AGH University of Science and Technology, MP 30059 Krak\u00f3w, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,2,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"4695","DOI":"10.1109\/TIP.2012.2214050","article-title":"No-reference image quality assessment in the spatial domain","volume":"21","author":"Mittal","year":"2012","journal-title":"IEEE Trans. Image Proc."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Nawa\u0142a, J., Leszczuk, M., Zajdel, M., and Baran, R. (2016). Software package for measurement of quality indicators working in no-reference model. Multimed. Tools Appl., 1\u201317.","DOI":"10.1007\/s11042-016-4195-3"},{"key":"ref_3","unstructured":"Zeng, H., Zhang, L., and Bovik, A.C. (2017). A Probabilistic Quality Representation Approach to Deep Blind Image Quality Prediction. arXiv preprint."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Lin, H., Hosu, V., and Saupe, D. (2019, January 5\u20137). KADID-10k: A large-scale artificially distorted IQA database. Proceedings of the 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), Berlin, Germany.","DOI":"10.1109\/QoMEX.2019.8743252"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.image.2014.10.009","article-title":"Image database TID2013: Peculiarities, results and perspectives","volume":"30","author":"Ponomarenko","year":"2015","journal-title":"Signal Proc. Image Commun."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1427","DOI":"10.1109\/TIP.2010.2042111","article-title":"Study of subjective and objective quality assessment of video","volume":"19","author":"Seshadrinathan","year":"2010","journal-title":"IEEE Trans. Image Proc."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Proc."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3440","DOI":"10.1109\/TIP.2006.881959","article-title":"A statistical evaluation of recent full reference image quality assessment algorithms","volume":"15","author":"Sheikh","year":"2006","journal-title":"IEEE Trans. Image Proc."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1109\/JSTSP.2012.2215306","article-title":"The influence of subjects and environment on audiovisual subjective tests: An international study","volume":"6","author":"Pinson","year":"2012","journal-title":"IEEE J. Sel. Topics Signal Proc."},{"key":"ref_10","unstructured":"ITU (2004). Objective Perceptual Video Quality Measurement Techniques for Digital Cable Television in the Presence of a Full Reference, ITU. ITU-T Recommendation J.144."},{"key":"ref_11","unstructured":"ITU (2008). Subjective Video Quality Assessment Methods for Multimedia Applications, ITU. ITU-T Recommendation P.910."},{"key":"ref_12","unstructured":"ITU (1996). Methods for Subjective Determination of Transmission Quality, ITU. ITU-T Recommendation P.800."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"390","DOI":"10.1109\/TIP.2014.2378061","article-title":"CID2013: A database for evaluating no-reference image quality assessment algorithms","volume":"24","author":"Virtanen","year":"2015","journal-title":"IEEE Trans. Image Proc."},{"key":"ref_14","unstructured":"Pinson, M.H. (2018). ITS4S: A Video Quality Dataset with Four-Second Unrepeated Scenes."},{"key":"ref_15","unstructured":"Pinson, M.H. (2019). ITS4S2: An Image Quality Dataset with Unrepeated Images from Consumer Cameras."},{"key":"ref_16","unstructured":"Pinson, M.H. (2019). ITS4S3: A Video Quality Dataset with Unrepeated Videos, Camera Impairments, and Public Safety Scenarios."},{"key":"ref_17","unstructured":"(2020, February 19). Consumer Digital Video Library. Available online: https:\/\/www.cdvl.org\/."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1109\/TIP.2015.2500021","article-title":"Massive online crowdsourced study of subjective and objective picture quality","volume":"25","author":"Ghadiyaram","year":"2016","journal-title":"IEEE Trans. Image Proc."},{"key":"ref_19","unstructured":"Ghadiyaram, D., Pan, J., Bovik, A.C., Moorthy, A.K., Panda, P., and Yang, K. (2020, February 20). LIVE-Qualcomm Mobile In-Capture Video Quality Database. Available online: http:\/\/live.ece.utexas.edu\/research\/incaptureDatabase\/index.html."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2061","DOI":"10.1109\/TCSVT.2017.2707479","article-title":"In-capture mobile video distortions: A study of subjective behavior and objective algorithms","volume":"28","author":"Ghadiyaram","year":"2018","journal-title":"IEEE Trans. Circ. Syst. Video Technol."},{"key":"ref_21","unstructured":"Lin, H., Hosu, V., and Saupe, D. (2018). KonIQ-10K: Towards an ecologically valid and large-scale IQA database. arXiv preprint."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1145\/2812802","article-title":"YFCC100M: The new data in multimedia research","volume":"59","author":"Thomee","year":"2016","journal-title":"Commun. ACM"},{"key":"ref_23","unstructured":"Hosu, V., Hahn, F., Jenadeleh, M., Lin, H., Men, H., Szir\u00e1nyi, T., Li, S., and Saupe, D. (2020, February 20). The Konstanz Natural Video Database. Available online: http:\/\/database.mmsp-kn.de\/konvid-1k-database.html."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Hosu, V., Hahn, F., Jenadeleh, M., Lin, H., Men, H., Szir\u00e1nyi, T., Li, S., and Saupe, D. (June, January 31). The Konstanz natural video database (KoNViD-1k). Proceedings of the 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), Erfurt, Germany.","DOI":"10.1109\/QoMEX.2017.7965673"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Saad, M.A., Pinson, M.H., Nicholas, D.G., Kets, N.V., Wallendael, G.V., Silva, R.D., Jaladi, R.V., and Corriveau, P.J. (2015, January 25\u201326). Impact of camera pixel count and monitor resolution perceptual image quality. Proceedings of the 2015 Colour and Visual Computing Symposium (CVCS), Gj\u00f8vik, Norway.","DOI":"10.1109\/CVCS.2015.7274887"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Ho\u00dffeld, T., Schatz, R., and Egger, S. (2011, January 7\u20139). SOS: The MOS is not enough!. Proceedings of the 2011 Third International Workshop on Quality of Multimedia Experience (QoMEX), Mechelen, Belgium.","DOI":"10.1109\/QoMEX.2011.6065690"},{"key":"ref_27","unstructured":"IEEE 1858-2016 (2020, February 20). Standard for Camera Phone Image Quality (CPIQ). Available online: https:\/\/standards.ieee.org\/standard\/1858-2016.html."},{"key":"ref_28","unstructured":"Video Quality Experts Group (2020, February 20). Video and Image Models for Consumer Content Evaluation, Available online: https:\/\/www.its.bldrdoc.gov\/vqeg\/projects\/vime.aspx."},{"key":"ref_29","unstructured":"Video and Image Models for consumer content Evaluation (2020, February 20). VIME Image Database | Flickr. Available online: https:\/\/www.flickr.com\/groups\/vime\/."},{"key":"ref_30","unstructured":"ITU (2014). Methods for the Subjective Assessment of Video Quality, Audio Quality and Audiovisual Quality of Internet Video and Distribution Quality Television in any Environment, ITU. Recommendation P.913."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"31","DOI":"10.5594\/J15853","article-title":"A multidimensional evaluation of the perceptual quality of television sets","volume":"105","author":"Teunissen","year":"1996","journal-title":"SMPTE J."},{"key":"ref_32","first-page":"1","article-title":"Statistical quality of experience analysis: On planning the sample size and statistical significance testing","volume":"27","author":"Barkowsky","year":"2018","journal-title":"J. Electr. Imaging"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ho\u00dffeld, T., Heegaard, P.E., and Varela, M. (2015, January 26\u201329). QoE beyond the MOS: Added value using quantiles and distributions. Proceedings of the 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX), Pylos-Nestoras, Greece.","DOI":"10.1109\/QoMEX.2015.7148142"},{"key":"ref_34","first-page":"1","article-title":"QoE beyond the MOS: An in-depth look at QoE via better metrics and their relation to MOS","volume":"1","author":"Heegaard","year":"2016","journal-title":"Qual. User Exp."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/6\/3\/7\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:01:29Z","timestamp":1760173289000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/6\/3\/7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,25]]},"references-count":34,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,3]]}},"alternative-id":["jimaging6030007"],"URL":"https:\/\/doi.org\/10.3390\/jimaging6030007","relation":{},"ISSN":["2313-433X"],"issn-type":[{"type":"electronic","value":"2313-433X"}],"subject":[],"published":{"date-parts":[[2020,2,25]]}}}