{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,1]],"date-time":"2025-10-01T16:21:40Z","timestamp":1759335700686,"version":"3.37.3"},"reference-count":61,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T00:00:00Z","timestamp":1692316800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T00:00:00Z","timestamp":1692316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["251654672"],"award-info":[{"award-number":["251654672"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010583","name":"Universit\u00e4t Konstanz","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100010583","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Qual User Exp"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Scale-invariance is an open problem in many computer vision subfields. For example, object labels should remain constant across scales, yet model predictions diverge in many cases. This problem gets harder for tasks where the ground-truth labels change with the presentation scale. In image quality assessment (IQA), down-sampling attenuates impairments, e.g., blurs or compression artifacts, which can positively affect the impression evoked in subjective studies. To accurately predict perceptual image quality, cross-resolution IQA methods must therefore account for resolution-dependent discrepancies induced by model inadequacies as well as for the perceptual label shifts in the ground truth. We present the first study of its kind that disentangles and examines the two issues separately via <jats:italic>KonX<\/jats:italic>, a novel, carefully crafted cross-resolution IQA database. This paper contributes the following: 1. Through <jats:italic>KonX<\/jats:italic>, we provide empirical evidence of label shifts caused by changes in the presentation resolution. 2. We show that objective IQA methods have a scale bias, which reduces their predictive performance. 3. We propose a multi-scale and multi-column deep neural network architecture that improves performance over previous state-of-the-art IQA models for this task. We thus both raise and address a novel research problem in image quality assessment.<\/jats:p>","DOI":"10.1007\/s41233-023-00061-8","type":"journal-article","created":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T09:02:05Z","timestamp":1692349325000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Konx: cross-resolution image quality assessment"],"prefix":"10.1007","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9534-126X","authenticated-orcid":false,"given":"Oliver","family":"Wiedemann","sequence":"first","affiliation":[]},{"given":"Vlad","family":"Hosu","sequence":"additional","affiliation":[]},{"given":"Shaolin","family":"Su","sequence":"additional","affiliation":[]},{"given":"Dietmar","family":"Saupe","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,8,18]]},"reference":[{"key":"61_CR1","doi-asserted-by":"publisher","first-page":"4041","DOI":"10.1109\/TIP.2020.2967829","volume":"29","author":"V Hosu","year":"2020","unstructured":"Hosu V, Lin H, Sziranyi T, Saupe D (2020) Koniq-10k: an ecologically valid database for deep learning of blind image quality assessment. IEEE Trans Image Process 29:4041\u20134056","journal-title":"IEEE Trans Image Process"},{"key":"61_CR2","unstructured":"Su S, Hosu V, Lin H, Zhang Y, Saupe D (2021) Koniq++: Boosting no-reference image quality assessment in the wild by jointly predicting image quality and defects. In: The 32nd British machine vision conference (BMVC)"},{"issue":"8","key":"61_CR3","doi-asserted-by":"publisher","first-page":"3998","DOI":"10.1109\/TIP.2018.2831899","volume":"27","author":"H Talebi","year":"2018","unstructured":"Talebi H, Milanfar P (2018) NIMA: neural image assessment. IEEE Trans Image Process 27(8):3998\u20134011","journal-title":"IEEE Trans Image Process"},{"key":"61_CR4","doi-asserted-by":"crossref","unstructured":"Bosse S, Maniry D, Wiegand T, Samek W (2016) A deep neural network for image quality assessment. In: International conference on image processing (ICIP). IEEE, pp. 3773\u20133777","DOI":"10.1109\/ICIP.2016.7533065"},{"key":"61_CR5","doi-asserted-by":"publisher","unstructured":"Li D, Jiang T, Jiang M (2020) Norm-in-Norm Loss with faster convergence and better performance for image quality assessment. Proceedings of the 28th ACM International conference on multimedia, 789\u2013797. Accessed 2021-11-07, https:\/\/doi.org\/10.1145\/3394171.3413804. arXiv: 2008.03889","DOI":"10.1145\/3394171.3413804"},{"key":"61_CR6","doi-asserted-by":"crossref","unstructured":"Ying Z, Niu H, Gupta P, Mahajan D, Ghadiyaram D, Bovik A (2020) From patches to pictures (paq-2-piq): Mapping the perceptual space of picture quality. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp. 3575\u20133585","DOI":"10.1109\/CVPR42600.2020.00363"},{"key":"61_CR7","unstructured":"Ghadiyaram D, Bovik A (2015) Live in the wild image quality challenge database"},{"key":"61_CR8","first-page":"57","volume":"30","author":"N Ponomarenko","year":"2015","unstructured":"Ponomarenko N, Jin L, Ieremeiev O, Lukin V, Egiazarian K, Astola J, Vozel B, Chehdi K, Carli M, Battisti F (2015) Image database tid2013: peculiarities, results and perspectives. Sign Process: Image Commun 30:57\u201377","journal-title":"Sign Process: Image Commun"},{"key":"61_CR9","unstructured":"Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning, pp. 6105\u20136114. PMLR"},{"key":"61_CR10","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp. 618\u2013626","DOI":"10.1109\/ICCV.2017.74"},{"key":"61_CR11","doi-asserted-by":"crossref","unstructured":"Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI Conference on artificial intelligence","DOI":"10.1609\/aaai.v31i1.11231"},{"issue":"4","key":"61_CR12","doi-asserted-by":"publisher","first-page":"600","DOI":"10.1109\/TIP.2003.819861","volume":"13","author":"Z Wang","year":"2004","unstructured":"Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600\u2013612","journal-title":"IEEE Trans Image Process"},{"key":"61_CR13","doi-asserted-by":"crossref","unstructured":"Sheikh HR, Bovik AC (2005) A visual information fidelity approach to video quality assessment. In: The first international workshop on video processing and quality metrics for consumer electronics, vol. 7","DOI":"10.1016\/B978-012119792-6\/50120-0"},{"key":"61_CR14","doi-asserted-by":"crossref","unstructured":"Kang L, Ye P, Li Y, Doermann D (2014) Convolutional neural networks for no-reference image quality assessment. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 1733\u20131740","DOI":"10.1109\/CVPR.2014.224"},{"key":"61_CR15","doi-asserted-by":"crossref","unstructured":"Wiedemann O, Hosu V, Lin H, Saupe D (2018) Disregarding the big picture: towards local image quality assessment. In: 2018 Tenth international conference on quality of multimedia experience (QoMEX), pp. 1\u20136. IEEE","DOI":"10.1109\/QoMEX.2018.8463384"},{"key":"61_CR16","doi-asserted-by":"crossref","unstructured":"Zhu H, Li L, Wu J, Dong W, Shi G (2020) Metaiqa: deep meta-learning for no-reference image quality assessment. In: Proceedings of the IEEE\/CVF Conference on computer vision and pattern recognition, pp. 14143\u201314152","DOI":"10.1109\/CVPR42600.2020.01415"},{"key":"61_CR17","doi-asserted-by":"crossref","unstructured":"Yang S, Jiang Q, Lin W, Wang Y (2019) Sgdnet: An end-to-end saliency-guided deep neural network for no-reference image quality assessment. In: Proceedings of the 27th ACM international conference on multimedia, pp. 1383\u20131391","DOI":"10.1145\/3343031.3350990"},{"key":"61_CR18","doi-asserted-by":"crossref","unstructured":"Pan D, Shi P, Hou M, Ying Z, Fu S, Zhang Y (2018) Blind predicting similar quality map for image quality assessment. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 6373\u20136382","DOI":"10.1109\/CVPR.2018.00667"},{"key":"61_CR19","doi-asserted-by":"crossref","unstructured":"Golestaneh SA, Dadsetan S, Kitani KM (2022) No-reference image quality assessment via transformers, relative ranking, and self-consistency. In: Proceedings of the IEEE\/CVF winter conference on applications of computer vision, pp. 1220\u20131230","DOI":"10.1109\/WACV51458.2022.00404"},{"key":"61_CR20","doi-asserted-by":"publisher","first-page":"4041","DOI":"10.1109\/TIP.2020.2967829","volume":"29","author":"V Hosu","year":"2020","unstructured":"Hosu V, Lin H, Sziranyi T, Saupe D (2020) Koniq-10k: an ecologically valid database for deep learning of blind image quality assessment. IEEE Trans Image Process 29:4041\u20134056","journal-title":"IEEE Trans Image Process"},{"key":"61_CR21","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1016\/j.patcog.2016.06.005","volume":"61","author":"N Van Noord","year":"2017","unstructured":"Van Noord N, Postma E (2017) Learning scale-variant and scale-invariant features for deep image classification. Patt Recogn 61:583\u2013592","journal-title":"Patt Recogn"},{"key":"61_CR22","doi-asserted-by":"crossref","unstructured":"Hii Y-L, See J, Kairanbay M, Wong L-K (2017) Multigap: Multi-pooled inception network with text augmentation for aesthetic prediction of photographs. In: International conference on image processing (ICIP), pp. 1722\u20131726. IEEE","DOI":"10.1109\/ICIP.2017.8296576"},{"key":"61_CR23","doi-asserted-by":"crossref","unstructured":"Hosu V, Goldlucke B, Saupe D (2019) Effective aesthetics prediction with multi-level spatially pooled features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9375\u20139383","DOI":"10.1109\/CVPR.2019.00960"},{"key":"61_CR24","doi-asserted-by":"crossref","unstructured":"Ke J, Wang Q, Wang Y, Milanfar P, Yang F (2021) Musiq: multi-scale image quality transformer. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp. 5148\u20135157","DOI":"10.1109\/ICCV48922.2021.00510"},{"key":"61_CR25","doi-asserted-by":"crossref","unstructured":"Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The Thrity-seventh asilomar conference on signals, systems & computers, IEEE, vol. 2, pp. 1398\u20131402","DOI":"10.1109\/ACSSC.2003.1292216"},{"key":"61_CR26","doi-asserted-by":"publisher","unstructured":"Temel D, AlRegib G (2015) Persim: multi-resolution image quality assessment in the perceptually uniform color domain. In: 2015 IEEE International conference on image processing (ICIP), pp. 1682\u20131686. https:\/\/doi.org\/10.1109\/ICIP.2015.7351087","DOI":"10.1109\/ICIP.2015.7351087"},{"key":"61_CR27","doi-asserted-by":"crossref","unstructured":"You J, Korhonen J (2021) Transformer for image quality assessment. In: 2021 IEEE International conference on image processing (ICIP), IEEE, pp. 1389\u20131393","DOI":"10.1109\/ICIP42928.2021.9506075"},{"issue":"2","key":"61_CR28","doi-asserted-by":"publisher","first-page":"374","DOI":"10.3390\/make3020019","volume":"3","author":"M Graziani","year":"2021","unstructured":"Graziani M, Lompech T, M\u00fcller H, Depeursinge A, Andrearczyk V (2021) On the scale invariance in state of the art CNNs trained on ImageNet. Mach Learn Knowl Extr 3(2):374\u2013391. https:\/\/doi.org\/10.3390\/make3020019","journal-title":"Mach Learn Knowl Extr"},{"key":"61_CR29","doi-asserted-by":"publisher","unstructured":"Liu W, Duanmu Z, Wang Z. End-to-end blind quality assessment of compressed videos using deep neural networks. In: ACM International Conference on Multimedia, pp. 546\u2013554. ACM. https:\/\/doi.org\/10.1145\/3240508.3240643","DOI":"10.1145\/3240508.3240643"},{"key":"61_CR30","unstructured":"Touvron H, Vedaldi A, Douze M, J\u00e9gou H (2020) Fixing the train-test resolution discrepancy: Fixefficientnet. arXiv preprint arXiv:2003.08237"},{"key":"61_CR31","doi-asserted-by":"crossref","unstructured":"Murray N, Marchesotti L, Perronnin F (2012) Ava: a large-scale database for aesthetic visual analysis. In: Computer vision and pattern recognition (CVPR). IEEE, pp. 2408\u20132415","DOI":"10.1109\/CVPR.2012.6247954"},{"key":"61_CR32","doi-asserted-by":"publisher","unstructured":"Su S, Yan Q, Zhu Y, Zhang C, Ge X, Sun J, Zhang Y. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In: Computer vision and pattern recognition (CVPR), pp. 3664\u20133673. IEEE. https:\/\/doi.org\/10.1109\/CVPR42600.2020.00372. https:\/\/ieeexplore.ieee.org\/document\/9156687\/","DOI":"10.1109\/CVPR42600.2020.00372"},{"key":"61_CR33","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1016\/j.patcog.2016.06.005","volume":"61","author":"N van Noord","year":"2016","unstructured":"van Noord N, Postma E (2016) Learning scale-variant and scale-invariant features for deep image classification. Patt Recong 61:583\u2013592. https:\/\/doi.org\/10.1016\/j.patcog.2016.06.005","journal-title":"Patt Recong"},{"key":"61_CR34","unstructured":"Kang D, Chan AB (2018) Crowd counting by adaptively fusing predictions from an image pyramid. In: British machine vision conference (BMVC), p. 89"},{"key":"61_CR35","doi-asserted-by":"publisher","unstructured":"Zhang Y, Zhou D, Chen S, Gao S, Ma Y. Single-image crowd counting via multi-column convolutional neural network. In: Conference on computer vision and pattern recognition (CVPR), pp. 589\u2013597. IEEE. https:\/\/doi.org\/10.1109\/CVPR.2016.70. http:\/\/ieeexplore.ieee.org\/document\/7780439\/","DOI":"10.1109\/CVPR.2016.70"},{"key":"61_CR36","doi-asserted-by":"publisher","unstructured":"Walach E, Wolf L (2016) Learning to count with cnn boosting. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) In: European conference on computer vision (ECCV). Lecture notes in computer science, vol. 9906, pp. 660\u2013676. Springer. https:\/\/doi.org\/10.1007\/978-3-319-46475-6_41","DOI":"10.1007\/978-3-319-46475-6_41"},{"key":"61_CR37","doi-asserted-by":"publisher","unstructured":"O\u00f1oro-Rubio D, L\u00f3pez-Sastre RJ (2016) Towards perspective-free object counting with deep learning. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) In: European conference on computer vision (ECCV). Lecture notes in computer science, vol. 9911, pp. 615\u2013629. Springer International Publishing. https:\/\/doi.org\/10.1007\/978-3-319-46478-7_38. Accessed 2022-02-27","DOI":"10.1007\/978-3-319-46478-7_38"},{"key":"61_CR38","unstructured":"Sheikh HR, Wang Z, Cormack L, Bovik AC (2005) Live image quality assessment database release 2. http:\/\/live.ece.utexas.edu\/research\/quality"},{"key":"61_CR39","doi-asserted-by":"crossref","unstructured":"Liu X, Pedersen M, Hardeberg JY (2014) Cid: Iq\u2013a new image quality database. In: International conference on image and signal processing, pp. 193\u2013202. Springer","DOI":"10.1007\/978-3-319-07998-1_22"},{"key":"61_CR40","doi-asserted-by":"crossref","unstructured":"Lin H, Hosu V, Saupe D (2019) Kadid-10k: a large-scale artificially distorted iqa database. In: International conference on quality of multimedia experience (QoMEX), IEEE, pp. 1\u20133","DOI":"10.1109\/QoMEX.2019.8743252"},{"key":"61_CR41","unstructured":"Lin H, Hosu V, Saupe D (2020) Deepfl-iqa: weak supervision for deep IQA feature learning. arXiv preprint arXiv:2001.08113"},{"key":"61_CR42","doi-asserted-by":"crossref","unstructured":"Fang Y, Zhu H, Zeng Y, Ma K, Wang Z (2020) Perceptual quality assessment of smartphone photography. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp. 3677\u20133686","DOI":"10.1109\/CVPR42600.2020.00373"},{"key":"61_CR43","unstructured":"Brunnstr\u00f6m K, Beker SA, De\u00a0Moor K, Dooms A, Egger S, Garcia M-N, Hossfeld T, Jumisko-Pyykk\u00f6 S, Keimel C, Larabi M-C, et al (2013) Qualinet white paper on definitions of quality of experience"},{"key":"61_CR44","doi-asserted-by":"crossref","unstructured":"Reiter U, Brunnstr\u00f6m K, De\u00a0Moor K, Larabi M-C, Pereira M, Pinheiro A, You J, Zgank A. Factors Influencing Quality of Experience. In: M\u00f6ller, S., Raake, A. (eds.) Quality of Experience. T-Labs Series in Telecommunication Services, pp. 55\u201372. Springer International Publishing","DOI":"10.1007\/978-3-319-02681-7_4"},{"issue":"6","key":"61_CR45","doi-asserted-by":"publisher","first-page":"652","DOI":"10.1109\/JSTSP.2012.2212417","volume":"6","author":"AK Moorthy","year":"2012","unstructured":"Moorthy AK, Bovik Choi LK, de Veciana ACG (2012) Video quality assessment on mobile devices: subjective, behavioral and objective studies. IEEE J Select Topics Signal Process 6(6):652\u2013671. https:\/\/doi.org\/10.1109\/JSTSP.2012.2212417","journal-title":"IEEE J Select Topics Signal Process"},{"issue":"11","key":"61_CR46","doi-asserted-by":"publisher","first-page":"2554","DOI":"10.1016\/j.ijleo.2013.10.092","volume":"125","author":"R Gong","year":"2013","unstructured":"Gong R, Xu H (2013) Impacts of appearance parameters on perceived image quality for mobile-phone displays. Optik 125(11):2554\u20132559. https:\/\/doi.org\/10.1016\/j.ijleo.2013.10.092","journal-title":"Optik"},{"key":"61_CR47","doi-asserted-by":"publisher","unstructured":"Rehman A, Zeng K, Wang Z (2015) Display device-adapted video quality-of-experience assessment. In: Rogowitz, B.E., Pappas, T.N., de Ridder, H. (eds.) Human vision and electronic imaging XX, vol. 9394, pp. 27\u201337. SPIE https:\/\/doi.org\/10.1117\/12.2077917. International Society for Optics and Photonics","DOI":"10.1117\/12.2077917"},{"key":"61_CR48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2016\/9621925","volume":"2016","author":"W Zou","year":"2016","unstructured":"Zou W, Song J, Yang F (2016) Perceived image quality on mobile phones with different screen resolution. Mobile Inform 2016:1\u201317. https:\/\/doi.org\/10.1155\/2016\/9621925","journal-title":"Mobile Inform"},{"key":"61_CR49","doi-asserted-by":"publisher","unstructured":"Sotelo R, Joskowicz J, Anedda M, Murroni M, Giusto DD (2017) Subjective video quality assessments for 4K UHDTV. In: 2017 IEEE International symposium on broadband multimedia systems and broadcasting (BMSB), pp. 1\u20136. IEEE. https:\/\/doi.org\/10.1109\/BMSB.2017.7986225","DOI":"10.1109\/BMSB.2017.7986225"},{"issue":"1","key":"61_CR50","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1007\/s41233-019-0027-3","volume":"4","author":"PA Kara","year":"2019","unstructured":"Kara PA, Robitza W, Pinter N, Martini MG, Raake A, Simon A (2019) Comparison of HD and UHD video quality with and without the influence of the labeling effect. Qual User Exper 4(1):4. https:\/\/doi.org\/10.1007\/s41233-019-0027-3","journal-title":"Qual User Exper"},{"key":"61_CR51","doi-asserted-by":"crossref","unstructured":"Saad MA, Pinson MH, Nicholas DG, Van\u00a0Kets N, Van\u00a0Wallendael G, Da\u00a0Silva R, Jaladi RV, Corriveau PJ (2015) Impact of camera pixel count and monitor resolution perceptual image quality. In: 2015 Colour and Visual Computing Symposium (CVCS). IEEE, pp. 1\u20136","DOI":"10.1109\/CVCS.2015.7274887"},{"key":"61_CR52","unstructured":"Rossi EA (2009) The Limits of Visual Resolution. Technical report, University of California, Berkeley"},{"issue":"9","key":"61_CR53","doi-asserted-by":"publisher","first-page":"2215","DOI":"10.1364\/JOSAA.25.002215","volume":"25","author":"YJ Kim","year":"2008","unstructured":"Kim YJ, Luo MR, Choe W, Kim HS, Park SO, Baek Y, Rhodes P, Lee S, Kim CY (2008) Factors affecting the psychophysical image quality evaluation of mobile phone displays: the case of transmissive liquid-crystal displays. J Opt Soci Am A 25(9):2215. https:\/\/doi.org\/10.1364\/JOSAA.25.002215","journal-title":"J Opt Soci Am A"},{"key":"61_CR54","doi-asserted-by":"crossref","unstructured":"Ho\u00dffeld T, Schatz R, Egger S (2011) Sos: The mos is not enough! In: 2011 Third International Workshop on Quality of Multimedia Experience, IEEE, pp. 131\u2013136","DOI":"10.1109\/QoMEX.2011.6065690"},{"key":"61_CR55","doi-asserted-by":"crossref","unstructured":"Hosu V, Hahn F, Jenadeleh M, Lin H, Men H, Szir\u00e1nyi T, Li S, Saupe D (2017) The konstanz natural video database (konvid-1k). In: 2017 Ninth international conference on quality of multimedia experience (QoMEX). IEEE, pp. 1\u20136","DOI":"10.1109\/QoMEX.2017.7965673"},{"issue":"2","key":"61_CR56","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1145\/2812802","volume":"59","author":"B Thomee","year":"2016","unstructured":"Thomee B, Shamma DA, Friedland G, Elizalde B, Ni K, Poland D, Borth D, Li L-J (2016) Yfcc100m: the new data in multimedia research. Commun ACM 59(2):64\u201373","journal-title":"Commun ACM"},{"key":"61_CR57","doi-asserted-by":"crossref","unstructured":"Hosu V, Lin H, Saupe D (2018) Expertise screening in crowdsourcing image quality. In: QoMEX 2018: Tenth international conference on quality of multimedia experience","DOI":"10.1109\/QoMEX.2018.8463427"},{"issue":"1","key":"61_CR58","doi-asserted-by":"publisher","first-page":"23","DOI":"10.20982\/tqmp.08.1.p023","volume":"8","author":"KA Hallgren","year":"2012","unstructured":"Hallgren KA (2012) Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quantit Meth Psychol 8(1):23","journal-title":"Tutor Quantit Meth Psychol"},{"issue":"2","key":"61_CR59","doi-asserted-by":"publisher","first-page":"420","DOI":"10.1037\/0033-2909.86.2.420","volume":"86","author":"PE Shrout","year":"1979","unstructured":"Shrout PE, Fleiss JL (1979) Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86(2):420","journal-title":"Psychol Bull"},{"key":"61_CR60","unstructured":"Touvron H, Vedaldi A, Douze M, Jegou H (2019) Fixing the train-test resolution discrepancy. In: Advances in neural information processing systems, vol. 32. Curran Associates, Inc"},{"key":"61_CR61","unstructured":"Dozat T (2016) Incorporating nesterov momentum into adam. In: Proceedings of the 4th International conference on learning representations (ICLR)"}],"container-title":["Quality and User Experience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41233-023-00061-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41233-023-00061-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41233-023-00061-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T10:06:02Z","timestamp":1734516362000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41233-023-00061-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,18]]},"references-count":61,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["61"],"URL":"https:\/\/doi.org\/10.1007\/s41233-023-00061-8","relation":{},"ISSN":["2366-0139","2366-0147"],"issn-type":[{"type":"print","value":"2366-0139"},{"type":"electronic","value":"2366-0147"}],"subject":[],"published":{"date-parts":[[2023,8,18]]},"assertion":[{"value":"23 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 August 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"8"}}