{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T04:29:41Z","timestamp":1772166581105,"version":"3.50.1"},"reference-count":56,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T00:00:00Z","timestamp":1727395200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T00:00:00Z","timestamp":1727395200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001871","name":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["10.54499\/CHIST-ERA\/0003\/2019"],"award-info":[{"award-number":["10.54499\/CHIST-ERA\/0003\/2019"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Image Video Proc."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The integration of Face Verification (FV) systems into multiple critical moments of daily life has become increasingly prevalent, raising concerns regarding the transparency and reliability of these systems. Consequently, there is a growing need for FV explainability tools to provide insights into the behavior of these systems. FV explainability tools that generate visual explanations, e.g., saliency maps, heatmaps, contour-based visualization maps, and face segmentation maps, show promise in enhancing FV transparency by highlighting the contributions of different face regions to the FV decision-making process. However, evaluating the performance of such explainability tools remains challenging due to the lack of standardized assessment metrics and protocols. In this context, this paper proposes a subjective performance assessment protocol for evaluating the explainability performance of visual explanation-based FV explainability tools through pairwise comparisons of their explanation outputs. The proposed protocol encompasses a set of key specifications designed to efficiently collect the subjects\u2019 preferences and estimate explainability performance scores, facilitating the relative assessment of the explainability tools. This protocol aims to address the current gap in evaluating the effectiveness of visual explanation-based FV explainability tools, providing a structured approach for assessing their performance and comparing with alternative tools. The proposed protocol is exercised and validated through an experiment conducted using two distinct heatmap-based FV explainability tools, notably FV-RISE and CorrRISE, taken as examples of visual explanation-based explainability tools, considering the various types of FV decisions, i.e., True Acceptance (TA), False Acceptance (FA), True Rejection (TR), and False Rejection (FR). A group of subjects with variety in age, gender, and ethnicity was tasked to express their preferences regarding the heatmap-based explanations generated by the two selected explainability tools. The subject preferences were collected and statistically processed to derive quantifiable scores, expressing the relative explainability performance of the assessed tools. The experimental results revealed that both assessed explainability tools exhibit comparable explainability performance for FA, TR, and FR decisions with CorrRISE performing slightly better than FV-RISE for TA decisions.<\/jats:p>","DOI":"10.1186\/s13640-024-00645-0","type":"journal-article","created":{"date-parts":[[2024,9,27]],"date-time":"2024-09-27T06:02:04Z","timestamp":1727416924000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Subjective performance assessment protocol for visual explanations-based face verification explainability"],"prefix":"10.1186","volume":"2024","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6016-7000","authenticated-orcid":false,"given":"Naima","family":"Bousnina","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jo\u00e3o","family":"Ascenso","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paulo Lobato","family":"Correia","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fernando","family":"Pereira","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,9,27]]},"reference":[{"issue":"113","key":"645_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2008\/579416","volume":"2008","author":"AK Jain","year":"2008","unstructured":"A.K. Jain, K. Nandakumar, A. Nagar, Biometric template security. EURASIP J. Adv. Signal Process. 2008(113), 1\u201317 (2008). https:\/\/doi.org\/10.1155\/2008\/579416","journal-title":"EURASIP J. Adv. Signal Process."},{"issue":"1","key":"645_CR2","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/TCSVT.2003.818349","volume":"14","author":"AK Jain","year":"2004","unstructured":"A.K. Jain, A. Ross, S. Prabhakar, An introduction to biometric recognition. IEEE Trans. Circuits Syst. Video Technol. 14(1), 4\u201320 (2004). https:\/\/doi.org\/10.1109\/TCSVT.2003.818349","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"645_CR3","doi-asserted-by":"crossref","unstructured":"B. Yalavarthi et al., Enhancing privacy in face analytics using fully homomorphic encryption (2024). arXiv:2404.16255v1","DOI":"10.1109\/FG59268.2024.10581983"},{"key":"645_CR4","doi-asserted-by":"crossref","unstructured":"M. Huber, A. T. Luu, P. Terh\u00f6rst, N. Damer, Efficient explainable face verification based on similarity score argument backpropagation (2023). arXiv:2304.13409v2","DOI":"10.1109\/WACV57701.2024.00467"},{"key":"645_CR5","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1007\/s43681-021-00077-w","volume":"2","author":"D Almeida","year":"2022","unstructured":"D. Almeida, K. Shmarko, E. Lomas, The ethics of facial recognition technologies, surveillance, and accountability in an age of artificial intelligence: a comparative analysis of US, EU, and UK Regulatory Frameworks. AI Ethics. 2, 377\u2013387 (2022). https:\/\/doi.org\/10.1007\/s43681-021-00077-w","journal-title":"AI Ethics."},{"key":"645_CR6","unstructured":"P. C. Neto et al., Causality-inspired taxonomy for explainable artificial intelligence (2024). arXiv:2208.09500v2"},{"key":"645_CR7","doi-asserted-by":"crossref","unstructured":"M. Knoche, T. Teepe, S. Hormann, G. Rigoll, Explainable model-agnostic similarity and confidence in face verification. In: Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), Waikoloa, HI, USA, 2023","DOI":"10.1109\/WACVW58289.2023.00078"},{"key":"645_CR8","doi-asserted-by":"crossref","unstructured":"D. Mery, B. Morris, On black-box explanation for face verification. In: IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2022","DOI":"10.1109\/WACV51458.2022.00126"},{"key":"645_CR9","doi-asserted-by":"crossref","unstructured":"N. Bousnina, J. Ascenso, P. L. Correia, F. Pereira, A RISE-based explainability method for genuine and impostor face verification. In: International Conference of the Biometrics Special Interest Group (BIOSIG), Darmstadt, Germany, 2023","DOI":"10.1109\/BIOSIG58226.2023.10345979"},{"key":"645_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2021.108102","author":"X Bai","year":"2021","unstructured":"X. Bai et al., Explainable deep learning for efficient and robust pattern recognition: a survey of recent developments. Pattern Recognit. (2021). https:\/\/doi.org\/10.1016\/j.patcog.2021.108102","journal-title":"Pattern Recognit."},{"key":"645_CR11","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","volume":"6","author":"A Adadi","year":"2018","unstructured":"A. Adadi, M. Berrada, Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access. 6, 52138\u201352160 (2018). https:\/\/doi.org\/10.1109\/ACCESS.2018.2870052","journal-title":"IEEE Access."},{"issue":"8","key":"645_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3390\/electronics8080832","volume":"8","author":"DV Carvalho","year":"2019","unstructured":"D.V. Carvalho, E.M. Pereira, J.S. Cardoso, Machine learning interpretability: a survey on methods and metrics. Electronics 8(8), 1\u201334 (2019). https:\/\/doi.org\/10.3390\/electronics8080832","journal-title":"Electronics"},{"issue":"13s","key":"645_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3583558","volume":"55","author":"M Nauta","year":"2023","unstructured":"M. Nauta et al., From anecdotal evidence to quantitative evaluation methods: a systematic review on evaluating Explainable AI. ACM Comput. Surv. 55(13s), 1\u201334 (2023). https:\/\/doi.org\/10.1145\/3583558","journal-title":"ACM Comput. Surv."},{"key":"645_CR14","unstructured":"J. R. Williford, B. B. May, J. Byrne, Explainable face recognition. In: Computer Vision\u2014ECCV 2020, Glasgow, UK, August 2020, ed. by A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm"},{"issue":"3s","key":"645_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3469288","volume":"17","author":"Y-S Lin","year":"2021","unstructured":"Y.-S. Lin et al., xCos: an explainable cosine metric for face verification task. ACM Trans. Multimedia Comput. Commun. Appl. 17(3s), 1\u201316 (2021). https:\/\/doi.org\/10.1145\/3469288","journal-title":"ACM Trans. Multimedia Comput. Commun. Appl."},{"key":"645_CR16","doi-asserted-by":"crossref","unstructured":"Y. Lu, Z. Xu, T. Ebrahimi, Towards visual saliency explanations of face verification (2023). arXiv:2305.08546v4","DOI":"10.1109\/WACV57701.2024.00466"},{"key":"645_CR17","doi-asserted-by":"crossref","unstructured":"B. Yin et al., Towards interpretable face recognition. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019","DOI":"10.1109\/ICCV.2019.00944"},{"key":"645_CR18","doi-asserted-by":"crossref","unstructured":"M. Winter, W. Bailer, G. Thallinger, Demystifying face-recognition with locally interpretable boosted features (LIBF). In: 10th European Workshop on Visual Information Processing (EUVIP), Lisbon, Portugal, 2022","DOI":"10.1109\/EUVIP53989.2022.9922905"},{"key":"645_CR19","doi-asserted-by":"crossref","unstructured":"Y. Lu, T. Ebrahimi, Explanation of face recognition via saliency maps. In: Applications of Digital Image Processing XLVI, ed. by A. G. Tescher and T. Ebrahimi, San Diego, CA, United States, 2023","DOI":"10.1117\/12.2677353"},{"key":"645_CR20","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/j.inffus.2021.05.009","volume":"76","author":"G Vilone","year":"2021","unstructured":"G. Vilone, L. Longo, Notions of explainability and evaluation approaches for explainable artificial intelligence. Inf. Fusion 76, 89\u2013106 (2021). https:\/\/doi.org\/10.1016\/j.inffus.2021.05.009","journal-title":"Inf. Fusion"},{"key":"645_CR21","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-319-98131-4_1","volume-title":"Explainable and interpretable models in computer vision and machine learning","author":"F Doshi-Velez","year":"2018","unstructured":"F. Doshi-Velez, B. Kim, Considerations for evaluation and generalization in interpretable machine learning, in Explainable and interpretable models in computer vision and machine learning. ed. by H.J. Escalante, S. Escalera, I. Guyon, X. Bar\u00f3, Y. G\u00fc\u00e7l\u00fct\u00fcrk, U. G\u00fc\u00e7l\u00fc, M. Van Gerven (Springer International Publishing, Cham, 2018), pp.3\u201317"},{"key":"645_CR22","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1007\/s44230-023-00038-y","volume":"3","author":"W Yang","year":"2023","unstructured":"W. Yang et al., Survey on explainable AI: from approaches, limitations and applications aspects. Hum.-Cent. Intell. Syst. 3, 161\u2013188 (2023). https:\/\/doi.org\/10.1007\/s44230-023-00038-y","journal-title":"Hum.-Cent. Intell. Syst."},{"key":"645_CR23","unstructured":"J. Colin, T. FEL, R. Cadene, T. Serre, What I cannot predict, I do not understand: A human-centered evaluation framework for explainability methods. In Advances in Neural Information Processing Systems, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Vol. 35 (Curran Associates, Inc., 2022), pp. 2832\u20132845"},{"key":"645_CR24","doi-asserted-by":"crossref","unstructured":"K. Sokol, J. E. Vogt, What does evaluation of explainable artificial intelligence actually tell us? A case for compositional and contextual validation of XAI building blocks. In: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, Honolulu HI USA, 2024","DOI":"10.1145\/3613905.3651047"},{"key":"645_CR25","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.inffus.2019.12.012","volume":"58","author":"A Barredo Arrieta","year":"2020","unstructured":"A. Barredo Arrieta et al., Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58, 82\u2013115 (2020). https:\/\/doi.org\/10.1016\/j.inffus.2019.12.012","journal-title":"Inf. Fusion"},{"issue":"10","key":"645_CR26","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1145\/323323","volume":"61","author":"ZC Lipton","year":"2018","unstructured":"Z.C. Lipton, The mythos of model interpretability. Commun. ACM 61(10), 36\u201343 (2018). https:\/\/doi.org\/10.1145\/323323","journal-title":"Commun. ACM"},{"key":"645_CR27","doi-asserted-by":"crossref","unstructured":"G. Castanon, J. Byrne, Visualizing and quantifying discriminative features for face recognition. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi\u2019an, China, 2018","DOI":"10.1109\/FG.2018.00013"},{"key":"645_CR28","doi-asserted-by":"crossref","unstructured":"Z. Xu, Y. Lu, T. Ebrahimi, Discriminative deep feature visualization for explainable face recognition. In: IEEE 25th International Workshop on Multimedia Signal Processing (MMSP), Poitiers, France, 2023","DOI":"10.1109\/MMSP59012.2023.10337692"},{"key":"645_CR29","doi-asserted-by":"crossref","unstructured":"D. Mery, True black-box explanation in facial analysis. In: IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW0), New Orleans, LA, USA, 2022","DOI":"10.1109\/CVPRW56347.2022.00166"},{"key":"645_CR30","doi-asserted-by":"publisher","first-page":"663","DOI":"10.1007\/s11277-022-10127-z","volume":"129","author":"A Rajpal","year":"2023","unstructured":"A. Rajpal, K. Sehra, R. Bagri, P. Sikka, XAI-FR: explainable AI-based face recognition using deep neural networks. Wirel. Pers. Commun. 129, 663\u2013680 (2023). https:\/\/doi.org\/10.1007\/s11277-022-10127-z","journal-title":"Wirel. Pers. Commun."},{"key":"645_CR31","doi-asserted-by":"crossref","unstructured":"H. Jiang, D. Zeng, Explainable face recognition based on accurate facial compositions. In: IEEE\/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada, 2021","DOI":"10.1109\/ICCVW54120.2021.00175"},{"issue":"8","key":"645_CR32","doi-asserted-by":"publisher","first-page":"2478","DOI":"10.1111\/j.1467-8659.2012.03188.x","volume":"31","author":"RK Mantiuk","year":"2012","unstructured":"R.K. Mantiuk, A. Tomaszewska, R. Mantiuk, Comparison of four subjective methods for image quality assessment. Comput. Graph. Forum 31(8), 2478\u20132491 (2012). https:\/\/doi.org\/10.1111\/j.1467-8659.2012.03188.x","journal-title":"Comput. Graph. Forum"},{"key":"645_CR33","unstructured":"G. B. Huang, M. Ramesh, T. Berg, E. Learned-Miller, Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In: Workshop on Faces in\u2019Real-Life\u2019Images: Detection, Alignment, and Recognition, Marseille, France, 2008"},{"key":"645_CR34","doi-asserted-by":"crossref","unstructured":"R. Correia, P. Correia, F. Pereira, Face verification explainability heatmap generation. In: International Conference of the Biometrics Special Interest Group (BIOSIG), Darmstadt, Germany, 2023","DOI":"10.1109\/BIOSIG58226.2023.10345992"},{"key":"645_CR35","doi-asserted-by":"crossref","unstructured":"O. M. Parkhi, A. Vedaldi, A. Zisserman, Deep face recognition. In: Proceedings of the British Machine Vision Conference, Swansea, UK, 2015","DOI":"10.5244\/C.29.41"},{"key":"645_CR36","doi-asserted-by":"crossref","unstructured":"A. Chattopadhay, A. Sarkar, P. Howlader, V. N. Balasubramanian, Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 2018","DOI":"10.1109\/WACV.2018.00097"},{"key":"645_CR37","unstructured":"K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: visualising image classification models and saliency maps. In: Proceedings of the 2nd International Conference on Learning Representations, Banff, AB, Canada, 2013"},{"key":"645_CR38","doi-asserted-by":"crossref","unstructured":"A. Stylianou, R. Souvenir, R. Pless, Visualizing deep similarity networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA, 2019","DOI":"10.1109\/WACV.2019.00220"},{"key":"645_CR39","doi-asserted-by":"crossref","unstructured":"S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kotsia, S. Zafeiriou, AgeDB: The first manually collected, in-the-wild age database. In: IEEE Conference on Computer Vision and Pattern Recognition Work-shops (CVPRW), Honolulu, HI, USA, 2017","DOI":"10.1109\/CVPRW.2017.250"},{"key":"645_CR40","doi-asserted-by":"crossref","unstructured":"S. Sengupta et al., Frontal to profile face verification in the wild. In: IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA, 2016","DOI":"10.1109\/WACV.2016.7477558"},{"key":"645_CR41","unstructured":"Standard ISO\/IEC 29170-2:2015, Information technology\u2014Advanced image coding and evaluation\u2014Part 2: Evaluation procedure for nearly lossless coding. (Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), 2015), https:\/\/www.iso.org\/standard\/66094.html. Accessed 23 February 2024"},{"key":"645_CR42","unstructured":"Recommendation ITU-T P.910, Subjective video quality assessment methods for multimedia applications. (ITU-T Telecommunication Standardization Sector of ITU, 2022), https:\/\/www.itu.int\/rec\/T-REC-P.910-202207-S. Accessed 21 February 2024"},{"key":"645_CR43","unstructured":"S. Bel, Color blindness test: Color deficiency testing plates, (Independently published, 2021), pp. 1\u201350"},{"key":"645_CR44","doi-asserted-by":"crossref","unstructured":"Z. Zhang et al., An improved pairwise comparison scaling method for subjective image quality assessment. In: IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Cagliari, Italy, 2017","DOI":"10.1109\/BMSB.2017.7986235"},{"issue":"3","key":"645_CR45","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1111\/1467-9876.00159","volume":"48","author":"ME Glickman","year":"1999","unstructured":"M.E. Glickman, Parameter estimation in large dynamic paired comparison experiments. J. R. Stat. Soc. C Appl. Stat. 48(3), 377\u2013394 (1999). https:\/\/doi.org\/10.1111\/1467-9876.00159","journal-title":"J. R. Stat. Soc. C Appl. Stat."},{"key":"645_CR46","doi-asserted-by":"publisher","first-page":"324","DOI":"10.2307\/2334029","volume":"39","author":"RA Bradley","year":"1952","unstructured":"R.A. Bradley, M.E. Terry, Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 324\u2013345 (1952). https:\/\/doi.org\/10.2307\/2334029","journal-title":"Biometrika"},{"issue":"4","key":"645_CR47","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1037\/h0070288","volume":"34","author":"LL Thurstone","year":"1927","unstructured":"L.L. Thurstone, A law of comparative judgment. Psychol. Rev. 34(4), 273\u2013286 (1927). https:\/\/doi.org\/10.1037\/h0070288","journal-title":"Psychol. Rev."},{"key":"645_CR48","doi-asserted-by":"crossref","unstructured":"J. Deng, J. Guo, N. Xue, S. Zafeiriou, ArcFace: additive angular margin loss for deep face recognition. In: IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019","DOI":"10.1109\/CVPR.2019.00482"},{"key":"645_CR49","doi-asserted-by":"crossref","unstructured":"K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016","DOI":"10.1109\/CVPR.2016.90"},{"key":"645_CR50","unstructured":"Distributed Arcface Training in Pytorch, https:\/\/github.com\/deepinsight\/insightface\/tree\/master\/recognition\/arcface_torch. Accessed 11 July 2024"},{"key":"645_CR51","doi-asserted-by":"crossref","unstructured":"J. Deng, J. Guo, E. Ververas, I. Kotsia, S. Zafeiriou, RetinaFace: single-shot multi-level face localisation in the wild. In: IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 2020","DOI":"10.1109\/CVPR42600.2020.00525"},{"key":"645_CR52","unstructured":"V. Petsiuk, A. Das, K. Saenko, RISE: Randomized input sampling for explanation of black-box models (2018). arXiv:1806.07421v3"},{"key":"645_CR53","unstructured":"T. Ho\u00dffeld et al., Best practices and recommendations for crowdsourced QoE lessons learned from the Qualinet WG2 task force \u201cCrowdsourcing\u201d. (COST Action IC1003 European Network on Quality of Experience in Multimedia Systems and Services (QUALINET), 2014), https:\/\/infoscience.epfl.ch\/record\/204797?ln=en. Accessed 23 February 2024"},{"key":"645_CR54","unstructured":"OpenJS Foundation and Node.js contributors, Node.Js. https:\/\/nodejs.org\/en. Accessed 23 February 2024"},{"key":"645_CR55","unstructured":"MongoDB, Inc., MongoDB. https:\/\/www.mongodb.com\/. Accessed 23 February 2024"},{"key":"645_CR56","unstructured":"Recommendation ITU-R BT.500-10, Methodology for the subjective assessment of the quality of television pictures. (ITU Radiocommunication Sector, 2000), https:\/\/www.itu.int\/rec\/R-REC-BT.500-10-200003-S\/en. Accessed 23 February 2024"}],"container-title":["EURASIP Journal on Image and Video Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13640-024-00645-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13640-024-00645-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13640-024-00645-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,28]],"date-time":"2024-11-28T15:04:15Z","timestamp":1732806255000},"score":1,"resource":{"primary":{"URL":"https:\/\/jivp-eurasipjournals.springeropen.com\/articles\/10.1186\/s13640-024-00645-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,27]]},"references-count":56,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["645"],"URL":"https:\/\/doi.org\/10.1186\/s13640-024-00645-0","relation":{"has-preprint":[{"id-type":"doi","id":"10.21203\/rs.3.rs-4009938\/v1","asserted-by":"object"}]},"ISSN":["1687-5281"],"issn-type":[{"value":"1687-5281","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,27]]},"assertion":[{"value":"4 March 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 August 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 September 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"33"}}