{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T03:07:01Z","timestamp":1772680021522,"version":"3.50.1"},"publisher-location":"Cham","reference-count":42,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031408366","type":"print"},{"value":"9783031408373","type":"electronic"}],"license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,22]],"date-time":"2023-08-22T00:00:00Z","timestamp":1692662400000},"content-version":"vor","delay-in-days":233,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Research in the field of explainable artificial intelligence has produced a vast amount of visual explanation methods for deep learning-based image classification in various domains of application. However, there is still a lack of domain-specific evaluation methods to assess an explanation\u2019s quality and a classifier\u2019s performance with respect to domain-specific requirements. In particular, evaluation methods could benefit from integrating human expertise into quality criteria and metrics. Such domain-specific evaluation methods can help to assess the robustness of deep learning models more precisely. In this paper, we present an approach for domain-specific evaluation of visual explanation methods in order to enhance the transparency of deep learning models and estimate their robustness accordingly. As an example use case, we apply our framework to facial expression recognition. We can show that the domain-specific evaluation is especially beneficial for challenging use cases such as facial expression recognition and provides application-grounded quality criteria that are not covered by standard evaluation methods. Our comparison of the domain-specific evaluation method with standard approaches thus shows that the quality of the expert knowledge is of great importance for assessing a model\u2019s performance precisely.<\/jats:p>","DOI":"10.1007\/978-3-031-40837-3_3","type":"book-chapter","created":{"date-parts":[[2023,8,21]],"date-time":"2023-08-21T23:02:25Z","timestamp":1692658945000},"page":"31-44","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Domain-Specific Evaluation of\u00a0Visual Explanations for\u00a0Application-Grounded Facial Expression Recognition"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9415-6254","authenticated-orcid":false,"given":"Bettina","family":"Finzel","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8694-762X","authenticated-orcid":false,"given":"Ines","family":"Rieger","sequence":"additional","affiliation":[]},{"given":"Simon","family":"Kuhn","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1301-0326","authenticated-orcid":false,"given":"Ute","family":"Schmid","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,8,22]]},"reference":[{"issue":"93","key":"3_CR1","first-page":"1","volume":"20","author":"M Alber","year":"2019","unstructured":"Alber, M., et al.: iNNvestigate neural networks! J. Mach. Learn. Res. 20(93), 1\u20138 (2019)","journal-title":"J. Mach. Learn. Res."},{"key":"3_CR2","unstructured":"Alvarez-Melis, D., Jaakkola, T.: On the robustness of interpretability methods. arXiv preprint arXiv:1806.08049 (2018)"},{"key":"3_CR3","doi-asserted-by":"publisher","unstructured":"Artelt, A., et al.: Evaluating robustness of counterfactual explanations. In: Proceedings of Symposium Series on Computational Intelligence, pp. 1\u20139. IEEE (2021). https:\/\/doi.org\/10.1109\/SSCI50451.2021.9660058","DOI":"10.1109\/SSCI50451.2021.9660058"},{"issue":"7","key":"3_CR4","doi-asserted-by":"publisher","first-page":"01","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), 01\u201309 (2015). https:\/\/doi.org\/10.1371\/journal.pone.0130140","journal-title":"PLoS ONE"},{"key":"3_CR5","doi-asserted-by":"publisher","unstructured":"Benitez-Quiroz, C.F., Srinivasan, R., Martinez, A.M.: EmotioNet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5562\u20135570. IEEE (2016). https:\/\/doi.org\/10.1109\/cvpr.2016.600","DOI":"10.1109\/cvpr.2016.600"},{"key":"3_CR6","unstructured":"Dhurandhar, A., et al.: Explanations based on the missing: towards contrastive explanations with pertinent negatives. In: Proceedings of Advances in Neural Information Processing Systems, vol. 31, pp. 590\u2013601 (2018)"},{"key":"3_CR7","doi-asserted-by":"crossref","unstructured":"Ekman, P., Friesen, W.V.: Facial Action Coding Systems. Consulting Psychologists Press (1978)","DOI":"10.1037\/t27734-000"},{"key":"3_CR8","unstructured":"Finzel, B., Kollmann, R., Rieger, I., Pahl, J., Schmid, U.: Deriving temporal prototypes from saliency map clusters for the analysis of deep-learning-based facial action unit classification. In: Proceedings of the LWDA 2021 Workshops: FGWM, KDML, FGWI-BIA, and FGIR. CEUR Workshop Proceedings, vol. 2993, pp. 86\u201397. CEUR-WS.org (2021)"},{"key":"3_CR9","unstructured":"Finzel, B., Tafler, D.E., Thaler, A.M., Schmid, U.: Multimodal explanations for user-centric medical decision support systems. In: Proceedings of the AAAI Fall Symposium on Human Partnership with Medical AI: Design, Operationalization, and Ethics (AAAI-HUMAN). CEUR Workshop Proceedings, vol. 3068. CEUR-WS.org (2021)"},{"issue":"5","key":"3_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3236009","volume":"51","author":"R Guidotti","year":"2018","unstructured":"Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5), 1\u201342 (2018). https:\/\/doi.org\/10.1145\/3236009","journal-title":"ACM Comput. Surv."},{"issue":"6","key":"3_CR11","doi-asserted-by":"publisher","first-page":"1815","DOI":"10.1109\/TPAMI.2019.2958341","volume":"43","author":"T Hassan","year":"2019","unstructured":"Hassan, T., et al.: Automatic detection of pain from facial expressions: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(6), 1815\u20131831 (2019)","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"issue":"34","key":"3_CR12","first-page":"1","volume":"24","author":"A Hedstr\u00f6m","year":"2023","unstructured":"Hedstr\u00f6m, A., et al.: Quantus: an explainable AI toolkit for responsible evaluation of neural network explanations and beyond. J. Mach. Learn. Res. 24(34), 1\u201311 (2023)","journal-title":"J. Mach. Learn. Res."},{"key":"3_CR13","series-title":"Communications in Computer and Information Science","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1007\/978-3-030-93736-2_33","volume-title":"Machine Learning and Principles and Practice of Knowledge Discovery in Databases","author":"A Holzinger","year":"2021","unstructured":"Holzinger, A.: The next frontier: AI we can really trust. In: Kamp, M., et al. (eds.) ECML PKDD 2021. CCIS, vol. 1524, pp. 427\u2013440. Springer, Cham (2021). https:\/\/doi.org\/10.1007\/978-3-030-93736-2_33"},{"key":"3_CR14","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/j.inffus.2021.10.007","volume":"79","author":"A Holzinger","year":"2022","unstructured":"Holzinger, A., et al.: Information fusion as an integrative cross-cutting enabler to achieve robust, explainable, and trustworthy medical artificial intelligence. Inf. Fusion 79, 263\u2013278 (2022). https:\/\/doi.org\/10.1016\/j.inffus.2021.10.007","journal-title":"Inf. Fusion"},{"key":"3_CR15","unstructured":"Hsieh, C., et al.: Evaluations and methods for explanation through robustness analysis. In: Proceedings of the 9th International Conference on Learning Representations (ICLR 2021). OpenReview.net (2021)"},{"key":"3_CR16","doi-asserted-by":"publisher","unstructured":"Jesus, S.M., et al.: How can I choose an explainer?: an application-grounded evaluation of post-hoc explanations. In: Proceedings of Conference on Fairness, Accountability, and Transparency (FAccT 2021), pp. 805\u2013815. ACM (2021). https:\/\/doi.org\/10.1145\/3442188.3445941","DOI":"10.1145\/3442188.3445941"},{"key":"3_CR17","doi-asserted-by":"publisher","unstructured":"Karasmanoglou, A., Antonakakis, M., Zervakis, M.E.: Heatmap-based explanation of YOLOv5 object detection with layer-wise relevance propagation. In: Proceedings of International Conference on Imaging Systems and Techniques, (IST), pp. 1\u20136. IEEE (2022). https:\/\/doi.org\/10.1109\/IST55454.2022.9827744","DOI":"10.1109\/IST55454.2022.9827744"},{"key":"3_CR18","first-page":"1755","volume":"10","author":"DE King","year":"2009","unstructured":"King, D.E.: Dlib-ML: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755\u20131758 (2009)","journal-title":"J. Mach. Learn. Res."},{"key":"3_CR19","doi-asserted-by":"publisher","unstructured":"Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., Lapuschkin, S.: Towards best practice in explaining neural network decisions with LRP. In: Proceedings of International Joint Conference on Neural Networks (IJCNN 2020), pp. 1\u20137. IEEE (2020). https:\/\/doi.org\/10.1109\/IJCNN48605.2020.9206975","DOI":"10.1109\/IJCNN48605.2020.9206975"},{"key":"3_CR20","unstructured":"Kollias, D., Zafeiriou, S.: Aff-wild2: extending the aff-wild database for affect recognition. arXiv preprint arXiv:1811.07770 (2018)"},{"key":"3_CR21","doi-asserted-by":"publisher","unstructured":"Kulesza, T., Stumpf, S., Burnett, M.M., Yang, S., Kwan, I., Wong, W.: Too much, too little, or just right? Ways explanations impact end users\u2019 mental models. In: 2013 IEEE Symposium on Visual Languages and Human Centric Computing, San Jose, CA, pp. 3\u201310. IEEE Computer Society (2013). https:\/\/doi.org\/10.1109\/VLHCC.2013.6645235","DOI":"10.1109\/VLHCC.2013.6645235"},{"issue":"6","key":"3_CR22","doi-asserted-by":"publisher","first-page":"813","DOI":"10.1002\/j.1532-2149.2013.00421.x","volume":"18","author":"M Kunz","year":"2014","unstructured":"Kunz, M., Lautenbacher, S.: The faces of pain: a cluster analysis of individual differences in facial activity patterns of pain. Eur. J. Pain 18(6), 813\u2013823 (2014)","journal-title":"Eur. J. Pain"},{"issue":"3","key":"3_CR23","doi-asserted-by":"publisher","first-page":"535","DOI":"10.1097\/j.pain.0000000000001424","volume":"160","author":"M Kunz","year":"2019","unstructured":"Kunz, M., Meixner, D., Lautenbacher, S.: Facial muscle movements encoding pain-a systematic review. Pain 160(3), 535\u2013549 (2019)","journal-title":"Pain"},{"key":"3_CR24","doi-asserted-by":"publisher","unstructured":"Lin, Y., Lee, W., Celik, Z.B.: What do you see?: evaluation of explainable artificial intelligence (XAI) interpretability through neural backdoors. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2021), Virtual Event, Singapore, pp. 1027\u20131035. ACM (2021). https:\/\/doi.org\/10.1145\/3447548.3467213","DOI":"10.1145\/3447548.3467213"},{"key":"3_CR25","doi-asserted-by":"publisher","unstructured":"Lucey, P., Cohn, J.F., Kanade, T., Saragih, J.M., Ambadar, Z., Matthews, I.A.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2010, San Francisco, CA, USA, pp. 94\u2013101. IEEE Computer Society (2010). https:\/\/doi.org\/10.1109\/CVPRW.2010.5543262","DOI":"10.1109\/CVPRW.2010.5543262"},{"key":"3_CR26","doi-asserted-by":"publisher","unstructured":"Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Matthews, I.A.: Painful data: the UNBC-McMaster shoulder pain expression archive database. In: Proceedings of the 9th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2011), Santa Barbara, CA, USA, pp. 57\u201364. IEEE Computer Society (2011). https:\/\/doi.org\/10.1109\/FG.2011.5771462","DOI":"10.1109\/FG.2011.5771462"},{"key":"3_CR27","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1016\/j.neucom.2019.03.082","volume":"355","author":"C Ma","year":"2019","unstructured":"Ma, C., Chen, L., Yong, J.: AU R-CNN: encoding expert prior knowledge into R-CNN for action unit detection. Neurocomputing 355, 35\u201347 (2019). https:\/\/doi.org\/10.1016\/j.neucom.2019.03.082","journal-title":"Neurocomputing"},{"key":"3_CR28","doi-asserted-by":"publisher","first-page":"112731","DOI":"10.1109\/ACCESS.2022.3214824","volume":"10","author":"M Malafaia","year":"2022","unstructured":"Malafaia, M., Silva, F., Neves, I., Pereira, T., Oliveira, H.P.: Robustness analysis of deep learning-based lung cancer classification using explainable methods. IEEE Access 10, 112731\u2013112741 (2022). https:\/\/doi.org\/10.1109\/ACCESS.2022.3214824","journal-title":"IEEE Access"},{"key":"3_CR29","unstructured":"Pahl, J., Rieger, I., Seuss, D.: Multi-label learning with missing values using combined facial action unit datasets. In: The Art of Learning with Missing Values Workshop at International Conference on Machine Learning (ICML 2020) abs\/2008.07234 (2020). https:\/\/arxiv.org\/abs\/2008.07234"},{"key":"3_CR30","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1007\/978-3-030-22999-3_12","volume-title":"Advances and Trends in Artificial Intelligence. From Theory to Practice","author":"I Rieger","year":"2019","unstructured":"Rieger, I., Hauenstein, T., Hettenkofer, S., Garbas, J.: Towards real-time head pose estimation: exploring parameter-reduced residual networks on in-the-wild datasets. In: Wotawa, F., Friedrich, G., Pill, I., Koitz-Hristov, R., Ali, M. (eds.) IEA\/AIE 2019. LNCS, vol. 11606, pp. 123\u2013134. Springer, Cham (2019). https:\/\/doi.org\/10.1007\/978-3-030-22999-3_12"},{"key":"3_CR31","unstructured":"Rieger, I., Kollmann, R., Finzel, B., Seuss, D., Schmid, U.: Verifying deep learning-based decisions for facial expression recognition. In: 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2020), Bruges, Belgium, pp. 139\u2013144 (2020). https:\/\/www.esann.org\/sites\/default\/files\/proceedings\/2020\/ES2020-49.pdf"},{"key":"3_CR32","doi-asserted-by":"crossref","unstructured":"Rieger, I., Pahl, J., Finzel, B., Schmid, U.: CorrLoss: integrating co-occurrence domain knowledge for affect recognition. In: Proceedings of the 26th International Conference on Pattern Recognition (ICPR 2022), pp. 798\u2013804. IEEE (2022)","DOI":"10.1109\/ICPR56361.2022.9956319"},{"key":"3_CR33","doi-asserted-by":"publisher","unstructured":"Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., M\u00fcller, K. (eds.): Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Lecture Notes in Computer Science, vol. 11700. Springer, Cham (2019). https:\/\/doi.org\/10.1007\/978-3-030-28954-6","DOI":"10.1007\/978-3-030-28954-6"},{"key":"3_CR34","doi-asserted-by":"publisher","unstructured":"Schlegel, U., Arnout, H., El-Assady, M., Oelke, D., Keim, D.A.: Towards a rigorous evaluation of XAI methods on time series. In: IEEE\/CVF International Conference on Computer Vision Workshop (ICCVW 2019), Seoul, Korea (South), pp. 4197\u20134201. IEEE (2019). https:\/\/doi.org\/10.1109\/ICCVW.2019.00516","DOI":"10.1109\/ICCVW.2019.00516"},{"key":"3_CR35","doi-asserted-by":"publisher","unstructured":"Schwalbe, G., Finzel, B.: A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts. Data Min. Knowl. Discov. 1\u201359 (2023). https:\/\/doi.org\/10.1007\/s10618-022-00867-8","DOI":"10.1007\/s10618-022-00867-8"},{"issue":"2","key":"3_CR36","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1007\/s11263-019-01228-7","volume":"128","author":"RR Selvaraju","year":"2020","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis. 128(2), 336\u2013359 (2020). https:\/\/doi.org\/10.1007\/s11263-019-01228-7","journal-title":"Int. J. Comput. Vis."},{"key":"3_CR37","doi-asserted-by":"publisher","unstructured":"Seuss, D., et al.: Emotion expression from different angles: a video database for facial expressions of actors shot by a camera array. In: Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII 2019), Cambridge, United Kingdom, pp. 35\u201341. IEEE (2019). https:\/\/doi.org\/10.1109\/ACII.2019.8925458","DOI":"10.1109\/ACII.2019.8925458"},{"key":"3_CR38","unstructured":"Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA (2015). https:\/\/arxiv.org\/abs\/1409.1556"},{"key":"3_CR39","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/j.inffus.2021.05.009","volume":"76","author":"G Vilone","year":"2021","unstructured":"Vilone, G., Longo, L.: Notions of explainability and evaluation approaches for explainable artificial intelligence. Inf. Fusion 76, 89\u2013106 (2021). https:\/\/doi.org\/10.1016\/j.inffus.2021.05.009","journal-title":"Inf. Fusion"},{"issue":"1","key":"3_CR40","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1109\/TAFFC.2019.2946774","volume":"13","author":"P Werner","year":"2022","unstructured":"Werner, P., Martinez, D.L., Walter, S., Al-Hamadi, A., Gruss, S., Picard, R.W.: Automatic recognition methods supporting pain assessment: a survey. IEEE Trans. Affect. Comput. 13(1), 530\u2013552 (2022). https:\/\/doi.org\/10.1109\/TAFFC.2019.2946774","journal-title":"IEEE Trans. Affect. Comput."},{"issue":"10","key":"3_CR41","doi-asserted-by":"publisher","first-page":"692","DOI":"10.1016\/j.imavis.2014.06.002","volume":"32","author":"X Zhang","year":"2014","unstructured":"Zhang, X., et al.: BP4D-spontaneous: a high-resolution spontaneous 3D dynamic facial expression database. Image Vis. Comput. 32(10), 692\u2013706 (2014)","journal-title":"Image Vis. Comput."},{"issue":"5","key":"3_CR42","doi-asserted-by":"publisher","first-page":"593","DOI":"10.3390\/electronics10050593","volume":"10","author":"J Zhou","year":"2021","unstructured":"Zhou, J., Gandomi, A.H., Chen, F., Holzinger, A.: Evaluating the quality of machine learning explanations: a survey on methods and metrics. Electronics 10(5), 593 (2021)","journal-title":"Electronics"}],"container-title":["Lecture Notes in Computer Science","Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-40837-3_3","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,13]],"date-time":"2024-03-13T12:32:46Z","timestamp":1710333166000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-40837-3_3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"ISBN":["9783031408366","9783031408373"],"references-count":42,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-40837-3_3","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023]]},"assertion":[{"value":"22 August 2023","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"CD-MAKE","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Cross-Domain Conference for Machine Learning and Knowledge Extraction","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Benevento","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Italy","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2023","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"28 August 2023","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1 September 2023","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"7","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"cd-make2023","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/cd-make.net\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"30","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"18","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"60% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"2","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"No","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}