{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T04:56:55Z","timestamp":1776401815361,"version":"3.51.2"},"publisher-location":"Cham","reference-count":50,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031440663","type":"print"},{"value":"9783031440670","type":"electronic"}],"license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,21]],"date-time":"2023-10-21T00:00:00Z","timestamp":1697846400000},"content-version":"vor","delay-in-days":293,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Analysis of how semantic concepts are represented within Convolutional Neural Networks (CNNs) is a widely used approach in Explainable Artificial Intelligence (XAI) for interpreting CNNs. A motivation is the need for transparency in safety-critical AI-based systems, as mandated in various domains like automated driving. However, to use the concept representations for safety-relevant purposes, like inspection or error retrieval, these must be of high quality and, in particular, stable. This paper focuses on two stability goals when working with concept representations in computer vision CNNs: stability of concept retrieval and of concept attribution. The guiding use-case is a post-hoc explainability framework for object detection (OD) CNNs, towards which existing concept analysis (CA) methods are successfully adapted. To address concept retrieval stability, we propose a novel metric that considers both concept separation and consistency, and is agnostic to layer and concept representation dimensionality. We then investigate impacts of concept abstraction level, number of concept training samples, CNN size, and concept representation dimensionality on stability. For concept attribution stability we explore the effect of gradient instability on gradient-based explainability methods. The results on various CNNs for classification and object detection yield the main findings that (1) the stability of concept retrieval can be enhanced through dimensionality reduction via data aggregation, and (2) in shallow layers where gradient instability is more pronounced, gradient smoothing techniques are advised. Finally, our approach provides valuable insights into selecting the appropriate layer and concept representation dimensionality, paving the way towards CA in safety-critical XAI applications.<\/jats:p>","DOI":"10.1007\/978-3-031-44067-0_26","type":"book-chapter","created":{"date-parts":[[2023,10,20]],"date-time":"2023-10-20T06:02:33Z","timestamp":1697781753000},"page":"499-524","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Evaluating the\u00a0Stability of\u00a0Semantic Concept Representations in\u00a0CNNs for\u00a0Robust Explainability"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2494-6285","authenticated-orcid":false,"given":"Georgii","family":"Mikriukov","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2690-2478","authenticated-orcid":false,"given":"Gesina","family":"Schwalbe","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5781-6575","authenticated-orcid":false,"given":"Christian","family":"Hellert","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9139-8947","authenticated-orcid":false,"given":"Korinna","family":"Bade","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,10,21]]},"reference":[{"key":"26_CR1","unstructured":"32, I.S.: ISO 26262-1:2018(En): Road Vehicles \u2013 Functional Safety \u2013 Part 1: Vocabulary (2018). https:\/\/www.iso.org\/standard\/68383.html"},{"key":"26_CR2","unstructured":"Abid, A., Yuksekgonul, M., Zou, J.: Meaningfully debugging model mistakes using conceptual counterfactual explanations. In: Proceedings of the 39th International Conference on Machine Learning, pp. 66\u201388. PMLR, June 2022"},{"key":"26_CR3","doi-asserted-by":"crossref","unstructured":"Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A.: Network dissection: quantifying interpretability of deep visual representations. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6541\u20136549 (2017)","DOI":"10.1109\/CVPR.2017.354"},{"key":"26_CR4","unstructured":"Bodria, F., Giannotti, F., Guidotti, R., Naretto, F., Pedreschi, D., Rinzivillo, S.: Benchmarking and survey of explanation methods for black box models. arXiv preprint arXiv:2102.13076 (2021)"},{"issue":"8","key":"26_CR5","doi-asserted-by":"publisher","first-page":"832","DOI":"10.3390\/electronics8080832","volume":"8","author":"DV Carvalho","year":"2019","unstructured":"Carvalho, D.V., Pereira, E.M., Cardoso, J.S.: Machine learning interpretability: a survey on methods and metrics. Electronics 8(8), 832 (2019). https:\/\/doi.org\/10.3390\/electronics8080832","journal-title":"Electronics"},{"issue":"12","key":"26_CR6","doi-asserted-by":"publisher","first-page":"772","DOI":"10.1038\/s42256-020-00265-z","volume":"2","author":"Z Chen","year":"2020","unstructured":"Chen, Z., Bei, Y., Rudin, C.: Concept whitening for interpretable image recognition. Nat. Mach. Intell. 2(12), 772\u2013782 (2020)","journal-title":"Nat. Mach. Intell."},{"key":"26_CR7","unstructured":"Chyung, C., Tsang, M., Liu, Y.: Extracting interpretable concept-based decision trees from CNNs. In: Proceedings of the 2019 ICML Workshop Human in the Loop Learning, vol. 1906.04664, June 2019. CoRR"},{"key":"26_CR8","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248\u2013255. IEEE (2009)","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"26_CR9","doi-asserted-by":"publisher","unstructured":"Esser, P., Rombach, R., Ommer, B.: A disentangling invertible interpretation network for explaining latent representations. In: Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition, pp. 9220\u20139229. IEEE, June 2020. https:\/\/doi.org\/10.1109\/CVPR42600.2020.00924","DOI":"10.1109\/CVPR42600.2020.00924"},{"key":"26_CR10","doi-asserted-by":"crossref","unstructured":"Fong, R., Vedaldi, A.: Net2Vec: quantifying and explaining how concepts are encoded by filters in deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8730\u20138738 (2018)","DOI":"10.1109\/CVPR.2018.00910"},{"key":"26_CR11","doi-asserted-by":"crossref","unstructured":"Ge, Y., et al.: A peek into the reasoning of neural networks: interpreting with structural visual concepts. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 2195\u20132204 (2021)","DOI":"10.1109\/CVPR46437.2021.00223"},{"key":"26_CR12","unstructured":"Ghorbani, A., Wexler, J., Zou, J.Y., Kim, B.: Towards automatic concept-based explanations. Adv. Neural Inf. Process. Syst. 32 (2019)"},{"key":"26_CR13","doi-asserted-by":"publisher","unstructured":"Goodman, B., Flaxman, S.: European union regulations on algorithmic decision-making and a right to explanation. AI Mag. 38(3), 50\u201357 (2017). https:\/\/doi.org\/10.1609\/aimag.v38i3.2741","DOI":"10.1609\/aimag.v38i3.2741"},{"key":"26_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2020.103865","volume":"123","author":"M Graziani","year":"2020","unstructured":"Graziani, M., Andrearczyk, V., Marchand-Maillet, S., M\u00fcller, H.: Concept attribution: explaining CNN decisions to physicians. Comput. Biol. Med. 123, 103865 (2020). https:\/\/doi.org\/10.1016\/j.compbiomed.2020.103865","journal-title":"Comput. Biol. Med."},{"key":"26_CR15","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1007\/978-3-030-02628-8_14","volume-title":"Understanding and Interpreting Machine Learning in Medical Image Computing Applications","author":"M Graziani","year":"2018","unstructured":"Graziani, M., Andrearczyk, V., M\u00fcller, H.: Regression concept vectors for bidirectional explanations in histopathology. In: Stoyanov, D., et al. (eds.) MLCN\/DLF\/IMIMIC -2018. LNCS, vol. 11038, pp. 124\u2013132. Springer, Cham (2018). https:\/\/doi.org\/10.1007\/978-3-030-02628-8_14"},{"key":"26_CR16","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770\u2013778 (2016)","DOI":"10.1109\/CVPR.2016.90"},{"key":"26_CR17","doi-asserted-by":"crossref","unstructured":"Howard, A., et al.: Searching for MobileNetV3. In: Proceedings of the IEEE\/CVF International Conference on Computer Vision, pp. 1314\u20131324 (2019)","DOI":"10.1109\/ICCV.2019.00140"},{"key":"26_CR18","unstructured":"Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: alexnet-level accuracy with 50x fewer parameters and $$<$$0.5 mb model size. arXiv preprint arXiv:1602.07360 (2016)"},{"key":"26_CR19","unstructured":"ISO\/TC 22\/SC 32: ISO 26262-8:2018(En): Road Vehicles \u2014 Functional Safety \u2014 Part 8: Supporting Processes, ISO 26262:2018(En), vol.\u00a08. International Organization for Standardization, second edn., December 2018"},{"key":"26_CR20","doi-asserted-by":"publisher","unstructured":"Jocher, G.: YOLOv5 in PyTorch, ONNX, CoreML, TFLite, October 2020. https:\/\/github.com\/ultralytics\/yolov5, https:\/\/doi.org\/10.5281\/zenodo.4154370","DOI":"10.5281\/zenodo.4154370"},{"key":"26_CR21","unstructured":"Kazhdan, D., Dimanov, B., Jamnik, M., Li\u00f2, P., Weller, A.: Now you see me (CME): concept-based model extraction. In: Proceedings of the 29th ACM International Conference Information and Knowledge Management Workshops. CEUR Workshop Proceedings, vol. 2699. CEUR-WS.org (2020)"},{"key":"26_CR22","unstructured":"Kazhdan, D., Dimanov, B., Terre, H.A., Jamnik, M., Li\u00f2, P., Weller, A.: Is disentanglement all you need? Comparing concept-based & disentanglement approaches. CoRR abs\/2104.06917 (2021)"},{"key":"26_CR23","unstructured":"Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.: Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). In: International Conference on Machine Learning, pp. 2668\u20132677. PMLR (2018)"},{"key":"26_CR24","unstructured":"Kirchknopf, A., Slijepcevic, D., Wunderlich, I., Breiter, M., Traxler, J., Zeppelzauer, M.: Explaining yolo: leveraging grad-cam to explain object detections. arXiv preprint arXiv:2211.12108 (2022)"},{"key":"26_CR25","unstructured":"Koh, P.W., et al.: Concept bottleneck models. In: International Conference on Machine Learning, pp. 5338\u20135348. PMLR (2020)"},{"key":"26_CR26","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1007\/978-3-319-10602-1_48","volume-title":"Computer Vision \u2013 ECCV 2014","author":"T-Y Lin","year":"2014","unstructured":"Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740\u2013755. Springer, Cham (2014). https:\/\/doi.org\/10.1007\/978-3-319-10602-1_48"},{"issue":"1","key":"26_CR27","doi-asserted-by":"publisher","first-page":"18","DOI":"10.3390\/e23010018","volume":"23","author":"P Linardatos","year":"2021","unstructured":"Linardatos, P., Papastefanopoulos, V., Kotsiantis, S.: Explainable AI: a review of machine learning interpretability methods. Entropy 23(1), 18 (2021). https:\/\/doi.org\/10.3390\/e23010018","journal-title":"Entropy"},{"key":"26_CR28","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1007\/978-3-319-46448-0_2","volume-title":"Computer Vision \u2013 ECCV 2016","author":"W Liu","year":"2016","unstructured":"Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21\u201337. Springer, Cham (2016). https:\/\/doi.org\/10.1007\/978-3-319-46448-0_2"},{"key":"26_CR29","unstructured":"Margeloiu, A., Ashman, M., Bhatt, U., Chen, Y., Jamnik, M., Weller, A.: Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289 (2021)"},{"key":"26_CR30","unstructured":"Pfau, J., Young, A.T., Wei, J., Wei, M.L., Keiser, M.J.: Robust semantic interpretability: revisiting concept activation vectors. In: Proceedings of the 2021 ICML Workshop Human Interpretability in Machine Learning, April 2021. CoRR"},{"key":"26_CR31","series-title":"Lecture Notes in Computer Science (Lecture Notes in Artificial Intelligence)","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1007\/978-3-030-58285-2_11","volume-title":"KI 2020: Advances in Artificial Intelligence","author":"J Rabold","year":"2020","unstructured":"Rabold, J., Schwalbe, G., Schmid, U.: Expressive explanations of DNNs by\u00a0combining concept analysis with ILP. In: Schmid, U., Kl\u00fcgl, F., Wolter, D. (eds.) KI 2020. LNCS (LNAI), vol. 12325, pp. 148\u2013162. Springer, Cham (2020). https:\/\/doi.org\/10.1007\/978-3-030-58285-2_11"},{"key":"26_CR32","unstructured":"Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)"},{"key":"26_CR33","unstructured":"Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 28 (2015)"},{"key":"26_CR34","doi-asserted-by":"publisher","first-page":"41758","DOI":"10.1109\/ACCESS.2022.3167702","volume":"10","author":"Y Sawada","year":"2022","unstructured":"Sawada, Y., Nakamura, K.: Concept bottleneck model with additional unsupervised concepts. IEEE Access 10, 41758\u201341765 (2022)","journal-title":"IEEE Access"},{"key":"26_CR35","series-title":"IFIP Advances in Information and Communication Technology","doi-asserted-by":"publisher","first-page":"374","DOI":"10.1007\/978-3-030-79150-6_30","volume-title":"Artificial Intelligence Applications and Innovations","author":"G Schwalbe","year":"2021","unstructured":"Schwalbe, G.: Verification of size invariance in DNN activations using concept embeddings. In: Maglogiannis, I., Macintyre, J., Iliadis, L. (eds.) AIAI 2021. IAICT, vol. 627, pp. 374\u2013386. Springer, Cham (2021). https:\/\/doi.org\/10.1007\/978-3-030-79150-6_30"},{"key":"26_CR36","unstructured":"Schwalbe, G.: Concept embedding analysis: a review, March 2022. arXiv:2203.13909 [cs, stat]"},{"key":"26_CR37","doi-asserted-by":"publisher","unstructured":"Schwalbe, G., Finzel, B.: A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts. Data Min. Knowl. Discov. (2023). https:\/\/doi.org\/10.1007\/s10618-022-00867-8","DOI":"10.1007\/s10618-022-00867-8"},{"key":"26_CR38","unstructured":"Schwalbe, G., Wirth, C., Schmid, U.: Concept embeddings for fuzzy logic verification of deep neural networks in perception tasks. arXiv preprint arXiv:2201.00572 (2022)"},{"key":"26_CR39","unstructured":"Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)"},{"key":"26_CR40","unstructured":"Smilkov, D., Thorat, N., Kim, B., Vi\u00e9gas, F., Wattenberg, M.: Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 (2017)"},{"key":"26_CR41","unstructured":"Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 3319\u20133328. PMLR (2017)"},{"key":"26_CR42","unstructured":"Tan, M., Le, Q.: Efficientnet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105\u20136114. PMLR (2019)"},{"issue":"4","key":"26_CR43","doi-asserted-by":"publisher","first-page":"97","DOI":"10.9785\/cri-2021-220402","volume":"22","author":"M Veale","year":"2021","unstructured":"Veale, M., Borgesius, F.Z.: Demystifying the draft EU artificial intelligence act-analysing the good, the bad, and the unclear elements of the proposed approach. Comput. Law Rev. Int. 22(4), 97\u2013112 (2021)","journal-title":"Comput. Law Rev. Int."},{"issue":"3","key":"26_CR44","doi-asserted-by":"publisher","first-page":"615","DOI":"10.3390\/make3030032","volume":"3","author":"G Vilone","year":"2021","unstructured":"Vilone, G., Longo, L.: Classification of explainable artificial intelligence methods through their output formats. Mach. Learn. Knowl. Extr. 3(3), 615\u2013661 (2021)","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"26_CR45","unstructured":"Wang, D., Cui, X., Wang, Z.J.: Chain: concept-harmonized hierarchical inference interpretation of deep convolutional neural networks. arXiv preprint arXiv:2002.01660 (2020)"},{"key":"26_CR46","doi-asserted-by":"publisher","unstructured":"Wu, W., et al.: Towards global explanations of convolutional neural networks with concept attribution. In: Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 8649\u20138658 (2020). https:\/\/doi.org\/10.1109\/CVPR42600.2020.00868","DOI":"10.1109\/CVPR42600.2020.00868"},{"key":"26_CR47","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Wang, W., Zhu, S.C.: Examining CNN representations with respect to dataset bias. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp. 4464\u20134473. AAAI Press (2018)","DOI":"10.1609\/aaai.v32i1.11833"},{"key":"26_CR48","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Wu, Y.N., Zhu, S.C.: Interpretable convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8827\u20138836 (2018)","DOI":"10.1109\/CVPR.2018.00920"},{"key":"26_CR49","doi-asserted-by":"crossref","unstructured":"Zhang, R., Madumal, P., Miller, T., Ehinger, K.A., Rubinstein, B.I.: Invertible concept-based explanations for CNN models with non-negative concept activation vectors. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11682\u201311690 (2021)","DOI":"10.1609\/aaai.v35i13.17389"},{"key":"26_CR50","doi-asserted-by":"crossref","unstructured":"Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223\u20132232 (2017)","DOI":"10.1109\/ICCV.2017.244"}],"container-title":["Communications in Computer and Information Science","Explainable Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-44067-0_26","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,13]],"date-time":"2024-02-13T06:08:54Z","timestamp":1707804534000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-44067-0_26"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"ISBN":["9783031440663","9783031440670"],"references-count":50,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-44067-0_26","relation":{},"ISSN":["1865-0929","1865-0937"],"issn-type":[{"value":"1865-0929","type":"print"},{"value":"1865-0937","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023]]},"assertion":[{"value":"21 October 2023","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"xAI","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"World Conference on Explainable Artificial Intelligence","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Lisbon","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Portugal","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2023","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26 July 2023","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"28 July 2023","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"xai2023","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"http:\/\/xaiworldconference.com\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"220","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"94","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"43% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"No","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}