{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,27]],"date-time":"2025-03-27T05:45:42Z","timestamp":1743054342640,"version":"3.40.3"},"publisher-location":"Cham","reference-count":50,"publisher":"Springer Nature Switzerland","isbn-type":[{"type":"print","value":"9783031656293"},{"type":"electronic","value":"9783031656309"}],"license":[{"start":{"date-parts":[[2024,1,1]],"date-time":"2024-01-01T00:00:00Z","timestamp":1704067200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"vor","delay-in-days":206,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The behavior of neural networks (NNs) on previously unseen types of data (out-of-distribution or OOD) is typically unpredictable. This can be dangerous if the network\u2019s output is used for decision making in a safety-critical system. Hence, detecting that an input is OOD is crucial for the safe application of the NN. Verification approaches do not scale to practical NNs, making runtime monitoring more appealing for practical use. While various monitors have been suggested recently, their optimization for a given problem, as well as comparison with each other and reproduction of results, remain challenging.<\/jats:p><jats:p>We present a tool for users and developers of NN monitors. It allows for (i)\u00a0application of various types of monitors from the literature to a given input NN, (ii)\u00a0optimization of the monitor\u2019s hyperparameters, and (iii)\u00a0experimental evaluation and comparison to other approaches. Besides, it facilitates the development of new monitoring approaches. We demonstrate the tool\u2019s usability on several use cases of different types of users as well as on a case study comparing different approaches from recent literature.<\/jats:p>","DOI":"10.1007\/978-3-031-65630-9_14","type":"book-chapter","created":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T22:01:56Z","timestamp":1721858516000},"page":"265-279","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Monitizer: Automating Design and\u00a0Evaluation of\u00a0Neural Network Monitors"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4532-8344","authenticated-orcid":false,"given":"Muqsit","family":"Azeem","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0003-3314-3358","authenticated-orcid":false,"given":"Marta","family":"Grobelna","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6078-4175","authenticated-orcid":false,"given":"Sudeep","family":"Kanav","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8122-2881","authenticated-orcid":false,"given":"Jan","family":"K\u0159et\u00ednsk\u00fd","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8630-3218","authenticated-orcid":false,"given":"Stefanie","family":"Mohr","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0006-6397-3100","authenticated-orcid":false,"given":"Sabine","family":"Rieder","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,7,25]]},"reference":[{"key":"14_CR1","unstructured":"Azeem, M., Grobelna, M., Kanav, S., K\u0159et\u00ednsk\u00fd, J., Mohr, S., Rieder, S.: Monitizer: Automating design and evaluation of neural network monitors. CoRR (2024). https:\/\/arxiv.org\/abs\/2405.10350"},{"key":"14_CR2","doi-asserted-by":"publisher","unstructured":"Azeem, M., Grobelna, M., Kanav, S., K\u0159et\u00ednsk\u00fd, J., Mohr, S., Rieder, S.: Reproduction package for article \u2018monitizer: automating design and evaluation of neural network monitors. In: Proceedings of CAV 2024, Zenodo (2024). https:\/\/doi.org\/10.5281\/zenodo.10933013","DOI":"10.5281\/zenodo.10933013"},{"key":"14_CR3","unstructured":"Bai, H., Canal, G., Du, X., Kwon, J., Nowak, R.D., Li, Y.: Feed two birds with one scone: exploiting wild data for both out-of-distribution generalization and detection. In: ICML 2023. PMLR, vol.\u00a0202, pp. 1454\u20131471. PMLR (2023), https:\/\/proceedings.mlr.press\/v202\/bai23a.html"},{"key":"14_CR4","unstructured":"Bai, J., Lu, F., Zhang, K., et\u00a0al.: ONNX: Open neural network exchange (2019). https:\/\/github.com\/onnx\/onnx"},{"key":"14_CR5","doi-asserted-by":"publisher","unstructured":"Breitenstein, J., Term\u00f6hlen, J., Lipinski, D., Fingscheidt, T.: systematization of corner cases for visual perception in automated driving. In: Proceedings of IV, pp. 1257\u20131264. IEEE (2020). https:\/\/doi.org\/10.1109\/IV47402.2020.9304789","DOI":"10.1109\/IV47402.2020.9304789"},{"key":"14_CR6","doi-asserted-by":"publisher","unstructured":"Casadio, M., Komendantskaya, E., Daggitt, M.L., Kokke, W., Katz, G., Amir, G., Refaeli, I.: Neural network robustness as a verification property: a principled case study. In: Proceedings of CAV, pp. 219\u2013231. Springer (2022). https:\/\/doi.org\/10.1007\/978-3-031-13185-1_11","DOI":"10.1007\/978-3-031-13185-1_11"},{"key":"14_CR7","doi-asserted-by":"publisher","unstructured":"Cheng, C., N\u00fchrenberg, G., Yasuoka, H.: Runtime monitoring neuron activation patterns. In: Proceedings of DATE, pp. 300\u2013303. IEEE (2019). https:\/\/doi.org\/10.23919\/DATE.2019.8714971","DOI":"10.23919\/DATE.2019.8714971"},{"key":"14_CR8","doi-asserted-by":"publisher","unstructured":"Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , Vedaldi, A.: Describing textures in the wild. In: Proceedings of CVPR (2014). https:\/\/doi.org\/10.1109\/CVPR.2014.461","DOI":"10.1109\/CVPR.2014.461"},{"key":"14_CR9","doi-asserted-by":"publisher","unstructured":"Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., Ha, D.: Deep learning for classical japanese literature. CoRR (2018). https:\/\/doi.org\/10.48550\/arXiv.1812.01718","DOI":"10.48550\/arXiv.1812.01718"},{"key":"14_CR10","unstructured":"Djurisic, A., Bozanic, N., Ashok, A., Liu, R.: Extremely simple activation shaping for out-of-distribution detection. In: Proceedings of ICLR. OpenReview.net (2023). https:\/\/openreview.net\/forum?id=ndYXTEL6cZz"},{"key":"14_CR11","doi-asserted-by":"publisher","unstructured":"Ferreira, R.S., Arlat, J., Guiochet, J., Waeselynck, H.: Benchmarking safety monitors for image classifiers with machine learning. In: PRDC 2021, pp. 7\u201316. IEEE (2021). https:\/\/doi.org\/10.1109\/PRDC53464.2021.00012","DOI":"10.1109\/PRDC53464.2021.00012"},{"key":"14_CR12","unstructured":"Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proc. ICML, pp. 1321\u20131330. PMLR (2017). https:\/\/proceedings.mlr.press\/v70\/guo17a.html"},{"key":"14_CR13","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1007\/978-3-030-88494-9_14","volume-title":"Runtime Verification","author":"V Hashemi","year":"2021","unstructured":"Hashemi, V., K\u0159et\u00ednsk\u00fd, J., Mohr, S., Seferis, E.: Gaussian-based runtime detection of\u00a0out-of-distribution inputs for neural networks. In: Feng, L., Fisman, D. (eds.) RV 2021. LNCS, vol. 12974, pp. 254\u2013264. Springer, Cham (2021). https:\/\/doi.org\/10.1007\/978-3-030-88494-9_14"},{"key":"14_CR14","doi-asserted-by":"publisher","unstructured":"Hashemi, V., Kret\u00ednsk\u00fd, J., Rieder, S., Schmidt, J.: Runtime monitoring for out-of-distribution detection in object detection neural networks. In: Proc. FM. LNCS, vol. 14000, pp. 622\u2013634. Springer (2023). https:\/\/doi.org\/10.1007\/978-3-031-27481-7_36","DOI":"10.1007\/978-3-031-27481-7_36"},{"key":"14_CR15","unstructured":"Hendrycks, D., et al.: Scaling out-of-distribution detection for real-world settings. In: Proc. ICML. PMLR, vol.\u00a0162, pp. 8759\u20138773. PMLR (2022). https:\/\/proceedings.mlr.press\/v162\/hendrycks22a.html"},{"key":"14_CR16","unstructured":"Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. In: ICLR. OpenReview.net (2019). https:\/\/openreview.net\/forum?id=HJz6tiCqYm"},{"key":"14_CR17","unstructured":"Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: Proc. ICLR. OpenReview.net (2017). https:\/\/openreview.net\/forum?id=Hkg4TI9xl"},{"key":"14_CR18","doi-asserted-by":"publisher","unstructured":"Henriksen, P., Lomuscio, A.R.: Efficient neural network verification via adaptive refinement and adversarial search. In: Proceedings of ECAI. FAIA, vol.\u00a0325, pp. 2513\u20132520. IOS Press (2020). https:\/\/doi.org\/10.3233\/FAIA200385","DOI":"10.3233\/FAIA200385"},{"key":"14_CR19","doi-asserted-by":"publisher","unstructured":"Henriksson, J., et al.: Towards structured evaluation of deep neural network supervisors. In: Proceedings of AITest, pp. 27\u201334. IEEE (2019). https:\/\/doi.org\/10.1109\/AITest.2019.00-12","DOI":"10.1109\/AITest.2019.00-12"},{"key":"14_CR20","doi-asserted-by":"publisher","unstructured":"Henzinger, T.A., Lukina, A., Schilling, C.: Outside the box: abstraction-based monitoring of neural networks. In: Proceedings of ECAI, FAIA, vol.\u00a0325, pp. 2433\u20132440. IOS Press (2020). https:\/\/doi.org\/10.3233\/FAIA200375","DOI":"10.3233\/FAIA200375"},{"key":"14_CR21","doi-asserted-by":"publisher","unstructured":"Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., Igel, C.: Detection of traffic signs in real-world images: the german traffic sign detection benchmark. In: Proceedings of IJCNN. pp.\u00a01\u20138. IEEE (2013). https:\/\/doi.org\/10.1109\/IJCNN.2013.6706807","DOI":"10.1109\/IJCNN.2013.6706807"},{"key":"14_CR22","doi-asserted-by":"publisher","unstructured":"Hsu, Y., Shen, Y., Jin, H., Kira, Z.: Generalized ODIN: detecting out-of-distribution image without learning from out-of-distribution data. In: Proceedings of CVPR, pp. 10948\u201310957. IEEE\/CVF (2020). https:\/\/doi.org\/10.1109\/CVPR42600.2020.01096","DOI":"10.1109\/CVPR42600.2020.01096"},{"key":"14_CR23","unstructured":"Huang, R., Geng, A., Li, Y.: On the importance of gradients for detecting distributional shifts in the wild. In: NeurIPS, vol.\u00a034, pp. 677\u2013689 (2021), https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2021\/hash\/063e26c670d07bb7c4d30e6fc69fe056-Abstract.html"},{"issue":"1","key":"14_CR24","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/s10703-021-00363-7","volume":"60","author":"G Katz","year":"2022","unstructured":"Katz, G., Barrett, C.W., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: a calculus for reasoning about deep neural networks. FMSD 60(1), 87\u2013116 (2022). https:\/\/doi.org\/10.1007\/s10703-021-00363-7","journal-title":"FMSD"},{"key":"14_CR25","doi-asserted-by":"publisher","unstructured":"Katz, G., et al.: The Marabou framework for verification and analysis of deep neural networks. In: Proceedings of CAV. LNCS, vol. 11561, pp. 443\u2013452. Springer (2019). https:\/\/doi.org\/10.1007\/978-3-030-25540-4_26","DOI":"10.1007\/978-3-030-25540-4_26"},{"key":"14_CR26","unstructured":"Katz-Samuels, J., Nakhleh, J.B., Nowak, R.D., Li, Y.: Training OOD detectors in their natural habitats. In: Proc. ICML. PMLR, vol.\u00a0162, pp. 10848\u201310865. PMLR (2022). https:\/\/proceedings.mlr.press\/v162\/katz-samuels22a.html"},{"key":"14_CR27","doi-asserted-by":"publisher","unstructured":"Kirchheim, K., Filax, M., Ortmeier, F.: PyTorch-OOD: a library for out-of-distribution detection based on PyTorch. In: CVPR Workshops 2022, pp. 4350\u20134359. IEEE\/CVF (2022). https:\/\/doi.org\/10.1109\/CVPRW56347.2022.00481","DOI":"10.1109\/CVPRW56347.2022.00481"},{"key":"14_CR28","unstructured":"Krizhevsky, A., Hinton, G., et\u00a0al.: Learning multiple layers of features from tiny images. Tech. rep., https:\/\/www.cs.toronto.edu\/~kriz\/learning-features-2009-TR.pdf"},{"key":"14_CR29","unstructured":"LeCun, Y., Cortes, C., Burges, C.: MNIST handwritten digit database 2"},{"key":"14_CR30","unstructured":"Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: NeurIPS, vol.\u00a031, pp. 7167\u20137177 (2018). https:\/\/proceedings.neurips.cc\/paper\/2018\/hash\/abdeb6f575ac5c6676b747bca8d09cc2-Abstract.html"},{"key":"14_CR31","unstructured":"Liang, S., Li, Y., Srikant, R.: Enhancing the reliability of out-of-distribution image detection in neural networks. In: Proceedings of ICLR. OpenReview.net (2018). https:\/\/openreview.net\/forum?id=H1VGkIxRZ"},{"key":"14_CR32","unstructured":"Liu, W., Wang, X., Owens, J., Li, Y.: Energy-based out-of-distribution detection. NeurIPS 33, 21464\u201321475 (2020). https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/hash\/f5496252609c43eb8a3d147ab9b9c006-Abstract.html"},{"key":"14_CR33","doi-asserted-by":"publisher","unstructured":"Mac\u00eado, D., Ren, T.I., Zanchettin, C., Oliveira, A.L., Ludermir, T.: Entropic out-of-distribution detection. In: Proceedings of (IJCNN), pp.\u00a01\u20138. IEEE (2021). https:\/\/doi.org\/10.1109\/IJCNN52387.2021.9533899","DOI":"10.1109\/IJCNN52387.2021.9533899"},{"key":"14_CR34","doi-asserted-by":"publisher","unstructured":"M\u00fcller, M.N., Brix, C., Bak, S., Liu, C., Johnson, T.T.: The third international verification of neural networks competition (VNN-COMP 2022): Summary and results. CoRR (2022). https:\/\/doi.org\/10.48550\/arXiv.2212.10376","DOI":"10.48550\/arXiv.2212.10376"},{"key":"14_CR35","doi-asserted-by":"publisher","unstructured":"M\u00fcller, M.N., Makarchuk, G., Singh, G., P\u00fcschel, M., Vechev, M.T.: PRIMA: general and precise neural network certification via scalable convex hull approximations. PACMPL 6(POPL), 1\u201333 (2022). https:\/\/doi.org\/10.1145\/3498704","DOI":"10.1145\/3498704"},{"key":"14_CR36","unstructured":"Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning"},{"key":"14_CR37","doi-asserted-by":"publisher","unstructured":"Olber, B., Radlak, K., Popowicz, A., Szczepankiewicz, M., Chachula, K.: Detection of out-of-distribution samples using binary neuron activation patterns. In: Proceedings of CVPR, pp. 3378\u20133387. IEEE\/CVF (2023). https:\/\/doi.org\/10.1109\/CVPR52729.2023.00329","DOI":"10.1109\/CVPR52729.2023.00329"},{"key":"14_CR38","doi-asserted-by":"publisher","unstructured":"Pinggera, P., Ramos, S., Gehrig, S., Franke, U., Rother, C., Mester, R.: Lost and found: detecting small road hazards for self-driving vehicles. In: Proceedings of IROS, pp. 1099\u20131106. IEEE (2016). https:\/\/doi.org\/10.1109\/IROS.2016.7759186","DOI":"10.1109\/IROS.2016.7759186"},{"key":"14_CR39","doi-asserted-by":"publisher","unstructured":"Ren, J., Fort, S., Liu, J., Roy, A.G., Padhy, S., Lakshminarayanan, B.: A simple fix to mahalanobis distance for improving near-ood detection. CoRR (2021). https:\/\/doi.org\/10.48550\/arXiv.2106.09022","DOI":"10.48550\/arXiv.2106.09022"},{"issue":"3","key":"14_CR40","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2015","unstructured":"Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211\u2013252 (2015). https:\/\/doi.org\/10.1007\/s11263-015-0816-y","journal-title":"Int. J. Comput. Vis."},{"key":"14_CR41","unstructured":"Sun, Y., Guo, C., Li, Y.: ReAct: Out-of-distribution detection with rectified activations. In: NeurIPS. vol.\u00a034, pp. 144\u2013157 (2021). https:\/\/proceedings.neurips.cc\/paper\/2021\/hash\/01894d6f048493d2cacde3c579c315a3-Abstract.html"},{"key":"14_CR42","doi-asserted-by":"publisher","unstructured":"Sun, Y., Li, Y.: DICE: Leveraging sparsification for out-of-distribution detection. In: Proceeding of ECCV. LNCS, vol. 13684, pp. 691\u2013708. Springer (2022). https:\/\/doi.org\/10.1007\/978-3-031-20053-3_40","DOI":"10.1007\/978-3-031-20053-3_40"},{"key":"14_CR43","unstructured":"Sun, Y., Ming, Y., Zhu, X., Li, Y.: Out-of-distribution detection with deep nearest neighbors. In: Proceedings of ICML, pp. 20827\u201320840. PMLR (2022). https:\/\/proceedings.mlr.press\/v162\/sun22d"},{"key":"14_CR44","doi-asserted-by":"publisher","unstructured":"Tajwar, F., Kumar, A., Xie, S.M., Liang, P.: No true state-of-the-art? OOD detection methods are inconsistent across datasets. CoRR (2021). https:\/\/doi.org\/10.48550\/arXiv.2109.05554","DOI":"10.48550\/arXiv.2109.05554"},{"key":"14_CR45","doi-asserted-by":"publisher","unstructured":"Wang, H., Li, Z., Feng, L., Zhang, W.: ViM: Out-of-distribution with virtual-logit matching. In: Proceedings of CVPR, pp. 4921\u20134930. IEEE\/CVF (2022). https:\/\/doi.org\/10.1109\/CVPR52688.2022.00487","DOI":"10.1109\/CVPR52688.2022.00487"},{"key":"14_CR46","unstructured":"Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms"},{"key":"14_CR47","unstructured":"Yang, J., et al.: OpenOOD: benchmarking generalized out-of-distribution detection. In: NeurIPS (2022). http:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/d201587e3a84fc4761eadc743e9b3f35-Abstract-Datasets_and_Benchmarks.html"},{"key":"14_CR48","doi-asserted-by":"publisher","unstructured":"Zhang, J., et al.: OpenOOD v1.5: Enhanced benchmark for out-of-distribution detection. CoRR (2023). https:\/\/doi.org\/10.48550\/arXiv.2306.09301","DOI":"10.48550\/arXiv.2306.09301"},{"key":"14_CR49","unstructured":"Zhang, J., et\u00a0al.: Out-of-distribution detection based on in-distribution data patterns memorization with modern hopfield energy. In: Proceedings of ICLR (2022). https:\/\/openreview.net\/forum?id=KkazG4lgKL"},{"key":"14_CR50","doi-asserted-by":"publisher","unstructured":"Zhang, Z., Xiang, X.: Decoupling maxlogit for out-of-distribution detection. In: Proceedings of CVPR, pp. 3388\u20133397. IEEE\/CVF (2023). https:\/\/doi.org\/10.1109\/CVPR52729.2023.00330","DOI":"10.1109\/CVPR52729.2023.00330"}],"container-title":["Lecture Notes in Computer Science","Computer Aided Verification"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-65630-9_14","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T22:04:12Z","timestamp":1721858652000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-65630-9_14"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"ISBN":["9783031656293","9783031656309"],"references-count":50,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-65630-9_14","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2024]]},"assertion":[{"value":"25 July 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"CAV","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Computer Aided Verification","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Montreal, QC","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Canada","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"24 July 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"27 July 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"36","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"cav2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"http:\/\/i-cav.org\/2024\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}