{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T19:14:17Z","timestamp":1776885257383,"version":"3.51.2"},"publisher-location":"Cham","reference-count":54,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783031131844","type":"print"},{"value":"9783031131851","type":"electronic"}],"license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,7]],"date-time":"2022-08-07T00:00:00Z","timestamp":1659830400000},"content-version":"vor","delay-in-days":218,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Deep Reinforcement Learning (DRL) has demonstrated its strength in developing intelligent systems. These systems shall be formally guaranteed to be trustworthy when applied to safety-critical domains, which is typically achieved by formal verification performed after training. This <jats:italic>train-then-verify<\/jats:italic> process has two limits: (i) trained systems are difficult to formally verify due to their continuous and infinite state space and inexplicable AI components (<jats:italic>i.e.<\/jats:italic>, deep neural networks), and (ii) the <jats:italic>ex post facto<\/jats:italic> detection of bugs increases both the time- and money-wise cost of training and deployment. In this paper, we propose a novel verification-in-the-loop training framework called <jats:sc>Trainify<\/jats:sc> for developing safe DRL systems driven by counterexample-guided abstraction and refinement. Specifically, <jats:sc>Trainify<\/jats:sc> trains a DRL system on a finite set of coarsely abstracted but efficiently verifiable state spaces. When verification fails, we refine the abstraction based on returned counterexamples and train again on the finer abstract states. The process is iterated until all predefined properties are verified against the trained system. We demonstrate the effectiveness of our framework on six classic control systems. The experimental results show that our framework yields more reliable DRL systems with provable guarantees without sacrificing system performance such as cumulative reward and robustness than conventional DRL approaches.<\/jats:p>","DOI":"10.1007\/978-3-031-13185-1_10","type":"book-chapter","created":{"date-parts":[[2022,8,6]],"date-time":"2022-08-06T19:29:09Z","timestamp":1659814149000},"page":"193-218","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Trainify: A CEGAR-Driven Training and\u00a0Verification Framework for\u00a0Safe Deep Reinforcement Learning"],"prefix":"10.1007","author":[{"given":"Peng","family":"Jin","sequence":"first","affiliation":[]},{"given":"Jiaxu","family":"Tian","sequence":"additional","affiliation":[]},{"given":"Dapeng","family":"Zhi","sequence":"additional","affiliation":[]},{"given":"Xuejun","family":"Wen","sequence":"additional","affiliation":[]},{"given":"Min","family":"Zhang","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,8,7]]},"reference":[{"key":"10_CR1","unstructured":"Abel, D.: A theory of abstraction in reinforcement learning. Dissertation, Brown University (2020)"},{"key":"10_CR2","doi-asserted-by":"crossref","unstructured":"Bacci, E., Giacobbe, M., Parker, D.: Verifying reinforcement learning up to infinity. In: IJCAI 2021, Montreal, Canada, pp. 2154\u20132160. ijcai.org (2021)","DOI":"10.24963\/ijcai.2021\/297"},{"key":"10_CR3","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1007\/978-3-030-57628-8_14","volume-title":"Formal Modeling and Analysis of Timed Systems","author":"E Bacci","year":"2020","unstructured":"Bacci, E., Parker, D.: Probabilistic guarantees for safe deep reinforcement learning. In: Bertrand, N., Jansen, N. (eds.) FORMATS 2020. LNCS, vol. 12288, pp. 231\u2013248. Springer, Cham (2020). https:\/\/doi.org\/10.1007\/978-3-030-57628-8_14"},{"key":"10_CR4","volume-title":"Principles of Model Checking","author":"C Baier","year":"2008","unstructured":"Baier, C., Katoen, J.P.: Principles of Model Checking. MIT Press, Cambridge (2008)"},{"key":"10_CR5","unstructured":"Brockman, G., et al.: OpenAI Gym (2016). arXiv:1606.01540"},{"key":"10_CR6","unstructured":"Casagrande, A.: pyModelChecking (2020). https:\/\/github.com\/albertocasagrande\/pyModelChecking"},{"key":"10_CR7","doi-asserted-by":"crossref","unstructured":"Cheng, R., Orosz, G., Murray, R.M., Burdick, J.W.: End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In: AAAI 2019, vol. 33, pp. 3387\u20133395. AAAI Press (2019)","DOI":"10.1609\/aaai.v33i01.33013387"},{"issue":"04","key":"10_CR8","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1142\/S012905410300190X","volume":"14","author":"E Clarke","year":"2003","unstructured":"Clarke, E., et al.: Abstraction and counterexample-guided refinement in model checking of hybrid systems. Int. J. Found. Comput. Sci. 14(04), 583\u2013604 (2003)","journal-title":"Int. J. Found. Comput. Sci."},{"key":"10_CR9","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1007\/10722167_15","volume-title":"Computer Aided Verification","author":"E Clarke","year":"2000","unstructured":"Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-guided abstraction refinement. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 154\u2013169. Springer, Heidelberg (2000). https:\/\/doi.org\/10.1007\/10722167_15"},{"key":"10_CR10","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10575-8","volume-title":"Handbook of model checking","author":"EM Clarke","year":"2018","unstructured":"Clarke, E.M., Henzinger, T.A., Veith, H., Bloem, R.: Handbook of model checking. Springer, Cham (2018). https:\/\/doi.org\/10.1007\/978-3-319-10575-8"},{"issue":"2","key":"10_CR11","doi-asserted-by":"publisher","first-page":"324","DOI":"10.1145\/234528.234740","volume":"28","author":"P Cousot","year":"1996","unstructured":"Cousot, P.: Abstract interpretation. ACM Comput. Surv. (CSUR) 28(2), 324\u2013328 (1996)","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"10_CR12","unstructured":"Du, S., Lee, J., Li, H., Wang, L., Zhai, X.: Gradient descent finds global minima of deep neural networks. In: ICML 2019, pp. 1675\u20131685. PMLR (2019)"},{"key":"10_CR13","doi-asserted-by":"crossref","unstructured":"Dutta, S., Chen, X., Sankaranarayanan, S.: Reachability analysis for neural feedback systems using regressive polynomial rule inference. In: Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control, pp. 157\u2013168 (2019)","DOI":"10.1145\/3302504.3311807"},{"issue":"1","key":"10_CR14","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1145\/4904.4999","volume":"33","author":"EA Emerson","year":"1986","unstructured":"Emerson, E.A., Halpern, J.Y.: \u201csometimes\u2019\u2019 and \u201cnot never\u2019\u2019 revisited: on branching versus linear time temporal logic. J. ACM (JACM) 33(1), 151\u2013178 (1986)","journal-title":"J. ACM (JACM)"},{"issue":"3","key":"10_CR15","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1016\/S0019-9958(84)80047-9","volume":"61","author":"EA Emerson","year":"1984","unstructured":"Emerson, E.A., Sistla, A.P.: Deciding full branching time logic. Inf. Control 61(3), 175\u2013201 (1984)","journal-title":"Inf. Control"},{"key":"10_CR16","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1007\/978-3-030-59152-6_30","volume-title":"Automated Technology for Verification and Analysis","author":"J Fan","year":"2020","unstructured":"Fan, J., Huang, C., Chen, X., Li, W., Zhu, Q.: ReachNN*: a tool for reachability analysis of neural-network controlled systems. In: Hung, D.V., Sokolsky, O. (eds.) ATVA 2020. LNCS, vol. 12302, pp. 537\u2013542. Springer, Cham (2020). https:\/\/doi.org\/10.1007\/978-3-030-59152-6_30"},{"issue":"3","key":"10_CR17","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1109\/JAS.2014.7004690","volume":"1","author":"A Faust","year":"2014","unstructured":"Faust, A., Ruymgaart, P., Salman, M., Fierro, R., Tapia, L.: Continuous action reinforcement learning for control-affine systems with unknown dynamics. IEEE\/CAA J. Automatica Sinica 1(3), 323\u2013336 (2014)","journal-title":"IEEE\/CAA J. Automatica Sinica"},{"key":"10_CR18","doi-asserted-by":"crossref","unstructured":"Fulton, N., Platzer, A.: Safe reinforcement learning via formal methods: toward safe control through proof and learning. In: AAAI 2018, pp. 6485\u20136492. AAAI Press (2018)","DOI":"10.1609\/aaai.v32i1.12107"},{"key":"10_CR19","unstructured":"Gallestey, E., Hokayem, P.: Lecture notes in nonlinear systems and control (2019)"},{"key":"10_CR20","doi-asserted-by":"crossref","unstructured":"Gilpin, L., Bau, D., Yuan, B.Z., et al.: Explaining explanations: an overview of interpretability of machine learning. In: DSAA 2018, pp. 80\u201389 (2018)","DOI":"10.1109\/DSAA.2018.00018"},{"issue":"5","key":"10_CR21","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1109\/MSPEC.2016.7459105","volume":"53","author":"L Gomes","year":"2016","unstructured":"Gomes, L.: When will Google\u2019s self-driving car really be ready? It depends on where you live and what you mean by \u201cready.\u2019\u2019. IEEE Spectr. 53(5), 13\u201314 (2016)","journal-title":"IEEE Spectr."},{"key":"10_CR22","doi-asserted-by":"crossref","unstructured":"Guttman, A.: R-trees: A dynamic index structure for spatial searching. In: SIGMOD 1984, pp. 47\u201357. ACM (1984)","DOI":"10.1145\/971697.602266"},{"key":"10_CR23","unstructured":"Hasanbeig, M., Kroening, D., Abate, A.: Towards verifiable and safe model-free reinforcement learning. In: CEUR Workshop Proceedings (2020)"},{"key":"10_CR24","doi-asserted-by":"crossref","unstructured":"Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., Meger, D.: Deep reinforcement learning that matters. In: AAAI 2018, pp. 3207\u20133214. AAAI Press (2018)","DOI":"10.1609\/aaai.v32i1.11694"},{"key":"10_CR25","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1007\/978-3-030-81685-8_11","volume-title":"Computer Aided Verification","author":"R Ivanov","year":"2021","unstructured":"Ivanov, R., Carpenter, T., Weimer, J., Alur, R., Pappas, G., Lee, I.: Verisig 2.0: verification of neural network controllers using Taylor model preconditioning. In: Silva, A., Leino, K.R.M. (eds.) CAV 2021. LNCS, vol. 12759, pp. 249\u2013262. Springer, Cham (2021). https:\/\/doi.org\/10.1007\/978-3-030-81685-8_11"},{"key":"10_CR26","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1007\/3-540-58555-9_113","volume-title":"FME \u201994: Industrial Benefit of Formal Methods","author":"D Jackson","year":"1994","unstructured":"Jackson, D.: Abstract model checking of infinite specifications. In: Naftalin, M., Denvir, T., Bertran, M. (eds.) FME 1994. LNCS, vol. 873, pp. 519\u2013531. Springer, Heidelberg (1994). https:\/\/doi.org\/10.1007\/3-540-58555-9_113"},{"issue":"3","key":"10_CR27","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1109\/87.491203","volume":"4","author":"M Jankovic","year":"1996","unstructured":"Jankovic, M., Fontaine, D., KokotoviC, P.V.: Tora example: cascade-and passivity-based control designs. IEEE Trans. Control Syst. Technol. 4(3), 292\u2013297 (1996)","journal-title":"IEEE Trans. Control Syst. Technol."},{"key":"10_CR28","doi-asserted-by":"publisher","first-page":"107","DOI":"10.29007\/9xgv","volume":"74","author":"TT Johnson","year":"2020","unstructured":"Johnson, T.T., Manzanas Lopez, D., Musau, P., et al.: Arch-comp20 category report: artificial intelligence and neural network control systems (AINNCS) for continuous and hybrid systems plants. EPiC Ser. Comput. 74, 107\u2013173 (2020)","journal-title":"EPiC Ser. Comput."},{"key":"10_CR29","doi-asserted-by":"crossref","unstructured":"Kazak, Y., Barrett, C., Katz, G., Schapira, M.: Verifying deep-RL-driven systems. In: 2019 Workshop on Network Meets AI & ML, pp. 83\u201389. ACM (2019)","DOI":"10.1145\/3341216.3342218"},{"key":"10_CR30","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"585","DOI":"10.1007\/978-3-642-22110-1_47","volume-title":"Computer Aided Verification","author":"M Kwiatkowska","year":"2011","unstructured":"Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585\u2013591. Springer, Heidelberg (2011). https:\/\/doi.org\/10.1007\/978-3-642-22110-1_47"},{"key":"10_CR31","unstructured":"Li, Y.: Deep reinforcement learning: an overview. arXiv preprint arXiv:1701.07274 (2017)"},{"key":"10_CR32","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., et al.: Continuous control with deep reinforcement learning. In: ICLR 2016. OpenReview.net (2016)"},{"key":"10_CR33","unstructured":"Lin, X., Zhu, H., Samanta, R., Jagannathan, S.: Art: abstraction refinement-guided training for provably correct neural networks. In: FMCAD, pp. 148\u2013157. AAAI Press (2020)"},{"key":"10_CR34","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing Atari with deep reinforcement learning. CoRR abs\/1312.5602 (2013)"},{"issue":"7540","key":"10_CR35","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529\u2013533 (2015)","journal-title":"Nature"},{"issue":"4","key":"10_CR36","doi-asserted-by":"publisher","first-page":"1294","DOI":"10.1109\/TCST.2015.2501351","volume":"24","author":"P Nilsson","year":"2015","unstructured":"Nilsson, P., Hussien, O., Balkan, A., et al.: Correct-by-construction adaptive cruise control: two approaches. IEEE Trans. Control Syst. Technol. 24(4), 1294\u20131307 (2015)","journal-title":"IEEE Trans. Control Syst. Technol."},{"issue":"1","key":"10_CR37","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/TIV.2016.2571067","volume":"1","author":"E Ohn-Bar","year":"2016","unstructured":"Ohn-Bar, E., Trivedi, M.M.: Looking at humans in the age of self-driving and highly automated vehicles. IEEE Trans. Intell. Veh. 1(1), 90\u2013104 (2016)","journal-title":"IEEE Trans. Intell. Veh."},{"key":"10_CR38","unstructured":"Pyeatt, L.D., Howe, A.E.: Decision tree function approximation in reinforcement learning. Technical report, ISAS 2011 (2011)"},{"key":"10_CR39","doi-asserted-by":"crossref","unstructured":"Schmidt, L.M., Kontes, G., Plinge, A., Mutschler, C.: Can you trust your autonomous car? interpretable and verifiably safe reinforcement learning. In: 2021 IEEE Intelligent Vehicles Symposium (IV), pp. 171\u2013178. IEEE (2021)","DOI":"10.1109\/IV48863.2021.9575328"},{"key":"10_CR40","unstructured":"Shalev-Shwartz, S., Shammah, S., Shashua, A.: Safe, multi-agent, reinforcement learning for autonomous driving. CoRR abs\/1610.03295 (2016). http:\/\/arxiv.org\/abs\/1610.03295"},{"key":"10_CR41","doi-asserted-by":"crossref","unstructured":"Sinclair, S., Wang, T., Jain, G., Banerjee, S., Yu, C.: Adaptive discretization for model-based reinforcement learning. In: NeurIPS 2020. vol. 31, pp. 3858\u20133871 (2020)","DOI":"10.1145\/3393691.3394176"},{"issue":"3","key":"10_CR42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3366703","volume":"3","author":"SR Sinclair","year":"2019","unstructured":"Sinclair, S.R., Banerjee, S., Yu, C.L.: Adaptive discretization for episodic reinforcement learning in metric spaces. Proc. ACM Measur. Anal. Comput. Syst. 3(3), 1\u201344 (2019)","journal-title":"Proc. ACM Measur. Anal. Comput. Syst."},{"key":"10_CR43","doi-asserted-by":"crossref","unstructured":"Singh, G., Gehr, T., P\u00fcschel, M., Vechev, M.: An abstract domain for certifying neural networks. In: POPL 2019, pp. 1\u201330. ACM (2019)","DOI":"10.1145\/3290354"},{"key":"10_CR44","unstructured":"Srinivasan, K., Eysenbach, B., Ha, S., Tan, J., Finn, C.: Learning to be safe: deep RL with a safety critic. arXiv preprint arXiv:2010.14603 (2020)"},{"key":"10_CR45","doi-asserted-by":"crossref","unstructured":"Stevia, P., Mindom, N., Nikanjam, A., Khomh, F., Mullins, J.: On assessing the safety of reinforcement learning algorithms using formal methods. arXiv preprint arXiv:2111.04865 (2021)","DOI":"10.1109\/QRS54544.2021.00037"},{"issue":"5s","key":"10_CR46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3358230","volume":"18","author":"HD Tran","year":"2019","unstructured":"Tran, H.D., Cai, F., Diego, M.L., Musau, P., Johnson, T.T., Koutsoukos, X.: Safety verification of cyber-physical systems with reinforcement learning control. ACM Trans. Emb. Comput. Syst. 18(5s), 1\u201322 (2019)","journal-title":"ACM Trans. Emb. Comput. Syst."},{"key":"10_CR47","unstructured":"Van Wesel, P., Goodloe, A.E.: Challenges in the verification of reinforcement learning algorithms. NASA STI Program (2017)"},{"key":"10_CR48","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","volume":"17","author":"P Virtanen","year":"2020","unstructured":"Virtanen, P., Gommers, R., Oliphant, T.E., et al.: SciPy 1.0: fundamental algorithms for scientific computing in python. Nat. Meth. 17, 261\u2013272 (2020)","journal-title":"Nat. Meth."},{"key":"10_CR49","unstructured":"Wang, Y., Huang, C., Wang, Z., Wang, Z., Zhu, Q.: Verification in the loop: correct-by-construction control learning with reach-avoid guarantees. arXiv preprint arXiv:2106.03245 (2021)"},{"issue":"11","key":"10_CR50","doi-asserted-by":"publisher","first-page":"5777","DOI":"10.1109\/TNNLS.2018.2808470","volume":"29","author":"W Xiang","year":"2018","unstructured":"Xiang, W., Tran, H.D., Johnson, T.T.: Output reachable set estimation and verification for multilayer neural networks. IEEE Trans. Neural Networks Learn. Syst. 29(11), 5777\u20135783 (2018)","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"10_CR51","unstructured":"Xiong, Z., Jagannathan, S.: Scalable synthesis of verified controllers in deep reinforcement learning. arXiv preprint arXiv:2104.10219 (2021)"},{"issue":"2","key":"10_CR52","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1142\/S2705078520500150","volume":"7","author":"RV Yampolskiy","year":"2020","unstructured":"Yampolskiy, R.V.: Unexplainability and incomprehensibility of AI. J. Artif. Intell. Conscious. 7(2), 277\u2013291 (2020)","journal-title":"J. Artif. Intell. Conscious."},{"key":"10_CR53","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"publisher","first-page":"467","DOI":"10.1007\/978-3-030-81685-8_22","volume-title":"Computer Aided Verification","author":"Z Yang","year":"2021","unstructured":"Yang, Z., et al.: An iterative scheme of safe reinforcement learning for nonlinear systems via barrier certificate generation. In: Silva, A., Leino, K.R.M. (eds.) CAV 2021. LNCS, vol. 12759, pp. 467\u2013490. Springer, Cham (2021). https:\/\/doi.org\/10.1007\/978-3-030-81685-8_22"},{"key":"10_CR54","doi-asserted-by":"crossref","unstructured":"Zhu, H., Xiong, Z., Magill, S., Jagannathan, S.: An inductive synthesis framework for verifiable reinforcement learning. In: PLDI 2019. pp. 686\u2013701. ACM (2019)","DOI":"10.1145\/3314221.3314638"}],"container-title":["Lecture Notes in Computer Science","Computer Aided Verification"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-13185-1_10","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T17:13:05Z","timestamp":1667495585000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-13185-1_10"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"ISBN":["9783031131844","9783031131851"],"references-count":54,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-13185-1_10","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022]]},"assertion":[{"value":"7 August 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"CAV","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Computer Aided Verification","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Haifa","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Israel","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2022","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"7 August 2022","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"10 August 2022","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"34","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"cav2022","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"http:\/\/i-cav.org\/2022\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Double-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"209","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"40","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"11","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"19% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3.9","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"9.7","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}