{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,26]],"date-time":"2025-11-26T05:03:59Z","timestamp":1764133439584,"version":"3.40.3"},"publisher-location":"Cham","reference-count":63,"publisher":"Springer International Publishing","isbn-type":[{"type":"print","value":"9783030714994"},{"type":"electronic","value":"9783030715007"}],"license":[{"start":{"date-parts":[[2021,1,1]],"date-time":"2021-01-01T00:00:00Z","timestamp":1609459200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,20]],"date-time":"2021-03-20T00:00:00Z","timestamp":1616198400000},"content-version":"vor","delay-in-days":78,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We present a synthetic benchmarking framework that targets the systematic evaluation of RV tools for message-based concurrent systems. Our tool can emulate various load profiles via configuration. It provides a multi-faceted view of measurements that is conducive to a comprehensive assessment of the overhead induced by runtime monitoring. The tool is able to generate significant loads to reveal edge case behaviour that may only emerge when the monitoring system is pushed to its limit. We evaluate our framework in two ways. First, we conduct sanity checks to assess the precision of the measurement mechanisms used, the repeatability of the results obtained, and the veracity of the behaviour emulated by our synthetic benchmark. We then showcase the utility of the features offered by our tool in a two-part RV case study.<\/jats:p>","DOI":"10.1007\/978-3-030-71500-7_1","type":"book-chapter","created":{"date-parts":[[2021,3,19]],"date-time":"2021-03-19T13:12:14Z","timestamp":1616159534000},"page":"3-23","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["On Benchmarking for Concurrent Runtime Verification"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2197-3018","authenticated-orcid":false,"given":"Luca","family":"Aceto","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2448-5394","authenticated-orcid":false,"given":"Duncan Paul","family":"Attard","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3829-7391","authenticated-orcid":false,"given":"Adrian","family":"Francalanza","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8362-3075","authenticated-orcid":false,"given":"Anna","family":"Ing\u00f3lfsd\u00f3ttir","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,3,20]]},"reference":[{"key":"1_CR1","doi-asserted-by":"crossref","unstructured":"Aceto, L., Achilleos, A., Francalanza, A., Ing\u00f3lfsd\u00f3ttir, A., Kjartansson, S.\u00d6.: Determinizing Monitors for HML with Recursion. JLAMP 111, 100515 (2020)","DOI":"10.1016\/j.jlamp.2019.100515"},{"key":"1_CR2","doi-asserted-by":"crossref","unstructured":"Agha, G., Mason, I.A., Smith, S.F., Talcott, C.L.: A Foundation for Actor Computation. JFP 7(1), 1\u201372 (1997)","DOI":"10.1017\/S095679689700261X"},{"key":"1_CR3","unstructured":"Apache Software Foundtation: Jmeter (2020), https:\/\/jmeter.apache.org"},{"key":"1_CR4","unstructured":"Attard, D.P.: detectEr (2020), https:\/\/github.com\/duncanatt\/detecter-inline"},{"key":"1_CR5","unstructured":"Attard, D.P., Cassar, I., Francalanza, A., Aceto, L., Ing\u00f3lfsd\u00f3ttir, A.: Introduction to Runtime Verification. In: Behavioural Types: from Theory to Tools, pp. 49\u201376. Automation, Control and Robotics, River (2017)"},{"key":"1_CR6","doi-asserted-by":"crossref","unstructured":"Attard, D.P., Francalanza, A.: A Monitoring Tool for a Branching-Time Logic. In: RV. LNCS, vol. 10012, pp. 473\u2013481 (2016)","DOI":"10.1007\/978-3-319-46982-9_31"},{"key":"1_CR7","doi-asserted-by":"crossref","unstructured":"Attard, D.P., Francalanza, A.: Trace Partitioning and Local Monitoring for Asynchronous Components. In: SEFM. LNCS, vol. 10469, pp. 219\u2013235 (2017)","DOI":"10.1007\/978-3-319-66197-1_14"},{"key":"1_CR8","unstructured":"Bartocci, E., Falcone, Y., Bonakdarpour, B., Colombo, C., Decker, N., Havelund, K., Joshi, Y., Klaedtke, F., Milewicz, R., Reger, G., Rosu, G., Signoles, J., Thoma, D., Zalinescu, E., Zhang, Y.: First International Competition on Runtime Verification: Rules, Benchmarks, Tools, and Final Results of CRV 2014. Int. J. Softw. Tools Technol. Transf. 21(1), 31\u201370 (2019)"},{"key":"1_CR9","doi-asserted-by":"crossref","unstructured":"Bartocci, E., Falcone, Y., Francalanza, A., Reger, G.: Introduction to Runtime Verification. In: Lectures on RV, LNCS, vol. 10457, pp. 1\u201333. Springer (2018)","DOI":"10.1007\/978-3-319-75632-5_1"},{"key":"1_CR10","doi-asserted-by":"crossref","unstructured":"Berkovich, S., Bonakdarpour, B., Fischmeister, S.: Runtime Verification with Minimal Intrusion through Parallelism. FMSD 46(3), 317\u2013348 (2015)","DOI":"10.1007\/s10703-015-0226-3"},{"key":"1_CR11","doi-asserted-by":"crossref","unstructured":"Blackburn, S.M., Garner, R., Hoffmann, C., Khan, A.M., McKinley, K.S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S.Z., Hirzel, M., Hosking, A.L., Jump, M., Lee, H.B., Moss, J.E.B., Phansalkar, A., Stefanovic, D., VanDrunen, T., von Dincklage, D., Wiedermann, B.: The DaCapo Benchmarks: Java Benchmarking Development and Analysis. In: OOPSLA. pp. 169\u2013190 (2006)","DOI":"10.1145\/1167515.1167488"},{"key":"1_CR12","doi-asserted-by":"crossref","unstructured":"Blessing, S., Fernandez-Reyes, K., Yang, A.M., Drossopoulou, S., Wrigstad, T.: Run, Actor, Run: Towards Cross-Actor Language Benchmarking. In: AGERE!@SPLASH. pp. 41\u201350 (2019)","DOI":"10.1145\/3358499.3361224"},{"key":"1_CR13","unstructured":"Bodden, E., Hendren, L.J., Lam, P., Lhot\u00e1k, O., Naeem, N.A.: Collaborative Runtime Verification with Tracematches. J. Log. Comput. 20(3), 707\u2013723 (2010)"},{"key":"1_CR14","doi-asserted-by":"crossref","unstructured":"Bonakdarpour, B., Finkbeiner, B.: The Complexity of Monitoring Hyperproperties. In: CSF. pp. 162\u2013174 (2018)","DOI":"10.1109\/CSF.2018.00019"},{"key":"1_CR15","doi-asserted-by":"crossref","unstructured":"Buyya, R., Broberg, J., Goscinski, A.M.: Cloud Computing: Principles and Paradigms. Wiley-Blackwell (2011)","DOI":"10.1002\/9780470940105"},{"key":"1_CR16","unstructured":"Cesarini, F., Thompson, S.: Erlang Programming: A Concurrent Approach to Software Development. O\u2019Reilly Media (2009)"},{"key":"1_CR17","unstructured":"Chappell, D.: Enterprise Service Bus: Theory in Practice. O\u2019Reilly Media (2004)"},{"key":"1_CR18","doi-asserted-by":"crossref","unstructured":"Chen, F., Rosu, G.: Mop: An Efficient and Generic Runtime Verification Framework. In: OOPSLA. pp. 569\u2013588 (2007)","DOI":"10.1145\/1297105.1297069"},{"key":"1_CR19","doi-asserted-by":"crossref","unstructured":"Chen, F., Rosu, G.: Parametric Trace Slicing and Monitoring. In: TACAS. LNCS, vol.\u00a05505, pp. 246\u2013261 (2009)","DOI":"10.1007\/978-3-642-00768-2_23"},{"key":"1_CR20","unstructured":"Ciemiewicz, D.M.: What Do You mean? - Revisiting Statistics for Web Response Time Measurements. In: CMG. pp. 385\u2013396 (2001)"},{"key":"1_CR21","doi-asserted-by":"crossref","unstructured":"Cornejo, O., Briola, D., Micucci, D., Mariani, L.: In the Field Monitoring of Interactive Application. In: ICSE-NIER. pp. 55\u201358 (2017)","DOI":"10.1109\/ICSE-NIER.2017.19"},{"key":"1_CR22","doi-asserted-by":"crossref","unstructured":"Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM 51(1), 107\u2013113 (2008)","DOI":"10.1145\/1327452.1327492"},{"key":"1_CR23","doi-asserted-by":"crossref","unstructured":"El-Hokayem, A., Falcone, Y.: Monitoring Decentralized Specifications. In: ISSTA. pp. 125\u2013135 (2017)","DOI":"10.1145\/3092703.3092723"},{"key":"1_CR24","doi-asserted-by":"crossref","unstructured":"Francalanza, A.: A Theory of Monitors (Extended Abstract). In: FoSSaCS. LNCS, vol.\u00a09634, pp. 145\u2013161 (2016)","DOI":"10.1007\/978-3-662-49630-5_9"},{"key":"1_CR25","doi-asserted-by":"crossref","unstructured":"Francalanza, A., Aceto, L., Achilleos, A., Attard, D.P., Cassar, I., Della Monica, D., Ing\u00f3lfsd\u00f3ttir, A.: A Foundation for Runtime Monitoring. In: RV. LNCS, vol. 10548, pp. 8\u201329 (2017)","DOI":"10.1007\/978-3-319-67531-2_2"},{"key":"1_CR26","doi-asserted-by":"crossref","unstructured":"Francalanza, A., Aceto, L., Ing\u00f3lfsd\u00f3ttir, A.: Monitorability for the Hennessy-Milner Logic with Recursion. FMSD 51(1), 87\u2013116 (2017)","DOI":"10.1007\/s10703-017-0273-z"},{"key":"1_CR27","doi-asserted-by":"crossref","unstructured":"Francalanza, A., P\u00e9rez, J.A., S\u00e1nchez, C.: Runtime Verification for Decentralised and Distributed Systems. In: Lectures on RV, LNCS, vol. 10457, pp. 176\u2013210. Springer (2018)","DOI":"10.1007\/978-3-319-75632-5_6"},{"key":"1_CR28","doi-asserted-by":"crossref","unstructured":"Francalanza, A., Xuereb, J.: On Implementing Symbolic Controllability. In: COORDINATION. LNCS, vol. 12134, pp. 350\u2013369 (2020)","DOI":"10.1007\/978-3-030-50029-0_22"},{"key":"1_CR29","doi-asserted-by":"crossref","unstructured":"Ghosh, S.: Distributed Systems: An Algorithmic Approach. CRC (2014)","DOI":"10.1201\/b17224"},{"key":"1_CR30","unstructured":"Gray, J.: The Benchmark Handbook for Database and Transaction Processing Systems. Morgan Kaufmann (1993)"},{"key":"1_CR31","doi-asserted-by":"crossref","unstructured":"Grove, D.A., Coddington, P.D.: Analytical Models of Probability Distributions for MPI Point-to-Point Communication Times on Distributed Memory Parallel Computers. In: ICA3PP. LNCS, vol.\u00a03719, pp. 406\u2013415 (2005)","DOI":"10.1007\/11564621_47"},{"key":"1_CR32","doi-asserted-by":"crossref","unstructured":"Harman, M., O\u2019Hearn, P.W.: From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis. In: SCAM. pp. 1\u201323 (2018)","DOI":"10.1109\/SCAM.2018.00009"},{"key":"1_CR33","unstructured":"Hoguin, L.: Cowboy (2020), https:\/\/ninenines.eu"},{"key":"1_CR34","unstructured":"Hoguin, L.: Ranch (2020), https:\/\/ninenines.eu"},{"key":"1_CR35","doi-asserted-by":"crossref","unstructured":"Imam, S.M., Sarkar, V.: Savina - An Actor Benchmark Suite: Enabling Empirical Evaluation of Actor Libraries. In: AGERE!@SPLASH. pp. 67\u201380 (2014)","DOI":"10.1145\/2687357.2687368"},{"key":"1_CR36","doi-asserted-by":"crossref","unstructured":"Jin, D., Meredith, P.O., Lee, C., Rosu, G.: JavaMOP: Efficient Parametric Runtime Monitoring Framework. In: ICSE. pp. 1427\u20131430 (2012)","DOI":"10.1109\/ICSE.2012.6227231"},{"key":"1_CR37","unstructured":"Kayser, B.: What is the expected distribution of website response times? (2017, last accessed, 19th Jan 2021), https:\/\/blog.newrelic.com\/engineering\/expected-distributions-website-response-times"},{"key":"1_CR38","doi-asserted-by":"crossref","unstructured":"Kim, M., Viswanathan, M., Kannan, S., Lee, I., Sokolsky, O.: Java-mac: A Run-Time Assurance Approach for Java Programs. FMSD 24(2), 129\u2013155 (2004)","DOI":"10.1023\/B:FORM.0000017719.43755.7c"},{"key":"1_CR39","unstructured":"Kshemkalyani, A.D.: Distributed Computing: Principles, Algorithms, and Systems. Cambridge University Press (2011)"},{"key":"1_CR40","doi-asserted-by":"crossref","unstructured":"Kuhtz, L., Finkbeiner, B.: LTL Path Checking is Efficiently Parallelizable. In: ICALP (2). LNCS, vol.\u00a05556, pp. 235\u2013246 (2009)","DOI":"10.1007\/978-3-642-02930-1_20"},{"key":"1_CR41","doi-asserted-by":"crossref","unstructured":"Larsen, K.G.: Proof Systems for Satisfiability in Hennessy-Milner Logic with Recursion. TCS 72(2&3), 265\u2013288 (1990)","DOI":"10.1016\/0304-3975(90)90038-J"},{"key":"1_CR42","unstructured":"Lightbend: Play framework (2020), https:\/\/www.playframework.com"},{"key":"1_CR43","doi-asserted-by":"crossref","unstructured":"Liu, Z., Niclausse, N., Jalpa-Villanueva, C.: Traffic Model and Performance Evaluation of Web Servers. Perform. Evaluation 46(2-3), 77\u2013100 (2001)","DOI":"10.1016\/S0166-5316(01)00046-3"},{"key":"1_CR44","unstructured":"Matthes, E.: Python Crash Course: A Hands-On, Project-Based Introduction to Programming. No Starch Press (2019)"},{"key":"1_CR45","doi-asserted-by":"crossref","unstructured":"Meredith, P.O., Jin, D., Griffith, D., Chen, F., Rosu, G.: An Overview of the MOP Runtime Verification Framework. STTT 14(3), 249\u2013289 (2012)","DOI":"10.1007\/s10009-011-0198-6"},{"key":"1_CR46","doi-asserted-by":"crossref","unstructured":"Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley (2011)","DOI":"10.1002\/9781119202486"},{"key":"1_CR47","doi-asserted-by":"crossref","unstructured":"Navabpour, S., Joshi, Y., Wu, C.W.W., Berkovich, S., Medhat, R., Bonakdarpour, B., Fischmeister, S.: RiTHM: A Tool for Enabling Time-Triggered Runtime Verification for C Programs. In: ESEC\/SIGSOFT FSE. pp. 603\u2013606. ACM (2013)","DOI":"10.1145\/2491411.2494596"},{"key":"1_CR48","doi-asserted-by":"crossref","unstructured":"Neykova, R., Yoshida, N.: Let it Recover: Multiparty Protocol-Induced Recovery. In: CC. pp. 98\u2013108 (2017)","DOI":"10.1145\/3033019.3033031"},{"key":"1_CR49","unstructured":"Niclausse, N.: Tsung (2017), http:\/\/tsung.erlang-projects.org"},{"key":"1_CR50","doi-asserted-by":"crossref","unstructured":"Nielsen, J.: Usability Engineering. Morgan Kaufmann (1993)","DOI":"10.1016\/B978-0-08-052029-2.50009-7"},{"key":"1_CR51","unstructured":"Odersky, M., Spoon, L., Venners, B.: Programming in Scala. Artima Inc. (2020)"},{"key":"1_CR52","doi-asserted-by":"crossref","unstructured":"Prokopec, A., Ros\u00e0, A., Leopoldseder, D., Duboscq, G., Tuma, P., Studener, M., Bulej, L., Zheng, Y., Villaz\u00f3n, A., Simon, D., W\u00fcrthinger, T., Binder, W.: Renaissance: Benchmarking Suite for Parallel Applications on the JVM. In: PLDI. pp. 31\u201347 (2019)","DOI":"10.1145\/3325986"},{"key":"1_CR53","unstructured":"Quick, K.: Thespian (2020), http:\/\/thespianpy.com"},{"key":"1_CR54","doi-asserted-by":"crossref","unstructured":"Reger, G., Cruz, H.C., Rydeheard, D.E.: MarQ: Monitoring at Runtime with QEA. In: TACAS. LNCS, vol.\u00a09035, pp. 596\u2013610 (2015)","DOI":"10.1007\/978-3-662-46681-0_55"},{"key":"1_CR55","unstructured":"Roestenburg, R., Bakker, R., Williams, R.: Akka in Action. Manning (2015)"},{"key":"1_CR56","doi-asserted-by":"crossref","unstructured":"Rossi, R.J.: Mathematical Statistics: An Introduction to Likelihood Based Inference. Wiley (2018)","DOI":"10.1002\/9781118771075"},{"key":"1_CR57","doi-asserted-by":"crossref","unstructured":"Scheffel, T., Schmitz, M.: Three-Valued Asynchronous Distributed Runtime Verification. In: MEMOCODE. pp. 52\u201361 (2014)","DOI":"10.1109\/MEMCOD.2014.6961843"},{"key":"1_CR58","unstructured":"Seow, S.C.: Designing and Engineering Time: The Psychology of Time Perception in Software. Addison-Wesley (2008)"},{"key":"1_CR59","doi-asserted-by":"crossref","unstructured":"Sewe, A., Mezini, M., Sarimbekov, A., Binder, W.: DaCapo con Scala: design and analysis of a Scala benchmark suite for the JVM. In: OOPSLA. pp. 657\u2013676 (2011)","DOI":"10.1145\/2076021.2048118"},{"key":"1_CR60","unstructured":"SPEC: SPECjvm2008 (2008), https:\/\/www.spec.org\/jvm2008"},{"key":"1_CR61","doi-asserted-by":"crossref","unstructured":"Tarkoma, S.: Overlay Networks: Toward Information Networking. Auerbach (2010)","DOI":"10.1201\/9781439813737"},{"key":"1_CR62","doi-asserted-by":"crossref","unstructured":"Welford, B.P.: Note on a Method for Calculating Corrected Sums of Squares and Products. Technometrics 4(3), 419\u2013420 (1962)","DOI":"10.1080\/00401706.1962.10490022"},{"key":"1_CR63","unstructured":"White, T.: Hadoop: The Definitive Guide: Storage and Analysis at Internet Scale. O\u2019Reilly Media (2015)"}],"container-title":["Lecture Notes in Computer Science","Fundamental Approaches to Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-030-71500-7_1","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,20]],"date-time":"2021-03-20T00:12:30Z","timestamp":1616199150000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/978-3-030-71500-7_1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"ISBN":["9783030714994","9783030715007"],"references-count":63,"URL":"https:\/\/doi.org\/10.1007\/978-3-030-71500-7_1","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2021]]},"assertion":[{"value":"20 March 2021","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"FASE","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Fundamental Approaches to Software Engineering","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Luxembourg City","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Luxembourg","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2021","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"27 March 2021","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1 April 2021","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"24","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"fase2021","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/etaps.org\/2021\/fase","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Double-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"52","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"16","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"31% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"5,5","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"The conference changed to an online format due to the COVID-19 pandemic.","order":10,"name":"additional_info_on_review_process","label":"Additional Info on Review Process","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}