{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,25]],"date-time":"2025-11-25T06:57:30Z","timestamp":1764053850461,"version":"3.40.3"},"publisher-location":"Cham","reference-count":55,"publisher":"Springer Nature Switzerland","isbn-type":[{"type":"print","value":"9783031308253"},{"type":"electronic","value":"9783031308260"}],"license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,20]],"date-time":"2023-04-20T00:00:00Z","timestamp":1681948800000},"content-version":"vor","delay-in-days":109,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Player eXperience (PX) testing has attracted attention in the game industry as video games become more complex and widespread. Understanding players\u2019 desires and their experience are key elements to guarantee the success of a game in the highly competitive market. Although a number of techniques have been introduced to measure the emotional aspect of the experience, automated testing of player experience still needs to be explored. This paper presents a framework for automated player experience testing by formulating emotion patterns\u2019 requirements and utilizing a computational model of players\u2019 emotions developed based on a psychological theory of emotions along with a model-based testing approach for test suite generation. We evaluate the strength of our framework by performing mutation test. The paper also evaluates the performance of a search-based generated test suite and LTL model checking-based test suite in revealing various variations of temporal and spatial emotion patterns. Results show the contribution of both algorithms in generating complementary test cases for revealing various emotions in different locations of a game level.<\/jats:p>","DOI":"10.1007\/978-3-031-30826-0_9","type":"book-chapter","created":{"date-parts":[[2023,4,19]],"date-time":"2023-04-19T18:02:59Z","timestamp":1681927379000},"page":"151-172","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Model-based Player Experience Testing with Emotion Pattern Verification"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7135-5605","authenticated-orcid":false,"given":"Saba Gholizadeh","family":"Ansari","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3421-4635","authenticated-orcid":false,"given":"I. S. W. B.","family":"Prasetya","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Davide","family":"Prandi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fitsum Meshesha","family":"Kifetew","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mehdi","family":"Dastani","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Frank","family":"Dignum","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gabriele","family":"Keller","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,4,20]]},"reference":[{"key":"9_CR1","doi-asserted-by":"crossref","unstructured":"Agarwal, A., Meyer, A.: Beyond usability: evaluating emotional response as an integral part of the user experience. In: CHI\u201909 Extended Abstracts on Human Factors in Computing Systems, pp. 2919\u20132930. ACM New York, NY, USA (2009)","DOI":"10.1145\/1520340.1520420"},{"key":"9_CR2","doi-asserted-by":"crossref","unstructured":"Alagar, V., Periyasamy, K.: Extended finite state machine. In: Specification of software systems, pp. 105\u2013128. Springer (2011)","DOI":"10.1007\/978-0-85729-277-3_7"},{"key":"9_CR3","doi-asserted-by":"crossref","unstructured":"Alves, R., Valente, P., Nunes, N.J.: The state of user experience evaluation practice. In: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. pp. 93\u2013102 (2014)","DOI":"10.1145\/2639189.2641208"},{"key":"9_CR4","doi-asserted-by":"crossref","unstructured":"Ammann, P.E., Black, P.E., Majurski, W.: Using model checking to generate tests from specifications. In: Proceedings second international conference on formal engineering methods (Cat. No. 98EX241). pp. 46\u201354. IEEE (1998)","DOI":"10.6028\/NIST.IR.6166"},{"key":"9_CR5","doi-asserted-by":"crossref","unstructured":"Anderson, J.R.: Arguments concerning representations for mental imagery. Psychological review 85(4), \u00a0249 (1978)","DOI":"10.1037\/0033-295X.85.4.249"},{"key":"9_CR6","doi-asserted-by":"crossref","unstructured":"Ansari, S.G.: Toward automated assessment of user experience in extended reality. In: 2020 IEEE 13th international conference on software testing, validation and verification (ICST). pp. 430\u2013432. IEEE (2020)","DOI":"10.1109\/ICST46399.2020.00056"},{"key":"9_CR7","doi-asserted-by":"crossref","unstructured":"Ariyurek, S., Betin-Can, A., Surer, E.: Automated video game testing using synthetic and humanlike agents. IEEE Transactions on Games 13(1), 50\u201367 (2019)","DOI":"10.1109\/TG.2019.2947597"},{"key":"9_CR8","unstructured":"Baier, C., Katoen, J.P.: Principles of model checking. MIT press (2008)"},{"key":"9_CR9","unstructured":"Bartneck, C.: Integrating the occ model of emotions in embodied characters. In: Proceeding of the Workshop on Virtual Conversational Characters: Applications, Methods, and Research Challenges (2002)"},{"key":"9_CR10","unstructured":"Bevan, N.: What is the difference between the purpose of usability and user experience evaluation methods. In: Proceedings of the Workshop UXEM. vol.\u00a09, pp.\u00a01\u20134. Citeseer (2009)"},{"key":"9_CR11","unstructured":"Callahan, J., Schneider, F., Easterbrook, S., et\u00a0al.: Automated software testing using model-checking. In: Proceedings 1996 SPIN workshop. vol.\u00a0353. Citeseer (1996)"},{"key":"9_CR12","doi-asserted-by":"crossref","unstructured":"Demeure, V., Niewiadomski, R., Pelachaud, C.: How is believability of a virtual agent related to warmth, competence, personification, and embodiment? Presence 20(5), 431\u2013448 (2011)","DOI":"10.1162\/PRES_a_00065"},{"key":"9_CR13","doi-asserted-by":"crossref","unstructured":"Desmet, P., Hekkert, P.: Framework of product experience. International journal of design 1(1) (2007)","DOI":"10.1016\/B978-008045089-6.50003-4"},{"key":"9_CR14","doi-asserted-by":"crossref","unstructured":"Drachen, A., Mirza-Babaei, P., Nacke, L.E.: Games user research. Oxford University Press (2018)","DOI":"10.1093\/oso\/9780198794844.001.0001"},{"key":"9_CR15","unstructured":"Elliott, C.D.: The affective reasoner: a process model of emotions in a multiagent system. Ph.D. thesis, Northwestern University (1992)"},{"key":"9_CR16","doi-asserted-by":"crossref","unstructured":"Ellsworth, P.C., Smith, C.A.: From appraisal to emotion: Differences among unpleasant feelings. Motivation and emotion 12(3), 271\u2013302 (1988)","DOI":"10.1007\/BF00993115"},{"key":"9_CR17","doi-asserted-by":"crossref","unstructured":"Fang, X., Chan, S., Brzezinski, J., Nair, C.: Development of an instrument to measure enjoyment of computer game play. INTL. Journal of human\u2013computer interaction 26(9), 868\u2013886 (2010)","DOI":"10.1080\/10447318.2010.496337"},{"key":"9_CR18","doi-asserted-by":"crossref","unstructured":"Ferdous, R., Kifetew, F., Prandi, D., Prasetya, I., Shirzadehhajimahmood, S., Susi, A.: Search-based automated play testing of computer games: A model-based approach. In: International Symposium on Search Based Software Engineering. pp. 56\u201371. Springer (2021)","DOI":"10.1007\/978-3-030-88106-1_5"},{"key":"9_CR19","doi-asserted-by":"crossref","unstructured":"Fraser, G., Arcuri, A.: Evosuite: automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. pp. 416\u2013419 (2011)","DOI":"10.1145\/2025113.2025179"},{"key":"9_CR20","doi-asserted-by":"crossref","unstructured":"Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. In: Software Engineering\u2014ESEC\/FSE\u201999. pp. 146\u2013162. Springer (1999)","DOI":"10.1007\/3-540-48166-4_10"},{"key":"9_CR21","doi-asserted-by":"crossref","unstructured":"Gholizadeh\u00a0Ansari, S., Prasetya, I.S.W.B., Dastani, M., Dignum, F., Keller, G.: An appraisal transition system for event-driven emotions in agent-based player experience testing. In: Engineering Multi-Agent Systems: 9th International Workshop, EMAS 2021, Virtual Event, May 3\u20134, 2021, Revised Selected Papers. pp. 156\u2013174. Springer Nature (2021)","DOI":"10.1007\/978-3-030-97457-2_9"},{"key":"9_CR22","doi-asserted-by":"crossref","unstructured":"Glover, F.: Tabu search\u2014part i. ORSA Journal on computing 1(3), 190\u2013206 (1989)","DOI":"10.1287\/ijoc.1.3.190"},{"key":"9_CR23","unstructured":"Goldberg, D.E.: Genetic algorithms. Pearson Education India (2006)"},{"key":"9_CR24","doi-asserted-by":"crossref","unstructured":"Gordillo, C., Bergdahl, J., Tollmar, K., Gissl\u00e9n, L.: Improving playtesting coverage via curiosity driven reinforcement learning agents. arXiv preprint arXiv:2103.13798 (2021)","DOI":"10.1109\/CoG52621.2021.9619048"},{"key":"9_CR25","doi-asserted-by":"crossref","unstructured":"Guckelsberger, C., Salge, C., Gow, J., Cairns, P.: Predicting player experience without the player. an exploratory study. In: Proceedings of the Annual Symposium on Computer-Human Interaction in Play. pp. 305\u2013315 (2017)","DOI":"10.1145\/3116595.3116631"},{"key":"9_CR26","doi-asserted-by":"crossref","unstructured":"Harman, M., Jones, B.F.: Search-based software engineering. Information and software Technology 43(14), 833\u2013839 (2001)","DOI":"10.1016\/S0950-5849(01)00189-6"},{"key":"9_CR27","doi-asserted-by":"crossref","unstructured":"Herzig, A., Lorini, E., Perrussel, L., Xiao, Z.: BDI logics for BDI architectures: old problems, new perspectives. KI-K\u00fcnstliche Intelligenz 31(1) (2017)","DOI":"10.1007\/s13218-016-0457-5"},{"key":"9_CR28","doi-asserted-by":"crossref","unstructured":"Holmg\u00e5rd, C., Green, M.C., Liapis, A., Togelius, J.: Automated playtesting with procedural personas through mcts with evolved heuristics. IEEE Transactions on Games 11(4), 352\u2013362 (2018)","DOI":"10.1109\/TG.2018.2808198"},{"key":"9_CR29","doi-asserted-by":"crossref","unstructured":"Holzmann, G.J.: The model checker spin. IEEE Transactions on software engineering 23(5), 279\u2013295 (1997)","DOI":"10.1109\/32.588521"},{"key":"9_CR30","doi-asserted-by":"crossref","unstructured":"Iftikhar, S., Iqbal, M.Z., Khan, M.U., Mahmood, W.: An automated model based testing approach for platform games. In: 2015 ACM\/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS). pp. 426\u2013435. IEEE (2015)","DOI":"10.1109\/MODELS.2015.7338274"},{"key":"9_CR31","doi-asserted-by":"crossref","unstructured":"Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., Walton, A.: Measuring and defining the experience of immersion in games. International journal of human-computer studies 66(9), 641\u2013661 (2008)","DOI":"10.1016\/j.ijhcs.2008.04.004"},{"key":"9_CR32","doi-asserted-by":"crossref","unstructured":"Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE transactions on software engineering 37(5), 649\u2013678 (2010)","DOI":"10.1109\/TSE.2010.62"},{"key":"9_CR33","unstructured":"Lazarus, R.S., Folkman, S.: Stress, appraisal, and coping. Springer publishing company (1984)"},{"key":"9_CR34","doi-asserted-by":"crossref","unstructured":"Lee, I., Kim, H., Lee, B.: Automated playtesting with a cognitive model of sensorimotor coordination. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 4920\u20134929 (2021)","DOI":"10.1145\/3474085.3475429"},{"key":"9_CR35","doi-asserted-by":"crossref","unstructured":"Lewis, C., Whitehead, J., Wardrip-Fruin, N.: What went wrong: a taxonomy of video game bugs. In: Proceedings of the fifth international conference on the foundations of digital games. pp. 108\u2013115 (2010)","DOI":"10.1145\/1822348.1822363"},{"key":"9_CR36","doi-asserted-by":"crossref","unstructured":"McMinn, P.: Search-based software test data generation: a survey. Software testing, Verification and reliability 14(2), 105\u2013156 (2004)","DOI":"10.1002\/stvr.294"},{"key":"9_CR37","doi-asserted-by":"crossref","unstructured":"Mirza-Babaei, P., Nacke, L.E., Gregory, J., Collins, N., Fitzpatrick, G.: How does it play better? exploring user testing and biometric storyboards in games user research. In: Proceedings of the SIGCHI conference on human factors in computing systems. pp. 1499\u20131508 (2013)","DOI":"10.1145\/2470654.2466200"},{"key":"9_CR38","doi-asserted-by":"crossref","unstructured":"Myers, G.J., Sandler, C., Badgett, T.: The art of software testing. John Wiley & Sons (2011)","DOI":"10.1002\/9781119202486"},{"key":"9_CR39","doi-asserted-by":"crossref","unstructured":"Nacke, L., Lindley, C.A.: Flow and immersion in first-person shooters: measuring the player\u2019s gameplay experience. In: Proceedings of the 2008 conference on future play: Research, play, share. pp. 81\u201388 (2008)","DOI":"10.1145\/1496984.1496998"},{"key":"9_CR40","doi-asserted-by":"crossref","unstructured":"Nacke, L.E.: Games user research and physiological game evaluation. In: Game user experience evaluation, pp. 63\u201386. Springer (2015)","DOI":"10.1007\/978-3-319-15985-0_4"},{"key":"9_CR41","doi-asserted-by":"crossref","unstructured":"Nikolik, B.: Test diversity. Information and Software Technology 48(11), 1083\u20131094 (2006)","DOI":"10.1016\/j.infsof.2006.02.001"},{"key":"9_CR42","unstructured":"Ochs, M., Pelachaud, C., Sadek, D.: An empathic virtual dialog agent to improve human-machine interaction. In: Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 1. pp. 89\u201396 (2008)"},{"key":"9_CR43","doi-asserted-by":"crossref","unstructured":"Ortony, A., Clore, G., Collins, A.: The cognitive structure of emotions. cam (bridge university press. Cambridge, England (1988)","DOI":"10.1017\/CBO9780511571299"},{"key":"9_CR44","doi-asserted-by":"crossref","unstructured":"Panichella, A., Kifetew, F.M., Tonella, P.: Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering 44(2), 122\u2013158 (2017)","DOI":"10.1109\/TSE.2017.2663435"},{"key":"9_CR45","doi-asserted-by":"crossref","unstructured":"Prasetya, I., Dastani, M., Prada, R., Vos, T.E., Dignum, F., Kifetew, F.: Aplib: Tactical agents for testing computer games. In: International Workshop on Engineering Multi-Agent Systems. pp. 21\u201341. Springer (2020)","DOI":"10.1007\/978-3-030-66534-0_2"},{"key":"9_CR46","doi-asserted-by":"crossref","unstructured":"Procci, K., Singer, A.R., Levy, K.R., Bowers, C.: Measuring the flow experience of gamers: An evaluation of the dfs-2. Computers in Human Behavior 28(6), 2306\u20132312 (2012)","DOI":"10.1016\/j.chb.2012.06.039"},{"key":"9_CR47","unstructured":"Reilly, W.S.: Believable social and emotional agents. Tech. rep., Carnegie-Mellon Univ Pittsburgh pa Dept of Computer Science (1996)"},{"key":"9_CR48","doi-asserted-by":"crossref","unstructured":"Roseman, I.J., Smith, C.A.: Appraisal theory. Appraisal processes in emotion: Theory, methods, research pp. 3\u201319 (2001)","DOI":"10.1093\/oso\/9780195130072.003.0001"},{"key":"9_CR49","doi-asserted-by":"crossref","unstructured":"Roseman, I.J., Spindel, M.S., Jose, P.E.: Appraisals of emotion-eliciting events: Testing a theory of discrete emotions. Journal of personality and social psychology 59(5), \u00a0899 (1990)","DOI":"10.1037\/0022-3514.59.5.899"},{"key":"9_CR50","doi-asserted-by":"crossref","unstructured":"Smith, C.A., Ellsworth, P.C.: Patterns of cognitive appraisal in emotion. Journal of personality and social psychology 48(4), \u00a0813 (1985)","DOI":"10.1037\/0022-3514.48.4.813"},{"key":"9_CR51","doi-asserted-by":"crossref","unstructured":"Stahlke, S.N., Mirza-Babaei, P.: Usertesting without the user: Opportunities and challenges of an ai-driven approach in games user research. Computers in Entertainment (CIE) 16(2), 1\u201318 (2018)","DOI":"10.1145\/3183568"},{"key":"9_CR52","unstructured":"Utting, M., Legeard, B.: Practical model-based testing: a tools approach. Elsevier (2010)"},{"key":"9_CR53","doi-asserted-by":"crossref","unstructured":"Vermeeren, A.P., Law, E.L.C., Roto, V., Obrist, M., Hoonhout, J., V\u00e4\u00e4n\u00e4nen-Vainio-Mattila, K.: User experience evaluation methods: current state and development needs. In: Proceedings of the 6th Nordic conference on human-computer interaction: Extending boundaries. pp. 521\u2013530 (2010)","DOI":"10.1145\/1868914.1868973"},{"key":"9_CR54","doi-asserted-by":"crossref","unstructured":"Zarembo, I.: Analysis of artificial intelligence applications for automated testing of video games. In: ENVIRONMENT. TECHNOLOGIES. RESOURCES. Proceedings of the International Scientific and Practical Conference. vol.\u00a02, pp. 170\u2013174 (2019)","DOI":"10.17770\/etr2019vol2.4158"},{"key":"9_CR55","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Borovikov, I., de\u00a0Mesentier\u00a0Silva, F., Beirami, A., Rupert, J., Somers, C., Harder, J., Kolen, J., Pinto, J., Pourabolghasem, R., et\u00a0al.: Winning is not everything: Enhancing game development with intelligent agents. IEEE Transactions on Games 12(2), 199\u2013212 (2020)","DOI":"10.1109\/TG.2020.2990865"}],"container-title":["Lecture Notes in Computer Science","Fundamental Approaches to Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-30826-0_9","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,11]],"date-time":"2023-12-11T00:27:54Z","timestamp":1702254474000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-30826-0_9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"ISBN":["9783031308253","9783031308260"],"references-count":55,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-30826-0_9","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2023]]},"assertion":[{"value":"20 April 2023","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"FASE","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Fundamental Approaches to Software Engineering","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Paris","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"France","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2023","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"22 April 2023","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"27 April 2023","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"fase2023","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/etaps.org\/2023\/fase","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Double-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"50","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"12","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"24% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"6-7","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"The proceedings also include 2 tool papers, 2 NIER papers, and 2 competition papers","order":10,"name":"additional_info_on_review_process","label":"Additional Info on Review Process","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}