{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T21:15:01Z","timestamp":1768338901920,"version":"3.49.0"},"publisher-location":"Cham","reference-count":45,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031572586","type":"print"},{"value":"9783031572593","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,1,1]],"date-time":"2024-01-01T00:00:00Z","timestamp":1704067200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,6]],"date-time":"2024-04-06T00:00:00Z","timestamp":1712361600000},"content-version":"vor","delay-in-days":96,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Given the scale and complexity of large online service systems and the diversity of environments in which the services are to be invoked, it is inevitable that those service systems contain bugs that affect the users. As a result, it is essential for service providers to discover issues in their systems based on information gathered from users. iFeedback is a state-of-the-art technique for user-feedback-based issue detection. While it has been deployed to help detect issues in real-world service systems, the accuracy of iFeedback\u2019s detection results is relatively low due to limitations in its design. In this paper, we propose the<jats:sc>SkyNet<\/jats:sc>technique and tool that analyzes both user feedback gathered via specific channels and public posts collected from social media platforms to more accurately detect issues in service systems. We have applied the tool to detect issues for three real-world, large-scale online service systems based on their historical data gathered over a ten-month period of time.<jats:sc>SkyNet<\/jats:sc>reported in total 2790 issues, among which 93.0% were confirmed by developers as reflecting real problems that deserve their close attention. It also detected 58 out of the 62 severe issues reported during the period, achieving a recall of 93.5% for severe issues. Such results suggest<jats:sc>SkyNet<\/jats:sc>is both effective and accurate in issue detection.<\/jats:p>","DOI":"10.1007\/978-3-031-57259-3_8","type":"book-chapter","created":{"date-parts":[[2024,4,5]],"date-time":"2024-04-05T13:01:39Z","timestamp":1712322099000},"page":"165-187","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Smart Issue Detection for Large-Scale Online Service Systems Using Multi-Channel Data"],"prefix":"10.1007","author":[{"given":"Liushan","family":"Chen","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6065-6958","authenticated-orcid":false,"given":"Yu","family":"Pei","sequence":"additional","affiliation":[]},{"given":"Mingyang","family":"Wan","sequence":"additional","affiliation":[]},{"given":"Zhihui","family":"Fei","sequence":"additional","affiliation":[]},{"given":"Tao","family":"Liang","sequence":"additional","affiliation":[]},{"given":"Guojun","family":"Ma","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,4,6]]},"reference":[{"key":"8_CR1","unstructured":"Albert pre-trained model for chinese. https:\/\/github.com\/brightmart\/albert_zh. Last accessed 19 May 2022."},{"key":"8_CR2","unstructured":"Cascading classifiers - wikipedia. https:\/\/en.wikipedia.org\/wiki\/Cascading_classifiers. Last accessed 19 May 2022."},{"key":"8_CR3","unstructured":"Github elasticsearch. https:\/\/github.com\/elastic\/elasticsearch. Last accessed 19 May 2022."},{"key":"8_CR4","unstructured":"Interquartile range. https:\/\/en.wikipedia.org\/wiki\/Interquartile_range. Last accessed 19 May 2022."},{"key":"8_CR5","unstructured":"Jieba - chinese text segmentation. https:\/\/github.com\/fxsjy\/jieba."},{"key":"8_CR6","unstructured":"Okapi bm25 - wikipedia. https:\/\/en.wikipedia.org\/wiki\/Okapi_BM25. Last accessed 19 May 2022."},{"key":"8_CR7","unstructured":"Tencent ai lab embedding corpora for chinese and english words and phrases. https:\/\/ai.tencent.com\/ailab\/nlp\/en\/embedding.html."},{"key":"8_CR8","unstructured":"tf-idf - wikipedia. https:\/\/en.wikipedia.org\/wiki\/Tf%e2%80%93idf. Last accessed 19 May 2022."},{"key":"8_CR9","unstructured":"S.\u00a0Arora, Y.\u00a0Liang, and T.\u00a0Ma. A simple but tough-to-beat baseline for sentence embeddings. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017."},{"key":"8_CR10","doi-asserted-by":"crossref","unstructured":"D.\u00a0R. Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 20(2):215\u2013232, 1958.","DOI":"10.1111\/j.2517-6161.1958.tb00292.x"},{"key":"8_CR11","unstructured":"J.\u00a0Devlin, M.-W. Chang, K.\u00a0Lee, and K.\u00a0Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018."},{"key":"8_CR12","doi-asserted-by":"crossref","unstructured":"A.\u00a0Di\u00a0Sorbo, S.\u00a0Panichella, C.\u00a0V. Alexandru, J.\u00a0Shimagaki, C.\u00a0A. Visaggio, G.\u00a0Canfora, and H.\u00a0C. Gall. What would users change in my app? summarizing app reviews for recommending software changes. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 499\u2013510, 2016.","DOI":"10.1145\/2950290.2950299"},{"key":"8_CR13","doi-asserted-by":"crossref","unstructured":"B.\u00a0Fu, J.\u00a0Lin, L.\u00a0Li, C.\u00a0Faloutsos, J.\u00a0Hong, and N.\u00a0Sadeh. Why people hate your app: Making sense of user feedback in a mobile app store. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1276\u20131284, 2013.","DOI":"10.1145\/2487575.2488202"},{"key":"8_CR14","doi-asserted-by":"crossref","unstructured":"C.\u00a0Gao, J.\u00a0Zeng, M.\u00a0R. Lyu, and I.\u00a0King. Online app review analysis for identifying emerging issues. In M.\u00a0Chaudron, I.\u00a0Crnkovic, M.\u00a0Chechik, and M.\u00a0Harman, editors, Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, pages 48\u201358. ACM, 2018.","DOI":"10.1145\/3180155.3180218"},{"key":"8_CR15","doi-asserted-by":"crossref","unstructured":"J.\u00a0Gu, C.\u00a0Luo, S.\u00a0Qin, B.\u00a0Qiao, Q.\u00a0Lin, H.\u00a0Zhang, Z.\u00a0Li, Y.\u00a0Dang, S.\u00a0Cai, W.\u00a0Wu, Y.\u00a0Zhou, M.\u00a0Chintalapati, and D.\u00a0Zhang. Efficient incident identification from multi-dimensional issue reports via meta-heuristic search. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2020, page 292-303, New York, NY, USA, 2020. Association for Computing Machinery.","DOI":"10.1145\/3368089.3409741"},{"key":"8_CR16","doi-asserted-by":"crossref","unstructured":"E.\u00a0Guzman, M.\u00a0Ibrahim, and M.\u00a0Glinz. A little bird told me: Mining tweets for requirements and software evolution. In 2017 IEEE 25th International Requirements Engineering Conference (RE), pages 11\u201320, 2017.","DOI":"10.1109\/RE.2017.88"},{"key":"8_CR17","doi-asserted-by":"crossref","unstructured":"S.\u00a0Hochreiter and J.\u00a0Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735\u20131780, 1997.","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"8_CR18","unstructured":"C.\u00a0Huang, G.\u00a0Min, Y.\u00a0Wu, Y.\u00a0Ying, K.\u00a0Pei, and Z.\u00a0Xiang. Time series anomaly detection for trustworthy services in cloud computing systems. IEEE Transactions on Big Data, 2017."},{"key":"8_CR19","doi-asserted-by":"crossref","unstructured":"T.\u00a0Johann, C.\u00a0Stanik, W.\u00a0Maalej, et\u00a0al. Safe: A simple approach for feature extraction from app descriptions and app reviews. In 2017 IEEE 25th international requirements engineering conference (RE), pages 21\u201330. IEEE, 2017.","DOI":"10.1109\/RE.2017.71"},{"key":"8_CR20","unstructured":"Z.\u00a0Lan, M.\u00a0Chen, S.\u00a0Goodman, K.\u00a0Gimpel, P.\u00a0Sharma, and R.\u00a0Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020."},{"key":"8_CR21","doi-asserted-by":"crossref","unstructured":"N.\u00a0Laptev, S.\u00a0Amizadeh, and I.\u00a0Flint. Generic and scalable framework for automated time-series anomaly detection. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1939\u20131947, 2015.","DOI":"10.1145\/2783258.2788611"},{"key":"8_CR22","unstructured":"L.\u00a0Li, X.\u00a0Zhang, X.\u00a0Zhao, H.\u00a0Zhang, Y.\u00a0Kang, P.\u00a0Zhao, B.\u00a0Qiao, S.\u00a0He, P.\u00a0Lee, J.\u00a0Sun, F.\u00a0Gao, L.\u00a0Yang, Q.\u00a0Lin, S.\u00a0Rajmohan, Z.\u00a0Xu, and D.\u00a0Zhang. Fighting the fog of war: Automated incident detection for cloud systems. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 131\u2013146. USENIX Association, July 2021."},{"key":"8_CR23","doi-asserted-by":"crossref","unstructured":"Z.\u00a0Li, Y.\u00a0Zhao, R.\u00a0Liu, and D.\u00a0Pei. Robust and rapid clustering of kpis for large-scale anomaly detection. In 2018 IEEE\/ACM 26th International Symposium on Quality of Service (IWQoS), pages 1\u201310. IEEE, 2018.","DOI":"10.1109\/IWQoS.2018.8624168"},{"key":"8_CR24","doi-asserted-by":"crossref","unstructured":"Q.\u00a0Lin, J.-G. Lou, H.\u00a0Zhang, and D.\u00a0Zhang. idice: Problem identification for emerging issues. In 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE), pages 214\u2013224, 2016.","DOI":"10.1145\/2884781.2884795"},{"key":"8_CR25","doi-asserted-by":"crossref","unstructured":"D.\u00a0Liu, Y.\u00a0Zhao, H.\u00a0Xu, Y.\u00a0Sun, D.\u00a0Pei, J.\u00a0Luo, X.\u00a0Jing, and M.\u00a0Feng. Opprentice: Towards practical and automatic anomaly detection through machine learning. In Proceedings of the 2015 Internet Measurement Conference, pages 211\u2013224, 2015.","DOI":"10.1145\/2815675.2815679"},{"key":"8_CR26","doi-asserted-by":"crossref","unstructured":"M.\u00a0Ma, S.\u00a0Zhang, D.\u00a0Pei, X.\u00a0Huang, and H.\u00a0Dai. Robust and rapid adaption for concept drift in software system anomaly detection. In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), pages 13\u201324. IEEE, 2018.","DOI":"10.1109\/ISSRE.2018.00013"},{"key":"8_CR27","unstructured":"T.\u00a0Mikolov, K.\u00a0Chen, G.\u00a0Corrado, and J.\u00a0Dean. Efficient estimation of word representations in vector space. In Y.\u00a0Bengio and Y.\u00a0LeCun, editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013."},{"key":"8_CR28","unstructured":"T.\u00a0Mikolov, I.\u00a0Sutskever, K.\u00a0Chen, G.\u00a0Corrado, and J.\u00a0Dean. Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546, 2013."},{"key":"8_CR29","doi-asserted-by":"crossref","unstructured":"D.\u00a0Pagano and W.\u00a0Maalej. User feedback in the appstore: An empirical study. In 2013 21st IEEE international requirements engineering conference (RE), pages 125\u2013134. IEEE, 2013.","DOI":"10.1109\/RE.2013.6636712"},{"key":"8_CR30","doi-asserted-by":"crossref","unstructured":"F.\u00a0Palomba, M.\u00a0Linares-V\u00e1squez, G.\u00a0Bavota, R.\u00a0Oliveto, M.\u00a0Di\u00a0Penta, D.\u00a0Poshyvanyk, and A.\u00a0De\u00a0Lucia. User reviews matter! tracking crowdsourced reviews to support evolution of successful apps. In 2015 IEEE international conference on software maintenance and evolution (ICSME), pages 291\u2013300. IEEE, 2015.","DOI":"10.1109\/ICSM.2015.7332475"},{"key":"8_CR31","doi-asserted-by":"crossref","unstructured":"S.\u00a0K. Panda and P.\u00a0K. Jana. Efficient task scheduling algorithms for heterogeneous multi-cloud environment. The Journal of Supercomputing, 71(4):1505\u20131533, 2015.","DOI":"10.1007\/s11227-014-1376-6"},{"key":"8_CR32","doi-asserted-by":"crossref","unstructured":"S.\u00a0Patro and K.\u00a0K. Sahu. Normalization: A preprocessing stage. arXiv preprint arXiv:1503.06462, 2015.","DOI":"10.17148\/IARJSET.2015.2305"},{"key":"8_CR33","doi-asserted-by":"crossref","unstructured":"M.\u00a0Raginsky, R.\u00a0M. Willett, C.\u00a0Horn, J.\u00a0Silva, and R.\u00a0F. Marcia. Sequential anomaly detection in the presence of noise and limited feedback. IEEE Transactions on Information Theory, 58(8):5544\u20135562, 2012.","DOI":"10.1109\/TIT.2012.2201375"},{"key":"8_CR34","unstructured":"K.\u00a0Ramasubramanian and J.\u00a0Moolayil. Applied Supervised Learning with R: Use machine learning libraries of R to build models that solve business problems and predict future trends. Packt Publishing, 2019."},{"key":"8_CR35","unstructured":"M.\u00a0Saberian and N.\u00a0Vasconcelos. Boosting algorithms for detector cascade learning. Journal of Machine Learning Research, 15:2569\u20132605, 2014."},{"key":"8_CR36","doi-asserted-by":"crossref","unstructured":"R.\u00a0Sathya, A.\u00a0Abraham, et\u00a0al. Comparison of supervised and unsupervised learning algorithms for pattern classification. International Journal of Advanced Research in Artificial Intelligence, 2(2):34\u201338, 2013.","DOI":"10.14569\/IJARAI.2013.020206"},{"key":"8_CR37","doi-asserted-by":"crossref","unstructured":"S.\u00a0Sra, S.\u00a0Nowozin, and S.\u00a0J. Wright. Optimization for machine learning. Mit Press, 2012.","DOI":"10.7551\/mitpress\/8996.001.0001"},{"key":"8_CR38","doi-asserted-by":"crossref","unstructured":"C.\u00a0Stanik, T.\u00a0Pietz, and W.\u00a0Maalej. Unsupervised topic discovery in user comments. In 2021 IEEE 29th International Requirements Engineering Conference (RE), pages 150\u2013161. IEEE, 2021.","DOI":"10.1109\/RE51729.2021.00021"},{"key":"8_CR39","doi-asserted-by":"crossref","unstructured":"Y.\u00a0Sun, Y.\u00a0Zhao, Y.\u00a0Su, D.\u00a0Liu, X.\u00a0Nie, Y.\u00a0Meng, S.\u00a0Cheng, D.\u00a0Pei, S.\u00a0Zhang, X.\u00a0Qu, et\u00a0al. Hotspot: Anomaly localization for additive kpis with multi-dimensional attributes. IEEE Access, 6:10909\u201310923, 2018.","DOI":"10.1109\/ACCESS.2018.2804764"},{"key":"8_CR40","doi-asserted-by":"crossref","unstructured":"D.\u00a0Tang, A.\u00a0Agarwal, D.\u00a0O\u2019Brien, and M.\u00a0Meyer. Overlapping experiment infrastructure: More, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 17\u201326, 2010.","DOI":"10.1145\/1835804.1835810"},{"key":"8_CR41","doi-asserted-by":"crossref","unstructured":"L.\u00a0Villarroel, G.\u00a0Bavota, B.\u00a0Russo, R.\u00a0Oliveto, and M.\u00a0Di\u00a0Penta. Release planning of mobile apps based on user reviews. In 2016 IEEE\/ACM 38th International Conference on Software Engineering (ICSE), pages 14\u201324. IEEE, 2016.","DOI":"10.1145\/2884781.2884818"},{"key":"8_CR42","doi-asserted-by":"crossref","unstructured":"P.\u00a0M. Vu, H.\u00a0V. Pham, T.\u00a0T. Nguyen, and T.\u00a0T. Nguyen. Phrase-based extraction of user opinions in mobile app reviews. In Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering, pages 726\u2013731, 2016.","DOI":"10.1145\/2970276.2970365"},{"key":"8_CR43","doi-asserted-by":"crossref","unstructured":"G.\u00a0Williams and A.\u00a0Mahmoud. Mining twitter feeds for software user requirements. In 2017 IEEE 25th International Requirements Engineering Conference (RE), pages 1\u201310. IEEE, 2017.","DOI":"10.1109\/RE.2017.14"},{"key":"8_CR44","doi-asserted-by":"crossref","unstructured":"H.\u00a0Xu, W.\u00a0Chen, N.\u00a0Zhao, Z.\u00a0Li, J.\u00a0Bu, Z.\u00a0Li, Y.\u00a0Liu, Y.\u00a0Zhao, D.\u00a0Pei, Y.\u00a0Feng, et\u00a0al. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 World Wide Web Conference, pages 187\u2013196, 2018.","DOI":"10.1145\/3178876.3185996"},{"key":"8_CR45","doi-asserted-by":"crossref","unstructured":"W.\u00a0Zheng, H.\u00a0Lu, Y.\u00a0Zhou, J.\u00a0Liang, H.\u00a0Zheng, and Y.\u00a0Deng. ifeedback: exploiting user feedback for real-time issue detection in large-scale online service systems. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE), pages 352\u2013363. IEEE, 2019.","DOI":"10.1109\/ASE.2019.00041"}],"container-title":["Lecture Notes in Computer Science","Fundamental Approaches to Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-57259-3_8","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,15]],"date-time":"2024-11-15T19:39:46Z","timestamp":1731699586000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-57259-3_8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"ISBN":["9783031572586","9783031572593"],"references-count":45,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-57259-3_8","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024]]},"assertion":[{"value":"6 April 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"FASE","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Fundamental Approaches to Software Engineering","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Luxembourg City","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Luxembourg","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"6 April 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"11 April 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"27","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"fase2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/etaps.org\/2024\/conferences\/fase\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"41","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"14","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"5","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"34% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3-4","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"4","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}