{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:12:35Z","timestamp":1750219955562,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,10,17]],"date-time":"2022-10-17T00:00:00Z","timestamp":1665964800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"funder":[{"name":"Natural Science Foundation of Hebei Province, China","award":["F2022208006"],"award-info":[{"award-number":["F2022208006"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62173195"],"award-info":[{"award-number":["62173195"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,10,17]]},"DOI":"10.1145\/3511808.3557167","type":"proceedings-article","created":{"date-parts":[[2022,10,16]],"date-time":"2022-10-16T01:29:57Z","timestamp":1665883797000},"page":"5044-5048","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["An In-depth Interactive and Visualized Platform for Evaluating and Analyzing MRC Models"],"prefix":"10.1145","author":[{"given":"Zhijing","family":"Wu","sequence":"first","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Jingliang","family":"Fang","sequence":"additional","affiliation":[{"name":"Tsinghua University & Hebei University of Science and Technology, Beijing, China"}]},{"given":"Hua","family":"Xu","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Kai","family":"Gao","sequence":"additional","affiliation":[{"name":"Hebei University of Science and Technology, Hebei, China"}]}],"member":"320","published-online":{"date-parts":[[2022,10,17]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.12.012"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-2607"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-5817"},{"key":"e_1_3_2_2_4_1","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2019","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2019 , Minneapolis, MN, USA, June 2--7 , 2019. 4171--4186. https:\/\/www.aclweb.org\/anthology\/N19--1423\/ Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL 2019, Minneapolis, MN, USA, June 2--7, 2019. 4171--4186. https:\/\/www.aclweb.org\/anthology\/N19--1423\/"},{"volume-title":"Explainable and interpretable models in computer vision and machine learning","author":"Doshi-Velez Finale","key":"e_1_3_2_2_5_1","unstructured":"Finale Doshi-Velez and Been Kim . 2018. Considerations for evaluation and generalization in interpretable machine learning . In Explainable and interpretable models in computer vision and machine learning . Springer , 3--17. Finale Doshi-Velez and Been Kim. 2018. Considerations for evaluation and generalization in interpretable machine learning. In Explainable and interpretable models in computer vision and machine learning. Springer, 3--17."},{"key":"e_1_3_2_2_6_1","volume-title":"Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber.","author":"Feng Shi","year":"2018","unstructured":"Shi Feng , Eric Wallace , Alvin Grissom II , Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018 . Pathologies of neural models make interpretations difficult. arXiv preprint arXiv:1804.07781 (2018). Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. arXiv preprint arXiv:1804.07781 (2018)."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-5801"},{"key":"e_1_3_2_2_8_1","volume-title":"TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing. CoRR","author":"Gui Tao","year":"2021","unstructured":"Tao Gui , Xiao Wang , Qi Zhang , Qin Liu , Yicheng Zou , Xin Zhou , Rui Zheng , Chong Zhang , Qinzhuo Wu , Jiacheng Ye , Zexiong Pang , Yongxin Zhang , Zhengyan Li , Ruotian Ma , Zichu Fei , Ruijian Cai , Jun Zhao , Xinwu Hu , Zhiheng Yan , Yiding Tan , Yuan Hu , Qiyuan Bian , Zhihua Liu , Bolin Zhu , Shan Qin , Xiaoyu Xing , Jinlan Fu , Yue Zhang , Minlong Peng , Xiaoqing Zheng , Yaqian Zhou , Zhongyu Wei , Xipeng Qiu , and Xuanjing Huang . 2021. TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing. CoRR , Vol. abs\/ 2103 .11441 ( 2021 ). showeprint[arXiv]2103.11441 https:\/\/arxiv.org\/abs\/2103.11441 Tao Gui, Xiao Wang, Qi Zhang, Qin Liu, Yicheng Zou, Xin Zhou, Rui Zheng, Chong Zhang, Qinzhuo Wu, Jiacheng Ye, Zexiong Pang, Yongxin Zhang, Zhengyan Li, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xinwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Bolin Zhu, Shan Qin, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, and Xuanjing Huang. 2021. TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing. CoRR, Vol. abs\/2103.11441 (2021). showeprint[arXiv]2103.11441 https:\/\/arxiv.org\/abs\/2103.11441"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1215"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1262"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00300"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1147"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.85"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1160"},{"key":"e_1_3_2_2_15_1","unstructured":"Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever etal 2019. Language models are unsupervised multitask learners. OpenAI blog Vol. 1 8 (2019) 9.  Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever et al. 2019. Language models are unsupervised multitask learners. OpenAI blog Vol. 1 8 (2019) 9."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1264"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.442"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.mrqa-1.15"},{"key":"e_1_3_2_2_19_1","volume-title":"Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020","author":"Schlegel Viktor","year":"2020","unstructured":"Viktor Schlegel , Marco Valentino , Andr\u00e9 Freitas , Goran Nenadic , and Riza Batista-Navarro . 2020 . A Framework for Evaluation of Machine Reading Comprehension Gold Standards . In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020 , Marseille, France, May 11--16 , 2020. 5359--5369. https:\/\/aclanthology.org\/2020.lrec-1.660\/ Viktor Schlegel, Marco Valentino, Andr\u00e9 Freitas, Goran Nenadic, and Riza Batista-Navarro. 2020. A Framework for Evaluation of Machine Reading Comprehension Gold Standards. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11--16, 2020. 5359--5369. https:\/\/aclanthology.org\/2020.lrec-1.660\/"},{"key":"e_1_3_2_2_20_1","volume-title":"Bidirectional Attention Flow for Machine Comprehension. In 5th International Conference on Learning Representations, ICLR 2017","author":"Seo Min Joon","year":"2017","unstructured":"Min Joon Seo , Aniruddha Kembhavi , Ali Farhadi , and Hannaneh Hajishirzi . 2017 . Bidirectional Attention Flow for Machine Comprehension. In 5th International Conference on Learning Representations, ICLR 2017 , Toulon, France, April 24--26 , 2017. https:\/\/openreview.net\/forum?id=HJ0UKP9ge Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional Attention Flow for Machine Comprehension. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24--26, 2017. https:\/\/openreview.net\/forum?id=HJ0UKP9ge"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.189"},{"key":"e_1_3_2_2_22_1","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022","author":"Sugawara Saku","year":"2022","unstructured":"Saku Sugawara , Nikita Nangia , Alex Warstadt , and Samuel R. Bowman . 2022. What Makes Reading Comprehension Questions Difficult? . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022 , Dublin, Ireland, May 22--27 , 2022 . 6951--6971. https:\/\/aclanthology.org\/2022.acl-long.479 Saku Sugawara, Nikita Nangia, Alex Warstadt, and Samuel R. Bowman. 2022. What Makes Reading Comprehension Questions Difficult?. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22--27, 2022. 6951--6971. https:\/\/aclanthology.org\/2022.acl-long.479"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6422"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.10957"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-2623"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1018"},{"key":"e_1_3_2_2_27_1","volume-title":"4th International Conference on Learning Representations, ICLR","author":"Weston Jason","year":"2016","unstructured":"Jason Weston , Antoine Bordes , Sumit Chopra , and Tom\u00e1 s Mikolov . 2016. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks . In 4th International Conference on Learning Representations, ICLR 2016 , San Juan, Puerto Rico , May 2--4, 2016, Conference Track Proceedings . http:\/\/arxiv.org\/abs\/1502.05698 Jason Weston, Antoine Bordes, Sumit Chopra, and Tom\u00e1 s Mikolov. 2016. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2--4, 2016, Conference Track Proceedings. http:\/\/arxiv.org\/abs\/1502.05698"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106075"},{"key":"e_1_3_2_2_29_1","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang , Peng Qi , Saizheng Zhang , Yoshua Bengio , William W. Cohen , Ruslan Salakhutdinov , and Christopher D. Manning . 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , Brussels, Belgium, October 31 - November 4, 2019 . 2369--2380. https:\/\/doi.org\/10.18653\/v1\/d18--1259 10.18653\/v1 Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2019. 2369--2380. https:\/\/doi.org\/10.18653\/v1\/d18--1259"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i16.17705"}],"event":{"name":"CIKM '22: The 31st ACM International Conference on Information and Knowledge Management","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Atlanta GA USA","acronym":"CIKM '22"},"container-title":["Proceedings of the 31st ACM International Conference on Information & Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511808.3557167","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3511808.3557167","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:06Z","timestamp":1750182546000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3511808.3557167"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,17]]},"references-count":30,"alternative-id":["10.1145\/3511808.3557167","10.1145\/3511808"],"URL":"https:\/\/doi.org\/10.1145\/3511808.3557167","relation":{},"subject":[],"published":{"date-parts":[[2022,10,17]]},"assertion":[{"value":"2022-10-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}