{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T01:01:06Z","timestamp":1774400466289,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":24,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,7,11]],"date-time":"2021-07-11T00:00:00Z","timestamp":1625961600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Engineering and Physical SciencesResearch Council","award":["EP\/V025708\/1"],"award-info":[{"award-number":["EP\/V025708\/1"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,11]]},"DOI":"10.1145\/3404835.3463262","type":"proceedings-article","created":{"date-parts":[[2021,7,12]],"date-time":"2021-07-12T02:41:54Z","timestamp":1626057714000},"page":"2335-2341","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":37,"title":["How Deep is your Learning: the DL-HARD Annotated Deep Learning Dataset"],"prefix":"10.1145","author":[{"given":"Iain","family":"Mackie","sequence":"first","affiliation":[{"name":"University of Glasgow, Glasgow, United Kingdom"}]},{"given":"Jeffrey","family":"Dalton","sequence":"additional","affiliation":[{"name":"University of Glasgow, Glasgow, United Kingdom"}]},{"given":"Andrew","family":"Yates","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Informatics, Saarbr\u00fccken, Germany"}]}],"member":"320","published-online":{"date-parts":[[2021,7,11]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Jonathan Berant A. Chou Roy Frostig and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In EMNLP."},{"key":"e_1_3_2_1_2_1","volume-title":"Lin (Eds.)","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877--1901. https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406522.3446027"},{"key":"e_1_3_2_1_4_1","volume-title":"Autoregressive Entity Retrieval. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=5k8F6UU39V","author":"Cao Nicola De","year":"2021","unstructured":"Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive Entity Retrieval. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=5k8F6UU39V"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3284102"},{"key":"e_1_3_2_1_6_1","volume-title":"Overview of the TREC 2020 deep learning track. In Text REtrieval Conference (TREC). TREC.","author":"Craswell Nick","year":"2021","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. In Text REtrieval Conference (TREC). TREC."},{"key":"e_1_3_2_1_7_1","volume-title":"Overview of the TREC 2019 deep learning track. In Text REtrieval Conference (TREC).","author":"Craswell Nick","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. Overview of the TREC 2019 deep learning track. In Text REtrieval Conference (TREC)."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609628"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Jeffrey Dalton Chenyan Xiong and J. Callan. 2020. TREC CAsT 2019: The Conversational Assistance Track Overview. Vol. abs\/2003.13624 (2020).","DOI":"10.6028\/NIST.SP.1266.cast-overview"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835499"},{"key":"e_1_3_2_1_12_1","volume-title":"Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics","author":"Kwiatkowski Tom","year":"2019","unstructured":"Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics (2019)."},{"key":"e_1_3_2_1_13_1","volume-title":"PARADE: Passage representation aggregation for document reranking. arXiv preprint arXiv:2008.09093","author":"Li Canjia","year":"2020","unstructured":"Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020. PARADE: Passage representation aggregation for document reranking. arXiv preprint arXiv:2008.09093 (2020)."},{"key":"e_1_3_2_1_14_1","volume-title":"MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPS.","author":"Nguyen Tri","year":"2016","unstructured":"Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPS."},{"key":"e_1_3_2_1_15_1","volume-title":"Passage Re-ranking with BERT. arXiv:1901.04085","author":"Nogueira Rodrigo","year":"2019","unstructured":"Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv:1901.04085 (2019)."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.63"},{"key":"e_1_3_2_1_17_1","first-page":"48","article-title":"Industry-scale Knowledge Graphs","volume":"17","author":"Noy Natasha","year":"2019","unstructured":"Natasha Noy, Yuqing Gao, A. Jain, A. Narayanan, A. Patterson, and J. Taylor. 2019. Industry-scale Knowledge Graphs: Lessons & Challenges. Queue, Vol. 17 (2019), 48--75.","journal-title":"Lessons & Challenges. Queue"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/1289189.1289276"},{"key":"e_1_3_2_1_19_1","first-page":"1","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, Vol. 21 (2020), 1--67.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1264"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401416"},{"key":"e_1_3_2_1_22_1","volume-title":"Proceedings of the Tenth Text REtrieval Conference (TREC","author":"Voorhees Ellen M.","year":"2001","unstructured":"Ellen M. Voorhees. 2001. Overview of the TREC 2001 Question Answering Track. In Proceedings of the Tenth Text REtrieval Conference (TREC 2001). Gaithersburg, Maryland, 42--51."},{"key":"e_1_3_2_1_23_1","unstructured":"Ledell Yu Wu F. Petroni Martin Josifoski Sebastian Riedel and Luke Zettlemoyer. 2020. Zero-shot Entity Linking with Dense Entity Retrieval. In EMNLP."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441667"}],"event":{"name":"SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval","location":"Virtual Event Canada","acronym":"SIGIR '21","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3404835.3463262","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3404835.3463262","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:17:44Z","timestamp":1750191464000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3404835.3463262"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,11]]},"references-count":24,"alternative-id":["10.1145\/3404835.3463262","10.1145\/3404835"],"URL":"https:\/\/doi.org\/10.1145\/3404835.3463262","relation":{},"subject":[],"published":{"date-parts":[[2021,7,11]]},"assertion":[{"value":"2021-07-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}