{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T19:52:16Z","timestamp":1793044336052,"version":"4.5.0"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,4,12]],"date-time":"2021-04-12T00:00:00Z","timestamp":1618185600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,4,12]],"date-time":"2021-04-12T00:00:00Z","timestamp":1618185600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The COVID-19 global pandemic has resulted in international efforts to understand, track, and mitigate the disease, yielding a significant corpus of COVID-19 and SARS-CoV-2-related publications across scientific disciplines. Throughout 2020, over 400,000 coronavirus-related publications have been collected through the COVID-19 Open Research Dataset. Here, we present CO-Search, a semantic, multi-stage, search engine designed to handle complex queries over the COVID-19 literature, potentially aiding overburdened health workers in finding scientific answers and avoiding misinformation during a time of crisis. CO-Search is built from two sequential parts: a hybrid semantic-keyword retriever, which takes an input query and returns a sorted list of the 1000 most relevant documents, and a re-ranker, which further orders them by relevance. The retriever is composed of a deep learning model (Siamese-BERT) that encodes query-level meaning, along with two keyword-based models (BM25, TF-IDF) that emphasize the most important words of a query. The re-ranker assigns a relevance score to each document, computed from the outputs of (1) a question\u2013answering module which gauges how much each document answers the query, and (2) an abstractive summarization module which determines how well a query matches a generated summary of the document. To account for the relatively limited dataset, we develop a text augmentation technique which splits the documents into pairs of paragraphs and the citations contained in them, creating millions of (citation title, paragraph) tuples for training the retriever. We evaluate our system (<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"http:\/\/einstein.ai\/covid\">http:\/\/einstein.ai\/covid<\/jats:ext-link>) on the data of the TREC-COVID information retrieval challenge, obtaining strong performance across multiple key information retrieval metrics.<\/jats:p>","DOI":"10.1038\/s41746-021-00437-0","type":"journal-article","created":{"date-parts":[[2021,4,12]],"date-time":"2021-04-12T10:06:33Z","timestamp":1618221993000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":96,"title":["COVID-19 information retrieval with deep-learning based semantic search, question answering, and abstractive summarization"],"prefix":"10.1038","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1937-9682","authenticated-orcid":false,"given":"Andre","family":"Esteva","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anuprit","family":"Kale","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Romain","family":"Paulus","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kazuma","family":"Hashimoto","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenpeng","family":"Yin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7830-6489","authenticated-orcid":false,"given":"Dragomir","family":"Radev","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Socher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,4,12]]},"reference":[{"key":"437_CR1","unstructured":"Thomala, L. L. Number of new hospital beds to be added in the designated hospitals after the coronavirus covid-19 outbreak in Wuhan, China as of February 2, 2020. Statista. https:\/\/www.statista.com\/statistics\/1095434\/china-changes-in-the-number-of-hospital-beds-in-designated-hospitals-after-coronavirus-outbreak-in-wuhan\/."},{"key":"437_CR2","unstructured":"Bogage, J. Tesla unveils ventilator prototype made with car parts on youtube. Wash. Post. https:\/\/www.washingtonpost.com\/business\/2020\/04\/06\/tesla-coronavirus-ventilators-musk\/ (2020)."},{"key":"437_CR3","unstructured":"Day, M. & Soper, S. Amazon is prioritizing essential products as online orders spike. Bloomberg. https:\/\/www.bloomberg.com\/news\/articles\/2020-03-17\/amazon-prioritizing-essentials-medical-goods-in-virus-response."},{"key":"437_CR4","unstructured":"Nicas, J. & Wakabayashi, D. Apple and google team up to \u2018contact trace\u2019 the coronavirus. N. Y. Times. https:\/\/www.nytimes.com\/2020\/04\/10\/technology\/apple-google-coronavirus-contact-tracing.html (2020)."},{"key":"437_CR5","unstructured":"Armitage, H. Stanford medicine launches national daily health survey to predict covid-19 surges, inform response efforts. Stanford Medicine News Center. http:\/\/med.stanford.edu\/news\/all-news\/2020\/04\/daily-health-survey-for-covid-19-launched0.html."},{"key":"437_CR6","doi-asserted-by":"crossref","unstructured":"Liu, C. et al. Research and Development on Therapeutic Agents and Vaccines for Covid-19 and Related Human Coronavirus Diseases. ACS Cent. Sci. 6, 315\u2013331 (2020).","DOI":"10.1021\/acscentsci.0c00272"},{"key":"437_CR7","unstructured":"Warwick, M., and F. Roshen. The global macroeconomic impacts of COVID-19: Seven scenarios. Centre for Applied Macroeconomic Analysis (CAMA) Working Paper 19\/2020, Australia National University (2020)."},{"key":"437_CR8","unstructured":"Thomas, P., Chaney, S. & Cutter, C. New covid-19 layoffs make job reductions permanent. Wall Street J. https:\/\/www.wsj.com\/articles\/new-covid-19-layoffs-make-job-reductions-permanent-11598654257 (2020)."},{"key":"437_CR9","unstructured":"Wang, L. L. et al. Cord-19: The covid-19 open research dataset. arXiv preprint arXiv:2004.10706 (2020)."},{"key":"437_CR10","doi-asserted-by":"crossref","unstructured":"Roberts, K. et al. TREC-Covid: rationale and structure of an information retrieval shared task for covid-19. J. Am. Med. Inform. Assoc. 27, 1431\u20131436 (2020).","DOI":"10.1093\/jamia\/ocaa091"},{"key":"437_CR11","doi-asserted-by":"crossref","unstructured":"Bullock, J., Luccioni, A., Pham, K. H., Lam, C. S. N. & Luengo-Oroz, M. Mapping the landscape of artificial intelligence applications against COVID-19. J. Artif. Intell. Res. 69, 807\u2013845 (2020).","DOI":"10.1613\/jair.1.12162"},{"key":"437_CR12","unstructured":"Tang, R. et al. Rapidly bootstrapping a question answering dataset for COVID-19. Preprint at arXiv:2004.11339 (2020)."},{"key":"437_CR13","unstructured":"Zhang, E., Gupta, N., Nogueira, R., Cho, K. & Lin, J. Rapidly deploying a neural search engine for the COVID-19 open research dataset: Preliminary thoughts and lessons learned. Preprint at arXiv:2004.05125 (2020)."},{"key":"437_CR14","unstructured":"Raffel, C. et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 1\u201367 (2020)"},{"key":"437_CR15","unstructured":"Bajaj, P. et al. MS Marco: a human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016)."},{"key":"437_CR16","unstructured":"MacAvaney, S., Cohan, A. & Goharian, N. SLEDGE: a simple yet effective baseline for coronavirus scientific knowledge search. Preprint at https:\/\/arxiv.org\/abs\/2005.02365 (2020)."},{"key":"437_CR17","doi-asserted-by":"crossref","unstructured":"Beltagy, I., Lo, K. & Cohan, A. Scibert: A pretrained language model for scientific text. In (eds. Inui, K., Jiang, J., Ng, V. & Wan, X.), EMNLP-IJCNLP, 3613\u20133618 (Association for Computational Linguistics, 2019).","DOI":"10.18653\/v1\/D19-1371"},{"key":"437_CR18","unstructured":"Devlin, J., Chang, M. W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"437_CR19","doi-asserted-by":"crossref","unstructured":"Su, D. et al. Caire-covid: a question answering and multi-document summarization system for COVID-19 research. Preprint at arXiv:2005.03975 (2020).","DOI":"10.18653\/v1\/2020.nlpcovid19-2.14"},{"key":"437_CR20","unstructured":"Dong, L. et al. Unified language model pre-training for natural language understanding andgeneration. In (eds Wallach, H. M. et al.) NeurIPS, 13042\u201313054 (Curran Associates, Inc., 2019)."},{"key":"437_CR21","doi-asserted-by":"crossref","unstructured":"Lewis, M. et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7871\u20137880 (Association for Computational Linguistics, Online, 2020).","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"437_CR22","doi-asserted-by":"crossref","unstructured":"Wang, X., Song, X., Guan, Y., Li, B. & Han, J. Comprehensive named entity recognition on CORD-19 with distant or weak supervision. Preprint at arXiv:2003.12218 (2020).","DOI":"10.1109\/BigData50022.2020.9378052"},{"key":"437_CR23","unstructured":"Ahamed, S. & Samad, M. Information mining for COVID-19 research from a large volume of scientific literature. Preprint at arXiv:2004.02085 (2020)."},{"key":"437_CR24","doi-asserted-by":"crossref","unstructured":"Mitra, B. & Craswell, N. Neural models for information retrieval. Preprint at arxiv: 1705.01509 (2017).","DOI":"10.1145\/3018661.3022755"},{"key":"437_CR25","doi-asserted-by":"publisher","first-page":"102067","DOI":"10.1016\/j.ipm.2019.102067","volume":"57","author":"J Guo","year":"2020","unstructured":"Guo, J. et al. A deep look into neural ranking models for information retrieval. Inf. Process. Manag. 57, 102067 (2020).","journal-title":"Inf. Process. Manag."},{"key":"437_CR26","unstructured":"Bendersky, M. et al. Rrf102: meeting the trec-covid challenge with a 100+ runs ensemble. Preprint at arxiv: 2010.00200 (2020)."},{"key":"437_CR27","doi-asserted-by":"crossref","unstructured":"Reimers, N. & Gurevych, I. Sentence-BERT: Sentence embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processingand the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3973\u20133983 (2019).","DOI":"10.18653\/v1\/D19-1410"},{"key":"437_CR28","doi-asserted-by":"publisher","unstructured":"Yang, P., Fang, H. & Lin, J. Anserini: Enabling the use of lucene for information retrieval research. In Proc. 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1253\u20131256. https:\/\/doi.org\/10.1145\/3077136.3080721 (2017).","DOI":"10.1145\/3077136.3080721"},{"key":"437_CR29","first-page":"2579","volume":"9","author":"Lvd Maaten","year":"2008","unstructured":"Maaten, L. v. d. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579\u20132605 (2008).","journal-title":"J. Mach. Learn. Res."},{"key":"437_CR30","unstructured":"Bia\u0142ecki, A., Muir, R., Ingersoll, G. & Imagination, L. Apache lucene 4. In SIGIR 2012 Workshop on Open Source Information Retrieval, Vol. 17. https:\/\/www.semanticscholar.org\/paper\/Apache-Lucene-4-Bialecki-Muir\/2795d9d165607b5ad6d8b9718373b82e55f41606 (2012)."},{"key":"437_CR31","unstructured":"Cormack, G. V., Clarke, C. L. & Buettcher, S. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In SIGIR 2009, 758\u2013759. https:\/\/www.semanticscholar.org\/paper\/Reciprocal-rank-fusion-outperforms-condorcet-and-Cormack-Clarke\/9e698010f9d8fa374e7f49f776af301dd200c548 (2009)."},{"key":"437_CR32","unstructured":"Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R. & Xiong, C. Learning to retrieve reasoning paths over wikipedia graph for question answering. In ICLR 2020. https:\/\/openreview.net\/forum?id=SJgVHkrYDH (2020)."},{"key":"437_CR33","doi-asserted-by":"crossref","unstructured":"Yang, Z. et al. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2369\u20132380 (Association for Computational Linguistics, Brussels, Belgium, 2018).","DOI":"10.18653\/v1\/D18-1259"},{"key":"437_CR34","doi-asserted-by":"crossref","unstructured":"Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019).","DOI":"10.18653\/v1\/D19-1259"},{"key":"437_CR35","doi-asserted-by":"crossref","unstructured":"Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016).","DOI":"10.18653\/v1\/D16-1264"},{"key":"437_CR36","first-page":"9","volume":"1","author":"A Radford","year":"2019","unstructured":"Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog 1, 9 (2019).","journal-title":"OpenAI Blog"},{"key":"437_CR37","doi-asserted-by":"crossref","unstructured":"Fan, A., Grangier, D. & Auli, M. Controllable Abstractive Summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, 45\u201354 (Association forComputational Linguistics, Melbourne, Australia, 2018).","DOI":"10.18653\/v1\/W18-2706"},{"key":"437_CR38","doi-asserted-by":"publisher","first-page":"416","DOI":"10.1007\/s10791-016-9282-6","volume":"19","author":"X Lu","year":"2016","unstructured":"Lu, X., Moffat, A. & Culpepper, J. S. The effect of pooling and evaluation depth on ir metrics. Inf. Retr. J. 19, 416\u2013445 (2016).","journal-title":"Inf. Retr. J."},{"key":"437_CR39","doi-asserted-by":"crossref","unstructured":"Ma, J., Korotkov, I., Yang, Y., Hall, K. & McDonald, R. Zero-shot neural retrieval via domain-targeted synthetic query generation. arXiv preprint arXiv:2004.14503 (2020).","DOI":"10.18653\/v1\/2021.eacl-main.92"},{"key":"437_CR40","unstructured":"Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. In (eds Bengio, Y. & LeCun, Y.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)."},{"key":"437_CR41","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"437_CR42","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3239571","volume":"10","author":"P Yang","year":"2018","unstructured":"Yang, P., Fang, H. & Lin, J. Anserini: reproducible ranking baselines using lucene. J. Data Inf. Qual. 10, 1\u201320 (2018).","journal-title":"J. Data Inf. Qual."},{"key":"437_CR43","unstructured":"Wolf, T. et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38\u201345 (Association for Computational Linguistics, Online, 2020)."},{"key":"437_CR44","unstructured":"Zhang, J., Zhao, Y., Saleh, M. & Liu, P. PEGASUS: Pre-training with extracted gap-sentences forabstractive summarization. In (eds III, H. D. & Singh, A.) Proceedings of the 37th International Conference on Machine Learning, vol. 119 of Proceedings of Machine Learning Research, 11328\u201311339 (PMLR, 2020)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00437-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00437-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00437-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T18:49:20Z","timestamp":1670093360000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00437-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,12]]},"references-count":44,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["437"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00437-0","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,12]]},"assertion":[{"value":"3 September 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 March 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 April 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"All authors were employees of Salesforce.com, Inc., at the time of manuscript preparation.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"68"}}