{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T04:30:15Z","timestamp":1776054615389,"version":"3.50.1"},"reference-count":12,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2023,1,17]],"date-time":"2023-01-17T00:00:00Z","timestamp":1673913600000},"content-version":"vor","delay-in-days":16,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["MRI 1625061"],"award-info":[{"award-number":["MRI 1625061"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DBI 1932765"],"award-info":[{"award-number":["DBI 1932765"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006754","name":"US Army Research Laboratory","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006754","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Motivation<\/jats:title><jats:p>Timetrees depict evolutionary relationships between species and the geological times of their divergence. Hundreds of research articles containing timetrees are published in scientific journals every year. The TimeTree (TT) project has been manually locating, curating and synthesizing timetrees from these articles for almost two decades into a TimeTree of Life, delivered through a unique, user-friendly web interface (timetree.org). The manual process of finding articles containing timetrees is becoming increasingly expensive and time-consuming. So, we have explored the effectiveness of text-mining approaches and developed optimizations to find research articles containing timetrees automatically.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>We have developed an optimized machine learning system to determine if a research article contains an evolutionary timetree appropriate for inclusion in the TT resource. We found that BERT classification fine-tuned on whole-text articles achieved an F1 score of 0.67, which we increased to 0.88 by text-mining article excerpts surrounding the mentioning of figures. The new method is implemented in the TimeTreeFinder (TTF) tool, which automatically processes millions of articles to discover timetree-containing articles. We estimate that the TTF tool would produce twice as many timetree-containing articles as those discovered manually, whose inclusion in the TT database would potentially double the knowledge accessible to a wider community. Manual inspection showed that the precision on out-of-distribution recently published articles is 87%. This automation will speed up the collection and curation of timetrees with much lower human and time costs.<\/jats:p><\/jats:sec><jats:sec><jats:title>Availability and implementation<\/jats:title><jats:p>https:\/\/github.com\/marija-stanojevic\/time-tree-classification.<\/jats:p><\/jats:sec><jats:sec><jats:title>Supplementary information<\/jats:title><jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p><\/jats:sec>","DOI":"10.1093\/bioinformatics\/btad035","type":"journal-article","created":{"date-parts":[[2023,1,16]],"date-time":"2023-01-16T20:34:01Z","timestamp":1673901241000},"source":"Crossref","is-referenced-by-count":3,"title":["Discovering research articles containing evolutionary timetrees by machine learning"],"prefix":"10.1093","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8227-6577","authenticated-orcid":false,"given":"Marija","family":"Stanojevic","sequence":"first","affiliation":[{"name":"Center for Data Analytics and Biomedical Informatics, Computer and Information Science Department, Temple University , Philadelphia, PA 19121, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jovan","family":"Andjelkovic","sequence":"additional","affiliation":[{"name":"Center for Data Analytics and Biomedical Informatics, Computer and Information Science Department, Temple University , Philadelphia, PA 19121, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adrienne","family":"Kasprowicz","sequence":"additional","affiliation":[{"name":"Department of Biology, Temple University , Philadelphia, PA 19121, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Louise A","family":"Huuki","sequence":"additional","affiliation":[{"name":"Department of Biology, Temple University , Philadelphia, PA 19121, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer","family":"Chao","sequence":"additional","affiliation":[{"name":"Center for Data Analytics and Biomedical Informatics, Computer and Information Science Department, Temple University , Philadelphia, PA 19121, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"S Blair","family":"Hedges","sequence":"additional","affiliation":[{"name":"Department of Biology, Temple University , Philadelphia, PA 19121, USA"},{"name":"Institute for Genomics and Evolutionary Medicine, Temple University , Philadelphia, PA 19121, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9918-8212","authenticated-orcid":false,"given":"Sudhir","family":"Kumar","sequence":"additional","affiliation":[{"name":"Department of Biology, Temple University , Philadelphia, PA 19121, USA"},{"name":"Institute for Genomics and Evolutionary Medicine, Temple University , Philadelphia, PA 19121, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zoran","family":"Obradovic","sequence":"additional","affiliation":[{"name":"Center for Data Analytics and Biomedical Informatics, Computer and Information Science Department, Temple University , Philadelphia, PA 19121, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2023,1,17]]},"reference":[{"key":"2023013106121732200_btad035-B1","first-page":"3615","author":"Beltagy","year":"2019"},{"key":"2023013106121732200_btad035-B2","author":"Devlin","year":"2018"},{"key":"2023013106121732200_btad035-B3","doi-asserted-by":"crossref","first-page":"2971","DOI":"10.1093\/bioinformatics\/btl505","article-title":"TimeTree: a public knowledge-base of divergence times among organisms","volume":"22","author":"Hedges","year":"2006","journal-title":"Bioinformatics"},{"key":"2023013106121732200_btad035-B4","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1108\/eb026526","article-title":"A statistical interpretation of term specificity and its application in retrieval","volume":"28","author":"Jones","year":"1972","journal-title":"J. Doc"},{"key":"2023013106121732200_btad035-B5","doi-asserted-by":"crossref","DOI":"10.1093\/molbev\/msac174","article-title":"TimeTree 5: an expanded resource for species divergence times","volume":"39","author":"Kumar","year":"2022","journal-title":"Mol. Biol. Evol"},{"key":"2023013106121732200_btad035-B6","first-page":"1188","volume-title":"International Conference on Machine Learning","author":"Le","year":"2014"},{"key":"2023013106121732200_btad035-B7","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2023013106121732200_btad035-B8","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1147\/rd.14.0309","article-title":"A statistical approach to mechanized encoding and searching of literary information","volume":"1","author":"Luhn","year":"1957","journal-title":"IBM J. Res. Dev"},{"key":"2023013106121732200_btad035-B9","author":"Mikolov","year":"2013"},{"key":"2023013106121732200_btad035-B10","article-title":"Distributed representations of words and phrases and their compositionality","volume":"26","author":"Mikolov","year":"2013","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"2023013106121732200_btad035-B11","author":"Sanh","year":"2019"},{"key":"2023013106121732200_btad035-B12","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J. R. Stat. Soc. Ser. B Methodol"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btad035\/48745036\/btad035.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/39\/1\/btad035\/48959142\/btad035.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/39\/1\/btad035\/48959142\/btad035.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,12]],"date-time":"2024-10-12T09:37:52Z","timestamp":1728725872000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btad035\/6989626"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2023,1,1]]},"references-count":12,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btad035","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,1,1]]},"published":{"date-parts":[[2023,1,1]]},"article-number":"btad035"}}