{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,1]],"date-time":"2026-01-01T14:07:15Z","timestamp":1767276435434},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2016,10,2]],"date-time":"2016-10-02T00:00:00Z","timestamp":1475366400000},"content-version":"vor","delay-in-days":480,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2015,6,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: Medical Subject Headings (MeSHs) are used by National Library of Medicine (NLM) to index almost all citations in MEDLINE, which greatly facilitates the applications of biomedical information retrieval and text mining. To reduce the time and financial cost of manual annotation, NLM has developed a software package, Medical Text Indexer (MTI), for assisting MeSH annotation, which uses k-nearest neighbors (KNN), pattern matching and indexing rules. Other types of information, such as prediction by MeSH classifiers (trained separately), can also be used for automatic MeSH annotation. However, existing methods cannot effectively integrate multiple evidence for MeSH annotation.<\/jats:p>\n               <jats:p>Methods: We propose a novel framework, MeSHLabeler, to integrate multiple evidence for accurate MeSH annotation by using \u2018learning to rank\u2019. Evidence includes numerous predictions from MeSH classifiers, KNN, pattern matching, MTI and the correlation between different MeSH terms, etc. Each MeSH classifier is trained independently, and thus prediction scores from different classifiers are incomparable. To address this issue, we have developed an effective score normalization procedure to improve the prediction accuracy.<\/jats:p>\n               <jats:p>Results: MeSHLabeler won the first place in Task 2A of 2014 BioASQ challenge, achieving the Micro F-measure of 0.6248 for 9,040 citations provided by the BioASQ challenge. Note that this accuracy is around 9.15% higher than 0.5724, obtained by MTI.<\/jats:p>\n               <jats:p>Availability and implementation: The software is available upon request.<\/jats:p>\n               <jats:p>Contact: \u00a0zhusf@fudan.edu.cn<\/jats:p>","DOI":"10.1093\/bioinformatics\/btv237","type":"journal-article","created":{"date-parts":[[2015,6,13]],"date-time":"2015-06-13T17:12:36Z","timestamp":1434215556000},"page":"i339-i347","source":"Crossref","is-referenced-by-count":56,"title":["MeSHLabeler: improving the accuracy of large-scale MeSH indexing by integrating diverse evidence"],"prefix":"10.1093","volume":"31","author":[{"given":"Ke","family":"Liu","sequence":"first","affiliation":[{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"},{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shengwen","family":"Peng","sequence":"additional","affiliation":[{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"},{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junqiu","family":"Wu","sequence":"additional","affiliation":[{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chengxiang","family":"Zhai","sequence":"additional","affiliation":[{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hiroshi","family":"Mamitsuka","sequence":"additional","affiliation":[{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shanfeng","family":"Zhu","sequence":"additional","affiliation":[{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"},{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"},{"name":"1 School of Computer Science and 2Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai, China, 3School of Information Science & Engineering, Central South University, Changsha 410083, China, 4Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA, 5Bioinformatics Center, Institute for Chemical Research, Kyoto University, Uji 611-0011, Japan and 6Center for Computational System Biology, Fudan University, Shanghai 200433, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2015,6,10]]},"reference":[{"key":"2023020115413452900_btv237-B1","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2004","journal-title":"J. Am. Med. Inform. Assoc."},{"issue":"Pt 1","key":"2023020115413452900_btv237-B2","first-page":"268","article-title":"The NLM indexing initiative\u2019s medical text indexer","volume":"107","author":"Aronson","year":"2004","journal-title":"Stud. Health Technol. Inform."},{"key":"2023020115413452900_btv237-B3","first-page":"1181","article-title":"Results of the BioASQ track of the question answering lab at CLEF 2014","author":"Balikas","year":"2014","journal-title":"Working Notes for CLEF 2014 Conference, Sheffield, UK, September 1518, 2014. CEUR Workshop Proceedings 1180, CEUR-WS.org 2014"},{"key":"2023020115413452900_btv237-B4","article-title":"From RankNet to LambdaRank to LambdaMART: an overview. Microsoft Research","author":"Burges","year":"2010","journal-title":"Technical Report."},{"key":"2023020115413452900_btv237-B5","doi-asserted-by":"crossref","DOI":"10.1145\/1961189.1961199","article-title":"LIBSVM: a library for support vector machines","volume":"2","author":"Chang","year":"2011","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"2023020115413452900_btv237-B6","first-page":"1871","article-title":"Liblinear: a library for large linear classification","volume":"9","author":"Fan","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"2023020115413452900_btv237-B7","doi-asserted-by":"crossref","first-page":"1265","DOI":"10.1109\/TSMCB.2012.2227998","article-title":"Efficient semi-supervised MEDLINE document clustering with MeSH semantic and global content constraints","volume":"43","author":"Gu","year":"2013","journal-title":"IEEE Trans. Cybern."},{"key":"2023020115413452900_btv237-B8","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1136\/amiajnl-2010-000055","article-title":"Recommending mesh terms for annotating biomedical articles","volume":"18","author":"Huang","year":"2011","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"2023020115413452900_btv237-B9","doi-asserted-by":"crossref","first-page":"2293","DOI":"10.1016\/j.ins.2011.01.029","article-title":"Enhanced clustering of biomedical documents using ensemble non-negative matrix factorization","volume":"181","author":"Huang","year":"2011","journal-title":"Inf. Sci."},{"key":"2023020115413452900_btv237-B10","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1007\/s10791-007-9027-7","article-title":"An empirical study of tokenization strategies for biomedical information retrieval","volume":"10","author":"Jiang","year":"2007","journal-title":"Inf. Retrieval"},{"key":"2023020115413452900_btv237-B11","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1145\/2110363.2110450","article-title":"MEDLINE MeSH indexing: lessons learned from machine learning and future directions","volume-title":"Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium","author":"Jimeno-Yepes","year":"2012"},{"key":"2023020115413452900_btv237-B12","doi-asserted-by":"crossref","first-page":"151","DOI":"10.5626\/JCSE.2012.6.2.151","article-title":"A one-size-fits-all indexing method does not exist: automatic selection based on meta-learning","volume":"6","author":"Jimeno-Yepes","year":"2012","journal-title":"JCSE"},{"key":"2023020115413452900_btv237-B13","first-page":"709","article-title":"Comparison and combination of several mesh indexing approaches","volume-title":"AMIA Annual Symposium Proceedings","author":"Jimeno-Yepes","year":"2013"},{"key":"2023020115413452900_btv237-B14","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1186\/1471-2105-14-208","article-title":"MeSH indexing based on automatically generated summaries","volume":"14","author":"Jimeno-Yepes","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023020115413452900_btv237-B15","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1186\/1471-2105-8-423","article-title":"Pubmed related articles: a probabilistic topic-based model for content similarity","volume":"8","author":"Lin","year":"2007","journal-title":"BMC Bioinformatics"},{"key":"2023020115413452900_btv237-B16","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-14267-3","volume-title":"Learning to Rank for Information Retrieval","author":"Liu","year":"2011"},{"key":"2023020115413452900_btv237-B17","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1007\/s10791-008-9074-8","article-title":"Evaluation of query expansion using MeSH in PubMed","volume":"12","author":"Lu","year":"2010","journal-title":"Inf. Retrieval"},{"key":"2023020115413452900_btv237-B18","article-title":"NCBI at the 2013 BioASQ challenge task: learning to rank for automatic MeSH indexing","author":"Mao","year":"2013","journal-title":"Microsoft Research Technical Report MSR-TR-2010-82."},{"key":"2023020115413452900_btv237-B19","author":"Mao","year":"2014"},{"key":"2023020115413452900_btv237-B20","author":"Mork","year":"2013"},{"key":"2023020115413452900_btv237-B21","author":"Mork","year":"2014"},{"issue":"Database issue","key":"2023020115413452900_btv237-B22","doi-asserted-by":"crossref","first-page":"D6","DOI":"10.1093\/nar\/gku1130","article-title":"Database resources of the national center for biotechnology information","volume":"43","author":"NCBI Resource Coordinators","year":"2015","journal-title":"Nucleic Acids Res."},{"issue":"Pt 1","key":"2023020115413452900_btv237-B23","first-page":"67","article-title":"The MeSH translation maintenance system: structure, interface design, and implementation","volume":"11","author":"Nelson","year":"2004","journal-title":"Stud. Health Technol. Inform."},{"key":"2023020115413452900_btv237-B24","author":"Partalas","year":"2013"},{"key":"2023020115413452900_btv237-B25","doi-asserted-by":"crossref","first-page":"658","DOI":"10.1093\/bioinformatics\/bti783","article-title":"Automatic assignment of biomedical categories: toward a generic approach","volume":"22","author":"Ruch","year":"2006","journal-title":"Bioinformatics"},{"key":"2023020115413452900_btv237-B26","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/s10791-008-9073-9","article-title":"Exploring criteria for successful query expansion in the genomic domain","volume":"12","author":"Stokes","year":"2010","journal-title":"Inf. Retrieval"},{"key":"2023020115413452900_btv237-B27","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1145\/1526709.1526738","article-title":"Large scale multi-label classification via metalabeler","volume-title":"Proceedings of the 18th international conference on World wide web","author":"Tang","year":"2009"},{"key":"2023020115413452900_btv237-B28","doi-asserted-by":"crossref","first-page":"1412","DOI":"10.1093\/bioinformatics\/btp249","article-title":"MeSH Up: effective MeSH text classification for improved document retrieval","volume":"25","author":"Trieschnigg","year":"2009","journal-title":"Bioinformatics"},{"key":"2023020115413452900_btv237-B29","author":"Tsoumakas","year":"2013"},{"key":"2023020115413452900_btv237-B30","doi-asserted-by":"crossref","first-page":"1819","DOI":"10.1109\/TKDE.2013.39","article-title":"A review on multi-label learning algorithms","volume":"26","author":"Zhang","year":"2014","journal-title":"IEEE Trans. Knowl.Data Eng."},{"key":"2023020115413452900_btv237-B31","doi-asserted-by":"crossref","first-page":"1944","DOI":"10.1093\/bioinformatics\/btp338","article-title":"Enhancing MEDLINE document clustering by incorporating mesh semantic similarity","volume":"25","author":"Zhu","year":"2009","journal-title":"Bioinformatics"},{"key":"2023020115413452900_btv237-B32","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1016\/j.ipm.2009.03.005","article-title":"Field independent probabilistic model for clustering multi-field documents","volume":"45","author":"Zhu","year":"2009","journal-title":"Inf. Process. Manage."}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/31\/12\/i339\/49014367\/bioinformatics_31_12_i339.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/31\/12\/i339\/49014367\/bioinformatics_31_12_i339.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T00:06:32Z","timestamp":1675296392000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/31\/12\/i339\/215654"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,6,10]]},"references-count":32,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2015,6,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btv237","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2015,6,15]]},"published":{"date-parts":[[2015,6,10]]}}}