{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T22:42:23Z","timestamp":1780699343806,"version":"3.54.1"},"reference-count":31,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2018,4,28]],"date-time":"2018-04-28T00:00:00Z","timestamp":1524873600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/academic.oup.com\/journals\/pages\/about_us\/legal\/notices"}],"funder":[{"DOI":"10.13039\/100000092","name":"U.S. National Library of Medicine","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To automatically recognize self-acknowledged limitations in clinical research publications to support efforts in improving research transparency.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>To develop our recognition methods, we used a set of 8431 sentences from 1197 PubMed Central articles. A subset of these sentences was manually annotated for training\/testing, and inter-annotator agreement was calculated. We cast the recognition problem as a binary classification task, in which we determine whether a given sentence from a publication discusses self-acknowledged limitations or not. We experimented with three methods: a rule-based approach based on document structure, supervised machine learning, and a semi-supervised method that uses self-training to expand the training set in order to improve classification performance. The machine learning algorithms used were logistic regression (LR) and support vector machines (SVM).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Annotators had good agreement in labeling limitation sentences (Krippendorff\u2019s \u03b1\u2009=\u20090.781). Of the three methods used, the rule-based method yielded the best performance with 91.5% accuracy (95% CI [90.1-92.9]), while self-training with SVM led to a small improvement over fully supervised learning (89.9%, 95% CI [88.4-91.4] vs 89.6%, 95% CI [88.1-91.1]).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>The approach presented can be incorporated into the workflows of stakeholders focusing on research transparency to improve reporting of limitations in clinical studies.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocy038","type":"journal-article","created":{"date-parts":[[2018,3,28]],"date-time":"2018-03-28T19:16:11Z","timestamp":1522264571000},"page":"855-861","source":"Crossref","is-referenced-by-count":16,"title":["Automatic recognition of self-acknowledged limitations in clinical research literature"],"prefix":"10.1093","volume":"25","author":[{"given":"Halil","family":"Kilicoglu","sequence":"first","affiliation":[{"name":"Lister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, Bethesda, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Graciela","family":"Rosemblat","sequence":"additional","affiliation":[{"name":"Lister Hill National Center for Biomedical Communications, U.S. National Library of Medicine, Bethesda, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mario","family":"Mali\u010dki","sequence":"additional","affiliation":[{"name":"Department of General Practice, Academic Medical Center, Amsterdam, The Netherlands"},{"name":"Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gerben","family":"ter Riet","sequence":"additional","affiliation":[{"name":"Department of General Practice, Academic Medical Center, Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2018,4,28]]},"reference":[{"issue":"8","key":"2020110612364062400_ocy038-B1","doi-asserted-by":"crossref","first-page":"e124.","DOI":"10.1371\/journal.pmed.0020124","article-title":"Why most published research findings are false","volume":"2","author":"Ioannidis","year":"2005","journal-title":"PLoS Med"},{"issue":"9","key":"2020110612364062400_ocy038-B2","doi-asserted-by":"crossref","first-page":"e2002173.","DOI":"10.1371\/journal.pbio.2002173","article-title":"Spin\u2019in published biomedical literature: a methodological systematic review","volume":"15","author":"Chiu","year":"2017","journal-title":"PLoS Biol"},{"issue":"1","key":"2020110612364062400_ocy038-B3","doi-asserted-by":"crossref","first-page":"21.","DOI":"10.1038\/s41562-016-0021","article-title":"A manifesto for reproducible science","volume":"1","author":"Munaf\u00f2","year":"2017","journal-title":"Nat Hum Behav"},{"issue":"4","key":"2020110612364062400_ocy038-B4","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.jclinepi.2006.09.011","article-title":"Limitations are not properly acknowledged in the scientific literature","volume":"60","author":"Ioannidis","year":"2007","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2020110612364062400_ocy038-B5","doi-asserted-by":"crossref","first-page":"11","DOI":"10.7326\/0003-4819-121-1-199407010-00003","article-title":"Manuscript quality before and after peer review and editing at Annals of Internal Medicine","volume":"121","author":"Goodman","year":"1994","journal-title":"Ann Intern Med"},{"issue":"11","key":"2020110612364062400_ocy038-B6","doi-asserted-by":"crossref","first-page":"e73623","DOI":"10.1371\/journal.pone.0073623","article-title":"All that glitters isn\u2019t gold: a survey on acknowledgment of limitations in biomedical studies","volume":"8","author":"ter Riet","year":"2013","journal-title":"PLoS One"},{"key":"2020110612364062400_ocy038-B7","doi-asserted-by":"crossref","first-page":"c332.","DOI":"10.1136\/bmj.c332","article-title":"CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials","volume":"340","author":"Schulz","year":"2010","journal-title":"BMJ"},{"issue":"6","key":"2020110612364062400_ocy038-B8","doi-asserted-by":"crossref","first-page":"e1000412","DOI":"10.1371\/journal.pbio.1000412","article-title":"Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research","volume":"8","author":"Kilkenny","year":"2010","journal-title":"PLoS Biol"},{"issue":"3","key":"2020110612364062400_ocy038-B9","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1016\/j.jclinepi.2006.06.016","article-title":"Reporting in randomized clinical trials improved after adoption of the CONSORT statement","volume":"60","author":"Kane","year":"2007","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"2020110612364062400_ocy038-B10","doi-asserted-by":"crossref","first-page":"60.","DOI":"10.1186\/2046-4053-1-60","article-title":"Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review","volume":"1","author":"Turner","year":"2012","journal-title":"Syst Rev"},{"key":"2020110612364062400_ocy038-B11","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbx057","article-title":"Biomedical text mining for research rigor and integrity: tasks, challenges, directions","author":"Kilicoglu","year":"2017","journal-title":"Brief Bioinform"},{"issue":"1","key":"2020110612364062400_ocy038-B12","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1186\/2046-4053-4-5","article-title":"Using text mining for study identification in systematic reviews: a systematic review of current approaches","volume":"4","author":"O\u2019Mara-Eves","year":"2015","journal-title":"Syst Rev"},{"issue":"1","key":"2020110612364062400_ocy038-B13","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1162\/coli.2007.33.1.63","article-title":"Answering clinical questions with knowledge-based and statistical techniques","volume":"33","author":"Demner-Fushman","year":"2007","journal-title":"Comput Linguist"},{"issue":"Suppl 2","key":"2020110612364062400_ocy038-B14","doi-asserted-by":"crossref","first-page":"S5.","DOI":"10.1186\/1471-2105-12-S2-S5","article-title":"Automatic classification of sentences to support Evidence Based Medicine","volume":"12","author":"Kim","year":"2011","journal-title":"BMC Bioinformatics"},{"key":"2020110612364062400_ocy038-B15","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1016\/j.jbi.2014.02.006","article-title":"Identifying scientific artefacts in biomedical literature: the Evidence Based Medicine use case","volume":"49","author":"Hassanzadeh","year":"2014","journal-title":"J Biomed Inform"},{"issue":"132","key":"2020110612364062400_ocy038-B16","first-page":"1","article-title":"Extracting PICO sentences from clinical trial reports using supervised distant supervision","volume":"17","author":"Wallace","year":"2016","journal-title":"J Mach Learn Res"},{"issue":"1","key":"2020110612364062400_ocy038-B17","doi-asserted-by":"crossref","first-page":"56.","DOI":"10.1186\/1472-6947-10-56","article-title":"ExaCT: automatic extraction of clinical trial characteristics from journal publications","volume":"10","author":"Kiritchenko","year":"2010","journal-title":"BMC Med Inform Decis Mak"},{"issue":"23","key":"2020110612364062400_ocy038-B18","doi-asserted-by":"crossref","first-page":"3306","DOI":"10.1093\/bioinformatics\/btr573","article-title":"Extraction of data deposition statements from the literature","volume":"27","author":"N\u00e9v\u00e9ol","year":"2011","journal-title":"Bioinformatics"},{"issue":"04","key":"2020110612364062400_ocy038-B19","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1055\/s-0038-1634945","article-title":"The unified medical language system","volume":"32","author":"Lindberg","year":"1993","journal-title":"Methods Inf Med"},{"issue":"1","key":"2020110612364062400_ocy038-B20","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1093\/jamia\/ocv044","article-title":"RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials","volume":"23","author":"Marshall","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2020110612364062400_ocy038-B21","author":"Teufel","year":"1999"},{"key":"2020110612364062400_ocy038-B22","author":"Teufel","year":"2009"},{"issue":"23","key":"2020110612364062400_ocy038-B23","doi-asserted-by":"crossref","first-page":"3174","DOI":"10.1093\/bioinformatics\/btp548","article-title":"Automatically classifying sentences in full-text biomedical articles into Introduction, Methods, Results and Discussion","volume":"25","author":"Agarwal","year":"2009","journal-title":"Bioinformatics"},{"key":"2020110612364062400_ocy038-B24","author":"Liakata","year":"2010"},{"key":"2020110612364062400_ocy038-B25","volume-title":"The Structure of Scientific Articles: Applications to Citation Indexing and Summarization","author":"Teufel","year":"2010"},{"issue":"3","key":"2020110612364062400_ocy038-B26","doi-asserted-by":"crossref","first-page":"e0148538.","DOI":"10.1371\/journal.pone.0148538","article-title":"Bio-SCoRes: a smorgasbord architecture for coreference resolution in biomedical text","volume":"11","author":"Kilicoglu","year":"2016","journal-title":"PLoS One"},{"key":"2020110612364062400_ocy038-B27","author":"Manning","year":"2014"},{"key":"2020110612364062400_ocy038-B28","first-page":"1530","volume-title":"Semi-Supervised Learning Literature Survey","author":"Zhu","year":"2005"},{"key":"2020110612364062400_ocy038-B29","first-page":"1871","article-title":"LIBLINEAR: a library for large linear classification","volume":"9","author":"Fan","year":"2008","journal-title":"J Mach Learn Res"},{"key":"2020110612364062400_ocy038-B30","author":"Zhou","year":"2012"},{"key":"2020110612364062400_ocy038-B31","volume-title":"Representation Learning: A Review and New Perspectives","author":"Bengio","year":"2012"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/7\/855\/34150197\/ocy038.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/7\/855\/34150197\/ocy038.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T17:45:13Z","timestamp":1604684713000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/25\/7\/855\/4990607"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,4,28]]},"references-count":31,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2018,4,28]]},"published-print":{"date-parts":[[2018,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocy038","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2018,7]]},"published":{"date-parts":[[2018,4,28]]}}}