{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T07:22:51Z","timestamp":1780384971707,"version":"3.54.1"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,4,15]],"date-time":"2021-04-15T00:00:00Z","timestamp":1618444800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2021,4,15]],"date-time":"2021-04-15T00:00:00Z","timestamp":1618444800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Semantic categorization analysis of clinical trials eligibility criteria based on natural language processing technology is crucial for the task of optimizing clinical trials design and building automated patient recruitment system. However, most of related researches focused on English eligibility criteria, and to the best of our knowledge, there are no researches studied the Chinese eligibility criteria. Thus in this study, we aimed to explore the semantic categories of Chinese eligibility criteria.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We downloaded the clinical trials registration files from the website of Chinese Clinical Trial Registry (ChiCTR) and extracted both the Chinese eligibility criteria and corresponding English eligibility criteria. We represented the criteria sentences based on the Unified Medical Language System semantic types and conducted the hierarchical clustering algorithm for the induction of semantic categories. Furthermore, in order to explore the classification performance of Chinese eligibility criteria with our developed semantic categories, we implemented multiple classification algorithms, include four baseline machine learning algorithms (LR, NB, kNN, SVM), three deep learning algorithms (CNN, RNN, FastText) and two pre-trained language models (BERT, ERNIE).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>We totally developed 44 types of semantic categories, summarized 8 topic groups, and investigated the average incidence and prevalence in 272 hepatocellular carcinoma related Chinese clinical trials. Compared with the previous proposed categories in English eligibility criteria, 13 novel categories are identified in Chinese eligibility criteria. The classification result shows that most of semantic categories performed quite well, the pre-trained language model ERNIE achieved best performance with macro-average F1 score of 0.7980 and micro-average F1 score of 0.8484.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>As a pilot study of Chinese eligibility criteria analysis, we developed the 44 semantic categories by hierarchical clustering algorithms for the first times, and validated the classification capacity with multiple classification algorithms.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-021-01487-w","type":"journal-article","created":{"date-parts":[[2021,4,15]],"date-time":"2021-04-15T16:03:24Z","timestamp":1618502604000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Semantic categorization of Chinese eligibility criteria in clinical trials using machine learning methods"],"prefix":"10.1186","volume":"21","author":[{"given":"Hui","family":"Zong","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinxuan","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zeyu","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zuofeng","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaoyan","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,4,15]]},"reference":[{"issue":"20","key":"1487_CR1","doi-asserted-by":"publisher","first-page":"1998","DOI":"10.1056\/NEJMsr1611785","volume":"375","author":"DA Zarin","year":"2016","unstructured":"Zarin DA, et al. Trial Reporting in ClinicalTrials.gov\u2014the final rule. N Engl J Med. 2016;375(20):1998\u20132004.","journal-title":"N Engl J Med"},{"issue":"2","key":"1487_CR2","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1056\/NEJMsr1414226","volume":"372","author":"DA Zarin","year":"2015","unstructured":"Zarin DA, Tse T, Sheehan J. The proposed rule for U.S. clinical trial registration and results submission. N Engl J Med. 2015;372(2):174\u201380.","journal-title":"N Engl J Med"},{"issue":"33","key":"1487_CR3","doi-asserted-by":"publisher","first-page":"3737","DOI":"10.1200\/JCO.2017.73.7916","volume":"35","author":"ES Kim","year":"2017","unstructured":"Kim ES, et al. Broadening eligibility criteria to make clinical trials more representative: American Society of Clinical Oncology and Friends of Cancer Research Joint Research Statement. J Clin Oncol. 2017;35(33):3737\u201344.","journal-title":"J Clin Oncol"},{"issue":"3","key":"1487_CR4","doi-asserted-by":"publisher","first-page":"451","DOI":"10.1016\/j.jbi.2009.12.004","volume":"43","author":"C Weng","year":"2010","unstructured":"Weng C, et al. Formal representation of eligibility criteria: a literature review. J Biomed Inform. 2010;43(3):451\u201367.","journal-title":"J Biomed Inform"},{"key":"1487_CR5","unstructured":"http:\/\/clinicaltrials.gov\/."},{"issue":"4","key":"1487_CR6","doi-asserted-by":"publisher","first-page":"781","DOI":"10.1093\/jamia\/ocw176","volume":"24","author":"K Zhang","year":"2017","unstructured":"Zhang K, Demner-Fushman D. Automated classification of eligibility criteria in clinical trials to facilitate patient-trial matching for specific patient populations. J Am Med Inform Assoc. 2017;24(4):781\u20137.","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"1487_CR7","doi-asserted-by":"publisher","first-page":"1163","DOI":"10.1093\/jamia\/ocz163","volume":"26","author":"A Stubbs","year":"2019","unstructured":"Stubbs A, et al. Cohort selection for clinical trials: n2c2 2018 shared task track 1. J Am Med Inform Assoc. 2019;26(11):1163\u201371.","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"1487_CR8","doi-asserted-by":"publisher","first-page":"1247","DOI":"10.1093\/jamia\/ocz149","volume":"26","author":"M Oleynik","year":"2019","unstructured":"Oleynik M, et al. Evaluating shallow and deep learning strategies for the 2018 n2c2 shared task on clinical text classification. J Am Med Inform Assoc. 2019;26(11):1247\u201354.","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"1487_CR9","doi-asserted-by":"publisher","first-page":"675","DOI":"10.1111\/cts.12764","volume":"13","author":"Z He","year":"2020","unstructured":"He Z, et al. Clinical trial generalizability assessment in the big data era: a review. Clin Transl Sci. 2020;13(4):675\u201384.","journal-title":"Clin Transl Sci"},{"issue":"33","key":"1487_CR10","doi-asserted-by":"publisher","first-page":"3781","DOI":"10.1200\/JCO.2017.74.4144","volume":"35","author":"L Gore","year":"2017","unstructured":"Gore L, et al. Modernizing clinical trial eligibility: recommendations of the American Society of Clinical Oncology-Friends of Cancer Research Minimum Age Working Group. J Clin Oncol. 2017;35(33):3781\u20137.","journal-title":"J Clin Oncol"},{"issue":"33","key":"1487_CR11","doi-asserted-by":"publisher","first-page":"3774","DOI":"10.1200\/JCO.2017.73.7338","volume":"35","author":"TS Uldrick","year":"2017","unstructured":"Uldrick TS, et al. Modernizing clinical trial eligibility criteria: recommendations of the American Society of Clinical Oncology-Friends of Cancer Research HIV Working Group. J Clin Oncol. 2017;35(33):3774\u201380.","journal-title":"J Clin Oncol"},{"issue":"33","key":"1487_CR12","doi-asserted-by":"publisher","first-page":"3760","DOI":"10.1200\/JCO.2017.74.0761","volume":"35","author":"NU Lin","year":"2017","unstructured":"Lin NU, et al. Modernizing clinical trial eligibility criteria: recommendations of the American Society of Clinical Oncology-Friends of Cancer Research Brain Metastases Working Group. J Clin Oncol. 2017;35(33):3760\u201373.","journal-title":"J Clin Oncol"},{"issue":"33","key":"1487_CR13","doi-asserted-by":"publisher","first-page":"3753","DOI":"10.1200\/JCO.2017.74.4102","volume":"35","author":"SM Lichtman","year":"2017","unstructured":"Lichtman SM, et al. Modernizing clinical trial eligibility criteria: recommendations of the American Society of Clinical Oncology-Friends of Cancer Research Organ Dysfunction, Prior or Concurrent Malignancy, and Comorbidities Working Group. J Clin Oncol. 2017;35(33):3753\u20139.","journal-title":"J Clin Oncol"},{"key":"1487_CR14","unstructured":"Rubin DL, et al. Tool support for authoring eligibility criteria for cancer trials. In: Proceedings of AMIA symposium; 1999. p. 369\u201373."},{"key":"1487_CR15","first-page":"2970","volume":"2016","author":"H Zhe","year":"2016","unstructured":"Zhe H, et al. Assessing the population representativeness of colorectal cancer treatment clinical trials. Conf Proc IEEE Eng Med Biol Soc. 2016;2016:2970\u20133.","journal-title":"Conf Proc IEEE Eng Med Biol Soc"},{"issue":"11","key":"1487_CR16","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1001\/jama.297.11.1233","volume":"297","author":"HG Van Spall","year":"2007","unstructured":"Van Spall HG, et al. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. 2007;297(11):1233\u201340.","journal-title":"JAMA"},{"issue":"2","key":"1487_CR17","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1197\/jamia.M2556","volume":"15","author":"DB Fridsma","year":"2008","unstructured":"Fridsma DB, et al. The BRIDG project: a technical report. J Am Med Inform Assoc. 2008;15(2):130\u20137.","journal-title":"J Am Med Inform Assoc"},{"issue":"(Database issue","key":"1487_CR18","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","volume":"32","author":"O Bodenreider","year":"2004","unstructured":"Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32((Database issue)):D267\u201370.","journal-title":"Nucleic Acids Res"},{"key":"1487_CR19","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1016\/j.jbi.2014.01.009","volume":"52","author":"T Hao","year":"2014","unstructured":"Hao T, et al. Clustering clinical trials with similar eligibility criteria features. J Biomed Inform. 2014;52:112\u201320.","journal-title":"J Biomed Inform"},{"key":"1487_CR20","unstructured":"Aronson AR. Effective mapping of biomedical text to the UMLS metathesaurus: the MetaMap program. In: Proceedings AMIA Symposium; 2001. p. 17\u201321."},{"key":"1487_CR21","first-page":"487","volume":"2010","author":"Z Luo","year":"2010","unstructured":"Luo Z, Johnson SB, Weng C. Semi-automatically inducing semantic classes of clinical research eligibility criteria using UMLS and hierarchical clustering. AMIA Annu Symp Proc. 2010;2010:487\u201391.","journal-title":"AMIA Annu Symp Proc"},{"issue":"6","key":"1487_CR22","doi-asserted-by":"publisher","first-page":"927","DOI":"10.1016\/j.jbi.2011.06.001","volume":"44","author":"Z Luo","year":"2011","unstructured":"Luo Z, Yetisgen-Yildiz M, Weng C. Dynamic categorization of clinical research eligibility criteria by hierarchical clustering. J Biomed Inform. 2011;44(6):927\u201335.","journal-title":"J Biomed Inform"},{"issue":"7","key":"1487_CR23","doi-asserted-by":"publisher","first-page":"e17832","DOI":"10.2196\/17832","volume":"8","author":"K Zeng","year":"2020","unstructured":"Zeng K, et al. An ensemble learning strategy for eligibility criteria text classification for clinical trial recruitment: algorithm development and validation. JMIR Med Inform. 2020;8(7):e17832.","journal-title":"JMIR Med Inform"},{"key":"1487_CR24","unstructured":"The China Conference on Health Information Processing (CHIP) 2019; 2019. http:\/\/www.cips-chip.org.cn:8088\/home."},{"issue":"1","key":"1487_CR25","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1186\/s13584-019-0293-9","volume":"8","author":"Z Wang","year":"2019","unstructured":"Wang Z. Data integration of electronic medical record under administrative decentralization of medical insurance and healthcare in China: a case study. Isr J Health Policy Res. 2019;8(1):24.","journal-title":"Isr J Health Policy Res"},{"issue":"Suppl 1","key":"1487_CR26","doi-asserted-by":"publisher","first-page":"i116","DOI":"10.1136\/amiajnl-2011-000321","volume":"18","author":"C Weng","year":"2011","unstructured":"Weng C, et al. EliXR: an approach to eligibility criteria extraction and representation. J Am Med Inform Assoc. 2011;18(Suppl 1):i116\u201324.","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"1487_CR27","doi-asserted-by":"publisher","first-page":"1062","DOI":"10.1093\/jamia\/ocx019","volume":"24","author":"T Kang","year":"2017","unstructured":"Kang T, et al. EliIE: an open-source information extraction system for clinical trial eligibility criteria. J Am Med Inform Assoc. 2017;24(6):1062\u201371.","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"1487_CR28","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1093\/jamia\/ocy178","volume":"26","author":"C Yuan","year":"2019","unstructured":"Yuan C, et al. Criteria2Query: a natural language interface to clinical databases for cohort definition. J Am Med Inform Assoc. 2019;26(4):294\u2013305.","journal-title":"J Am Med Inform Assoc"},{"key":"1487_CR29","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1186\/s12911-015-0149-3","volume":"15","author":"Y Ni","year":"2015","unstructured":"Ni Y, et al. Increasing the efficiency of trial-patient matching: automated clinical trial eligibility pre-screening for pediatric oncology patients. BMC Med Inform Decis Mak. 2015;15:28.","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"1487_CR30","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1136\/amiajnl-2014-002887","volume":"22","author":"Y Ni","year":"2015","unstructured":"Ni Y, et al. Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department. J Am Med Inform Assoc. 2015;22(1):166\u201378.","journal-title":"J Am Med Inform Assoc"},{"key":"1487_CR31","unstructured":"Chinese Clinical Trials Registry. http:\/\/www.chictr.org.cn"},{"key":"1487_CR32","unstructured":"Scikit-learn: Machine Learning in Python. https:\/\/scikit-learn.org\/stable\/."},{"key":"1487_CR33","unstructured":"Xiao, H. bert-as-service. 2018. https:\/\/github.com\/hanxiao\/bert-as-service."},{"key":"1487_CR34","doi-asserted-by":"crossref","unstructured":"Kim Y. Convolutional neural networks for sentence classification. arXiv e-prints, 2014. arXiv:1408.5882.","DOI":"10.3115\/v1\/D14-1181"},{"key":"1487_CR35","doi-asserted-by":"crossref","unstructured":"Joulin A et al. Bag of tricks for efficient text classification. arXiv e-prints, 2016. arXiv:1607.01759.","DOI":"10.18653\/v1\/E17-2068"},{"key":"1487_CR36","unstructured":"Devlin J, et al. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv e-prints, 2018. arXiv:1810.04805."},{"key":"1487_CR37","unstructured":"Sun Y, et al. ERNIE: enhanced representation through knowledge integration. arXiv e-prints, 2019. arXiv:1904.09223."},{"key":"1487_CR38","doi-asserted-by":"publisher","first-page":"4915828","DOI":"10.1155\/2017\/4915828","volume":"2017","author":"Y Lei","year":"2017","unstructured":"Lei Y, et al. Interactive K-means clustering method based on user behavior for different analysis target in medicine. Comput Math Methods Med. 2017;2017:4915828.","journal-title":"Comput Math Methods Med"},{"issue":"6","key":"1487_CR39","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1016\/j.ijmedinf.2011.02.003","volume":"80","author":"M Cuggia","year":"2011","unstructured":"Cuggia M, Besana P, Glasspool D. Comparing semi-automatic systems for recruitment of patients to clinical trials. Int J Med Inform. 2011;80(6):371\u201388.","journal-title":"Int J Med Inform"},{"issue":"2","key":"1487_CR40","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1093\/jamiaopen\/ooy028","volume":"1","author":"RA Bush","year":"2018","unstructured":"Bush RA, et al. A systematic review of the use of the electronic health record for patient identification, communication, and clinical support in palliative care. JAMIA Open. 2018;1(2):294\u2013303.","journal-title":"JAMIA Open"},{"key":"1487_CR41","doi-asserted-by":"publisher","first-page":"134","DOI":"10.1186\/1472-6947-13-134","volume":"13","author":"F Kopcke","year":"2013","unstructured":"Kopcke F, et al. Evaluating predictive modeling algorithms to assess patient eligibility for clinical trials from routine data. BMC Med Inform Decis Mak. 2013;13:134.","journal-title":"BMC Med Inform Decis Mak"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01487-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-021-01487-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-021-01487-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,4,15]],"date-time":"2021-04-15T16:05:33Z","timestamp":1618502733000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-021-01487-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,15]]},"references-count":41,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["1487"],"URL":"https:\/\/doi.org\/10.1186\/s12911-021-01487-w","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,15]]},"assertion":[{"value":"20 December 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 April 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 April 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no conflicts of interest.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"128"}}