{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T11:57:26Z","timestamp":1774526246417,"version":"3.50.1"},"reference-count":66,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,4,23]],"date-time":"2024-04-23T00:00:00Z","timestamp":1713830400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,23]],"date-time":"2024-04-23T00:00:00Z","timestamp":1713830400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Federal Ministry of Health (BMG), Germany","award":["KINBIOTICS"],"award-info":[{"award-number":["KINBIOTICS"]}]},{"name":"Ministry of Culture and Science of the State of North Rhine-Westphalia, Germany","award":["NW21-059A (SAIL)"],"award-info":[{"award-number":["NW21-059A (SAIL)"]}]},{"name":"Ministry of Culture and Science of the State of North Rhine-Westphalia, Germany","award":["NW21-059A (SAIL)"],"award-info":[{"award-number":["NW21-059A (SAIL)"]}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["Transregional Collaborative Research Centre TRR 318 \"Constructing Explainability\" (Projects B1 and C5)"],"award-info":[{"award-number":["Transregional Collaborative Research Centre TRR 318 \"Constructing Explainability\" (Projects B1 and C5)"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005721","name":"Universit\u00e4t Bielefeld","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005721","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Biomed Semant"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Systematic reviews of Randomized Controlled Trials (RCTs) are an important part of the evidence-based medicine paradigm. However, the creation of such systematic reviews by clinical experts is costly as well as time-consuming, and results can get quickly outdated after publication. Most RCTs are structured based on the Patient, Intervention, Comparison, Outcomes (PICO) framework and there exist many approaches which aim to extract PICO elements automatically. The automatic extraction of PICO information from RCTs has the potential to significantly speed up the creation process of systematic reviews and this way also benefit the field of evidence-based medicine.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Previous work has addressed the extraction of PICO elements as the task of identifying relevant text spans or sentences, but without populating a structured representation of a trial. In contrast, in this work, we treat PICO elements as structured templates with slots to do justice to the complex nature of the information they represent. We present two different approaches to extract this structured information from the abstracts of RCTs. The first approach is an extractive approach based on our previous work that is extended to capture full document representations as well as by a clustering step to infer the number of instances of each template type. The second approach is a generative approach based on a seq2seq model that encodes the abstract describing the RCT and uses a decoder to infer a structured representation of a trial including its arms, treatments, endpoints and outcomes. Both approaches are evaluated with different base models on a manually annotated dataset consisting of RCT abstracts on an existing dataset comprising 211 annotated clinical trial abstracts for Type 2 Diabetes and Glaucoma. For both diseases, the extractive approach (with ) reached the best <jats:inline-formula><jats:alternatives><jats:tex-math>$$F_1$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:msub>\n                      <mml:mi>F<\/mml:mi>\n                      <mml:mn>1<\/mml:mn>\n                    <\/mml:msub>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula> score, i.e. 0.547 (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\pm 0.006$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mrow>\n                      <mml:mo>\u00b1<\/mml:mo>\n                      <mml:mn>0.006<\/mml:mn>\n                    <\/mml:mrow>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula>) for type 2 diabetes and 0.636 (<jats:inline-formula><jats:alternatives><jats:tex-math>$$\\pm 0.006$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mrow>\n                      <mml:mo>\u00b1<\/mml:mo>\n                      <mml:mn>0.006<\/mml:mn>\n                    <\/mml:mrow>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula>) for glaucoma. Generally, the <jats:inline-formula><jats:alternatives><jats:tex-math>$$F_1$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:msub>\n                      <mml:mi>F<\/mml:mi>\n                      <mml:mn>1<\/mml:mn>\n                    <\/mml:msub>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula> scores were higher for glaucoma than for type 2 diabetes and the standard deviation was higher for the generative approach.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>In our experiments, both approaches show promising performance extracting structured PICO information from RCTs, especially considering that most related work focuses on the far easier task of predicting less structured objects. In our experimental results, the extractive approach performs best in both cases, although the lead is greater for glaucoma than for type 2 diabetes. For future work, it remains to be investigated how the base model size affects the performance of both approaches in comparison. Although the extractive approach currently leaves more room for direct improvements, the generative approach might benefit from larger models.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s13326-024-00305-2","type":"journal-article","created":{"date-parts":[[2024,4,23]],"date-time":"2024-04-23T11:02:17Z","timestamp":1713870137000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Comparing generative and extractive approaches to information extraction from abstracts describing randomized clinical trials"],"prefix":"10.1186","volume":"15","author":[{"given":"Christian","family":"Witte","sequence":"first","affiliation":[]},{"given":"David M.","family":"Schmidt","sequence":"additional","affiliation":[]},{"given":"Philipp","family":"Cimiano","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,4,23]]},"reference":[{"issue":"9","key":"305_CR1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1000326","volume":"7","author":"H Bastian","year":"2010","unstructured":"Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9): e1000326. https:\/\/doi.org\/10.1371\/journal.pmed.1000326.","journal-title":"PLoS Med."},{"issue":"7050","key":"305_CR2","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1136\/bmj.313.7050.170c","volume":"313","author":"DL Sackett","year":"1996","unstructured":"Sackett DL, Rosenberg WM, Gray JM, Haynes RB, Richardson WS. Evidence based medicine. BMJ. 1996;313(7050):170.","journal-title":"BMJ."},{"issue":"12","key":"305_CR3","doi-asserted-by":"publisher","first-page":"1461","DOI":"10.1001\/jama.286.12.1461","volume":"286","author":"PG Shekelle","year":"2001","unstructured":"Shekelle PG, Ortiz E, Rhodes S, Morton SC, Eccles MP, Grimshaw JM, et al. Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA. 2001;286(12):1461\u20137.","journal-title":"JAMA."},{"issue":"4","key":"305_CR4","doi-asserted-by":"publisher","first-page":"224","DOI":"10.7326\/0003-4819-147-4-200708210-00179","volume":"147","author":"KG Shojania","year":"2007","unstructured":"Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis Ann Intern Med. 2007;147(4):224\u201333.","journal-title":"A survival analysis Ann Intern Med."},{"key":"305_CR5","doi-asserted-by":"publisher","unstructured":"Beller EM, Chen JKH, Wang ULH, Glasziou PP. Are systematic reviews up-to-date at the time of publication? Syst Rev. 2013;2:36. https:\/\/doi.org\/10.1186\/2046-4053-2-36.","DOI":"10.1186\/2046-4053-2-36"},{"key":"305_CR6","unstructured":"Koch G. No improvement\u2013still less than half of the Cochrane reviews are up to date. In: XIV Cochrane Colloquium. Dublin; 2006."},{"key":"305_CR7","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1186\/2046-4053-3-74","volume":"3","author":"G Tsafnat","year":"2014","unstructured":"Tsafnat G, Glasziou P, Choong MK, et al. Systematic review automation technologies Syst Rev. 2014;3:74. https:\/\/doi.org\/10.1186\/2046-4053-3-74.","journal-title":"Systematic review automation technologies. Syst Rev."},{"key":"305_CR8","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1186\/s13643-018-0740-7","volume":"7","author":"E Beller","year":"2018","unstructured":"Beller E, Clark J, Tsafnat G, et al. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7:77. https:\/\/doi.org\/10.1186\/s13643-018-0740-7.","journal-title":"Syst Rev."},{"key":"305_CR9","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1186\/s13643-019-0975-y","volume":"8","author":"AM O\u2019Connor","year":"2019","unstructured":"O\u2019Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Shemilt I, Thomas J, et al. Still moving toward automation of the systematic review process: a summary of discussions at the third meeting of the International Collaboration for Automation of Systematic Reviews (ICASR). Syst Rev. 2019;8:57. https:\/\/doi.org\/10.1186\/s13643-019-0975-y.","journal-title":"Syst Rev."},{"issue":"1","key":"305_CR10","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1186\/s13326-022-00270-8","volume":"13","author":"O Sanchez-Graillet","year":"2022","unstructured":"Sanchez-Graillet O, Witte C, Grimm F, Grautoff S, Ell B, Cimiano P. Synthesizing evidence from clinical trials with dynamic interactive argument trees. J Biomed Semant. 2022;13(1):16. https:\/\/doi.org\/10.1186\/s13326-022-00270-8.","journal-title":"J Biomed Semant."},{"issue":"1","key":"305_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1472-6947-10-29","volume":"10","author":"F Boudin","year":"2010","unstructured":"Boudin F, Nie JY, Bartlett JC, Grad R, Pluye P, Dawes M. Combining classifiers for robust PICO element detection. BMC Med Inform Decis Mak. 2010;10(1):1\u20136.","journal-title":"BMC Med Inform Decis Mak."},{"key":"305_CR12","doi-asserted-by":"publisher","unstructured":"Jin D, Szolovits P. Pico. Element detection in medical text via long short-term memory neural networks. In: Proceedings of the BioNLP 2018 workshop. Melbourne: Association for Computational Linguistics; 2018. p. 67\u201375.\u00a0https:\/\/aclanthology.org\/W18-2308. https:\/\/doi.org\/10.18653\/v1\/W18-2308.","DOI":"10.18653\/v1\/W18-2308"},{"key":"305_CR13","unstructured":"Trenta A, Hunter A, Riedel S. Extraction of evidence tables from abstracts of randomized clinical trials using a maximum entropy classifier and global constraints. 2015. arXiv preprint arXiv:1509.05209."},{"key":"305_CR14","doi-asserted-by":"publisher","unstructured":"Witte C, Cimiano P. Intra-Template Entity Compatibility based Slot-Filling for Clinical Trial Information Extraction. In: Proceedings of the 21st Workshop on Biomedical Language Processing. Dublin: Association for Computational Linguistics; 2022. p. 178\u2013192.\u00a0https:\/\/aclanthology.org\/2022.bionlp-1.18. https:\/\/doi.org\/10.18653\/v1\/2022.bionlp-1.18.","DOI":"10.18653\/v1\/2022.bionlp-1.18"},{"key":"305_CR15","unstructured":"Sanchez-Graillet O, Cimiano P, Witte C, Ell B. C-TrO: An Ontology for Summarization and Aggregation of the Level of Evidence in Clinical Trials. In: Proc. of the 5th Joint Ontology Workshops (JOWO): Ontologies and Data in the Life Sciences. 2019.\u00a0https:\/\/ceur-ws.org\/Vol-2518\/paper-ODLS7.pdf."},{"key":"305_CR16","unstructured":"Beltagy I, Peters ME, Cohan A. Longformer: The long-document transformer. 2020. arXiv preprint arXiv:2004.05150."},{"key":"305_CR17","doi-asserted-by":"publisher","unstructured":"Chung HW, Hou L, Longpre S, Zoph B, Tay Y, Fedus W, et\u00a0al. Scaling Instruction-Finetuned Language Models. CoRR. 2022. https:\/\/doi.org\/10.48550\/ARXIV.2210.11416. arXiv:2210.11416","DOI":"10.48550\/ARXIV.2210.11416"},{"key":"305_CR18","doi-asserted-by":"publisher","unstructured":"Cabot PLH, Navigli R. REBEL: Relation extraction by end-to-end language generation. In: Findings of the Association for Computational Linguistics: EMNLP 2021.\u00a0Punta Cana: Association for Computational Linguistics; 2021. p. 2370\u20132381. https:\/\/aclanthology.org\/2021.findings-emnlp.204.https:\/\/doi.org\/10.18653\/v1\/2021.findings-emnlp.204.","DOI":"10.18653\/v1\/2021.findings-emnlp.204"},{"key":"305_CR19","doi-asserted-by":"crossref","unstructured":"Josifoski M, De\u00a0Cao N, Peyrard M, West R. GenIE: generative information extraction. 2021. arXiv preprint arXiv:2112.08340.","DOI":"10.18653\/v1\/2022.naacl-main.342"},{"key":"305_CR20","doi-asserted-by":"crossref","unstructured":"Lu Y, Lin H, Xu J, Han X, Tang J, Li A, et\u00a0al. Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction. CoRR. 2021. arXiv:2106.09232.","DOI":"10.18653\/v1\/2021.acl-long.217"},{"key":"305_CR21","doi-asserted-by":"crossref","unstructured":"Sanchez-Graillet O, Witte C, Grimm F, Cimiano P. An annotated corpus of clinical trial publications supporting schema-based relational information extraction. J Biomed Semant. 2021. Under Review.","DOI":"10.1186\/s13326-022-00271-7"},{"key":"305_CR22","unstructured":"Hsu I, Huang K, Boschee E, Miller S, Natarajan P, Chang K, et\u00a0al. Event Extraction as Natural Language Generation. CoRR. 2021. arXiv:2108.12724."},{"key":"305_CR23","doi-asserted-by":"publisher","unstructured":"Yang H, Sui D, Chen Y, Liu K, Zhao J, Wang T. Document-Level Event Extraction via Parallel Prediction Networks. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Online: Association for Computational Linguistics; 2021. p. 6298\u20136308. https:\/\/doi.org\/10.18653\/v1\/2021.acl-long.492.","DOI":"10.18653\/v1\/2021.acl-long.492"},{"key":"305_CR24","doi-asserted-by":"publisher","unstructured":"Giorgi J, Bader GD, Wang B. A sequence-to-sequence approach for document-level relation extraction. BioNLP 2022@ ACL 2022.\u00a0Dublin: Association for Computational Linguistics; 2022. p.\u00a010\u201325.\u00a0https:\/\/aclanthology.org\/2022.bionlp-1.2. https:\/\/doi.org\/10.18653\/v1\/2022.bionlp-1.2.","DOI":"10.18653\/v1\/2022.bionlp-1.2"},{"key":"305_CR25","doi-asserted-by":"publisher","unstructured":"Du X, Rush A, Cardie C. GRIT: Generative Role-filler Transformers for Document-level Event Entity Extraction. In: Merlo P, Tiedemann J, Tsarfaty R, editors. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Online: Association for Computational Linguistics; 2021. p. 634\u2013644. https:\/\/doi.org\/10.18653\/v1\/2021.eacl-main.52. https:\/\/aclanthology.org\/2021.eacl-main.52.","DOI":"10.18653\/v1\/2021.eacl-main.52"},{"key":"305_CR26","doi-asserted-by":"publisher","unstructured":"Du X, Rush A, Cardie C. Template Filling with Generative Transformers. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics; 2021. p. 909\u2013914. https:\/\/doi.org\/10.18653\/v1\/2021.naacl-main.70.","DOI":"10.18653\/v1\/2021.naacl-main.70"},{"key":"305_CR27","doi-asserted-by":"publisher","unstructured":"Wang XD, Weber L, Leser U. Biomedical Event Extraction as Multi-turn Question Answering. In: Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis. Online: Association for Computational Linguistics; 2020. p. 88\u201396. https:\/\/doi.org\/10.18653\/v1\/2020.louhi-1.10.","DOI":"10.18653\/v1\/2020.louhi-1.10"},{"key":"305_CR28","doi-asserted-by":"publisher","unstructured":"Ramponi A, Van Der\u00a0Goot R, Lombardo R, Plank B. Biomedical Event Extraction as Sequence Labeling. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics; 2020. p. 5357\u20135367. https:\/\/doi.org\/10.18653\/v1\/2020.emnlp-main.431.","DOI":"10.18653\/v1\/2020.emnlp-main.431"},{"issue":"1","key":"305_CR29","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1186\/s12859-020-3376-2","volume":"21","author":"L Zhu","year":"2020","unstructured":"Zhu L, Zheng H. Biomedical Event Extraction with a Novel Combination Strategy Based on Hybrid Deep Neural Networks. BMC Bioinformatics. 2020;21(1):47. https:\/\/doi.org\/10.1186\/s12859-020-3376-2.","journal-title":"BMC Bioinformatics."},{"key":"305_CR30","doi-asserted-by":"publisher","unstructured":"Huang KH, Yang M, Peng N. Biomedical Event Extraction with Hierarchical Knowledge Graphs. In: Cohn T, He Y, Liu Y, editors. Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics; 2020. p. 1277\u20131285. https:\/\/doi.org\/10.18653\/v1\/2020.findings-emnlp.114. https:\/\/aclanthology.org\/2020.findings-emnlp.114.","DOI":"10.18653\/v1\/2020.findings-emnlp.114"},{"issue":"19","key":"305_CR31","doi-asserted-by":"publisher","first-page":"4910","DOI":"10.1093\/bioinformatics\/btaa540","volume":"36","author":"HL Trieu","year":"2020","unstructured":"Trieu HL, Tran TT, Duong KNA, Nguyen A, Miwa M, Ananiadou S. DeepEventMine: End-to-End Neural Nested Event Extraction from Biomedical Texts. Bioinformatics. 2020;36(19):4910\u20137. https:\/\/doi.org\/10.1093\/bioinformatics\/btaa540.","journal-title":"Bioinformatics."},{"key":"305_CR32","doi-asserted-by":"publisher","unstructured":"Jiang Y, Kavuluru R. End-to-End $$n$$-ary Relation Extraction for Combination Drug Therapies. 2023. https:\/\/doi.org\/10.48550\/arXiv.2303.16886. arXiv:2303.16886.","DOI":"10.48550\/arXiv.2303.16886"},{"issue":"1","key":"305_CR33","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1093\/jamia\/ocz100","volume":"27","author":"Y Kim","year":"2020","unstructured":"Kim Y, Meystre SM. Ensemble Method-Based Extraction of Medication and Related Information from Clinical Texts. J Am Med Inform Assoc. 2020;27(1):31\u20138. https:\/\/doi.org\/10.1093\/jamia\/ocz100.","journal-title":"J Am Med Inform Assoc."},{"key":"305_CR34","doi-asserted-by":"publisher","unstructured":"Stylianou N, Kosmoliaptsis P, Vlahavas I. Improved Biomedical Entity Recognition via Longer Context Modeling. In: Maglogiannis I, Macintyre J, Iliadis L, editors. Artificial Intelligence Applications and Innovations. vol. 627. Cham: Springer International Publishing; 2021. p. 45\u201356. https:\/\/doi.org\/10.1007\/978-3-030-79150-6_4.","DOI":"10.1007\/978-3-030-79150-6_4"},{"key":"305_CR35","doi-asserted-by":"publisher","unstructured":"Farnsworth S, Gurdin G, Vargas J, Mulyar A, Lewinski N, McInnes BT. Extracting Experimental Parameter Entities from Scientific Articles. J Biomed Inform. 2022Feb;126: 103970. https:\/\/doi.org\/10.1016\/j.jbi.2021.103970.","DOI":"10.1016\/j.jbi.2021.103970"},{"key":"305_CR36","doi-asserted-by":"publisher","unstructured":"Tseo Y, Salkola MI, Mohamed A, Kumar A, Abnousi F. Information Extraction of Clinical Trial Eligibility Criteria. 2020. https:\/\/doi.org\/10.48550\/arXiv.2006.07296. arXiv:2006.07296.","DOI":"10.48550\/arXiv.2006.07296"},{"key":"305_CR37","doi-asserted-by":"publisher","unstructured":"Abaho M, Bollegala D, Williamson PR, Dodd S. Assessment of contextualised representations in detecting outcome phrases in clinical trials. CoRR. 2022. https:\/\/doi.org\/10.48550\/ARXIV.2203.03547. arXiv:2203.03547.","DOI":"10.48550\/ARXIV.2203.03547"},{"key":"305_CR38","doi-asserted-by":"publisher","unstructured":"Abaho M, Bollegala D, Williamson P, Dodd S. Position-based Prompting for Health Outcome Generation. In: Demner-Fushman D, Cohen KB, Ananiadou S, Tsujii J, editors. Proceedings of the 21st Workshop on Biomedical Language Processing. Dublin: Association for Computational Linguistics; 2022. p. 26\u201336. https:\/\/doi.org\/10.18653\/v1\/2022.bionlp-1.3. https:\/\/aclanthology.org\/2022.bionlp-1.3.","DOI":"10.18653\/v1\/2022.bionlp-1.3"},{"key":"305_CR39","doi-asserted-by":"publisher","unstructured":"Abaho M, Bollegala D, Williamson P, Dodd S. Detect and Classify \u2013 Joint Span Detection and Classification for Health Outcomes. In: Moens MF, Huang X, Specia L, Yih SWt, editors. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana: Association for Computational Linguistics; 2021. p. 8709\u20138721. https:\/\/doi.org\/10.18653\/v1\/2021.emnlp-main.686. https:\/\/aclanthology.org\/2021.emnlp-main.686.","DOI":"10.18653\/v1\/2021.emnlp-main.686"},{"key":"305_CR40","unstructured":"Ganguly D, Gleize M, Hou Y, Jochim C, Bonin F, Pascale A, et\u00a0al. Outcome Prediction from Behaviour Change Intervention Evaluations using a Combination of Node and Word Embedding. AMIA Ann Symp Proc. 2021;2021:486\u201395. Published online 2022 Feb 21."},{"key":"305_CR41","doi-asserted-by":"publisher","unstructured":"Papanikolaou Y, Staib M, Grace JJ, Bennett F. Slot Filling for Biomedical Information Extraction. In: Demner-Fushman D, Cohen KB, Ananiadou S, Tsujii J, editors. Proceedings of the 21st Workshop on Biomedical Language Processing, BioNLP@ACL 2022, Dublin, Ireland, May 26, 2022. Association for Computational Linguistics; 2022. p. 82\u201390. https:\/\/doi.org\/10.18653\/v1\/2022.bionlp-1.7.","DOI":"10.18653\/v1\/2022.bionlp-1.7"},{"key":"305_CR42","doi-asserted-by":"publisher","unstructured":"Dhrangadhariya A, M\u00fcller H. Not so Weak PICO: Leveraging Weak Supervision for Participants, Interventions, and Outcomes Recognition for Systematic Review Automation. JAMIA Open. 2023;6(1):ooac107. https:\/\/doi.org\/10.1093\/jamiaopen\/ooac107.","DOI":"10.1093\/jamiaopen\/ooac107"},{"key":"305_CR43","unstructured":"Nye BE, DeYoung J, Lehman E, Nenkova A, Marshall IJ, Wallace BC. Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations. CoRR. 2020. arXiv:2010.03550."},{"key":"305_CR44","unstructured":"Wallace BC, Kuiper J, Sharma A, Zhu MB, Marshall IJ. Extracting PICO Sentences from Clinical Trial Reports using Supervised Distant Supervision. J Mach Learn Res. 2016;17:132:1\u201325. http:\/\/jmlr.org\/papers\/v17\/15-404.html."},{"key":"305_CR45","doi-asserted-by":"publisher","unstructured":"Liu S, Sun Y, Li B, Wang W, Bourgeois FT, Dunn AG. Sent2Span: Span Detection for PICO Extraction in the Biomedical Text without Span Annotations. In: Moens MF, Huang X, Specia L, Yih SWt, editors. Findings of the Association for Computational Linguistics: EMNLP 2021. Punta Cana: Association for Computational Linguistics; 2021. p. 1705\u20131715. https:\/\/doi.org\/10.18653\/v1\/2021.findings-emnlp.147. https:\/\/aclanthology.org\/2021.findings-emnlp.147.","DOI":"10.18653\/v1\/2021.findings-emnlp.147"},{"issue":"12","key":"305_CR46","doi-asserted-by":"publisher","first-page":"3856","DOI":"10.1093\/bioinformatics\/btaa256","volume":"36","author":"D Jin","year":"2020","unstructured":"Jin D, Szolovits P. Advancing PICO element detection in biomedical text via deep neural networks. Bioinform. 2020;36(12):3856\u201362. https:\/\/doi.org\/10.1093\/bioinformatics\/btaa256.","journal-title":"Bioinform."},{"key":"305_CR47","doi-asserted-by":"publisher","unstructured":"Kang T, Zou S, Weng C. Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature. In: Ohno-Machado L, S\u00e9roussi B, editors. MEDINFO 2019: Health and Wellbeing e-Networks for All - Proceedings of the 17th World Congress on Medical and Health Informatics, Lyon, France, 25-30 August 2019. vol. 264 of Studies in Health Technology and Informatics. IOS Press; 2019. p. 188\u2013192. https:\/\/doi.org\/10.3233\/SHTI190209.","DOI":"10.3233\/SHTI190209"},{"issue":"1","key":"305_CR48","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1186\/s12911-018-0699-2","volume":"18","author":"S Chabou","year":"2018","unstructured":"Chabou S, Iglewski M. Combination of Conditional Random Field with a Rule Based Method in the Extraction of PICO Elements. BMC Med Inform Decis Mak. 2018;18(1):128. https:\/\/doi.org\/10.1186\/s12911-018-0699-2.","journal-title":"BMC Med Inform Decis Mak."},{"key":"305_CR49","doi-asserted-by":"publisher","unstructured":"Yuan X, Xiaoli L, Shilei L, Qinwen S, Ke L. Extracting PICO Elements From RCT Abstracts Using 1-2gram Analysis And Multitask Classification. In: Proceedings of the Third International Conference on Medical and Health Informatics 2019 - ICMHI 2019. Xiamen: ACM Press; 2019. p. 194\u2013199. https:\/\/doi.org\/10.1145\/3340037.3340043.","DOI":"10.1145\/3340037.3340043"},{"key":"305_CR50","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2020.101949","volume":"108","author":"N Stylianou","year":"2020","unstructured":"Stylianou N, Razis G, Goulis DG, Vlahavas I. EBM+: Advancing Evidence-Based Medicine via Two Level Automatic Identification of Populations, Interventions, Outcomes in Medical Literature. Artif Intell Med. 2020;108: 101949. https:\/\/doi.org\/10.1016\/j.artmed.2020.101949.","journal-title":"Artif Intell Med."},{"key":"305_CR51","doi-asserted-by":"publisher","unstructured":"Jin D, Szolovits P. PICO Element Detection in Medical Text via Long Short-Term Memory Neural Networks. In: Proceedings of the BioNLP 2018 Workshop. Melbourne: Association for Computational Linguistics; 2018. p. 67\u201375. https:\/\/doi.org\/10.18653\/v1\/W18-2308.","DOI":"10.18653\/v1\/W18-2308"},{"issue":"10","key":"305_CR52","doi-asserted-by":"publisher","DOI":"10.2196\/19810","volume":"22","author":"M Afzal","year":"2020","unstructured":"Afzal M, Alam F, Malik KM, Malik GM. Clinical Context-Aware Biomedical Text Summarization Using Deep Neural Network: Model Development and Validation. J Med Internet Res. 2020;22(10): e19810. https:\/\/doi.org\/10.2196\/19810.","journal-title":"J Med Internet Res."},{"key":"305_CR53","doi-asserted-by":"publisher","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of NAACL-HLT.\u00a0Minneapolis: Association for Computational Linguistics; 2019. p. 4171\u20134186.\u00a0https:\/\/aclanthology.org\/N19-1423. https:\/\/doi.org\/10.18653\/v1\/N19-1423.","DOI":"10.18653\/v1\/N19-1423"},{"key":"305_CR54","doi-asserted-by":"publisher","unstructured":"Schmidt L, Weeds J, Higgins JPT. Data Mining in Clinical Trial Text: Transformers for Classification and Question Answering Tasks. In: Cabitza F, Fred ALN, Gamboa H, editors. Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020) - Volume 5: HEALTHINF, Valletta, Malta, February 24-26, 2020. SCITEPRESS. 2020. p. 83\u201394. https:\/\/doi.org\/10.5220\/0008945700830094.","DOI":"10.5220\/0008945700830094"},{"key":"305_CR55","unstructured":"Zhang T, Yu Y, Mei J, Tang Z, Zhang X, Li S. Unlocking the Power of Deep PICO Extraction: Step-wise Medical NER Identification. CoRR. 2020. arXiv:2005.06601."},{"key":"305_CR56","doi-asserted-by":"publisher","first-page":"102661","DOI":"10.1016\/j.artmed.2023.102661","volume":"144","author":"J Whitton","year":"2023","unstructured":"Whitton J, Hunter A. Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations. Artif Intell Med. 2023;144:102661. https:\/\/doi.org\/10.1016\/j.artmed.2023.102661.","journal-title":"Artif Intell Med."},{"key":"305_CR57","doi-asserted-by":"publisher","unstructured":"Dhrangadhariya A, Aguilar G, Solorio T, Hilfiker R, M\u00fcller H. End-to-End Fine-Grained Neural Entity Recognition of Patients, Interventions, Outcomes. In: Candan KS, Ionescu B, Goeuriot L, Larsen B, M\u00fcller H, Joly A, et\u00a0al., editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction. vol. 12880. Cham: Springer International Publishing; 2021. p. 65\u201377. https:\/\/doi.org\/10.1007\/978-3-030-85251-1_6.","DOI":"10.1007\/978-3-030-85251-1_6"},{"key":"305_CR58","unstructured":"Text chunking using transformation-based learning. In: Natural language processing using very large corpora. Springer. p. 157\u2013176."},{"key":"305_CR59","unstructured":"Agarap AF. Deep learning using rectified linear units (relu). 2018. arXiv preprint arXiv:1803.08375."},{"key":"305_CR60","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et\u00a0al. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998-6008.\u00a0https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html."},{"key":"305_CR61","doi-asserted-by":"publisher","unstructured":"Chen MX, Firat O, Bapna A, Johnson M, Macherey W, Foster G, et\u00a0al. The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).\u00a0Melbourne: Association for Computational Linguistics; 2018. p. 76\u201386.\u00a0https:\/\/aclanthology.org\/P18-1008. https:\/\/doi.org\/10.18653\/v1\/P18-1008.","DOI":"10.18653\/v1\/P18-1008"},{"issue":"1","key":"305_CR62","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3419106","volume":"2","author":"T Shi","year":"2021","unstructured":"Shi T, Keneshloo Y, Ramakrishnan N, Reddy CK. Neural abstractive text summarization with sequence-to-sequence models. ACM Trans Data Sci. 2021;2(1):1\u201337.","journal-title":"ACM Trans Data Sci."},{"key":"305_CR63","doi-asserted-by":"publisher","unstructured":"Akiba T, Sano S, Yanase T, Ohta T, Koyama M. Optuna: A Next-generation Hyperparameter Optimization Framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.\u00a0New York: Association for Computing Machinery; 2019.\u00a0https:\/\/doi.org\/10.1145\/3292500.3330701.","DOI":"10.1145\/3292500.3330701"},{"issue":"3","key":"305_CR64","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1111\/jdi.12585","volume":"8","author":"RR Shankar","year":"2017","unstructured":"Shankar RR, Bao Y, Han P, Hu J, Ma J, Peng Y, et al. Sitagliptin added to stable insulin therapy with or without metformin in Chinese patients with type 2 diabetes. J Diabetes Investig. 2017;8(3):321\u20139.","journal-title":"J Diabetes Investig."},{"key":"305_CR65","doi-asserted-by":"publisher","unstructured":"Thomas J, Noel-Storr A, Marshall I, et\u00a0al. Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol. 2017;91:31\u20137. https:\/\/doi.org\/10.1016\/j.jclinepi.2017.08.011.","DOI":"10.1016\/j.jclinepi.2017.08.011"},{"key":"305_CR66","doi-asserted-by":"publisher","unstructured":"Schmidt DM, Witte C, Cimiano P. ag-sc\/Clinical-Trial-Information-Extraction: Initial release. Zenodo; 2023. https:\/\/doi.org\/10.5281\/zenodo.10419786.","DOI":"10.5281\/zenodo.10419786"}],"container-title":["Journal of Biomedical Semantics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-024-00305-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s13326-024-00305-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-024-00305-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,23]],"date-time":"2024-04-23T11:03:11Z","timestamp":1713870191000},"score":1,"resource":{"primary":{"URL":"https:\/\/jbiomedsem.biomedcentral.com\/articles\/10.1186\/s13326-024-00305-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,23]]},"references-count":66,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["305"],"URL":"https:\/\/doi.org\/10.1186\/s13326-024-00305-2","relation":{},"ISSN":["2041-1480"],"issn-type":[{"value":"2041-1480","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,23]]},"assertion":[{"value":"5 January 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 April 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"3"}}