{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T08:34:27Z","timestamp":1770798867330,"version":"3.50.0"},"reference-count":23,"publisher":"Oxford University Press (OUP)","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Objective This paper presents an automatic, active learning-based system for the extraction of medical concepts from clinical free-text reports. Specifically, (1) the contribution of active learning in reducing the annotation effort and (2) the robustness of incremental active learning framework across different selection criteria and data sets are determined.<\/jats:p><jats:p>Materials and methods The comparative performance of an active learning framework and a fully supervised approach were investigated to study how active learning reduces the annotation effort while achieving the same effectiveness as a supervised approach. Conditional random fields as the supervised method, and least confidence and information density as 2 selection criteria for active learning framework were used. The effect of incremental learning vs standard learning on the robustness of the models within the active learning framework with different selection criteria was also investigated. The following 2 clinical data sets were used for evaluation: the Informatics for Integrating Biology and the Bedside\/Veteran Affairs (i2b2\/VA) 2010 natural language processing challenge and the Shared Annotated Resources\/Conference and Labs of the Evaluation Forum (ShARe\/CLEF) 2013 eHealth Evaluation Lab.<\/jats:p><jats:p>Results The annotation effort saved by active learning to achieve the same effectiveness as supervised learning is up to 77%, 57%, and 46% of the total number of sequences, tokens, and concepts, respectively. Compared with the random sampling baseline, the saving is at least doubled.<\/jats:p><jats:p>Conclusion Incremental active learning is a promising approach for building effective and robust medical concept extraction models while significantly reducing the burden of manual annotation.<\/jats:p>","DOI":"10.1093\/jamia\/ocv069","type":"journal-article","created":{"date-parts":[[2015,8,8]],"date-time":"2015-08-08T02:30:14Z","timestamp":1439001014000},"page":"289-296","source":"Crossref","is-referenced-by-count":46,"title":["Active learning: a step towards automating medical concept extraction"],"prefix":"10.1093","volume":"23","author":[{"given":"Mahnoosh","family":"Kholghi","sequence":"first","affiliation":[{"name":"Science and Engineering Faculty, Queensland University of Technology, Brisbane 4000, Queensland, Australia."},{"name":"The Australian e-Health Research Centre, CSIRO, Brisbane 4029, Queensland, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Laurianne","family":"Sitbon","sequence":"additional","affiliation":[{"name":"Science and Engineering Faculty, Queensland University of Technology, Brisbane 4000, Queensland, Australia."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guido","family":"Zuccon","sequence":"additional","affiliation":[{"name":"Science and Engineering Faculty, Queensland University of Technology, Brisbane 4000, Queensland, Australia."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anthony","family":"Nguyen","sequence":"additional","affiliation":[{"name":"The Australian e-Health Research Centre, CSIRO, Brisbane 4029, Queensland, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2015,8,7]]},"reference":[{"issue":"5","key":"2020110613010809600_ocv069-B1","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1136\/amiajnl-2013-002214","article-title":"Natural language processing: algorithms and tools to extract computable information from EHRs and from the biomedical literature","volume":"20","author":"Ohno-Machado","year":"2013","journal-title":"J Am Med Inform Assoc."},{"key":"2020110613010809600_ocv069-B2","first-page":"117","article-title":"Automatic extraction of cancer characteristics from free-text pathology reports for cancer notifications","author":"Nguyen","year":"2011","journal-title":"Stud Health Technol Inform."},{"key":"2020110613010809600_ocv069-B3","first-page":"300","article-title":"Automatic classification of free-text radiology reports to identify limb fractures using machine learning and the SNOMED CT ontology","volume":"2013","author":"Zuccon","year":"2013","journal-title":"AMIA Summit Clin Res Inform."},{"issue":"5","key":"2020110613010809600_ocv069-B4","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","article-title":"2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text","volume":"18","author":"Uzuner","year":"2011","journal-title":"J Am Med Inform Assoc."},{"issue":"5","key":"2020110613010809600_ocv069-B5","doi-asserted-by":"crossref","first-page":"544","DOI":"10.1136\/amiajnl-2011-000464","article-title":"Natural language processing: an introduction","volume":"18","author":"Nadkarni","year":"2011","journal-title":"J Am Med Inform Assoc."},{"key":"2020110613010809600_ocv069-B6","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-031-01560-1","article-title":"Synthesis Lectures on Artificial Intelligence and Machine Learning","volume-title":"Active Learning","author":"Settles","year":"2012"},{"key":"2020110613010809600_ocv069-B7","article-title":"Annotating named entities in clinical text by combining pre-annotation and active learning","author":"Skeppstedt","journal-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013)"},{"key":"2020110613010809600_ocv069-B8","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1007\/978-3-642-40802-1_24","article-title":"Overview of the ShARe\/CLEF eHealth Evaluation Lab 2013","volume-title":"Information Access Evaluation. Multilinguality, Multimodality, and Visualization","author":"Suominen","year":"2013"},{"key":"2020110613010809600_ocv069-B9","article-title":"Factors influencing robustness and effectiveness of conditional random fields in active learning frameworks","volume-title":"Proceedings of the 12th Australasian Data Mining Conference (AusDM 2014)","author":"Kholghi"},{"key":"2020110613010809600_ocv069-B10","first-page":"282","article-title":"Conditional random fields: probabilistic models for segmenting and labeling sequence data","volume-title":"Proceedings of the Eighteenth International Conference on Machine Learning (ICML)","author":"Lafferty","year":"2001"},{"key":"2020110613010809600_ocv069-B11","article-title":"Task 1: ShARe\/CLEF ehealth evaluation lab 2013","author":"Pradhan"},{"key":"2020110613010809600_ocv069-B12","first-page":"148","article-title":"Heterogenous uncertainty sampling for supervised learning","volume-title":"Proceedings of the 18th International Conference on Machine Learning","author":"Lewis","year":"1994"},{"key":"2020110613010809600_ocv069-B13","article-title":"An analysis of active learning strategies for sequence labeling tasks","author":"Settles","journal-title":"In: Proceedings of the Conference on Empirical Methods in Natural"},{"issue":"2","key":"2020110613010809600_ocv069-B14","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1016\/j.jbi.2011.11.003","article-title":"Applying active learning to assertion classification of concepts in clinical text","volume":"45","author":"Chen","year":"2012","journal-title":"J Biomed Inform."},{"key":"2020110613010809600_ocv069-B15","first-page":"530","article-title":"Semi-supervised active learning for modeling medical concepts from free text","volume-title":"Proceedings of the Sixth International Conference on Machine Learning and Applications","author":"Rosales","year":"2007"},{"issue":"6","key":"2020110613010809600_ocv069-B16","doi-asserted-by":"crossref","first-page":"1302","DOI":"10.1007\/s11390-012-1306-0","article-title":"A unified active learning framework for biomedical relation extraction","volume":"27","author":"Zhang","year":"2012","journal-title":"J Comput Sci Technol."},{"issue":"97.55","key":"2020110613010809600_ocv069-B17","first-page":"90","article-title":"De-identifying health records by means of active learning","volume":"97","author":"Bostr\u00f6m","year":"2012","journal-title":"Recall (micro)."},{"issue":"5","key":"2020110613010809600_ocv069-B18","doi-asserted-by":"crossref","first-page":"809","DOI":"10.1136\/amiajnl-2011-000648","article-title":"Active learning for clinical text classification: is it better than random sampling?","volume":"19","author":"Figueroa","year":"2012","journal-title":"J Am Med Inform Assoc."},{"key":"2020110613010809600_ocv069-B19","doi-asserted-by":"crossref","first-page":"173","DOI":"10.3115\/977035.977059","article-title":"Representing text chunks","volume-title":"Proceedings of the Ninth Conference on European Chapter of the Association for Computational Linguistics","author":"Sang","year":"1999"},{"key":"2020110613010809600_ocv069-B20","article-title":"A simple pipeline application for identifying and negating SNOMED clinical terminology in free text","volume-title":"Proceedings of the Health Informatics Conference (HIC)","author":"Nguyen"},{"key":"2020110613010809600_ocv069-B21","doi-asserted-by":"crossref","DOI":"10.21236\/ADA440382","article-title":"Reducing labeling effort for structured prediction tasks","volume-title":"Proceedings of the National Conference on Artificial Intelligence (AAAI)","author":"Culotta"},{"key":"2020110613010809600_ocv069-B22","article-title":"MALLET: A Machine Learning for Language Toolkit","author":"McCallum"},{"key":"2020110613010809600_ocv069-B23","first-page":"1","article-title":"Active learning with real annotation costs","volume-title":"Proceedings of the NIPS Workshop on Cost-Sensitive Learning","author":"Settles","year":"2008"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/23\/2\/289\/34147527\/ocv069.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/23\/2\/289\/34147527\/ocv069.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,12]],"date-time":"2023-08-12T23:57:27Z","timestamp":1691884647000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/23\/2\/289\/2572381"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,8,7]]},"references-count":23,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2015,8,7]]},"published-print":{"date-parts":[[2016,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocv069","relation":{},"ISSN":["1527-974X","1067-5027"],"issn-type":[{"value":"1527-974X","type":"electronic"},{"value":"1067-5027","type":"print"}],"subject":[],"published-other":{"date-parts":[[2016,3]]},"published":{"date-parts":[[2015,8,7]]}}}