{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T14:06:20Z","timestamp":1760709980139,"version":"3.37.3"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2019,10,12]],"date-time":"2019-10-12T00:00:00Z","timestamp":1570838400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100014438","name":"Business Finland","doi-asserted-by":"crossref","award":["644\/31\/2015"],"award-info":[{"award-number":["644\/31\/2015"]}],"id":[{"id":"10.13039\/501100014438","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100002341","name":"Academy of Finland","doi-asserted-by":"publisher","award":["315376"],"award-info":[{"award-number":["315376"]}],"id":[{"id":"10.13039\/501100002341","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>This study focuses on the task of automatically assigning standardized (topical) subject headings to free-text sentences in clinical nursing notes. The underlying motivation is to support nurses when they document patient care by developing a computer system that can assist in incorporating suitable subject headings that reflect the documented topics. Central in this study is performance evaluation of several text classification methods to assess the feasibility of developing such a system.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Seven text classification methods are evaluated using a corpus of approximately 0.5 million nursing notes (5.5 million sentences) with 676 unique headings extracted from a Finnish university hospital. Several of these methods are based on artificial neural networks. Evaluation is first done in an automatic manner for all methods, then a manual error analysis is done on a sample.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We find that a method based on a bidirectional long short-term memory network performs best with an average recall of 0.5435 when allowed to suggest 1 subject heading per sentence and 0.8954 when allowed to suggest 10 subject headings per sentence. However, other methods achieve comparable results. The manual analysis indicates that the predictions are better than what the automatic evaluation suggests.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>The results indicate that several of the tested methods perform well in suggesting the most appropriate subject headings on sentence level. Thus, we find it feasible to develop a text classification system that can support the use of standardized terminologies and save nurses time and effort on care documentation.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocz150","type":"journal-article","created":{"date-parts":[[2019,8,4]],"date-time":"2019-08-04T11:11:09Z","timestamp":1564917069000},"page":"81-88","source":"Crossref","is-referenced-by-count":9,"title":["Supporting the use of standardized nursing terminologies with automatic subject heading prediction: a comparison of sentence-level text classification methods"],"prefix":"10.1093","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1418-7892","authenticated-orcid":false,"given":"Hans","family":"Moen","sequence":"first","affiliation":[{"name":"Department of Future Technologies, University of Turku, Turku, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kai","family":"Hakala","sequence":"additional","affiliation":[{"name":"Department of Future Technologies, University of Turku, Turku, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Laura-Maria","family":"Peltonen","sequence":"additional","affiliation":[{"name":"Department of Nursing Science, University of Turku, Turku, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Henry","family":"Suhonen","sequence":"additional","affiliation":[{"name":"Department of Nursing Science, University of Turku, Turku, Finland"},{"name":"Department of Nursing, Turku University Hospital, Turku, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Filip","family":"Ginter","sequence":"additional","affiliation":[{"name":"Department of Future Technologies, University of Turku, Turku, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tapio","family":"Salakoski","sequence":"additional","affiliation":[{"name":"Department of Future Technologies, University of Turku, Turku, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sanna","family":"Salanter\u00e4","sequence":"additional","affiliation":[{"name":"Department of Nursing Science, University of Turku, Turku, Finland"},{"name":"Department of Nursing, Turku University Hospital, Turku, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2019,10,12]]},"reference":[{"issue":"6","key":"2020110613100955700_ocz150-B1","first-page":"287","article-title":"The influence of integrated electronic medical records and computerized nursing notes on nurses\u2019 time spent in documentation","volume":"30","author":"Yee","year":"2012","journal-title":"Comput Inform Nurs"},{"issue":"2","key":"2020110613100955700_ocz150-B2","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1016\/j.jcrc.2007.06.003","article-title":"Quantifying the volume of documented clinical information in critical illness","volume":"23","author":"Manor-Shulman","year":"2008","journal-title":"J Crit Care"},{"issue":"4","key":"2020110613100955700_ocz150-B3","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1111\/scs.12094","article-title":"Impacts of structuring nursing records: a systematic review","volume":"28","author":"Saranto","year":"2014","journal-title":"Scand J Caring Sci"},{"issue":"3","key":"2020110613100955700_ocz150-B4","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1016\/j.ijmedinf.2013.11.006","article-title":"Impacts of structuring the electronic health record: A systematic review protocol and results of previous reviews","volume":"83","author":"Hypp\u00f6nen","year":"2014","journal-title":"Int J Med Inform"},{"issue":"8","key":"2020110613100955700_ocz150-B5","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1016\/j.ijmedinf.2010.05.002","article-title":"Evaluation of electronic nursing documentation\u2014nursing process model and standardized terminologies as keys to visible and transparent nursing","volume":"79","author":"H\u00e4yrinen","year":"2010","journal-title":"Int J Med Inform"},{"issue":"6","key":"2020110613100955700_ocz150-B6","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1136\/jamia.1998.0050571","article-title":"The granularity of medical narratives and its effect on the speed and completeness of information retrieval","volume":"5","author":"Tange","year":"1998","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2020110613100955700_ocz150-B7","doi-asserted-by":"crossref","first-page":"29.","DOI":"10.1007\/s10916-016-0677-0","article-title":"Literature review of patient record structures from the physician\u2019s perspective","volume":"41","author":"Forsvik","year":"2017","journal-title":"J Med Syst"},{"issue":"1","key":"2020110613100955700_ocz150-B8","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/j.pec.2016.07.031","article-title":"Patient-centered communication in the era of electronic health records: What does the evidence say?","volume":"100","author":"Rathert","year":"2017","journal-title":"Patient Educ Couns"},{"issue":"6","key":"2020110613100955700_ocz150-B9","doi-asserted-by":"crossref","first-page":"806","DOI":"10.1197\/jamia.M3037","article-title":"Evaluation of a method to identify and categorize section headers in clinical documents","volume":"16","author":"Denny","year":"2009","journal-title":"J Am Med Inform Assoc"},{"first-page":"744","year":"2010","author":"Li","key":"2020110613100955700_ocz150-B10"},{"key":"2020110613100955700_ocz150-B11","first-page":"636","article-title":"Developing a section labeler for clinical documents","volume":"2014","author":"Haug","year":"2014","journal-title":"AMIA Annu Symp Proc"},{"issue":"2","key":"2020110613100955700_ocz150-B12","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1016\/j.artmed.2015.04.007","article-title":"An empirical evaluation of supervised learning approaches in assigning diagnosis codes to electronic medical records","volume":"65","author":"Kavuluru","year":"2015","journal-title":"Artif Intell Med"},{"issue":"11","key":"2020110613100955700_ocz150-B13","doi-asserted-by":"crossref","first-page":"956","DOI":"10.1016\/j.ijmedinf.2015.08.004","article-title":"Automatic ICD-10 classification of cancers from free-text death certificates","volume":"84","author":"Koopman","year":"2015","journal-title":"Int J Med Inform"},{"key":"2020110613100955700_ocz150-B14","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.jbi.2013.11.008","article-title":"Development and evaluation of RapTAT: a machine learning system for concept mapping of phrases from medical narratives","volume":"48","author":"Gobbel","year":"2014","journal-title":"J Biomed Inform"},{"issue":"1","key":"2020110613100955700_ocz150-B15","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1197\/jamia.M2408","article-title":"Identifying patient smoking status from medical discharge records","volume":"15","author":"Uzuner","year":"2008","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2020110613100955700_ocz150-B16","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1197\/jamia.M3115","article-title":"Recognizing obesity and comorbidities in sparse data","volume":"16","author":"Uzuner","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2020110613100955700_ocz150-B17","doi-asserted-by":"crossref","first-page":"S67","DOI":"10.1016\/j.jbi.2015.07.001","article-title":"Identifying risk factors for heart disease over time: overview of 2014 i2b2\/UTHealth shared task Track 2","volume":"58","author":"Stubbs","year":"2015","journal-title":"J Biomed Inform"},{"key":"2020110613100955700_ocz150-B18","first-page":"649","volume-title":"Advances in Neural Information Processing Systems","author":"Zhang","year":"2015"},{"first-page":"1422","year":"2015","author":"Tang","key":"2020110613100955700_ocz150-B19"},{"year":"2016","author":"Joulin","key":"2020110613100955700_ocz150-B20"},{"first-page":"1014","year":"2016","author":"Limsopatham","key":"2020110613100955700_ocz150-B21"},{"issue":"8","key":"2020110613100955700_ocz150-B22","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"issue":"10","key":"2020110613100955700_ocz150-B23","doi-asserted-by":"crossref","first-page":"2451","DOI":"10.1162\/089976600300015015","article-title":"Learning to forget: continual prediction with LSTM","volume":"12","author":"Gers","year":"2000","journal-title":"Neural Comput"},{"issue":"11","key":"2020110613100955700_ocz150-B24","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc IEEE"},{"key":"2020110613100955700_ocz150-B25","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1162\/tacl_a_00051","article-title":"Enriching word vectors with subword information","volume":"5","author":"Bojanowski","year":"2017","journal-title":"Trans Assoc Comput Linguist"},{"key":"2020110613100955700_ocz150-B26","first-page":"169","volume-title":"Advances in Kernel Methods\u2013Support Vector Learning","author":"Joachims","year":"1999"},{"key":"2020110613100955700_ocz150-B27","first-page":"137","volume-title":"European Conference on Machine Learning","author":"Joachims","year":"1998"},{"issue":"5","key":"2020110613100955700_ocz150-B28","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1016\/0306-4573(88)90021-0","article-title":"Term-weighting approaches in automatic text retrieval","volume":"24","author":"Salton","year":"1988","journal-title":"Inf Process Manag"},{"key":"2020110613100955700_ocz150-B29","first-page":"661\u20137","volume-title":"NIPS\u201900 Proceedings of the 13th International Conference on Neural Information Processing Systems","author":"Williams","year":"2000"},{"key":"2020110613100955700_ocz150-B30","first-page":"1177","volume-title":"Advances in Neural Information Processing Systems 20","author":"Rahimi","year":"2008"},{"issue":"3","key":"2020110613100955700_ocz150-B31","first-page":"18","article-title":"Classification and regression by randomForest","volume":"2","author":"Liaw","year":"2002","journal-title":"R News"},{"key":"2020110613100955700_ocz150-B32","first-page":"3111","volume-title":"NIPS\u201913 Proceedings of the 26th International Conference on Neural Information Processing Systems, Volume 2","author":"Mikolov","year":"2013"},{"issue":"2\u20133","key":"2020110613100955700_ocz150-B33","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1080\/00437956.1954.11659520","article-title":"Distributional structure","volume":"10","author":"Harris","year":"1954","journal-title":"Word"},{"first-page":"807","year":"2014","author":"Kaewphan","key":"2020110613100955700_ocz150-B34"},{"first-page":"3120","year":"2018","author":"Zeng","key":"2020110613100955700_ocz150-B35"},{"key":"2020110613100955700_ocz150-B36","first-page":"301","volume-title":"European Conference on Computer Vision","author":"Krause","year":"2016"},{"year":"2014","author":"Hannun","key":"2020110613100955700_ocz150-B37"},{"year":"2017","author":"Xie","key":"2020110613100955700_ocz150-B38"},{"first-page":"94","year":"2018","author":"Moen","key":"2020110613100955700_ocz150-B39"},{"first-page":"2227","year":"2018","author":"Peters","key":"2020110613100955700_ocz150-B40"},{"year":"2019","author":"Devlin","key":"2020110613100955700_ocz150-B41"},{"first-page":"900","year":"2018","author":"Song","key":"2020110613100955700_ocz150-B42"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/1\/81\/34152059\/ocz150.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/1\/81\/34152059\/ocz150.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T19:31:48Z","timestamp":1604691108000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/1\/81\/5586511"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,12]]},"references-count":42,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,10,12]]},"published-print":{"date-parts":[[2020,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocz150","relation":{},"ISSN":["1527-974X"],"issn-type":[{"type":"electronic","value":"1527-974X"}],"subject":[],"published-other":{"date-parts":[[2020,1]]},"published":{"date-parts":[[2019,10,12]]}}}