{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T01:34:47Z","timestamp":1779932087801,"version":"3.53.1"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2020,9,1]],"date-time":"2020-09-01T00:00:00Z","timestamp":1598918400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"K1 COMET Competence Centre CBmed"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>Machine learning models trained on electronic health records have achieved high prognostic accuracy in test datasets, but little is known about their embedding into clinical workflows. We implemented a random forest\u2013based algorithm to identify hospitalized patients at high risk for delirium, and evaluated its performance in a clinical setting.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>Delirium was predicted at admission and recalculated on the evening of admission. The defined prediction outcome was a delirium coded for the recent hospital stay. During 7 months of prospective evaluation, 5530 predictions were analyzed. In addition, 119 predictions for internal medicine patients were compared with ratings of clinical experts in a blinded and nonblinded setting.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>During clinical application, the algorithm achieved a sensitivity of 74.1% and a specificity of 82.2%. Discrimination on prospective data (area under the receiver-operating characteristic curve = 0.86) was as good as in the test dataset, but calibration was poor. The predictions correlated strongly with delirium risk perceived by experts in the blinded (r\u2009=\u20090.81) and nonblinded (r\u2009=\u20090.62) settings. A major advantage of our setting was the timely prediction without additional data entry.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>The implemented machine learning algorithm achieved a stable performance predicting delirium in high agreement with expert ratings, but improvement of calibration is needed. Future research should evaluate the acceptance of implemented machine learning algorithms by health professionals.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Our study provides new insights into the implementation process of a machine learning algorithm into a clinical workflow and demonstrates its predictive power for delirium.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocaa113","type":"journal-article","created":{"date-parts":[[2020,5,20]],"date-time":"2020-05-20T11:29:29Z","timestamp":1589974169000},"page":"1383-1392","source":"Crossref","is-referenced-by-count":70,"title":["Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study"],"prefix":"10.1093","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1287-594X","authenticated-orcid":false,"given":"Stefanie","family":"Jauk","sequence":"first","affiliation":[{"name":"Department of Information and Process Management, Steierm\u00e4rkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria"},{"name":"Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diether","family":"Kramer","sequence":"additional","affiliation":[{"name":"Department of Information and Process Management, Steierm\u00e4rkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Birgit","family":"Gro\u00dfauer","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine, Steierm\u00e4rkische Krankenanstaltengesellschaft m.b.H. (KAGes) LKH Graz II, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Susanne","family":"Rienm\u00fcller","sequence":"additional","affiliation":[{"name":"Department of Internal Medicine, Steierm\u00e4rkische Krankenanstaltengesellschaft m.b.H. (KAGes) LKH Graz II, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Avian","sequence":"additional","affiliation":[{"name":"Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Berghold","sequence":"additional","affiliation":[{"name":"Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Werner","family":"Leodolter","sequence":"additional","affiliation":[{"name":"Department of Information and Process Management, Steierm\u00e4rkische Krankenanstaltengesellschaft m.b.H. (KAGes), Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefan","family":"Schulz","sequence":"additional","affiliation":[{"name":"Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2020,9,24]]},"reference":[{"issue":"2","key":"2020122718095897800_ocaa113-B1","doi-asserted-by":"crossref","first-page":"e1001381","DOI":"10.1371\/journal.pmed.1001381","article-title":"Prognosis Research Strategy (PROGRESS) 3: prognostic model research","volume":"10","author":"Steyerberg","year":"2013","journal-title":"PLoS Med"},{"issue":"4","key":"2020122718095897800_ocaa113-B2","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1136\/svn-2017-000101","article-title":"Artificial intelligence in healthcare: past, present and future","volume":"2","author":"Jiang","year":"2017","journal-title":"Stroke Vasc Neurol"},{"issue":"1","key":"2020122718095897800_ocaa113-B3","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1093\/jamia\/ocw042","article-title":"Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review","volume":"24","author":"Goldstein","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2020122718095897800_ocaa113-B4","doi-asserted-by":"crossref","first-page":"e0174944","DOI":"10.1371\/journal.pone.0174944","article-title":"Can machine-learning improve cardiovascular risk prediction using routine clinical data?","volume":"12","author":"Weng","year":"2017","journal-title":"PLoS One"},{"key":"2020122718095897800_ocaa113-B5","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.csbj.2014.11.005","article-title":"Machine learning applications in cancer prognosis and prediction","volume":"13","author":"Kourou","year":"2015","journal-title":"Comput Struct Biotechnol J"},{"issue":"12","key":"2020122718095897800_ocaa113-B6","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1016\/S2213-2600(18)30300-X","article-title":"Machine learning for real-time prediction of complications in critical care: a retrospective study","volume":"6","author":"Meyer","year":"2018","journal-title":"Lancet Respir Med"},{"key":"2020122718095897800_ocaa113-B7","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1038\/s41746-018-0029-1","article-title":"Scalable and accurate deep learning with electronic health records","volume":"1","author":"Rajkomar","year":"2018","journal-title":"NPJ Digit Med"},{"issue":"1","key":"2020122718095897800_ocaa113-B8","doi-asserted-by":"crossref","first-page":"3","DOI":"10.23876\/j.krcp.2017.36.1.3","article-title":"Medical big data: promise and challenges","volume":"36","author":"Lee","year":"2017","journal-title":"Kidney Res Clin Pract"},{"issue":"7","key":"2020122718095897800_ocaa113-B9","doi-asserted-by":"crossref","first-page":"1148","DOI":"10.1377\/hlthaff.2014.0352","article-title":"Implementing electronic health care predictive analytics: considerations and challenges","volume":"33","author":"Amarasingham","year":"2014","journal-title":"Health Aff (Millwood)"},{"key":"2020122718095897800_ocaa113-B10","doi-asserted-by":"crossref","DOI":"10.1186\/s13012-017-0644-2","article-title":"What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation","volume":"12","author":"Liberati","year":"2017","journal-title":"Implement Sci"},{"issue":"2","key":"2020122718095897800_ocaa113-B11","doi-asserted-by":"crossref","first-page":"54","DOI":"10.3390\/healthcare6020054","article-title":"A systematic review on healthcare analytics: application and theoretical perspective of data mining","volume":"6","author":"Islam","year":"2018","journal-title":"Healthcare (Basel)"},{"issue":"7","key":"2020122718095897800_ocaa113-B12","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1377\/hlthaff.2014.0041","article-title":"Big data in health care: using analytics to identify and manage high-risk and high-cost patients","volume":"33","author":"Bates","year":"2014","journal-title":"Health Aff (Millwood)"},{"issue":"9920","key":"2020122718095897800_ocaa113-B13","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1016\/S0140-6736(13)60688-1","article-title":"Delirium in elderly people","volume":"383","author":"Inouye","year":"2014","journal-title":"Lancet"},{"issue":"4","key":"2020122718095897800_ocaa113-B14","doi-asserted-by":"crossref","first-page":"512","DOI":"10.1001\/jamainternmed.2014.7779","article-title":"Effectiveness of multicomponent nonpharmacological delirium interventions: a meta-analysis","volume":"175","author":"Hshieh","year":"2015","journal-title":"JAMA Intern Med"},{"issue":"9","key":"2020122718095897800_ocaa113-B15","doi-asserted-by":"crossref","first-page":"669","DOI":"10.1056\/NEJM199903043400901","article-title":"A multicomponent intervention to prevent delirium in hospitalized older patients","volume":"340","author":"Inouye","year":"1999","journal-title":"N Engl J Med"},{"issue":"1","key":"2020122718095897800_ocaa113-B16","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1891\/rtnp.17.1.31.53169","article-title":"The Delirium Observation Screening Scale: a screening instrument for delirium","volume":"17","author":"Schuurmans","year":"2003","journal-title":"Res Theory Nurs Pract"},{"issue":"12","key":"2020122718095897800_ocaa113-B17","doi-asserted-by":"crossref","first-page":"941","DOI":"10.7326\/0003-4819-113-12-941","article-title":"Clarifying confusion: the confusion assessment method. A new method for detection of delirium","volume":"113","author":"Inouye","year":"1990","journal-title":"Ann Intern Med"},{"key":"2020122718095897800_ocaa113-B18","first-page":"32","article-title":"Development and validation of a multivariable prediction model for the occurrence of delirium in hospitalized gerontopsychiatry and internal medicine patients","volume":"236","author":"Kramer","year":"2017","journal-title":"Stud Health Technol Inform"},{"issue":"12","key":"2020122718095897800_ocaa113-B19","doi-asserted-by":"crossref","first-page":"2383","DOI":"10.1111\/jgs.13138","article-title":"Risk prediction models for postoperative delirium: a systematic review and meta-analysis","volume":"62","author":"van Meenen","year":"2014","journal-title":"J Am Geriatr Soc"},{"issue":"3","key":"2020122718095897800_ocaa113-B20","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1093\/bja\/aew476","article-title":"Risk prediction models for delirium in the intensive care unit after cardiac surgery: a systematic review and independent external validation","volume":"118","author":"Lee","year":"2017","journal-title":"Br J Anaesth"},{"issue":"4","key":"2020122718095897800_ocaa113-B21","doi-asserted-by":"crossref","first-page":"e019223","DOI":"10.1136\/bmjopen-2017-019223","article-title":"Systematic review of prediction models for delirium in the older adult inpatient","volume":"8","author":"Lindroth","year":"2018","journal-title":"BMJ Open"},{"issue":"5","key":"2020122718095897800_ocaa113-B22","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1016\/j.genhosppsych.2015.05.003","article-title":"Predicting delirium: a review of risk-stratification models","volume":"37","author":"Newman","year":"2015","journal-title":"Gen Hosp Psychiatry"},{"issue":"12","key":"2020122718095897800_ocaa113-B23","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/s10916-018-1109-0","article-title":"Prediction of incident delirium using a random forest classifier","volume":"42","author":"Corradi","year":"2018","journal-title":"J Med Syst"},{"issue":"4","key":"2020122718095897800_ocaa113-B24","doi-asserted-by":"crossref","first-page":"e181018","DOI":"10.1001\/jamanetworkopen.2018.1018","article-title":"Development and validation of an electronic health record\u2013based machine learning model to estimate delirium risk in newly hospitalized patients without known cognitive impairment","volume":"1","author":"Wong","year":"2018","journal-title":"JAMA Netw Open"},{"issue":"12","key":"2020122718095897800_ocaa113-B25","doi-asserted-by":"crossref","first-page":"e3072","DOI":"10.1097\/MD.0000000000003072","article-title":"DELirium prediction based on hospital information (Delphi) in general surgery patients","volume":"95","author":"Kim","year":"2016","journal-title":"Medicine (Baltimore)"},{"issue":"1","key":"2020122718095897800_ocaa113-B26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ijmedinf.2008.09.002","article-title":"STARE-HI -statement on reporting of evaluation studies in health informatics","volume":"78","author":"Talmon","year":"2009","journal-title":"Int J Med Inform"},{"issue":"4","key":"2020122718095897800_ocaa113-B27","doi-asserted-by":"crossref","first-page":"500","DOI":"10.1007\/s11606-017-4204-x","article-title":"Identifying older adults at risk of delirium following elective surgery: a systematic review and meta-analysis","volume":"33","author":"Watt","year":"2018","journal-title":"J Gen Intern Med"},{"key":"2020122718095897800_ocaa113-B28","first-page":"97","article-title":"On the representation of machine learning results for delirium prediction in a hospital information system in routine care","author":"Veeranki","year":"2018","journal-title":"Stud Health Technol Inform"},{"issue":"3","key":"2020122718095897800_ocaa113-B29","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"key":"2020122718095897800_ocaa113-B30","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1007\/978-3-030-03623-2_4","volume-title":"Anticipation, Agency and Complexity. Cham, Switzerland","author":"Sabetta","year":"2019"},{"key":"2020122718095897800_ocaa113-B31","author":"DELIRIUM: Diagnosis, Prevention and Management. London, United Kingdom: National Clinical","year":"2010"},{"issue":"4","key":"2020122718095897800_ocaa113-B32","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1093\/ageing\/afl005","article-title":"Occurrence and outcome of delirium in medical in-patients: a systematic literature review","volume":"35","author":"Siddiqi","year":"2006","journal-title":"Age Ageing"},{"issue":"10","key":"2020122718095897800_ocaa113-B33","doi-asserted-by":"crossref","first-page":"898","DOI":"10.1007\/s12630-010-9355-8","article-title":"Hospital administrative database underestimates delirium rate after cardiac surgery","volume":"57","author":"Katznelson","year":"2010","journal-title":"Can J Anaesth"},{"key":"2020122718095897800_ocaa113-B34","first-page":"65","article-title":"Information adapted machine learning models for prediction in clinical workflow","volume":"260","author":"Jauk","year":"2019","journal-title":"Stud Health Technol Inform"},{"issue":"1","key":"2020122718095897800_ocaa113-B35","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1186\/s12873-018-0170-9","article-title":"Assessment of the Feasibility of automated, real-time clinical decision support in the emergency department using electronic health record data","volume":"18","author":"Perry","year":"2018","journal-title":"BMC Emerg Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/9\/1383\/35176718\/ocaa113.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/9\/1383\/35176718\/ocaa113.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,27]],"date-time":"2020-12-27T23:10:38Z","timestamp":1609110638000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/9\/1383\/5910737"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,1]]},"references-count":35,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2020,9,24]]},"published-print":{"date-parts":[[2020,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa113","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,9]]},"published":{"date-parts":[[2020,9,1]]}}}