{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,20]],"date-time":"2026-06-20T01:32:23Z","timestamp":1781919143979,"version":"3.54.5"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,3,1]],"date-time":"2021-03-01T00:00:00Z","timestamp":1614556800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,1]],"date-time":"2021-03-01T00:00:00Z","timestamp":1614556800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100010109","name":"Medical University of Graz","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100010109","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Med Syst"],"published-print":{"date-parts":[[2021,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Early identification of patients with life-threatening risks such as delirium is crucial in order to initiate preventive actions as quickly as possible. Despite intense research on machine learning for the prediction of clinical outcomes, the acceptance of the integration of such complex models in clinical routine remains unclear. The aim of this study was to evaluate user acceptance of an already implemented machine learning-based application predicting the risk of delirium for in-patients. We applied a mixed methods design to collect opinions and concerns from health care professionals including physicians and nurses who regularly used the application. The evaluation was framed by the Technology Acceptance Model assessing perceived ease of use, perceived usefulness, actual system use and output quality of the application. Questionnaire results from\u00a047 nurses and physicians as well as qualitative results of four expert group meetings rated the overall usefulness of the delirium prediction positively. For healthcare professionals, the visualization and presented information was understandable, the application was easy to use and the additional information for delirium management was appreciated. The application did not increase their workload, but the actual system use was still low during the pilot study. Our study provides insights into the user acceptance of a machine learning-based application supporting delirium management in hospitals. In order to improve quality and safety in healthcare, computerized decision support should predict actionable events and be highly accepted by users.<\/jats:p>","DOI":"10.1007\/s10916-021-01727-6","type":"journal-article","created":{"date-parts":[[2021,3,1]],"date-time":"2021-03-01T15:03:46Z","timestamp":1614611026000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":47,"title":["Technology Acceptance of a Machine Learning Algorithm Predicting Delirium in a Clinical Setting: a Mixed-Methods Study"],"prefix":"10.1007","volume":"45","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1287-594X","authenticated-orcid":false,"given":"Stefanie","family":"Jauk","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diether","family":"Kramer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Avian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Berghold","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Werner","family":"Leodolter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefan","family":"Schulz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,3,1]]},"reference":[{"key":"1727_CR1","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.artmed.2015.07.003","volume":"65","author":"N Peek","year":"2015","unstructured":"Peek N, Combi C, Marin R, et al. Thirty years of artificial intelligence in medicine (AIME) conferences: A review of research themes. Artif Intell Med 2015;65:61\u201373. https:\/\/doi.org\/10.1016\/j.artmed.2015.07.003","journal-title":"Artif Intell Med"},{"key":"1727_CR2","doi-asserted-by":"publisher","first-page":"363","DOI":"10.1136\/jamia.1996.97084510","volume":"3","author":"EW Coiera","year":"1996","unstructured":"Coiera EW. Artificial Intelligence in Medicine: The Challenges Ahead. J Am Med Inform Assoc 1996;3:363\u20136. https:\/\/doi.org\/10.1136\/jamia.1996.97084510","journal-title":"J Am Med Inform Assoc"},{"key":"1727_CR3","doi-asserted-by":"publisher","unstructured":"Rajkomar A, Oren E, Chen K, et al. Scalable and accurate deep learning with electronic health records. Npj Digit Med 2018;1. https:\/\/doi.org\/10.1038\/s41746-018-0029-1","DOI":"10.1038\/s41746-018-0029-1"},{"key":"1727_CR4","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1038\/s41591-018-0307-0","volume":"25","author":"J He","year":"2019","unstructured":"He J, Baxter SL, Xu J, et al. The practical implementation of artificial intelligence technologies in medicine. Nat Med 2019;25:30\u20136. https:\/\/doi.org\/10.1038\/s41591-018-0307-0","journal-title":"Nat Med"},{"key":"1727_CR5","doi-asserted-by":"publisher","unstructured":"Vollmer S, Mateen BA, Bohner G, et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ 2020;l6927. https:\/\/doi.org\/10.1136\/bmj.l6927","DOI":"10.1136\/bmj.l6927"},{"key":"1727_CR6","doi-asserted-by":"publisher","unstructured":"Watson J, Hutyra CA, Clancy SM, et al. Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers? JAMIA Open 2020;ooz046. https:\/\/doi.org\/10.1093\/jamiaopen\/ooz046","DOI":"10.1093\/jamiaopen\/ooz046"},{"key":"1727_CR7","doi-asserted-by":"publisher","first-page":"1148","DOI":"10.1377\/hlthaff.2014.0352","volume":"33","author":"R Amarasingham","year":"2014","unstructured":"Amarasingham R, Patzer RE, Huesch M, et al. Implementing Electronic Health Care Predictive Analytics: Considerations And Challenges. Health Aff (Millwood) 2014;33:1148\u201354. https:\/\/doi.org\/10.1377\/hlthaff.2014.0352","journal-title":"Health Aff (Millwood)"},{"key":"1727_CR8","doi-asserted-by":"publisher","unstructured":"Magrabi F, Ammenwerth E, McNair J, et al. Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications: A Position Paper from the IMIA Technology Assessment & Quality Development in Health Informatics Working Group and the EFMI Working Group for Assessment of Health Information Systems. Yearb Med Inform Published Online First: 25 April 2019. https:\/\/doi.org\/10.1055\/s-0039-1677903","DOI":"10.1055\/s-0039-1677903"},{"key":"1727_CR9","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1136\/bmjqs-2018-008370","volume":"28","author":"R Challen","year":"2019","unstructured":"Challen R, Denny J, Pitt M, et al. Artificial intelligence, bias and clinical safety. BMJ Qual Saf 2019;28:231\u20137. https:\/\/doi.org\/10.1136\/bmjqs-2018-008370","journal-title":"BMJ Qual Saf"},{"key":"1727_CR10","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1093\/fampra\/cmn020","volume":"25","author":"H Varonen","year":"2008","unstructured":"Varonen H, Kortteisto T, Kaila M, et al. What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians. Fam Pract 2008;25:162\u20137. https:\/\/doi.org\/10.1093\/fampra\/cmn020","journal-title":"Fam Pract"},{"key":"1727_CR11","doi-asserted-by":"publisher","unstructured":"Liberati EG, Ruggiero F, Galuppo L, et al. What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci 2017;12. https:\/\/doi.org\/10.1186\/s13012-017-0644-2","DOI":"10.1186\/s13012-017-0644-2"},{"key":"1727_CR12","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1197\/jamia.M3170","volume":"17","author":"A Moxey","year":"2010","unstructured":"Moxey A, Robertson J, Newby D, et al. Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc 2010;17:25\u201333. https:\/\/doi.org\/10.1197\/jamia.M3170","journal-title":"J Am Med Inform Assoc"},{"key":"1727_CR13","doi-asserted-by":"publisher","unstructured":"Brennan M, Puri S, Ozrazgat-Baslanti T, et al. Comparing clinical judgment with the MySurgeryRisk algorithm for preoperative risk assessment: A pilot usability study. Surgery Published Online First: 18 February 2019. https:\/\/doi.org\/10.1016\/j.surg.2019.01.002","DOI":"10.1016\/j.surg.2019.01.002"},{"key":"1727_CR14","doi-asserted-by":"publisher","unstructured":"Bihorac A, Ozrazgat-Baslanti T, Ebadi A, et al. MySurgeryRisk: Development and Validation of a Machine-learning Risk Algorithm for Major Complications and Death After Surgery. Ann Surg 2018;1. https:\/\/doi.org\/10.1097\/SLA.0000000000002706","DOI":"10.1097\/SLA.0000000000002706"},{"key":"1727_CR15","doi-asserted-by":"publisher","first-page":"1477","DOI":"10.1097\/CCM.0000000000003803","volume":"47","author":"JC Ginestra","year":"2019","unstructured":"Ginestra JC, Giannini HM, Schweickert WD, et al. Clinician Perception of a Machine Learning\u2013Based Early Warning System Designed to Predict Severe Sepsis and Septic Shock*: Crit Care Med 2019;47:1477\u201384. https:\/\/doi.org\/10.1097\/CCM.0000000000003803","journal-title":"Crit Care Med"},{"key":"1727_CR16","doi-asserted-by":"publisher","first-page":"1485","DOI":"10.1097\/CCM.0000000000003891","volume":"47","author":"HM Giannini","year":"2019","unstructured":"Giannini HM, Ginestra JC, Chivers C, et al. A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice*. Crit Care Med 2019;47:1485\u201392. https:\/\/doi.org\/10.1097\/CCM.0000000000003891","journal-title":"Crit Care Med"},{"key":"1727_CR17","doi-asserted-by":"publisher","first-page":"1383","DOI":"10.1093\/jamia\/ocaa113","volume":"27","author":"S Jauk","year":"2020","unstructured":"Jauk S, Kramer D, Gro\u00dfauer B, et al. Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study. J Am Med Inform Assoc 2020;27:1383\u201392. https:\/\/doi.org\/10.1093\/jamia\/ocaa113","journal-title":"J Am Med Inform Assoc"},{"key":"1727_CR18","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1016\/S0140-6736(13)60688-1","volume":"383","author":"SK Inouye","year":"2014","unstructured":"Inouye SK, Westendorp RG, Saczynski JS. Delirium in elderly people. The Lancet 2014;383:911\u201322. https:\/\/doi.org\/10.1016\/S0140-6736(13)60688-1","journal-title":"The Lancet"},{"key":"1727_CR19","doi-asserted-by":"publisher","first-page":"1663","DOI":"10.1016\/j.athoracsur.2015.12.074","volume":"101","author":"CH Brown 4th","year":"2016","unstructured":"Brown CH 4th, Laflam A, Max L, et al. The Impact of Delirium After Cardiac Surgical Procedures on Postoperative Resource Use. Ann Thorac Surg 2016;101:1663\u20139. https:\/\/doi.org\/10.1016\/j.athoracsur.2015.12.074","journal-title":"Ann Thorac Surg"},{"key":"1727_CR20","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1001\/jamainternmed.2014.7779","volume":"175","author":"TT Hshieh","year":"2015","unstructured":"Hshieh TT, Yue J, Oh E, et al. Effectiveness of Multicomponent Nonpharmacological Delirium Interventions: A Meta-analysis. JAMA Intern Med 2015;175:512. https:\/\/doi.org\/10.1001\/jamainternmed.2014.7779","journal-title":"JAMA Intern Med"},{"key":"1727_CR21","doi-asserted-by":"publisher","first-page":"669","DOI":"10.1056\/NEJM199903043400901","volume":"340","author":"SK Inouye","year":"1999","unstructured":"Inouye SK, Bogardus Jr ST, Charpentier PA, et al. A multicomponent intervention to prevent delirium in hospitalized older patients. N Engl J Med 1999;340:669\u2013676.","journal-title":"N Engl J Med"},{"key":"1727_CR22","doi-asserted-by":"publisher","first-page":"319","DOI":"10.2307\/249008","volume":"13","author":"FD Davis","year":"1989","unstructured":"Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Q 1989;13:319. https:\/\/doi.org\/10.2307\/249008","journal-title":"MIS Q"},{"key":"1727_CR23","doi-asserted-by":"publisher","first-page":"982","DOI":"10.1287\/mnsc.35.8.982","volume":"35","author":"FD Davis","year":"1989","unstructured":"Davis FD, Bagozzi RP, Warshaw PR. User Acceptance of Computer Technology: A Comparison of Two Theoretical Models. Manag Sci 1989;35:982\u20131003. https:\/\/doi.org\/10.1287\/mnsc.35.8.982","journal-title":"Manag Sci"},{"key":"1727_CR24","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1016\/j.jbi.2009.07.002","volume":"43","author":"RJ Holden","year":"2010","unstructured":"Holden RJ, Karsh B-T. The Technology Acceptance Model: Its past and its future in health care. J Biomed Inform 2010;43:159\u201372. https:\/\/doi.org\/10.1016\/j.jbi.2009.07.002","journal-title":"J Biomed Inform"},{"key":"1727_CR25","volume-title":"Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research","author":"M Fishbein","year":"1975","unstructured":"Fishbein M, Ajzen I. Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research. MA: : Addison-Wesley 1975."},{"key":"1727_CR26","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1287\/mnsc.46.2.186.11926","volume":"46","author":"V Venkatesh","year":"2000","unstructured":"Venkatesh V, Davis FD. A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Manag Sci 2000;46:186\u2013204.","journal-title":"Manag Sci"},{"key":"1727_CR27","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1016\/j.im.2006.05.003","volume":"43","author":"WR King","year":"2006","unstructured":"King WR, He J. A meta-analysis of the technology acceptance model. Inf Manage 2006;43:740\u201355. https:\/\/doi.org\/10.1016\/j.im.2006.05.003","journal-title":"Inf Manage"},{"key":"1727_CR28","doi-asserted-by":"publisher","unstructured":"Veeranki S, Hayn D, Eggerth A, et al. On the Representation of Machine Learning Results for Delirium Prediction in a Hospital Information System in Routine Care. Stud Health Technol Inform 2018;97\u2013100. https:\/\/doi.org\/10.3233\/978-1-61499-880-8-97","DOI":"10.3233\/978-1-61499-880-8-97"},{"key":"1727_CR29","volume-title":"Latent Trait Models under IRT","author":"D Rizopoulos","year":"2018","unstructured":"Rizopoulos D. Latent Trait Models under IRT. 2018."},{"key":"1727_CR30","doi-asserted-by":"publisher","unstructured":"Geerligs L, Rankin NM, Shepherd HL, et al. Hospital-based interventions: a systematic review of staff-reported barriers and facilitators to implementation processes. Implement Sci IS 2018;13. https:\/\/doi.org\/10.1186\/s13012-018-0726-9","DOI":"10.1186\/s13012-018-0726-9"},{"key":"1727_CR31","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1080\/136455799295014","volume":"2","author":"S Michie","year":"1999","unstructured":"Michie S, Marteau T. Non-response bias in prospective studies of patients and health care professionals. Int J Soc Res Methodol 1999;2:203\u201312. https:\/\/doi.org\/10.1080\/136455799295014","journal-title":"Int J Soc Res Methodol"},{"key":"1727_CR32","doi-asserted-by":"publisher","DOI":"10.2196\/12875","volume":"21","author":"H Hypp\u00f6nen","year":"2019","unstructured":"Hypp\u00f6nen H, Kaipio J, Heponiemi T, et al. Developing the National Usability-Focused Health Information System Scale for Physicians: Validation Study. J Med Internet Res 2019;21:e12875. https:\/\/doi.org\/10.2196\/12875","journal-title":"J Med Internet Res"}],"updated-by":[{"DOI":"10.1007\/s10916-021-01728-5","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T00:00:00Z","timestamp":1615334400000}}],"container-title":["Journal of Medical Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10916-021-01727-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10916-021-01727-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10916-021-01727-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,31]],"date-time":"2021-07-31T00:02:33Z","timestamp":1627689753000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10916-021-01727-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,1]]},"references-count":32,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,4]]}},"alternative-id":["1727"],"URL":"https:\/\/doi.org\/10.1007\/s10916-021-01727-6","relation":{"correction":[{"id-type":"doi","id":"10.1007\/s10916-021-01728-5","asserted-by":"object"}]},"ISSN":["0148-5598","1573-689X"],"issn-type":[{"value":"0148-5598","type":"print"},{"value":"1573-689X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,1]]},"assertion":[{"value":"22 December 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 February 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 March 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 March 2021","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1007\/s10916-021-01728-5","URL":"https:\/\/doi.org\/10.1007\/s10916-021-01728-5","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The study received approval from the Ethics Committee of the Medical University of Graz (30\u2013146 ex 17\/18).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"48"}}