{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:44:22Z","timestamp":1785545062692,"version":"3.56.0"},"reference-count":33,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n The successful implementation of decision support systems promises to enhance high-quality care. However, the successful implementation of a clinical decision support system (CDSS) depends on user acceptance and adoption. A machine learning (ML)\u2013based CDSS to assist primary care professionals treating urinary tract infections (UTIs) was implemented, and usability and usefulness were assessed through a questionnaire.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to assess the system\u2019s usability by examining users\u2019 experiences with the software. A secondary goal was to assess users\u2019 attitudes toward evidence-based practice and innovation in health care.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n In collaboration with the Netherlands Institute for Health Services Research (NIVEL) and Leiden University Medical Center (LUMC), Pacmed Ltd developed the CDSS. The cohort was mostly recruited at the care group level; practices within participating care groups were required to participate. Health insurers partly funded the research. Practitioners participated in the implementation study for 4 months. A survey based on the Unified Theory of Acceptance and Use of Technology (UTAUT) was sent to 263 general practitioners and assistants shortly after the implementation period. Furthermore, usage data were analyzed.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Of the 34 participating practices that used the software, 30 (88%) submitted at least one survey response, with a mean of 2.23 responses per practice (SD 1.43). The CDSS was used throughout the pilot period, and 31 practices continued using the tool, with 9% dropping out during the first 8 weeks. Sixty-seven percent of respondents trusted the tool\u2019s output, and 73% found it understandable how the algorithm came to predictions. Sixty-five percent of respondents indicated that the information provided was useful in addition to the available guidelines, and 52% agreed that it supported their decision-making. However, many respondents were uncertain whether the tool improved patient care (46%) or patient outcomes (66%). Forty-eight percent of respondents found the software easy to integrate into their clinical workflow.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The CDSS was perceived as trustworthy and easy to use. However, users were unable to determine whether the CDSS improved patient outcomes. In addition, the CDSS development could have benefited from including assistants as well as general practitioners more in the design phase of the software. Because assistants play an important role in UTI care, designing the software to better fit existing workflows may reduce the perceived time investment associated with using the tool. Finally, respondents reported strong motivation to contribute to further research in this field and indicated willingness to embrace change in health care delivery, which may also reflect selection bias in our sample.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/80527","type":"journal-article","created":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T16:50:09Z","timestamp":1771347009000},"page":"e80527-e80527","source":"Crossref","is-referenced-by-count":1,"title":["Usability and Usefulness of Machine Learning\u2013Based Clinical Decision Support Software in Primary Care: Survey of Users in a Prospective Observational Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9189-6839","authenticated-orcid":false,"given":"Willem Ernst","family":"Herter","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1841-0254","authenticated-orcid":false,"given":"Janine","family":"Khuc","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7719-6182","authenticated-orcid":false,"given":"Tobias N","family":"Bonten","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2234-4819","authenticated-orcid":false,"given":"Robert A","family":"Verheij","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0368-5426","authenticated-orcid":false,"given":"Mattijs E","family":"Numans","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8607-9199","authenticated-orcid":false,"given":"Niels H","family":"Chavannes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,5,27]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1197\/jamia.M3170","article-title":"Computerized clinical decision support for prescribing: provision does not guarantee uptake","volume":"17","author":"Moxey","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"R2","doi-asserted-by":"publisher","DOI":"10.2196\/25759","article-title":"Role of artificial intelligence applications in real-life clinical practice: systematic review","volume":"23","author":"Yin","journal-title":"J Med Internet Res"},{"key":"R3","unstructured":"NHG-standaard urineweginfecties (derde herziening). Huisarts Wet. URL: https:\/\/pure.amsterdamumc.nl\/en\/publications\/nhg-standaard-urineweginfecties-derde-herziening\/ [Accessed 05-04-2026]"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1136\/amiajnl-2014-002886","article-title":"Effects of computer-aided clinical decision support systems in improving antibiotic prescribing by primary care providers: a systematic review","volume":"22","author":"Holstiege","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1186\/1748-5908-4-54","article-title":"Why don\u2019t physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners","volume":"4","author":"Lugtenberg","journal-title":"Implementation Sci"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2014.10.006","article-title":"Using electronic health records for clinical research: the case of the EHR4CR project","volume":"53","author":"De Moor","journal-title":"J Biomed Inform"},{"issue":"6","key":"R7","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1038\/nrg3208","article-title":"Mining electronic health records: towards better research applications and clinical care","volume":"13","author":"Jensen","journal-title":"Nat Rev Genet"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejim.2017.06.017","article-title":"The rise of artificial intelligence and the uncertain future for physicians","volume":"48","author":"Krittanawong","journal-title":"Eur J Intern Med"},{"issue":"5","key":"R9","doi-asserted-by":"publisher","DOI":"10.2196\/27795","article-title":"Impact of a machine learning-based decision support system for urinary tract infections: prospective observational study in 36 primary care practices","volume":"10","author":"Herter","journal-title":"JMIR Med Inform"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6963-12-349","article-title":"Clinical decision support must be useful, functional is not enough: a qualitative study of computer-based clinical decision support in primary care","volume":"12","author":"Kortteisto","journal-title":"BMC Health Serv Res"},{"issue":"12","key":"R11","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0167683","article-title":"The effects of clinical decision support systems on medication safety: an overview","volume":"11","author":"Jia","journal-title":"PLOS ONE"},{"issue":"2","key":"R12","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0170974","article-title":"Effectiveness and usage of a decision support system to improve stroke prevention in general practice: a cluster randomized controlled trial","volume":"12","author":"Arts","journal-title":"PLOS ONE"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1111\/j.1365-2125.2007.02995.x","article-title":"The effects of electronic prescribing on the quality of prescribing","volume":"65","author":"Donyai","journal-title":"Br J Clin Pharmacol"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","first-page":"88","DOI":"10.5694\/j.1326-5377.2006.tb00479.x","article-title":"General practitioners\u2019 use of computers for prescribing and electronic health records: results from a national survey","volume":"185","author":"McInnes","journal-title":"Med J Aust"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1177\/1460458212472333","article-title":"General practitioners\u2019 attitudes towards decision-supported prescribing: an analysis of the Dutch primary care sector","volume":"19","author":"Meulendijk","journal-title":"Health Informatics J"},{"issue":"10","key":"R16","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1097\/CIN.0000000000000185","article-title":"Use of simulation to study nurses\u2019 acceptance and nonacceptance of clinical decision support suggestions","volume":"33","author":"Sousa","journal-title":"Comput Inform Nurs"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1089\/tmj.2011.0066","article-title":"Using a modified technology acceptance model to evaluate healthcare professionals\u2019 adoption of a new telemonitoring system","volume":"18","author":"Gagnon","journal-title":"Telemed J E Health"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2012.10.049","article-title":"Modeling the longitudinality of user acceptance of technology with an evidence-adaptive clinical decision support system","volume":"57","author":"Johnson","journal-title":"Decis Support Syst"},{"issue":"4","key":"R19","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0193187","article-title":"Acceptance and barriers pertaining to a general practice decision support system for multiple clinical conditions: a mixed methods evaluation","volume":"13","author":"Arts","journal-title":"PLOS ONE"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","first-page":"953","DOI":"10.1016\/j.dss.2012.09.018","article-title":"Factors influencing decision support system acceptance","volume":"54","author":"Shibl","journal-title":"Decis Support Syst"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1186\/s13012-014-0179-8","article-title":"Barriers, facilitators and views about next steps to implementing supports for evidence-informed decision-making in health systems: a qualitative study","volume":"9","author":"Ellen","journal-title":"Implement Sci"},{"issue":"7494","key":"R22","doi-asserted-by":"publisher","first-page":"765","DOI":"10.1136\/bmj.38398.500764.8F","article-title":"Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success","volume":"330","author":"Kawamoto","journal-title":"BMJ"},{"issue":"2","key":"R23","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1111\/j.1471-1842.2008.00777.x","article-title":"The attitudes of health care staff to information technology: a comprehensive review of the research literature","volume":"25","author":"Ward","journal-title":"Health Info Libraries J"},{"issue":"2","key":"R24","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1093\/jamia\/ocw105","article-title":"Automation bias and verification complexity: a systematic review","volume":"24","author":"Lyell","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"R25","doi-asserted-by":"publisher","first-page":"909","DOI":"10.1016\/j.acra.2004.05.012","article-title":"Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography","volume":"11","author":"Alberdi","journal-title":"Acad Radiol"},{"key":"R26","unstructured":"Netherlands institute for health services research. NIVEL. URL: https:\/\/www.nivel.nl\/en [Accessed 05-04-2026]"},{"issue":"3","key":"R27","doi-asserted-by":"publisher","first-page":"425","DOI":"10.2307\/30036540","article-title":"User acceptance of information technology: toward a unified view1","volume":"27","author":"Venkatesh","journal-title":"MIS Q"},{"issue":"3","key":"R28","doi-asserted-by":"publisher","first-page":"319","DOI":"10.2307\/249008","article-title":"Perceived usefulness, perceived ease of use, and user acceptance of information technology","volume":"13","author":"Davis","journal-title":"MIS Q"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2296-12-98","article-title":"Perceived barriers to guideline adherence: a survey among general practitioners","volume":"12","author":"Lugtenberg","journal-title":"BMC Fam Pract"},{"key":"R30","unstructured":"Saris WE Oberski DL Revilla MA et al. The development of the program SQP 20 for the prediction of the quality of survey questions. e-Repositori (UPF). 2011. URL: http:\/\/repositori.upf.edu\/handle\/10230\/28334 [Accessed 05-04-2026]"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"Saris WE Gallhofer IN . Design, Evaluation, and Analysis of Questionnaires for Survey Research. John Wiley & Sons, Ltd; 2007:377. [doi: 10.1002\/9780470165195]","DOI":"10.1002\/9780470165195"},{"key":"R32","unstructured":"Velden LFJ Kasteleijn A Kenens RJ . Cijfers uit de registratie van huisartsen, peiling 2016 (nivel). URL: https:\/\/www.nivel.nl\/sites\/default\/files\/bestanden\/cijfers-uit-de-registratie-van-huisartsen-peiling-januari-2016.pdf [Accessed 15-04-2026]"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","DOI":"10.2196\/28639","article-title":"Human factors and technological characteristics influencing the interaction of medical professionals with artificial intelligence-enabled clinical decision support systems: literature review","volume":"9","author":"Knop","journal-title":"JMIR Hum Factors"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e80527","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:26:41Z","timestamp":1785536801000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e80527"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,27]]},"references-count":33,"URL":"https:\/\/doi.org\/10.2196\/80527","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,27]]},"article-number":"v14i1e80527"}}