{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T07:45:19Z","timestamp":1792136719186,"version":"4.4.0"},"reference-count":35,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:p>Digital health tools, platforms, and artificial intelligence\u2013 or machine learning\u2013based clinical decision support systems are increasingly part of health delivery approaches, with an ever-greater degree of system interaction. Critical to the successful deployment of these tools is their functional integration into existing clinical routines and workflows. This depends on system interoperability and on intuitive and safe user interface design. The importance of minimizing emergent workflow stress through human factors research and purposeful design for integration cannot be overstated. Usability of tools in practice is as important as algorithm quality. Regulatory and health technology assessment frameworks recognize the importance of these factors to a certain extent, but their focus remains mainly on the individual product rather than on emergent system and workflow effects. The measurement of performance and user experience has so far been performed in ad hoc, nonstandardized ways by individual actors using their own evaluation approaches. We propose that a standard framework for system-level and holistic evaluation could be built into interacting digital systems to enable systematic and standardized system-wide, multiproduct, postmarket surveillance and technology assessment. Such a system could be made available to developers through regulatory or assessment bodies as an application programming interface and could be a requirement for digital tool certification, just as interoperability is. This would enable health systems and tool developers to collect system-level data directly from real device use cases, enabling the controlled and safe delivery of systematic quality assessment or improvement studies suitable for the complexity and interconnectedness of clinical workflows using developing digital health technologies.<\/jats:p>","DOI":"10.2196\/50158","type":"journal-article","created":{"date-parts":[[2023,12,20]],"date-time":"2023-12-20T11:00:32Z","timestamp":1703070032000},"page":"e50158","source":"Crossref","is-referenced-by-count":25,"title":["Holistic Human-Serving Digitization of Health Care Needs Integrated Automated System-Level Assessment Tools"],"prefix":"10.2196","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5710-1975","authenticated-orcid":false,"given":"Cindy","family":"Welzel","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2898-422X","authenticated-orcid":false,"given":"Fabienne","family":"Cotte","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4498-107X","authenticated-orcid":false,"given":"Magdalena","family":"Wekenborg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0017-8891","authenticated-orcid":false,"given":"Baptiste","family":"Vasey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3210-8273","authenticated-orcid":false,"given":"Peter","family":"McCulloch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1997-1689","authenticated-orcid":false,"given":"Stephen","family":"Gilbert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2023,12,20]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.1276"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.2196\/30545"},{"key":"ref3","unstructured":"Interoperability standards in digital health\u2014a white paper from the medical technology industryMedTech Europe2023-05-05https:\/\/www.medtecheurope.org\/resource-library\/interoperability-standards-in-digital-health-a-white-paper-from-the-medical-technology-industry\/"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0262-2"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1017\/S0266462323000090"},{"key":"ref6","unstructured":"The MDR's usability\/human factors requirementsJohner Institute2023-04-06https:\/\/www.johner-institute.com\/articles\/regulatory-affairs\/and-more\/mdr-usability-requirements\/"},{"key":"ref7","unstructured":"Applying human factors and usability engineering to medical devicesU.S. Food and Drug Administration20192023-04-06https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/applying-human-factors-and-usability-engineering-medical-devices"},{"key":"ref8","unstructured":"Digital Technology Assessment CriteriaNHS Transformation Directorate2023-03-22https:\/\/transform.england.nhs.uk\/key-tools-and-info\/digital-technology-assessment-criteria-dtac\/"},{"key":"ref9","unstructured":"Marketing submission recommendations for a predetermined change control plan for artificial intelligence\/machine learning (AI\/ML)-enabled device software functionsU.S. Food and Drug Administration20232023-05-05https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/marketing-submission-recommendations-predetermined-change-control-plan-artificial"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2018.12777"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1093\/eurpub\/ckz160"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1177\/08404704221139219"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2021.635112"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.2196\/28199"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.2196\/26402"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.2196\/32340"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2013-001875"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-018-3094-x"},{"key":"ref19","author":"Brod, C","year":"1984","journal-title":"Technostress: The Human Cost of the Computer Revolution"},{"key":"ref20","year":"1991","journal-title":"IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries, 610"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-019-0158-1"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1016\/j.psychres.2018.01.002"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.3390\/designs7010009"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare11030296"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000222"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-022-01772-9"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.29337\/ijdh.49"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.2105\/AJPH.2014.302256"},{"issue":"Suppl","key":"ref29","first-page":"207","volume":"12","author":"Vatanpour, H","year":"2013","journal-title":"Iran J Pharm Res"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1016\/j.mcpdig.2023.05.010"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.2196\/43682"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1016\/j.ibmed.2023.100102"},{"key":"ref33","first-page":"264","author":"Paterson, KG","year":"2010","journal-title":"Information Security and Privacy. ACISP 2010. Lecture Notes in Computer Science, vol 6168"},{"key":"ref34","unstructured":"FHIR, phase I\u2014DiGA toolkit 1.0.0MIO2023-04-06https:\/\/mio.kbv.de\/pages\/viewpage.action?pageId=94508825"},{"key":"ref35","unstructured":"Proposal for a regulation of the European parliament and of the council on the European health data spaceEUR-Lex20222023-02-06https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX%3A52022PC0197"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2023\/1\/e50158","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:32:37Z","timestamp":1782840757000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2023\/1\/e50158"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,20]]},"references-count":35,"URL":"https:\/\/doi.org\/10.2196\/50158","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.50158","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,20]]}}}