{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,26]],"date-time":"2026-09-26T01:43:46Z","timestamp":1790387026607,"version":"4.1.0"},"reference-count":49,"publisher":"Georg Thieme Verlag KG","issue":"01","funder":[{"name":"Advance-CTR Grant (National Institute of Health","award":["U54GM115677"],"award-info":[{"award-number":["U54GM115677"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2022,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>\n          Background\u2003Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools for clinicians. These models have the potential to improve care, but are challenging to internally validate, implement, and maintain over the long term. Principles of ML operations (MLOps) may inform development of infrastructure to support the entire ML lifecycle, from feature selection to long-term model deployment and retraining.<\/jats:p><jats:p>\n          Objectives\u2003This study aimed to present the conceptual prototypes for a novel predictive model management system and to evaluate the acceptability of the system among three groups of end users.<\/jats:p><jats:p>\n          Methods\u2003Based on principles of user-centered software design, human-computer interaction, and ethical design, we created graphical prototypes of a web-based MLOps interface to support the construction, deployment, and maintenance of models using EHR data. To assess the acceptability of the interface, we conducted semistructured user interviews with three groups of users (health informaticians, clinical and data stakeholders, chief information officers) and evaluated preliminary usability using the System Usability Scale (SUS). We subsequently revised prototypes based on user input and developed user case studies.<\/jats:p><jats:p>\n          Results\u2003Our prototypes include design frameworks for feature selection, model training, deployment, long-term maintenance, visualization over time, and cross-functional collaboration. Users were able to complete 71% of prompted tasks without assistance. The average SUS score of the initial prototype was 75.8 out of 100, translating to a percentile range of 70 to 79, a letter grade of B, and an adjective rating of \u201cgood.\u201d We reviewed persona-based case studies that illustrate functionalities of this novel prototype.<\/jats:p><jats:p>\n          Conclusion\u2003The initial graphical prototypes of this MLOps system are preliminarily usable and demonstrate an unmet need within the clinical informatics landscape.<\/jats:p>","DOI":"10.1055\/s-0041-1740923","type":"journal-article","created":{"date-parts":[[2022,2,16]],"date-time":"2022-02-16T23:56:32Z","timestamp":1645055792000},"page":"056-066","source":"Crossref","is-referenced-by-count":6,"title":["A Graphical Toolkit for Longitudinal Dataset Maintenance and Predictive Model Training in Health Care"],"prefix":"10.1055","volume":"13","author":[{"given":"Eric","family":"Bai","sequence":"first","affiliation":[{"name":"Warren Alpert Medical School, Brown University, Providence, Rhode Island, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sophia L.","family":"Song","sequence":"first","affiliation":[{"name":"Warren Alpert Medical School, Brown University, Providence, Rhode Island, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hamish S. F.","family":"Fraser","sequence":"additional","affiliation":[{"name":"Brown University Center for Biomedical Informatics, Providence, Rhode Island, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Megan L.","family":"Ranney","sequence":"additional","affiliation":[{"name":"Brown-Lifespan Center for Digital Health, Providence, Rhode Island, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"194","published-online":{"date-parts":[[2022,2,16]]},"reference":[{"issue":"04","key":"ref1","doi-asserted-by":"crossref","first-page":"e38","DOI":"10.2196\/medinform.5359","article-title":"Challenges and opportunities of big data in health care: a systematic review","volume":"4","author":"C S Kruse","year":"2016","journal-title":"JMIR Med Inform"},{"issue":"07","key":"ref2","doi-asserted-by":"crossref","first-page":"1115","DOI":"10.1377\/hlthaff.2014.0147","article-title":"Creating value in health care through big data: opportunities and policy implications","volume":"33","author":"J Roski","year":"2014","journal-title":"Health Aff (Millwood)"},{"issue":"05","key":"ref3","doi-asserted-by":"crossref","first-page":"e262","DOI":"10.1016\/S1470-2045(19)30149-4","article-title":"Big data and machine learning algorithms for health-care delivery","volume":"20","author":"K Y Ngiam","year":"2019","journal-title":"Lancet Oncol"},{"issue":"15","key":"ref5","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1001\/jama.2011.1515","article-title":"Risk prediction models for hospital readmission: a systematic review","volume":"306","author":"D Kansagara","year":"2011","journal-title":"JAMA"},{"issue":"10","key":"ref6","doi-asserted-by":"crossref","first-page":"e0140271","DOI":"10.1371\/journal.pone.0140271","article-title":"Development, validation and deployment of a real time 30 day hospital readmission risk assessment tool in the maine healthcare information exchange","volume":"10","author":"S Hao","year":"2015","journal-title":"PLoS One"},{"issue":"02","key":"ref7","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1055\/s-0041-1726422","article-title":"Effect of a real-time risk score on 30-day readmission reduction in Singapore","volume":"12","author":"C X Wu","year":"2021","journal-title":"Appl Clin Inform"},{"issue":"03","key":"ref9","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1055\/s-0040-1710525","article-title":"A review of predictive analytics solutions for sepsis patients","volume":"11","author":"A K Teng","year":"2020","journal-title":"Appl Clin Inform"},{"key":"ref10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1017\/S0033291720004997","article-title":"Prediction of suicide attempts in a prospective cohort study with a nationally representative sample of the US population","author":"C DS Machado","year":"2021","journal-title":"Psychol Med"},{"issue":"01","key":"ref11","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1037\/ser0000099","article-title":"Development and applications of the Veterans Health Administration's Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide","volume":"14","author":"E M Oliva","year":"2017","journal-title":"Psychol Serv"},{"issue":"10","key":"ref12","doi-asserted-by":"crossref","first-page":"e0224502","DOI":"10.1371\/journal.pone.0224502","article-title":"Deep-learning-based risk stratification for mortality of patients with acute myocardial infarction","volume":"14","author":"J M Kwon","year":"2019","journal-title":"PLoS One"},{"issue":"04","key":"ref13","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1055\/s-0040-1715892","article-title":"A web application for adrenal incidentaloma identification, tracking, and management using machine learning","volume":"11","author":"W Bala","year":"2020","journal-title":"Appl Clin Inform"},{"key":"ref14","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1038\/s41746-020-00318-y","article-title":"Developing a delivery science for artificial intelligence in healthcare","volume":"3","author":"R C Li","year":"2020","journal-title":"NPJ Digit Med"},{"issue":"11","key":"ref15","doi-asserted-by":"crossref","first-page":"e16323","DOI":"10.2196\/16323","article-title":"The last mile: where artificial intelligence meets reality","volume":"21","author":"E Coiera","year":"2019","journal-title":"J Med Internet Res"},{"issue":"01","key":"ref16","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1186\/1741-7015-9-103","article-title":"Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting","volume":"9","author":"G S Collins","year":"2011","journal-title":"BMC Med"},{"issue":"01","key":"ref17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12916-014-0241-z","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement","volume":"13","author":"G S Collins","year":"2015","journal-title":"BMC Med"},{"issue":"03","key":"ref18","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1016\/j.jclinepi.2012.06.020","article-title":"A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods","volume":"66","author":"G S Collins","year":"2013","journal-title":"J Clin Epidemiol"},{"issue":"07","key":"ref19","doi-asserted-by":"crossref","first-page":"e15182","DOI":"10.2196\/15182","article-title":"Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study","volume":"8","author":"M P Sendak","year":"2020","journal-title":"JMIR Med Inform"},{"issue":"23","key":"ref20","doi-asserted-by":"crossref","first-page":"2161","DOI":"10.1056\/NEJMp1401111","article-title":"Learning from big health care data","volume":"370","author":"S Schneeweiss","year":"2014","journal-title":"N Engl J Med"},{"issue":"03","key":"ref21","doi-asserted-by":"crossref","first-page":"25","DOI":"10.3390\/informatics7030025","article-title":"Clinical implementation of predictive models embedded within electronic health record systems: a systematic review","volume":"7","author":"T C Lee","year":"2020","journal-title":"Informatics (MDPI)"},{"issue":"01","key":"ref22","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1093\/jamia\/ocw042","article-title":"Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review","volume":"24","author":"B A Goldstein","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"ref24","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-35333-9_14","volume-title":"Empirical analysis of hidden technical debt patterns in machine learning software","author":"M Alahdab","year":"2019"},{"issue":"07","key":"ref27","doi-asserted-by":"crossref","first-page":"363","DOI":"10.3390\/info11070363","article-title":"Applying DevOps practices of continuous automation for machine learning","volume":"11","author":"I Karamitsos","year":"2020","journal-title":"Information (Basel)"},{"issue":"01","key":"ref38","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1186\/s12911-016-0244-0","article-title":"Investigating the roots of successful IT adoption processes - an empirical study exploring the shared awareness-knowledge of Directors of Nursing and Chief Information Officers","volume":"16","author":"J D Liebe","year":"2016","journal-title":"BMC Med Inform Decis Mak"},{"issue":"05","key":"ref39","doi-asserted-by":"crossref","first-page":"709","DOI":"10.1093\/jamia\/ocaa021","article-title":"\u201cHow did you get to this number?\u201d Stakeholder needs for implementing predictive analytics: a pre-implementation qualitative study","volume":"27","author":"N C Benda","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"08","key":"ref41","doi-asserted-by":"crossref","first-page":"969","DOI":"10.1093\/jamia\/ocy032","article-title":"Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data","volume":"25","author":"J M Reps","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"ref42","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1177\/1937586720908508","article-title":"Respect for autonomy: seeking the roles of healthcare design from the principle of biomedical ethics","volume":"13","author":"L Zhu","year":"2020","journal-title":"HERD"},{"issue":"01","key":"ref43","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1111\/j.1945-1474.2009.00003.x","article-title":"The application of design principles to innovate clinical care delivery","volume":"31","author":"M D Brennan","year":"2009","journal-title":"J Healthc Qual"},{"issue":"06","key":"ref44","doi-asserted-by":"crossref","first-page":"703","DOI":"10.1177\/1740774520946314","article-title":"User-centered design principles in the development of clinical research tools","volume":"17","author":"A Walden","year":"2020","journal-title":"Clin Trials"},{"issue":"01","key":"ref45","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1177\/1460458212448890","article-title":"Design principles for achieving integrated healthcare information systems","volume":"19","author":"T B Jensen","year":"2013","journal-title":"Health Informatics J"},{"key":"ref48","volume-title":"Designing the User Interface: Strategies for Effective Human-Computer Interaction","author":"B Shneiderman","year":"2017"},{"issue":"07","key":"ref50","doi-asserted-by":"crossref","first-page":"501","DOI":"10.21037\/atm.2020.03.63","article-title":"Bridging the \u201clast mile\u201d gap between AI implementation and operation: \u201cdata awareness\u201d that matters","volume":"8","author":"F Cabitza","year":"2020","journal-title":"Ann Transl Med"},{"issue":"06","key":"ref52","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1080\/10447310802205776","article-title":"An empirical evaluation of the system usability scale","volume":"24","author":"A Bangor","year":"2008","journal-title":"Int J Hum Comput Interact"},{"issue":"08","key":"ref53","doi-asserted-by":"crossref","first-page":"545","DOI":"10.1080\/10447318.2015.1064665","article-title":"Psychometric evaluation of the EMO and the SUS in the context of a large-sample unmoderated usability study","volume":"31","author":"J R Lewis","year":"2015","journal-title":"Int J Hum Comput Interact"},{"issue":"01","key":"ref55","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1177\/1541931213571043","article-title":"Validation of the system usability scale (SUS): SUS in the wild","volume":"57","author":"S C Peres","year":"2013","journal-title":"Proc Hum Fact Ergon Soc Annu Meet"},{"key":"ref56","doi-asserted-by":"publisher","DOI":"10.2200\/S00706ED1V01Y201602HCI034","article-title":"Qualitative HCI research: Going behind the scenes","author":"A Blandford","year":"2016"},{"key":"ref57","volume-title":"A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices","author":"J Sauro","year":"2011"},{"key":"ref58","volume-title":"Quantifying the User Experience: Practical Statistics for User Research","author":"J Sauro","year":"2016"},{"issue":"01","key":"ref59","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.cmpb.2012.10.018","article-title":"Alternatives to relational database: comparison of NoSQL and XML approaches for clinical data storage","volume":"110","author":"K K-Y Lee","year":"2013","journal-title":"Comput Methods Programs Biomed"},{"issue":"01","key":"ref61","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1137\/141000671","article-title":"Julia: a fresh approach to numerical computing","volume":"59","author":"J Bezanson","year":"2017","journal-title":"SIAM Rev"},{"key":"ref62","volume-title":"Python 3 Reference Manual","author":"G Drake FL Rossum","year":"2009"},{"issue":"23","key":"ref64","doi-asserted-by":"crossref","first-page":"2171","DOI":"10.1056\/NEJMp2102616","article-title":"HIPAA and the leak of \u201cDeidentified\u201d EHR data","volume":"384","author":"K D Mandl","year":"2021","journal-title":"N Engl J Med"},{"issue":"02","key":"ref65","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1016\/j.jand.2020.11.012","article-title":"Managing HIPAA compliance includes legal and ethical considerations","volume":"121","author":"T Peregrin","year":"2021","journal-title":"J Acad Nutr Diet"},{"issue":"01","key":"ref66","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1007\/s10916-006-7405-0","article-title":"Challenges associated with privacy in health care industry: implementation of HIPAA and the security rules","volume":"30","author":"Y B Choi","year":"2006","journal-title":"J Med Syst"},{"issue":"08","key":"ref68","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"A Wong","year":"2021","journal-title":"JAMA Intern Med"},{"key":"ref69","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1016\/j.jacr.2020.11.008","article-title":"Failures hiding in success for artificial intelligence in radiology","volume":"18","author":"S Purkayastha","year":"2021","journal-title":"J Am Coll Radiol"},{"key":"ref70","doi-asserted-by":"crossref","first-page":"103611","DOI":"10.1016\/j.jbi.2020.103611","article-title":"Detection of calibration drift in clinical prediction models to inform model updating","volume":"112","author":"S E Davis","year":"2020","journal-title":"J Biomed Inform"},{"issue":"08","key":"ref71","doi-asserted-by":"crossref","first-page":"100129","DOI":"10.1016\/j.patter.2020.100129","article-title":"Extensions of the external validation for checking learned model interpretability and generalizability","volume":"1","author":"S Y Ho","year":"2020","journal-title":"Patterns (N Y)"},{"issue":"01","key":"ref72","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1109\/JPROC.2020.3004555","article-title":"A comprehensive survey on transfer learning","volume":"109","author":"F Zhuang","year":"2021","journal-title":"Proc IEEE"},{"issue":"02","key":"ref73","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1007\/s41666-019-00062-3","article-title":"Transfer learning for clinical time series analysis using deep neural networks","volume":"4","author":"P Gupta","year":"2020","journal-title":"J Healthc Inform Res"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0041-1740923.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,16]],"date-time":"2022-02-16T23:57:08Z","timestamp":1645055828000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0041-1740923"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1]]},"references-count":49,"journal-issue":{"issue":"01","published-online":{"date-parts":[[2022,1,5]]},"published-print":{"date-parts":[[2022,1]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0041-1740923","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"value":"1869-0327","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1]]}}}