{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T18:15:03Z","timestamp":1772820903646,"version":"3.50.1"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2024,10,29]],"date-time":"2024-10-29T00:00:00Z","timestamp":1730160000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Incorporating Learning Effects into Medical Device Active Safety Surveillance Methods"},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01 HL149948"],"award-info":[{"award-number":["R01 HL149948"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Heart, Lung, and Blood Institute Career Development","award":["K01HL159315"],"award-info":[{"award-number":["K01HL159315"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>Traditional methods for medical device post-market surveillance often fail to accurately account for operator learning effects, leading to biased assessments of device safety. These methods struggle with non-linearity, complex learning curves, and time-varying covariates, such as physician experience. To address these limitations, we sought to develop a machine learning (ML) framework to detect and adjust for operator learning effects.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>A gradient-boosted decision tree ML method was used to analyze synthetic datasets that replicate the complexity of clinical scenarios involving high-risk medical devices. We designed this process to detect learning effects using a risk-adjusted cumulative sum method, quantify the excess adverse event rate attributable to operator inexperience, and adjust for these alongside patient factors in evaluating device safety signals. To maintain integrity, we employed blinding between data generation and analysis teams. Synthetic data used underlying distributions and patient feature correlations based on clinical data from the Department of Veterans Affairs between 2005 and 2012. We generated 2494 synthetic datasets with widely varying characteristics including number of patient features, operators and institutions, and the operator learning form. Each dataset contained a hypothetical study device, Device B, and a reference device, Device A. We evaluated accuracy in identifying learning effects and identifying and estimating the strength of the device safety signal. Our approach also evaluated different clinically relevant thresholds for safety signal detection.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Our framework accurately identified the presence or absence of learning effects in 93.6% of datasets and correctly determined device safety signals in 93.4% of cases. The estimated device odds ratios\u2019 95% confidence intervals were accurately aligned with the specified ratios in 94.7% of datasets. In contrast, a comparative model excluding operator learning effects significantly underperformed in detecting device signals and in accuracy. Notably, our framework achieved 100% specificity for clinically relevant safety signal thresholds, although sensitivity varied with the threshold applied.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>A machine learning framework, tailored for the complexities of post-market device evaluation, may provide superior performance compared to standard parametric techniques when operator learning is present.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Demonstrating the capacity of ML to overcome complex evaluative challenges, our framework addresses the limitations of traditional statistical methods in current post-market surveillance processes. By offering a reliable means to detect and adjust for learning effects, it may significantly improve medical device safety evaluation.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae273","type":"journal-article","created":{"date-parts":[[2024,10,29]],"date-time":"2024-10-29T22:00:02Z","timestamp":1730239202000},"page":"206-217","source":"Crossref","is-referenced-by-count":2,"title":["A machine learning framework to adjust for learning effects in medical device safety evaluation"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5171-8475","authenticated-orcid":false,"given":"Jejo D","family":"Koola","sequence":"first","affiliation":[{"name":"Department of Medicine, University of California San Diego , San Diego, CA 92093,","place":["United States"]}]},{"given":"Karthik","family":"Ramesh","sequence":"additional","affiliation":[{"name":"School of Medicine, University of California San Diego , San Diego, CA 92093,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6671-3474","authenticated-orcid":false,"given":"Jialin","family":"Mao","sequence":"additional","affiliation":[{"name":"Department of Population Health Sciences, Weill Cornell Medicine , New York, NY 10065,","place":["United States"]}]},{"given":"Minyoung","family":"Ahn","sequence":"additional","affiliation":[{"name":"Jacobs School of Engineering, University of California San Diego , San Diego, CA 92093,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0792-8867","authenticated-orcid":false,"given":"Sharon E","family":"Davis","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}]},{"given":"Usha","family":"Govindarajulu","sequence":"additional","affiliation":[{"name":"Center for Biostatistics, Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai , New York, NY 10029,","place":["United States"]}]},{"given":"Amy M","family":"Perkins","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]},{"name":"Geriatric Research Education and Clinical Care Center, Tennessee Valley Healthcare System VA , Nashville, TN 37212,","place":["United States"]}]},{"given":"Dax","family":"Westerman","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]}]},{"given":"Henry","family":"Ssemaganda","sequence":"additional","affiliation":[{"name":"Comparative Effectiveness Research Institute, Lahey Hospital and Medical Center , Burlington, MA 01803,","place":["United States"]}]},{"given":"Theodore","family":"Speroff","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8005-7327","authenticated-orcid":false,"given":"Lucila","family":"Ohno-Machado","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale School of Medicine , New Haven, CT 06510,","place":["United States"]}]},{"given":"Craig R","family":"Ramsay","sequence":"additional","affiliation":[{"name":"Health Services Research Unit, University of Aberdeen , Aberdeen AB25 2ZD,","place":["United Kingdom"]}]},{"given":"Art","family":"Sedrakyan","sequence":"additional","affiliation":[{"name":"Department of Population Health Sciences, Weill Cornell Medicine , New York, NY 10065,","place":["United States"]}]},{"given":"Frederic S","family":"Resnic","sequence":"additional","affiliation":[{"name":"Comparative Effectiveness Research Institute, Lahey Hospital and Medical Center , Burlington, MA 01803,","place":["United States"]},{"name":"Division of Cardiovascular Medicine, Lahey Hospital and Medical Center , Burlington, MA 01805,","place":["United States"]}]},{"given":"Michael E","family":"Matheny","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, TN 37203,","place":["United States"]},{"name":"Department of Biostatistics, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]},{"name":"Geriatric Research Education and Clinical Care Center, Tennessee Valley Healthcare System VA , Nashville, TN 37212,","place":["United States"]},{"name":"Department of Medicine, Vanderbilt University Medical Center , Nashville, TN 37232,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2024,10,29]]},"reference":[{"key":"2024121616122626700_ocae273-B1","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1001\/jama.291.3.325","article-title":"Surveillance of medical device\u2013related hazards and adverse events in hospitalized patients","volume":"291","author":"Samore","year":"2004","journal-title":"JAMA"},{"key":"2024121616122626700_ocae273-B2","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1056\/NEJMp1000447","article-title":"Modernizing device regulation","volume":"362","author":"Garber","year":"2010","journal-title":"N Engl J Med"},{"key":"2024121616122626700_ocae273-B3","doi-asserted-by":"publisher","first-page":"981","DOI":"10.1056\/NEJMp068305","article-title":"Unanswered questions\u2013drug-eluting stents and the risk of late thrombosis","volume":"356","author":"Maisel","year":"2007","journal-title":"N Engl J Med"},{"key":"2024121616122626700_ocae273-B4","doi-asserted-by":"publisher","first-page":"892","DOI":"10.1016\/j.hrthm.2007.03.041","article-title":"Early failure of a small-diameter high-voltage implantable cardioverter-defibrillator lead","volume":"4","author":"Hauser","year":"2007","journal-title":"Heart Rhythm"},{"key":"2024121616122626700_ocae273-B5","author":"Schulte","year":"2017"},{"key":"2024121616122626700_ocae273-B6","doi-asserted-by":"publisher","first-page":"977","DOI":"10.1016\/j.ahj.2004.03.001","article-title":"Part I: identifying holes in the safety net","volume":"147","author":"O'Shea","year":"2004","journal-title":"Am Heart J"},{"key":"2024121616122626700_ocae273-B7","doi-asserted-by":"publisher","first-page":"17","DOI":"10.3233\/978-1-60750-872-4-17","volume-title":"Information Exchange for Medical Devices","author":"Gross","year":"1996"},{"key":"2024121616122626700_ocae273-B8","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1001\/jama.2016.8708","article-title":"Need for a national evaluation system for health technology","volume":"316","author":"Shuren","year":"2016","journal-title":"JAMA"},{"key":"2024121616122626700_ocae273-B9","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1016\/j.jcin.2011.09.017","article-title":"Quantifying the learning curve in the use of a novel vascular closure device: an analysis of the NCDR (national cardiovascular data registry) CathPCI registry","volume":"5","author":"Resnic","year":"2012","journal-title":"JACC Cardiovasc Interv"},{"key":"2024121616122626700_ocae273-B10","doi-asserted-by":"publisher","first-page":"4178","DOI":"10.1111\/jocs.15945","article-title":"The learning curve of robotic coronary arterial bypass surgery: a report from the STS database","volume":"36","author":"Patrick","year":"2021","journal-title":"J Card Surg"},{"key":"2024121616122626700_ocae273-B11","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1007\/s00464-018-6473-9","article-title":"A systematic review of the learning curve in robotic surgery: range and heterogeneity","volume":"33","author":"Kassite","year":"2019","journal-title":"Surg Endosc"},{"key":"2024121616122626700_ocae273-B12","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1080\/00325481.2014.996113","article-title":"Learning curves for cardiothoracic and vascular surgical procedures\u2014a systematic review","volume":"127","author":"Arora","year":"2015","journal-title":"Postgrad Med"},{"key":"2024121616122626700_ocae273-B13","author":"Center for Devices and Radiological Health","year":"2020"},{"key":"2024121616122626700_ocae273-B14","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1186\/s12874-023-01913-9","article-title":"Simulating complex patient populations with hierarchical learning effects to support methods development for post-market surveillance","volume":"23","author":"Davis","year":"2023","journal-title":"BMC Med Res Methodol"},{"key":"2024121616122626700_ocae273-B15","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1191\/1740774504cn042oa","article-title":"Statistical evaluation of learning curve effects in surgical trials","volume":"1","author":"Cook","year":"2004","journal-title":"Clin Trials"},{"key":"2024121616122626700_ocae273-B16","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1145\/2939672.2939785","author":"Chen","year":"2016"},{"key":"2024121616122626700_ocae273-B17","author":"Team RC","year":"2013"},{"issue":"3","key":"2024121616122626700_ocae273-B18","doi-asserted-by":"publisher","first-page":"230","DOI":"10.2307\/1270606","article-title":"Grouped data-sequential probability ratio tests and cumulative sum control charts","volume":"38","author":"Steiner","year":"1996","journal-title":"Technometrics"},{"key":"2024121616122626700_ocae273-B19","doi-asserted-by":"publisher","first-page":"e6649949","DOI":"10.1155\/2021\/6649949","article-title":"ARL estimation of the control chart of log likelihood ratios\u2019 sum for Markov sequence","volume":"2021","author":"Guo","year":"2021","journal-title":"J Math"},{"key":"2024121616122626700_ocae273-B20","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1146\/annurev.publhealth.21.1.121","article-title":"Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches","volume":"21","author":"Little","year":"2000","journal-title":"Annu Rev Public Health"},{"key":"2024121616122626700_ocae273-B21","doi-asserted-by":"publisher","first-page":"2867","DOI":"10.1002\/sim.4322","article-title":"Subgroup identification from randomized clinical trial data","volume":"30","author":"Foster","year":"2011","journal-title":"Stat Med"},{"key":"2024121616122626700_ocae273-B22","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1198\/016214504000001880","article-title":"Causal inference using potential outcomes","volume":"100","author":"Rubin","year":"2005","journal-title":"J Am Stat Assoc"},{"key":"2024121616122626700_ocae273-B23","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1002\/sam.11220","article-title":"Causal inference of interaction effects with inverse propensity weighting, G-computation and tree-based standardization","volume":"7","author":"Kang","year":"2014","journal-title":"Stat Anal"},{"key":"2024121616122626700_ocae273-B24","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1080\/02664763.2011.602056","article-title":"Tree-structured analysis of treatment effects with large observational data","volume":"39","author":"Kang","year":"2012","journal-title":"J Appl Stat"},{"key":"2024121616122626700_ocae273-B25","doi-asserted-by":"publisher","first-page":"757","DOI":"10.6339\/JDS.2012.10(4).1087","article-title":"Tree-structured assessment of causal odds ratio with large observational study data sets","volume":"10","author":"Kang","year":"2021","journal-title":"J Data Sci"},{"key":"2024121616122626700_ocae273-B26","volume-title":"Foundational and Applied Statistics for Biologists Using R","author":"Aho","year":"2013"},{"key":"2024121616122626700_ocae273-B27","year":"2016"},{"key":"2024121616122626700_ocae273-B28","volume-title":"Advances in Neural Information Processing Systems","author":"Lundberg","year":"2017"},{"key":"2024121616122626700_ocae273-B29","doi-asserted-by":"publisher","first-page":"404","DOI":"10.1016\/j.jtcvs.2010.10.029","article-title":"Learning curve analysis of mitral valve repair using telemanipulative technology","volume":"142","author":"Charland","year":"2011","journal-title":"J Thorac Cardiovasc Surg"},{"key":"2024121616122626700_ocae273-B30","doi-asserted-by":"publisher","first-page":"773","DOI":"10.1016\/j.jtcvs.2016.04.028","article-title":"Learning curves for transapical transcatheter aortic valve replacement in the PARTNER-I trial: technical performance, success, and safety","volume":"152","author":"Suri","year":"2016","journal-title":"J Thorac Cardiovasc Surg"},{"key":"2024121616122626700_ocae273-B31","doi-asserted-by":"publisher","first-page":"E94","DOI":"10.1002\/ccd.25038","article-title":"Learning curve for intracranial angioplasty and stenting in single center","volume":"83","author":"Cai","year":"2014","journal-title":"Catheter Cardiovasc Interv"},{"key":"2024121616122626700_ocae273-B32","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1097\/imi.0000000000000017","article-title":"Defining the learning curve for robotic-assisted endoscopic harvesting of the left internal mammary artery","volume":"8","author":"Hemli","year":"2013","journal-title":"Innovations (Phila)"},{"key":"2024121616122626700_ocae273-B33","doi-asserted-by":"publisher","first-page":"2764","DOI":"10.1002\/sim.7309","article-title":"Learning curve estimation in medical devices and procedures: hierarchical modeling","volume":"36","author":"Govindarajulu","year":"2017","journal-title":"Stat Med"},{"key":"2024121616122626700_ocae273-B34","doi-asserted-by":"publisher","first-page":"2541","DOI":"10.1056\/NEJMsa1901109","article-title":"Procedural volume and outcomes for transcatheter aortic-valve replacement","volume":"380","author":"Vemulapalli","year":"2019","journal-title":"N Engl J Med"},{"key":"2024121616122626700_ocae273-B35","doi-asserted-by":"publisher","first-page":"165","DOI":"10.2147\/MDER.S138158","article-title":"Can machine learning complement traditional medical device surveillance? A case study of dual-chamber implantable cardioverter\u2013defibrillators","volume":"10","author":"Ross","year":"2017","journal-title":"Med Devices (Auckl)"},{"key":"2024121616122626700_ocae273-B36","doi-asserted-by":"publisher","first-page":"1569","DOI":"10.1016\/j.jsurg.2019.05.009","article-title":"Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective","volume":"76","author":"Hopkins","year":"2019","journal-title":"J Surg Educ"},{"key":"2024121616122626700_ocae273-B37","doi-asserted-by":"publisher","DOI":"10.1115\/1.4049545","article-title":"Application-specific learning curve with a modern computer-assisted orthopedic surgery system for joint arthroplasty","volume":"15","author":"Dai","year":"2021","journal-title":"J Med Dev"},{"key":"2024121616122626700_ocae273-B38","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1002\/ccd.26120","article-title":"Learning curves for transfemoral transcatheter aortic valve replacement in the PARTNER-I trial: technical performance","volume":"87","author":"Alli","year":"2016","journal-title":"Catheter Cardiovasc Interv"},{"key":"2024121616122626700_ocae273-B39","doi-asserted-by":"publisher","first-page":"1951","DOI":"10.1253\/circj.CJ-18-0211","article-title":"Learning curve for transcatheter aortic valve implantation under a controlled introduction system \u2015 initial analysis of a Japanese nationwide registry","volume":"82","author":"Handa","year":"2018","journal-title":"Circ J"},{"key":"2024121616122626700_ocae273-B40","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1016\/j.jacc.2017.04.056","article-title":"Procedural experience for transcatheter aortic valve replacement and relation to outcomes","volume":"70","author":"Carroll","year":"2017","journal-title":"J Am Coll Cardiol"},{"key":"2024121616122626700_ocae273-B41","doi-asserted-by":"publisher","first-page":"342","DOI":"10.1177\/2168479015578155","article-title":"Ethical considerations for increased transparency and reproducibility in the retrospective analysis of health care data","volume":"49","author":"Rotelli","year":"2015","journal-title":"Ther Innov Regul Sci"},{"key":"2024121616122626700_ocae273-B42","doi-asserted-by":"publisher","first-page":"931","DOI":"10.1213\/ANE.0b013e31824a0b5b","article-title":"Publication bias, retrospective bias, and reproducibility of significant results in observational studies","volume":"114","author":"Shafer","year":"2012","journal-title":"Anesth Analg"},{"key":"2024121616122626700_ocae273-B43","doi-asserted-by":"publisher","first-page":"e1002106","DOI":"10.1371\/journal.pbio.1002106","article-title":"The extent and consequences of P-hacking in science","volume":"13","author":"Head","year":"2015","journal-title":"PLoS Biol"},{"key":"2024121616122626700_ocae273-B44","doi-asserted-by":"publisher","first-page":"418","DOI":"10.2345\/0899-8205-54.6.418","article-title":"RESEARCH: evaluation of orthopedic hip device recalls by the FDA from 2007 to 2017","volume":"54","author":"Peters","year":"2020","journal-title":"Biomed Instrum Technol"},{"key":"2024121616122626700_ocae273-B45","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1001\/jamainternmed.2016.6936","article-title":"National registry data and record linkage to inform postmarket surveillance of prosthetic aortic valve models over 15 Years","volume":"177","author":"Hickey","year":"2017","journal-title":"JAMA Intern Med"},{"key":"2024121616122626700_ocae273-B46","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1016\/j.cct.2011.02.001","article-title":"An evaluation of a distributed medical device safety surveillance system: the DELTA network study","volume":"32","author":"Vidi","year":"2011","journal-title":"Contemp Clin Trials"},{"key":"2024121616122626700_ocae273-B47","doi-asserted-by":"publisher","author":"Koola","year":"2024","DOI":"10.5281\/zenodo.10620930"},{"key":"2024121616122626700_ocae273-B48","doi-asserted-by":"publisher","author":"Koola","year":"2024","DOI":"10.5281\/zenodo.10620950"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/1\/206\/61202051\/ocae273.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/1\/206\/61202051\/ocae273.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T16:12:45Z","timestamp":1734365565000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/1\/206\/7849696"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,29]]},"references-count":48,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,10,29]]},"published-print":{"date-parts":[[2025,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae273","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,1]]},"published":{"date-parts":[[2024,10,29]]}}}