{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T10:30:21Z","timestamp":1769855421749,"version":"3.49.0"},"reference-count":53,"publisher":"Oxford University Press (OUP)","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Objective: To answer a \u201cgrand challenge\u201d in clinical decision support, the authors produced a recommender system that automatically data-mines inpatient decision support from electronic medical records (EMR), analogous to Netflix or Amazon.com\u2019s product recommender.<\/jats:p><jats:p>Materials and Methods: EMR data were extracted from 1 year of hospitalizations (&amp;gt;18K patients with &amp;gt;5.4M structured items including clinical orders, lab results, and diagnosis codes). Association statistics were counted for the \u223c1.5K most common items to drive an order recommender. The authors assessed the recommender\u2019s ability to predict hospital admission orders and outcomes based on initial encounter data from separate validation patients.<\/jats:p><jats:p>Results: Compared to a reference benchmark of using the overall most common orders, the recommender using temporal relationships improves precision at 10 recommendations from 33% to 38% ( P \u2009&amp;lt;\u200910 \u221210 ) for hospital admission orders. Relative risk-based association methods improve inverse frequency weighted recall from 4% to 16% ( P \u2009&amp;lt;\u200910 \u221216 ). The framework yields a prediction receiver operating characteristic area under curve (c-statistic) of 0.84 for 30\u2009day mortality, 0.84 for 1 week need for ICU life support, 0.80 for 1 week hospital discharge, and 0.68 for 30-day readmission.<\/jats:p><jats:p>Discussion: Recommender results quantitatively improve on reference benchmarks and qualitatively appear clinically reasonable. The method assumes that aggregate decision making converges appropriately, but ongoing evaluation is necessary to discern common behaviors from \u201ccorrect\u201d ones.<\/jats:p><jats:p>Conclusions: Collaborative filtering recommender algorithms generate clinical decision support that is predictive of real practice patterns and clinical outcomes. Incorporating temporal relationships improves accuracy. Different evaluation metrics satisfy different goals (predicting likely events vs. \u201cinteresting\u201d suggestions).<\/jats:p>","DOI":"10.1093\/jamia\/ocv091","type":"journal-article","created":{"date-parts":[[2015,7,22]],"date-time":"2015-07-22T02:00:48Z","timestamp":1437530448000},"page":"339-348","source":"Crossref","is-referenced-by-count":37,"title":["OrderRex: clinical order decision support and outcome predictions by data-mining electronic medical records"],"prefix":"10.1093","volume":"23","author":[{"given":"Jonathan H","family":"Chen","sequence":"first","affiliation":[{"name":"Center for Innovation to Implementation (Ci2i), Veterans Affairs Palo Alto Health Care System, Palo Alto, CA, USA"},{"name":"Center for Primary Care and Outcomes Research (PCOR), Stanford University, Stanford, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tanya","family":"Podchiyska","sequence":"additional","affiliation":[{"name":"Biomedical Informatics Training Program, Stanford University, Stanford, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Russ B","family":"Altman","sequence":"additional","affiliation":[{"name":"Departments of Bioengineering, Genetics, and Medicine, Stanford University, Stanford, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2015,7,21]]},"reference":[{"key":"2020110613022421600_ocv091-B1","volume-title":"Crossing the Quality Chasm: A New Health System for the 21st Century","author":"Richardson","year":"2001"},{"issue":"4","key":"2020110613022421600_ocv091-B2","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1016\/j.ahj.2013.12.003","article-title":"Eliminating the \u2018expensive\u2019 adjective for clinical trials","volume":"167","author":"Lauer","year":"2014","journal-title":"Am Heart J,"},{"key":"2020110613022421600_ocv091-B3","article-title":"Health information technology: standards, implementation specifications, and certification criteria for electronic health record technology","author":"Office of the National Coordinator for Health Information Technology (ONC)","year":"2014"},{"issue":"7","key":"2020110613022421600_ocv091-B4","doi-asserted-by":"crossref","first-page":"1229","DOI":"10.1377\/hlthaff.2014.0099","article-title":"A \u2018green button\u2019 for using aggregate patient data at the point of care","volume":"33","author":"Longhurst","year":"2014","journal-title":"Health Aff."},{"issue":"19","key":"2020110613022421600_ocv091-B5","doi-asserted-by":"crossref","first-page":"1758","DOI":"10.1056\/NEJMp1108726","article-title":"Evidence-based medicine in the EMR era","volume":"365","author":"Frankovich","year":"2011","journal-title":"N Engl J Med."},{"key":"2020110613022421600_ocv091-B6","first-page":"149","volume-title":"Best Care at Lower Cost: the Path to Continuously Learning Health Care in America","author":"Smith","year":"2012"},{"issue":"7","key":"2020110613022421600_ocv091-B7","doi-asserted-by":"crossref","first-page":"1163","DOI":"10.1377\/hlthaff.2014.0053","article-title":"Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system","volume":"33","author":"Krumholz","year":"2014","journal-title":"Health Aff."},{"issue":"9","key":"2020110613022421600_ocv091-B8","doi-asserted-by":"crossref","first-page":"759","DOI":"10.1097\/MLR.0b013e3182a67209","article-title":"Big data meets the electronic medical record: a commentary on \u2018identifying patients at increased risk for unplanned readmission\u2019","volume":"51","author":"de Lissovoy","year":"2013","journal-title":"Med Care."},{"issue":"12","key":"2020110613022421600_ocv091-B9","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1001\/archinte.163.12.1409","article-title":"Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review","volume":"163","author":"Kaushal","year":"2003","journal-title":"Arch Intern Med."},{"issue":"5","key":"2020110613022421600_ocv091-B10","first-page":"364","article-title":"A randomized trial of \u2018corollary orders\u2019 to prevent errors of omission","volume":"4","author":"Overhage","year":"1997","journal-title":"JAMIA."},{"issue":"6","key":"2020110613022421600_ocv091-B11","first-page":"523","article-title":"Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality","volume":"10","author":"Bates","year":"2003","journal-title":"JAMIA."},{"issue":"2","key":"2020110613022421600_ocv091-B12","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1016\/j.jbi.2007.09.003","article-title":"Grand challenges in clinical decision support","volume":"41","author":"Sittig","year":"2008","journal-title":"J Biomed Inform."},{"key":"2020110613022421600_ocv091-B13","first-page":"34","article-title":"Mining for clinical expertise in (undocumented) order sets to power an order suggestion system","volume":"2013","author":"Chen","year":"2013","journal-title":"AMIA Summits Transl Sci Proc."},{"key":"2020110613022421600_ocv091-B14","first-page":"206","article-title":"Automated physician order recommendations and outcome predictions by data-mining electronic medical records","author":"Chen","year":"2014","journal-title":"AMIA Summits Transl Sci Proc."},{"issue":"1","key":"2020110613022421600_ocv091-B15","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/MIC.2003.1167344","article-title":"Amazon.com recommendations: item-to-item collaborative filtering","volume":"7","author":"Linden","year":"2003","journal-title":"IEEE Internet Comput."},{"issue":"1","key":"2020110613022421600_ocv091-B16","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1080\/14639230010028786","article-title":"Discovery of association rules in medical data","volume":"26","author":"Doddi","year":"2001","journal-title":"Med Inform Internet Med."},{"key":"2020110613022421600_ocv091-B17","first-page":"387","article-title":"A method to compute treatment suggestions from local order entry data","volume":"2010","author":"Klann","year":"2010","journal-title":"AMIA Annu Symp Proc."},{"key":"2020110613022421600_ocv091-B18","first-page":"333","article-title":"A recommendation algorithm for automating corollary order generation","volume":"2009","author":"Klann","year":"2009","journal-title":"AMIA Annu Symp Proc."},{"key":"2020110613022421600_ocv091-B19","first-page":"819","article-title":"Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system","volume":"2006","author":"Wright","year":"2006","journal-title":"AMIA Annu Symp Proc."},{"issue":"e2","key":"2020110613022421600_ocv091-B20","first-page":"e304","article-title":"Paving the COWpath: data-driven design of pediatric order sets","volume":"21","author":"Zhang","year":"2014","journal-title":"JAMIA."},{"key":"2020110613022421600_ocv091-B21","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1016\/j.jbi.2013.12.005","article-title":"Decision support from local data: creating adaptive order menus from past clinician behavior","volume":"48","author":"Klann","year":"2014","journal-title":"J Biomed Inform ."},{"issue":"4","key":"2020110613022421600_ocv091-B22","first-page":"449","article-title":"Automatic detection of omissions in medication lists","volume":"18","author":"Hasan","year":"2011","journal-title":"JAMIA."},{"key":"2020110613022421600_ocv091-B23","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/j.jbi.2014.09.003","article-title":"The use of sequential pattern mining to predict next prescribed medications","volume":"53","author":"Wright","year":"2014","journal-title":"J Biomed Inform."},{"key":"2020110613022421600_ocv091-B24","first-page":"1469","article-title":"Data-driven order set generation and evaluation in the pediatric environment","volume":"2012","author":"Zhang","year":"2012","journal-title":"AMIA Annu Symp Proc."},{"key":"2020110613022421600_ocv091-B25","first-page":"391","article-title":"STRIDE\u2013An integrated standards-based translational research informatics platform","volume":"2009","author":"Lowe","year":"2009","journal-title":"AMIA Annu Symp Proc."},{"issue":"2","key":"2020110613022421600_ocv091-B26","first-page":"244","article-title":"Automated mapping of pharmacy orders from two electronic health record systems to RxNorm within the STRIDE clinical data warehouse","volume":"2009","author":"Hernandez","year":"2009","journal-title":"AMIA Annu Symp Proc."},{"issue":"1","key":"2020110613022421600_ocv091-B27","doi-asserted-by":"crossref","first-page":"32","DOI":"10.4338\/ACI-2009-12-RA-0023","article-title":"Distribution of problems, medications and lab results in electronic health records: the pareto principle at work","volume":"1","author":"Wright","year":"2010","journal-title":"Appl Clin Inform."},{"key":"2020110613022421600_ocv091-B28","doi-asserted-by":"crossref","DOI":"10.1145\/253260.253325","article-title":"Dynamic itemset counting and implication rules for market basket data","volume-title":"SIGMOD 1997, Proceedings ACM SIGMOD International Conference on Management of Data","author":"Brin"},{"key":"2020110613022421600_ocv091-B29","volume-title":"Foundations of Statistical Natural Language Processing","author":"Manning"},{"issue":"5","key":"2020110613022421600_ocv091-B30","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1186\/gm151","article-title":"Likelihood ratios for genome medicine","volume":"2","author":"Morgan","year":"2010","journal-title":"Genome Med."},{"key":"2020110613022421600_ocv091-B31"},{"issue":"19","key":"2020110613022421600_ocv091-B32","first-page":"1","article-title":"Evaluating recommendation systems","volume":"12","author":"Shani","year":"2011","journal-title":"Recomm Syst Handb."},{"key":"2020110613022421600_ocv091-B33","first-page":"240","article-title":"ACCF\/AHA guideline for the management of heart failure: A report of the american college of cardiology foundation\/american heart association task force on practice guidelines,\u201d Circulation","volume":"128","author":"Yancy","journal-title":"2013"},{"key":"2020110613022421600_ocv091-B34","doi-asserted-by":"crossref","first-page":"1651","DOI":"10.1002\/hep.26359","article-title":"AASLD PRACTICE GUIDELINE Management of Adult Patients with Ascites Due to Cirrhosis: Update 2012","volume":"57","author":"Runyon","year":"2013","journal-title":"Hepatology ."},{"key":"2020110613022421600_ocv091-B35","first-page":"1","article-title":"Hepatic encephalopathy in chronic liver disease","volume":"2014","author":"Vilstrup","year":"2014","journal-title":"Pract Guidel by AASLD EASL ."},{"key":"2020110613022421600_ocv091-B36","first-page":"1","article-title":"Esophageal varices","volume":"2014","author":"LaBrecque","year":"2014","journal-title":"World Gastroenterol Organ Glob Guidel."},{"key":"2020110613022421600_ocv091-B37","doi-asserted-by":"crossref","first-page":"2493","DOI":"10.1093\/eurheartj\/ehp297","article-title":"Guidelines for the diagnosis and treatment of pulmonary hypertension","volume":"30","author":"Gali","year":"2009","journal-title":"Eur Heart J."},{"issue":"13","key":"2020110613022421600_ocv091-B38","first-page":"1049","article-title":"Modeling the severity of illness of ICU patients","volume":"272","author":"Lemeshow","year":"1994","journal-title":"A systems update. JAMA."},{"issue":"2","key":"2020110613022421600_ocv091-B39","doi-asserted-by":"crossref","first-page":"R54","DOI":"10.1186\/cc7781","article-title":"Risk stratification of early admission to the intensive care unit of patients with no major criteria of severe community-acquired pneumonia: development of an international prediction rule","volume":"13","author":"Renaud","year":"2009","journal-title":"Crit Care."},{"key":"2020110613022421600_ocv091-B40","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1161\/01.CIR.91.3.677","article-title":"Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery. Steering Committee of the Provincial Adult Cardiac Care Network of Ontario","volume":"91","author":"Tu","year":"1995","journal-title":"Circulation."},{"key":"2020110613022421600_ocv091-B41","doi-asserted-by":"crossref","first-page":"477","DOI":"10.1016\/j.injury.2004.09.039","article-title":"The injury severity score or the new injury severity score for predicting intensive care unit admission and hospital length of stay?","volume":"36","author":"Lavoie","year":"2005","journal-title":"Injury"},{"issue":"4","key":"2020110613022421600_ocv091-B42","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1097\/00005650-199104000-00001","article-title":"The relationship between severity of illness and hospital length of stay and mortality","volume":"29","author":"Horn","year":"1991","journal-title":"Med Care."},{"issue":"11","key":"2020110613022421600_ocv091-B43","doi-asserted-by":"crossref","first-page":"981","DOI":"10.1097\/MLR.0b013e3181ef60d9","article-title":"An automated model to identify heart failure patients at risk for 30-day readmission or death using electronic medical record data","volume":"48","author":"Amarasingham","year":"2010","journal-title":"Med Care."},{"issue":"6","key":"2020110613022421600_ocv091-B44","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1503\/cmaj.091117","article-title":"Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community","volume":"182","author":"van Walraven","year":"2010","journal-title":"CMAJ."},{"issue":"15","key":"2020110613022421600_ocv091-B45","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1001\/jama.2011.1515","article-title":"Risk prediction models for hospital readmission: a systematic review","volume":"306","author":"Kansagara","year":"2011","journal-title":"JAMA."},{"key":"2020110613022421600_ocv091-B46","volume-title":"Google AdWords Click Through Rates Per Position","author":"Divecha","year":"2009"},{"key":"2020110613022421600_ocv091-B47","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1145\/1879141.1879193","article-title":"The impact of YouTube recommendation system on video views","author":"Zhou","year":"2010","journal-title":"Proc 10th ACM SIGCOMM Conf Internet Meas"},{"key":"2020110613022421600_ocv091-B48","first-page":"78","article-title":"Setting goals and choosing metrics for recommender system evaluations","volume":"811","author":"Schr\u00f6der","year":"2011","journal-title":"In CEUR Workshop Proc ."},{"issue":"1","key":"2020110613022421600_ocv091-B49","doi-asserted-by":"crossref","first-page":"34","DOI":"10.2307\/2082796","article-title":"Information aggregation, rationality, and the Condorcet jury theorem","volume":"90","author":"Austen-Smith","year":"1996","journal-title":"Am Polit Sci Rev."},{"key":"2020110613022421600_ocv091-B50","volume-title":"The Wisdom of Crowds: Why the Many are Smarter than the Few and How Collective Wisom Shapes Business, Economies, Societies, and Nations","author":"Surowiecki","year":"2004"},{"key":"2020110613022421600_ocv091-B51","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1007\/BF00116037","article-title":"The strength of weak learnability","volume":"227","author":"Schapire","year":"1990","journal-title":"Mach Learn."},{"issue":"5","key":"2020110613022421600_ocv091-B52","doi-asserted-by":"crossref","first-page":"e63499","DOI":"10.1371\/journal.pone.0063499","article-title":"Practice-based evidence: profiling the safety of cilostazol by text-mining of clinical notes","volume":"8","author":"Leeper","year":"2013","journal-title":"PLoS One."},{"key":"2020110613022421600_ocv091-B53","first-page":"435","article-title":"Data-mining electronic medical records for clinical order recommendations: wisdom of the crowd or tyranny of the mob?","volume":"2015","author":"Chen","year":"2015","journal-title":"AMIA Summits Transl Sci Proc."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/23\/2\/339\/34147543\/ocv091.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/23\/2\/339\/34147543\/ocv091.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,18]],"date-time":"2022-05-18T18:40:04Z","timestamp":1652899204000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/23\/2\/339\/2572407"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,7,21]]},"references-count":53,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2015,7,21]]},"published-print":{"date-parts":[[2016,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocv091","relation":{},"ISSN":["1527-974X","1067-5027"],"issn-type":[{"value":"1527-974X","type":"electronic"},{"value":"1067-5027","type":"print"}],"subject":[],"published-other":{"date-parts":[[2016,3]]},"published":{"date-parts":[[2015,7,21]]}}}