{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T00:29:25Z","timestamp":1773966565357,"version":"3.50.1"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T00:00:00Z","timestamp":1734480000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T00:00:00Z","timestamp":1734480000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000272","name":"National Institute for Health and Care Research","doi-asserted-by":"publisher","award":["2020-10-001"],"award-info":[{"award-number":["2020-10-001"]}],"id":[{"id":"10.13039\/501100000272","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"Abstract<\/jats:title>\n Background<\/jats:title>\n Numerous tools based on electronic health record (EHR) data that predict risk of unscheduled care and mortality exist. These are often criticised due to lack of external validation, potential for low predictive ability and the use of thresholds that can lead to large numbers being escalated for assessment that would not have an adverse outcome leading to unsuccessful active case management. Evidence supports the importance of clinical judgement in risk prediction particularly when ruling out disease. The aim of this pilot study was to explore performance analysis of a digitally driven risk stratification model combined with GP clinical judgement to identify patients with escalating urgent care and mortality events.<\/jats:p>\n <\/jats:sec>\n Methods<\/jats:title>\n Clinically risk stratified cohort study of 6 GP practices in a deprived, multi-ethnic UK city. Initial digital driven risk stratification into Escalated and Non-escalated groups used 7 risk factors. The Escalated group underwent stratification using GP global clinical judgement (GCJ) into Concern and No concern groupings.<\/jats:p>\n <\/jats:sec>\n Results<\/jats:title>\n 3968 out of 31,392 patients were data stratified into the Escalated group and further categorised into No concern (n<\/jats:italic>\u2009=\u20093450 (10.9%)) or Concern (n<\/jats:italic>\u2009=\u2009518 (1.7%)) by GPs.\u00a0The 30-day combined event rate (unscheduled care or death) per 1,000 was 19.0 in the whole population, 67.8 in the Escalated group and 168.0 in the Concern group (p<\/jats:italic>\u2009<\u20090.001).\u00a0The de-escalation effect of GP assessment into No Concern versus Concern was strongly negatively predictive (OR 0.25 (95%CI 0.19\u20130.33; p<\/jats:italic>\u2009<\u20090.001)).<\/jats:p>\n The whole population ROC for\u00a0the global approach (Non-escalated, GP No Concern, GP Concern) was\u00a00.614 (0.592\u20140.637), p<\/jats:italic>\u2009<\u20090.001,\u00a0and the\u00a0increase in the ROC area under the curve for 30-day events\u00a0was all focused here (+\u20090.4% (0.3\u20130.6%, p<\/jats:italic>\u2009<\u20090.001), translating into a specific ROC c-statistic for\u00a0GP GCJ\u00a0of 0.603 ((0.565\u20140.642), p<\/jats:italic>\u2009<\u20090.001).<\/jats:p>\n <\/jats:sec>\n Conclusions<\/jats:title>\n The digital only component of the model performed well but adding GP clinical judgement significantly improved risk prediction, particularly by adding negative predictive value.<\/jats:p>\n <\/jats:sec>","DOI":"10.1186\/s12911-024-02797-5","type":"journal-article","created":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T11:41:22Z","timestamp":1734522082000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Improving event prediction using general practitioner clinical judgement in a digital risk stratification model: a pilot study"],"prefix":"10.1186","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0278-6898","authenticated-orcid":false,"given":"Emma","family":"Parry","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kamran","family":"Ahmed","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elizabeth","family":"Guest","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vijay","family":"Klaire","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abdool","family":"Koodaruth","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Prasadika","family":"Labutale","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dawn","family":"Matthews","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonathan","family":"Lampitt","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0506-3652","authenticated-orcid":false,"given":"Alan","family":"Nevill","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gillian","family":"Pickavance","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mona","family":"Sidhu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kate","family":"Warren","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3460-6759","authenticated-orcid":false,"given":"Baldev M.","family":"Singh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,12,18]]},"reference":[{"key":"2797_CR1","unstructured":"Lewis G, Curry N, Bardsley M. Choosing a predictive risk model: a guide for commissioners in England. London 2011."},{"key":"2797_CR2","unstructured":"Blunt I. Focus on preventable admissions: Trends in emergency admissions for ambulatory care sensitive conditions, 2001 to 2003. London 2013."},{"key":"2797_CR3","doi-asserted-by":"publisher","first-page":"751","DOI":"10.1097\/MLR.0000000000000171","volume":"52","author":"E Wallace","year":"2014","unstructured":"Wallace E, Stuart E, Vaughan N, et al. Risk prediction models to predict emergency hospital admission in community-dwelling adults. Med Care. 2014;52:751\u201365.","journal-title":"Med Care"},{"key":"2797_CR4","doi-asserted-by":"publisher","first-page":"e740","DOI":"10.3399\/bjgp20X712793","volume":"70","author":"M Kingston","year":"2020","unstructured":"Kingston M, Griffiths R, Hutchings H, et al. Emergency admission risk stratification tools in UK primary care: a cross-sectional survey of availability and use. Br J Gen Pract. 2020;70:e740\u20138.","journal-title":"Br J Gen Pract"},{"key":"2797_CR5","doi-asserted-by":"publisher","first-page":"e001667","DOI":"10.1136\/bmjopen-2012-001667","volume":"2","author":"J Billings","year":"2012","unstructured":"Billings J, Blunt I, Steventon A, et al. Development of a predictive model to identify inpatients at risk of re-admission within 30 days of discharge (PARR-30). BMJ Open. 2012;2:e001667.","journal-title":"BMJ Open"},{"key":"2797_CR6","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1136\/bmj.38870.657917.AE","volume":"333","author":"J Billings","year":"2006","unstructured":"Billings J, Dixon J, Mijanovich T, et al. Case finding for patients at risk of readmission to hospital: development of algorithm to identify high risk patients. BMJ. 2006;333:327.","journal-title":"BMJ"},{"key":"2797_CR7","unstructured":"Dixon J, Curry N. Combined predictive model: Final report and technical documentation. London 2006."},{"key":"2797_CR8","unstructured":"Georghiou T, Blunt I, Steventon A, et al. Predictive risk and health care: an overview. London 2011."},{"key":"2797_CR9","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1136\/bmjqs-2018-007976","volume":"28","author":"H Snooks","year":"2019","unstructured":"Snooks H, Bailey-Jones K, Burge-Jones D, et al. Effects and costs of implementing predictive risk stratification in primary care: a randomised stepped wedge trial. BMJ Qual Saf. 2019;28:697\u2013705.","journal-title":"BMJ Qual Saf"},{"key":"2797_CR10","doi-asserted-by":"publisher","first-page":"1416","DOI":"10.1001\/archinte.168.13.1416","volume":"168","author":"PT Donnan","year":"2008","unstructured":"Donnan PT. Development and Validation of a Model for Predicting Emergency Admissions Over the Next Year (PEONY): a UK Historical Cohort Study. Arch Intern Med. 2008;168:1416.","journal-title":"Arch Intern Med"},{"key":"2797_CR11","doi-asserted-by":"publisher","first-page":"742","DOI":"10.3399\/bjgp10X532387","volume":"60","author":"G-J Geersing","year":"2010","unstructured":"Geersing G-J, Janssen KJ, Oudega R, et al. Diagnostic classification in patients with suspected deep venous thrombosis: physicians\u2019 judgement or a decision rule? Br J Gen Pract. 2010;60:742\u20138.","journal-title":"Br J Gen Pract"},{"key":"2797_CR12","first-page":"790","volume":"55","author":"J Munro","year":"2005","unstructured":"Munro J, Sampson F, Nicholl J. The impact of NHS Direct on the demand for out-of-hours primary and emergency care. Br J Gen Pract. 2005;55:790\u20132.","journal-title":"Br J Gen Pract"},{"key":"2797_CR13","doi-asserted-by":"publisher","first-page":"e0132340","DOI":"10.1371\/journal.pone.0132340","volume":"10","author":"J Stokes","year":"2015","unstructured":"Stokes J, Panagioti M, Alam R, et al. Effectiveness of case management for \u2018at risk\u2019 patients in primary care: a systematic review and meta-analysis. PLoS ONE. 2015;10:e0132340.","journal-title":"PLoS ONE"},{"key":"2797_CR14","unstructured":"Purdy S, Paranjothy S, Huntley A, et al. Interventions to reduce unplanned hospital admission: a series of systematic reviews. Bristol 2012. https:\/\/www.bristol.ac.uk\/medialibrary\/sites\/primaryhealthcare\/migrated\/documents\/nplannedadmissions.pdf. Accessed 31 Jan 2023."},{"key":"2797_CR15","first-page":"119","volume":"25","author":"GC Pope","year":"2004","unstructured":"Pope GC, Kautter J, Ellis RP, et al. Risk adjustment of Medicare capitation payments using the CMS-HCC model. Health Care Financ Rev. 2004;25:119\u201341.","journal-title":"Health Care Financ Rev"},{"key":"2797_CR16","doi-asserted-by":"publisher","first-page":"1348","DOI":"10.1136\/bmj.316.7141.1348","volume":"316","author":"R Bernabei","year":"1998","unstructured":"Bernabei R, Landi F, Gambassi G, et al. Randomised trial of impact of model of integrated care and case management for older people living in the community. BMJ. 1998;316:1348\u201351.","journal-title":"BMJ"},{"key":"2797_CR17","volume-title":"How Doctors Think: Clinical Judgment and the Practice of Medicine","author":"K Montgomery","year":"2006","unstructured":"Montgomery K. How Doctors Think: Clinical Judgment and the Practice of Medicine. New York: Oxford University Press; 2006."},{"key":"2797_CR18","doi-asserted-by":"publisher","first-page":"b946","DOI":"10.1136\/bmj.b946","volume":"338","author":"C Heneghan","year":"2009","unstructured":"Heneghan C, Glasziou P, Thompson M, et al. Diagnostic strategies used in primary care. BMJ. 2009;338:b946\u2013b946.","journal-title":"BMJ"},{"key":"2797_CR19","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1177\/0272989X8600600103","volume":"6","author":"WM Tierney","year":"1986","unstructured":"Tierney WM, Fitzgerald J, McHenry R, et al. Physicians\u2019 Estimates of the Probability of Myocardial Infarction in Emergency Boom Patients with chest Pain. Med Decis Making. 1986;6:12\u20137.","journal-title":"Med Decis Making"},{"key":"2797_CR20","doi-asserted-by":"publisher","first-page":"872","DOI":"10.1136\/emermed-2014-203832","volume":"31","author":"R Body","year":"2014","unstructured":"Body R, Cook G, Burrows G, et al. Can emergency physicians \u2018rule in\u2019 and \u2018rule out\u2019 acute myocardial infarction with clinical judgement? Emerg Med J. 2014;31:872\u20136.","journal-title":"Emerg Med J"},{"key":"2797_CR21","doi-asserted-by":"publisher","first-page":"e748","DOI":"10.3399\/bjgp15X687385","volume":"65","author":"J Haasenritter","year":"2015","unstructured":"Haasenritter J, Donner-Banzhoff N, B\u00f6sner S. Chest pain for coronary heart disease in general practice: clinical judgement and a clinical decision rule. Br J Gen Pract. 2015;65:e748\u201353.","journal-title":"Br J Gen Pract"},{"key":"2797_CR22","doi-asserted-by":"publisher","first-page":"e786","DOI":"10.3399\/bjgp19X706037","volume":"69","author":"M Pentzek","year":"2019","unstructured":"Pentzek M, Wagner M, Abholz H-H, et al. The value of the GP\u2019s clinical judgement in predicting dementia: a multicentre prospective cohort study among patients in general practice. Br J Gen Pract. 2019;69:e786\u201393.","journal-title":"Br J Gen Pract"},{"key":"2797_CR23","doi-asserted-by":"publisher","first-page":"e128","DOI":"10.2196\/resprot.5039","volume":"4","author":"G Luo","year":"2015","unstructured":"Luo G, Stone BL, Sakaguchi F, et al. Using computational approaches to improve risk-stratified patient management: rationale and methods. JMIR Res Protoc. 2015;4:e128.","journal-title":"JMIR Res Protoc"},{"key":"2797_CR24","doi-asserted-by":"crossref","unstructured":"Parry E, Ahmed K, Evans S, et al. General practitioner assessment of unmet need in a complex multimorbid population using a data driven and clinical triage system. BJGP Open. 2023;BJGPO.2023.0078.","DOI":"10.3399\/BJGPO.2023.0078"},{"key":"2797_CR25","doi-asserted-by":"publisher","first-page":"1080","DOI":"10.1136\/bmj.330.7499.1080","volume":"330","author":"CJ Gill","year":"2005","unstructured":"Gill CJ, Sabin L, Schmid CH. Why clinicians are natural bayesians. BMJ. 2005;330:1080\u20133.","journal-title":"BMJ"},{"key":"2797_CR26","doi-asserted-by":"publisher","first-page":"e046556","DOI":"10.1136\/bmjopen-2020-046556","volume":"11","author":"BM Singh","year":"2021","unstructured":"Singh BM, Bateman J, Viswanath A, et al. Risk of COVID-19 hospital admission and COVID-19 mortality during the first COVID-19 wave with a special emphasis on ethnic minorities: an observational study of a single, deprived, multiethnic UK health economy. BMJ Open. 2021;11:e046556.","journal-title":"BMJ Open"},{"key":"2797_CR27","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1111\/j.1365-2753.2010.01560.x","volume":"17","author":"GS Kienle","year":"2011","unstructured":"Kienle GS, Kiene H. Clinical judgement and the medical profession. J Eval Clin Pract. 2011;17:621\u20137.","journal-title":"J Eval Clin Pract"},{"key":"2797_CR28","doi-asserted-by":"crossref","unstructured":"Girwar S-AM, Jabroer R, Fiocco M, et al. A systematic review of risk stratification tools internationally used in primary care settings. Health Sci Rep. 2021;4:e329.","DOI":"10.1002\/hsr2.329"},{"key":"2797_CR29","doi-asserted-by":"publisher","first-page":"1688","DOI":"10.1001\/jama.2011.1515","volume":"306","author":"D Kansagara","year":"2011","unstructured":"Kansagara D, Englander H, Salanitro A, et al. Risk Prediction Models for Hospital Readmission. JAMA. 2011;306:1688.","journal-title":"JAMA"},{"key":"2797_CR30","doi-asserted-by":"publisher","first-page":"11222","DOI":"10.1038\/s41598-019-47712-5","volume":"9","author":"Y Li","year":"2019","unstructured":"Li Y, Sperrin M, Belmonte M, et al. Do population-level risk prediction models that use routinely collected health data reliably predict individual risks? Sci Rep. 2019;9:11222.","journal-title":"Sci Rep"},{"key":"2797_CR31","doi-asserted-by":"publisher","first-page":"134","DOI":"10.1186\/s12916-019-1368-8","volume":"17","author":"A Pate","year":"2019","unstructured":"Pate A, Emsley R, Ashcroft DM, et al. The uncertainty with using risk prediction models for individual decision making: an exemplar cohort study examining the prediction of cardiovascular disease in English primary care. BMC Med. 2019;17:134.","journal-title":"BMC Med"},{"key":"2797_CR32","doi-asserted-by":"publisher","first-page":"e0128233","DOI":"10.1371\/journal.pone.0128233","volume":"10","author":"S Sanders","year":"2015","unstructured":"Sanders S, Doust J, Glasziou P. A systematic review of studies comparing diagnostic clinical prediction rules with clinical judgment. PLoS ONE. 2015;10:e0128233.","journal-title":"PLoS ONE"},{"key":"2797_CR33","doi-asserted-by":"publisher","first-page":"556","DOI":"10.1016\/j.jval.2017.01.003","volume":"20","author":"W Hollingworth","year":"2017","unstructured":"Hollingworth W, Busby J, Butler CC, et al. The diagnosis of urinary tract infection in young children (DUTY) study clinical rule: economic evaluation. Value in Health. 2017;20:556\u201366.","journal-title":"Value in Health"},{"key":"2797_CR34","first-page":"127","volume":"50","author":"AA Montgomery","year":"2000","unstructured":"Montgomery AA, Fahey T, MacKintosh C, et al. Estimation of cardiovascular risk in hypertensive patients in primary care. Br J Gen Pract. 2000;50:127\u20138.","journal-title":"Br J Gen Pract"},{"key":"2797_CR35","doi-asserted-by":"publisher","first-page":"1085","DOI":"10.1016\/j.jclinepi.2008.04.008","volume":"61","author":"DB Toll","year":"2008","unstructured":"Toll DB, Janssen KJM, Vergouwe Y, et al. Validation, updating and impact of clinical prediction rules: A review. J Clin Epidemiol. 2008;61:1085\u201394.","journal-title":"J Clin Epidemiol"},{"key":"2797_CR36","unstructured":"Operational Research and Evaluation Unit. Risk stratification: Learning and Impact Study. NHS England. https:\/\/imperialcollegehealthpartners.com\/wp-content\/uploads\/2018\/07\/ORE__Risk_stratification_learning_and_impact_study.pdf. 2017."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02797-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-024-02797-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02797-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T12:03:13Z","timestamp":1734523393000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-024-02797-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,18]]},"references-count":36,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["2797"],"URL":"https:\/\/doi.org\/10.1186\/s12911-024-02797-5","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,18]]},"assertion":[{"value":"7 December 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 November 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 December 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Ethics approval and informed consent was not deemed necessary for this study according to our Institutional Review Board (Royal Wolverhampton NHS Trust Research and Development Department).The systems were designed and the data accrued for a wider programme relating to service reconfiguration in our local health economy and as such the GP assessments were part of routine care.No selection or randomisation was applied, interventions were part of indicated clinical care and thus research ethical approval was not deemed necessary as confirmed within local governance processes. All methods were carried out in accordance with relevant guidelines and regulations and in accordance with the Declaration of Helsinki.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"382"}}