{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T21:09:09Z","timestamp":1777669749371,"version":"3.51.4"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100013342","name":"NIHR Imperial Biomedical Research Centre","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100013342","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Digital and online symptom checkers are an increasingly adopted class of health technologies that enable patients to input their symptoms and biodata to produce a set of likely diagnoses and associated triage advice. However, concerns regarding the accuracy and safety of these symptom checkers have been raised. This systematic review evaluates the accuracy of symptom checkers in providing diagnoses and appropriate triage advice. MEDLINE and Web of Science were searched for studies that used either real or simulated patients to evaluate online or digital symptom checkers. The primary outcomes were the diagnostic and triage accuracy of the symptom checkers. The QUADAS-2 tool was used to assess study quality. Of the 177 studies retrieved, 10 studies met the inclusion criteria. Researchers evaluated the accuracy of symptom checkers using a variety of medical conditions, including ophthalmological conditions, inflammatory arthritides and HIV. A total of 50% of the studies recruited real patients, while the remainder used simulated cases. The diagnostic accuracy of the primary diagnosis was low across included studies (range: 19\u201337.9%) and varied between individual symptom checkers, despite consistent symptom data input. Triage accuracy (range: 48.8\u201390.1%) was typically higher than diagnostic accuracy. Overall, the diagnostic and triage accuracy of symptom checkers are variable and of low accuracy. Given the increasing push towards adopting this class of technologies across numerous health systems, this study demonstrates that reliance upon symptom checkers could pose significant patient safety hazards. Large-scale primary studies, based upon real-world data, are warranted to demonstrate the adequate performance of these technologies in a manner that is non-inferior to current best practices. Moreover, an urgent assessment of how these systems are regulated and implemented is required.<\/jats:p>","DOI":"10.1038\/s41746-022-00667-w","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T15:02:33Z","timestamp":1660748553000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":149,"title":["The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1726-2923","authenticated-orcid":false,"given":"William","family":"Wallace","sequence":"first","affiliation":[]},{"given":"Calvin","family":"Chan","sequence":"additional","affiliation":[]},{"given":"Swathikan","family":"Chidambaram","sequence":"additional","affiliation":[]},{"given":"Lydia","family":"Hanna","sequence":"additional","affiliation":[]},{"given":"Fahad Mujtaba","family":"Iqbal","sequence":"additional","affiliation":[]},{"given":"Amish","family":"Acharya","sequence":"additional","affiliation":[]},{"given":"Pasha","family":"Normahani","sequence":"additional","affiliation":[]},{"given":"Hutan","family":"Ashrafian","sequence":"additional","affiliation":[]},{"given":"Sheraz R.","family":"Markar","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4595-8402","authenticated-orcid":false,"given":"Viknesh","family":"Sounderajah","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7815-7989","authenticated-orcid":false,"given":"Ara","family":"Darzi","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,8,17]]},"reference":[{"key":"667_CR1","doi-asserted-by":"publisher","first-page":"h3480","DOI":"10.1136\/bmj.h3480","volume":"351","author":"HL Semigran","year":"2015","unstructured":"Semigran, H. L., Linder, J. A., Gidengil, C. & Mehrotra, A. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ 351, h3480 (2015).","journal-title":"BMJ"},{"key":"667_CR2","doi-asserted-by":"publisher","first-page":"e202","DOI":"10.2196\/jmir.6755","volume":"19","author":"J Mueller","year":"2017","unstructured":"Mueller, J. et al. Web use for symptom appraisal of physical health conditions: a systematic review. J. Med. Internet Res. 19, e202 (2017).","journal-title":"J. Med. Internet Res."},{"key":"667_CR3","unstructured":"Prescott, C. Internet access \u2013 households and individuals, Great Britain: 2020. Office for National Statistics https:\/\/www.ons.gov.uk\/peoplepopulationandcommunity\/householdcharacteristics\/homeinternetandsocialmediausage\/bulletins\/internetaccesshouseholdsandindividuals\/2020 (2020)."},{"key":"667_CR4","doi-asserted-by":"publisher","first-page":"298","DOI":"10.31486\/toj.18.0068","volume":"18","author":"AC Berry","year":"2018","unstructured":"Berry, A. C. Online symptom checker applications: syndromic surveillance for international health. Ochsner J. 18, 298\u2013299 (2018).","journal-title":"Ochsner J."},{"key":"667_CR5","doi-asserted-by":"publisher","first-page":"004695802091030","DOI":"10.1177\/0046958020910305","volume":"57","author":"D McIntyre","year":"2020","unstructured":"McIntyre, D. & Chow, C. K. Waiting time as an indicator for health services under strain: a narrative review. Inquiry 57, 004695802091030 (2020).","journal-title":"Inquiry"},{"key":"667_CR6","doi-asserted-by":"publisher","first-page":"e100114","DOI":"10.1136\/bmjhci-2019-100114","volume":"27","author":"K Gottliebsen","year":"2020","unstructured":"Gottliebsen, K. & Petersson, G. Limited evidence of benefits of patient operated intelligent primary care triage tools: findings of a literature review. BMJ Heal. Care Inform. 27, e100114 (2020).","journal-title":"BMJ Heal. Care Inform."},{"key":"667_CR7","unstructured":"NHS 111 powered by Babylon; Outcomes evaluation. https:\/\/assets.babylonhealth.com\/nhs\/NHS-111-Evaluation-of-outcomes.pdf (2017)."},{"key":"667_CR8","unstructured":"NHS England. 1 in 4 GP appointments potentially avoidable. NHS England https:\/\/www.england.nhs.uk\/2015\/10\/gp-appointments\/."},{"key":"667_CR9","doi-asserted-by":"publisher","DOI":"10.1186\/1741-7015-11-258","volume":"11","author":"P McHale","year":"2013","unstructured":"McHale, P. et al. Who uses emergency departments inappropriately and when \u2013 a national cross-sectional study using a monitoring data system. BMC Med. 11, 258 (2013).","journal-title":"BMC Med."},{"key":"667_CR10","doi-asserted-by":"publisher","first-page":"2263","DOI":"10.1016\/S0140-6736(18)32819-8","volume":"392","author":"H Fraser","year":"2018","unstructured":"Fraser, H., Coiera, E. & Wong, D. Safety of patient-facing digital symptom checkers. Lancet 392, 2263\u20132264 (2018).","journal-title":"Lancet"},{"key":"667_CR11","unstructured":"Blanchard, S. NHS-backed G. P. chatbot is branded a \u2018public health danger\u2019. Daily Mail Online https:\/\/www.dailymail.co.uk\/health\/article-6751393\/NHS-backed-GP-chatbot-branded-public-health-danger.html (2019)."},{"key":"667_CR12","unstructured":"Das, S. It\u2019s hysteria, not a heart attack, GP app Babylon tells women. The Sunday Times https:\/\/www.thetimes.co.uk\/article\/its-hysteria-not-a-heart-attack-gp-app-tells-women-gm2vxbrqk# (2019)."},{"key":"667_CR13","unstructured":"Adegoke, Y. \u2018Calm down dear, it\u2019s only an aneurysm\u2019 \u2013 why doctors need to take women\u2019s pain seriously. The Guardian https:\/\/www.theguardian.com\/commentisfree\/2019\/oct\/16\/doctors-women-pain-heart-attack-hysteria (2019)."},{"key":"667_CR14","doi-asserted-by":"publisher","first-page":"e027743","DOI":"10.1136\/bmjopen-2018-027743","volume":"9","author":"D Chambers","year":"2019","unstructured":"Chambers, D. et al. Digital and online symptom checkers and health assessment\/triage services for urgent health problems: systematic review. BMJ Open 9, e027743 (2019).","journal-title":"BMJ Open"},{"key":"667_CR15","doi-asserted-by":"publisher","DOI":"10.1186\/s12891-016-1189-2","volume":"17","author":"L Powley","year":"2016","unstructured":"Powley, L., McIlroy, G., Simons, G. & Raza, K. Are online symptoms checkers useful for patients with inflammatory arthritis? BMC Musculoskelet. Disord. 17, 362 (2016).","journal-title":"BMC Musculoskelet. Disord."},{"key":"667_CR16","first-page":"60","volume":"5","author":"JC Nazario Arancibia","year":"2019","unstructured":"Nazario Arancibia, J. C. et al. Evaluation of a diagnostic decision support system for the triage of patients in a hospital emergency department. Int. J. Interact. Multimed. Artif. Intell. 5, 60\u201367 (2019).","journal-title":"Int. J. Interact. Multimed. Artif. Intell."},{"key":"667_CR17","doi-asserted-by":"publisher","first-page":"690","DOI":"10.1001\/jamaophthalmol.2019.0571","volume":"137","author":"C Shen","year":"2019","unstructured":"Shen, C., Nguyen, M., Gregor, A., Isaza, G. & Beattie, A. Accuracy of a popular online symptom checker for ophthalmic diagnoses. JAMA Ophthalmol. 137, 690 (2019).","journal-title":"JAMA Ophthalmol."},{"key":"667_CR18","doi-asserted-by":"publisher","first-page":"514","DOI":"10.5694\/mja2.50600","volume":"212","author":"MG Hill","year":"2020","unstructured":"Hill, M. G., Sim, M. & Mills, B. The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. Med. J. Aust. 212, 514\u2013518 (2020).","journal-title":"Med. J. Aust."},{"key":"667_CR19","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1177\/1024907919842486","volume":"27","author":"SWY Yu","year":"2020","unstructured":"Yu, S. W. Y. et al. Triage accuracy of online symptom checkers for accident and emergency department patients. Hong. Kong J. Emerg. Med. 27, 217\u2013222 (2020).","journal-title":"Hong. Kong J. Emerg. Med."},{"key":"667_CR20","doi-asserted-by":"publisher","first-page":"168","DOI":"10.2186\/jpr.JPOR_2019_499","volume":"65","author":"Y Yoshida","year":"2021","unstructured":"Yoshida, Y. & Thomas Clark, G. Accuracy of online symptom checkers for diagnosis of orofacial pain and oral medicine disease. J. Prosthodont. Res. 65, 168\u2013190 (2021).","journal-title":"J. Prosthodont. Res."},{"key":"667_CR21","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1007\/s11552-014-9707-x","volume":"10","author":"MGJS Hageman","year":"2015","unstructured":"Hageman, M. G. J. S., Anderson, J., Blok, R., Bossen, J. K. J. & Ring, D. Internet self-diagnosis in hand surgery. Hand 10, 565\u2013569 (2015).","journal-title":"Hand"},{"key":"667_CR22","doi-asserted-by":"publisher","first-page":"e104","DOI":"10.1017\/S0950268819000268","volume":"147","author":"AC Berry","year":"2019","unstructured":"Berry, A. C. et al. Online symptom checker diagnostic and triage accuracy for HIV and hepatitis C. Epidemiol. Infect. 147, e104 (2019).","journal-title":"Epidemiol. Infect."},{"key":"667_CR23","doi-asserted-by":"publisher","first-page":"e040269","DOI":"10.1136\/bmjopen-2020-040269","volume":"10","author":"S Gilbert","year":"2020","unstructured":"Gilbert, S. et al. How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open 10, e040269 (2020).","journal-title":"BMJ Open"},{"key":"667_CR24","unstructured":"Revisiting the \u00abSwiss Cheese\u00bb Model of Accidents. Eurocontrol Experimental Centre https:\/\/www.eurocontrol.int\/publication\/revisiting-swiss-cheese-model-accidents (2006)."},{"key":"667_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41077-016-0006-1","volume":"1","author":"B Williams","year":"2016","unstructured":"Williams, B. & Song, J. J. Y. Are simulated patients effective in facilitating development of clinical competence for healthcare students? A scoping review. Adv. Simul. 1, 1\u20139 (2016).","journal-title":"Adv. Simul."},{"key":"667_CR26","doi-asserted-by":"publisher","first-page":"1715","DOI":"10.1001\/jama.283.13.1715","volume":"283","author":"J Peabody","year":"2000","unstructured":"Peabody, J., Luck, J., Glassman, P., Dresselhaus, T. & Lee, M. Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. JAMA 283, 1715\u20131722 (2000).","journal-title":"JAMA"},{"key":"667_CR27","doi-asserted-by":"publisher","first-page":"e053566","DOI":"10.1136\/bmjopen-2021-053566","volume":"12","author":"A El-Osta","year":"2022","unstructured":"El-Osta, A. et al. What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study. BMJ Open 12, e053566 (2022).","journal-title":"BMJ Open"},{"key":"667_CR28","doi-asserted-by":"publisher","first-page":"e0254088","DOI":"10.1371\/journal.pone.0254088","volume":"16","author":"A Ceney","year":"2021","unstructured":"Ceney, A. et al. Accuracy of online symptom checkers and the potential impact on service utilisation. PLoS One 16, e0254088 (2021).","journal-title":"PLoS One"},{"key":"667_CR29","doi-asserted-by":"publisher","first-page":"m815","DOI":"10.1136\/bmj.m815","volume":"368","author":"G Iacobucci","year":"2020","unstructured":"Iacobucci, G. Row over Babylon\u2019s chatbot shows lack of regulation. BMJ 368, m815 (2020).","journal-title":"BMJ"},{"key":"667_CR30","unstructured":"FDA. Software as a Medical Device (SaMD) (2018)."},{"key":"667_CR31","doi-asserted-by":"publisher","first-page":"844669","DOI":"10.3389\/fpubh.2022.844669","volume":"10","author":"J Knitza","year":"2022","unstructured":"Knitza, J. et al. Patient\u2019s perception of digital symptom assessment technologies in rheumatology: results from a multicentre study. Front. public Heal. 10, 844669 (2022).","journal-title":"Front. public Heal."},{"key":"667_CR32","doi-asserted-by":"crossref","unstructured":"Kleinert, S. et al. A real-world rheumatology registry and research consortium: The German RheumaDatenRhePort (RHADAR) registry. J. Med. Internet Res. 23, e28164 (2021).","DOI":"10.2196\/preprints.28164"},{"key":"667_CR33","doi-asserted-by":"crossref","unstructured":"Knitza, J. et al. Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial. Arthritis Res. Ther. 23, 112 (2021).","DOI":"10.1186\/s13075-021-02498-8"},{"key":"667_CR34","doi-asserted-by":"publisher","first-page":"12592","DOI":"10.1073\/pnas.1919012117","volume":"117","author":"AJ Larrazabal","year":"2020","unstructured":"Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H. & Ferrante, E. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc. Natl Acad. Sci. USA 117, 12592\u201312594 (2020).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"667_CR35","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1145\/3458723","volume":"64","author":"T Gebru","year":"2018","unstructured":"Gebru, T. et al. Datasheets for datasets. Commun. ACM 64, 86\u201392 (2018).","journal-title":"Commun. ACM"},{"key":"667_CR36","first-page":"29","volume":"37","author":"N Rostamzadeh","year":"2022","unstructured":"Rostamzadeh, N. et al. Healthsheet: development of a transparency artifact for health datasets. J. ACM 37, 29 (2022).","journal-title":"J. ACM"},{"key":"667_CR37","unstructured":"STANDING Together Working Group. STANDING Together. https:\/\/www.datadiversity.org\/ (2022)."},{"key":"667_CR38","doi-asserted-by":"publisher","first-page":"1860","DOI":"10.1001\/jamainternmed.2016.6001","volume":"176","author":"HL Semigran","year":"2016","unstructured":"Semigran, H. L., Levine, D. M., Nundy, S. & Mehrotra, A. Comparison of physician and computer diagnostic accuracy. JAMA Intern. Med. 176, 1860\u20131861 (2016).","journal-title":"JAMA Intern. Med."},{"key":"667_CR39","doi-asserted-by":"crossref","unstructured":"North, F., Ward, W. J., Varkey, P. & Tulledge-Scheitel, S. M. Should you search the internet for information about your acute symptom? Telemed. J. E. Health. 18, 213\u2013218. https:\/\/home.liebertpub.com\/tmj (2012).","DOI":"10.1089\/tmj.2011.0127"},{"key":"667_CR40","doi-asserted-by":"publisher","first-page":"1143","DOI":"10.1136\/bmj.39003.640567.AE","volume":"333","author":"H Tang","year":"2006","unstructured":"Tang, H. & Ng, J. H. K. Googling for a diagnosis\u2014use of Google as a diagnostic aid: internet based study. BMJ 333, 1143\u20131145 (2006).","journal-title":"BMJ"},{"key":"667_CR41","doi-asserted-by":"publisher","first-page":"807","DOI":"10.1038\/s41591-020-0941-1","volume":"26","author":"V Sounderajah","year":"2020","unstructured":"Sounderajah, V. et al. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group. Nat. Med. 26, 807\u2013808 (2020).","journal-title":"Nat. Med."},{"key":"667_CR42","doi-asserted-by":"publisher","first-page":"1364","DOI":"10.1038\/s41591-020-1034-x","volume":"26","author":"X Liu","year":"2020","unstructured":"Liu, X. et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat. Med. 26, 1364\u20131374 (2020).","journal-title":"Nat. Med."},{"key":"667_CR43","doi-asserted-by":"publisher","first-page":"n71","DOI":"10.1136\/bmj.n71","volume":"372","author":"MJ Page","year":"2021","unstructured":"Page, M. J. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, n71 (2021).","journal-title":"BMJ"},{"key":"667_CR44","doi-asserted-by":"publisher","first-page":"529","DOI":"10.7326\/0003-4819-155-8-201110180-00009","volume":"155","author":"PF Whiting","year":"2011","unstructured":"Whiting, P. F. et al. Quadas-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann. Intern. Med. 155, 529\u2013536 (2011).","journal-title":"Ann. Intern. Med."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00667-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00667-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00667-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T03:14:37Z","timestamp":1669346077000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-022-00667-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,17]]},"references-count":44,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["667"],"URL":"https:\/\/doi.org\/10.1038\/s41746-022-00667-w","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.12.21.21268167","asserted-by":"object"}]},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,17]]},"assertion":[{"value":"3 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 July 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 August 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"118"}}