{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T05:40:21Z","timestamp":1777614021791,"version":"3.51.4"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,9,24]],"date-time":"2022-09-24T00:00:00Z","timestamp":1663977600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,24]],"date-time":"2022-09-24T00:00:00Z","timestamp":1663977600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001646","name":"Robert Bosch Stiftung","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001646","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Medical decision support systems (CDSSs) are increasingly used in medicine, but their utility in daily medical practice is difficult to evaluate. One variant of CDSS is a generator of differential diagnoses (DDx generator). We performed a feasibility study on three different, publicly available data sets of medical cases in order to identify the frequency in which two different DDx generators provide helpful information (either by providing a list of differential diagnosis or recognizing the expert diagnosis if available) for a given case report.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>Used data sets were<jats:italic>n<\/jats:italic>\u2009=\u2009105 cases from a web-based forum of telemedicine with real life cases from Afghanistan (Afghan data set; AD),<jats:italic>n<\/jats:italic>\u2009=\u2009124 cases discussed in a web-based medical forum (Coliquio data set; CD). Both websites are restricted for medical professionals only. The third data set consisted 50 special case reports published in the New England Journal of Medicine (NEJM). After keyword extraction, data were entered into two different DDx generators (IsabelHealth (IH), Memem7 (M7)) to examine differences in target diagnosis recognition and physician-rated usefulness between DDx generators.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>Both DDx generators detected the target diagnosis equally successfully (all cases: M7, 83\/170 (49%); IH 90\/170 (53%), NEJM: M7, 28\/50 (56%); IH, 34\/50 (68%); differences n.s.). Differences occurred in AD, where detection of an expert diagnosis was less successful with IH than with M7 (29.7% vs. 54.1%,<jats:italic>p<\/jats:italic>\u2009=\u20090.003). In contrast, in CD IH performed significantly better than M7 (73.9% vs. 32.6%,<jats:italic>p<\/jats:italic>\u2009=\u20090.021). Congruent identification of target diagnosis occurred in only 46\/170 (27.1%) of cases. However, a qualitative analysis of the DDx results revealed useful complements from using the two systems in parallel.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>Both DDx systems IsabelHealth and Memem7 provided substantial help in finding a helpful list of differential diagnoses or identifying the target diagnosis either in standard cases or complicated and rare cases. Our pilot study highlights the need for different levels of complexity and types of real-world medical test cases, as there are significant differences between DDx generators away from traditional case reports. Combining different results from DDx generators seems to be a possible approach for future review and use of the systems.<\/jats:p><\/jats:sec>","DOI":"10.1186\/s12911-022-01988-2","type":"journal-article","created":{"date-parts":[[2022,9,24]],"date-time":"2022-09-24T17:02:53Z","timestamp":1664038973000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Evaluation of medical decision support systems (DDX generators) using real medical cases of varying complexity and origin"],"prefix":"10.1186","volume":"22","author":[{"given":"P.","family":"Fritz","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"A.","family":"Kleinhans","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"R.","family":"Raoufi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"A.","family":"Sediqi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"N.","family":"Schmid","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"S.","family":"Schricker","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M.","family":"Schanz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"C.","family":"Fritz-Kuisle","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"P.","family":"Dalquen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"H.","family":"Firooz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"G.","family":"Stauch","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"M. D.","family":"Alscher","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,9,24]]},"reference":[{"issue":"7494","key":"1988_CR1","doi-asserted-by":"publisher","first-page":"765","DOI":"10.1136\/bmj.38398.500764.8F","volume":"330","author":"K Kawamoto","year":"2005","unstructured":"Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765.","journal-title":"BMJ"},{"key":"1988_CR2","first-page":"1046","volume":"2018","author":"OA Usman","year":"2018","unstructured":"Usman OA, Oshiro C, Chambers JG, Tu SW, Martins S, Robinson A, Goldstein MK. Selecting test cases from the electronic health record for software testing of knowledge-based clinical decision support systems. AMIA Annu Symp Proc. 2018;2018:1046\u201355.","journal-title":"AMIA Annu Symp Proc"},{"key":"1988_CR3","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1016\/j.jbi.2018.05.004","volume":"82","author":"SV Kovalchuk","year":"2018","unstructured":"Kovalchuk SV, Funkner AA, Metsker OG, Yakovlev AN. Simulation of patient flow in multiple healthcare units using process and data mining techniques for model identification. J Biomed Inform. 2018;82:128\u201342.","journal-title":"J Biomed Inform"},{"issue":"7","key":"1988_CR4","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0235410","volume":"15","author":"EW Breitbart","year":"2020","unstructured":"Breitbart EW, Choudhury K, Andersen AD, Bunde H, Breitbart M, Sideri AM, Fengler S, Zibert JR. Improved patient satisfaction and diagnostic accuracy in skin diseases with a visual clinical decision support system-a feasibility study with general practitioners. PLoS ONE. 2020;15(7): e0235410.","journal-title":"PLoS ONE"},{"key":"1988_CR5","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1016\/j.compbiomed.2018.07.018","volume":"101","author":"RM Olsen","year":"2018","unstructured":"Olsen RM, Aasvang EK, Meyhoff CS, Dissing Sorensen HB. Towards an automated multimodal clinical decision support system at the post anesthesia care unit. Comput Biol Med. 2018;101:15\u201321.","journal-title":"Comput Biol Med"},{"issue":"3","key":"1988_CR6","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1111\/j.1365-2559.2012.04229.x","volume":"61","author":"J Miedema","year":"2012","unstructured":"Miedema J, Marron JS, Niethammer M, Borland D, Woosley J, Coposky J, Wei S, Reisner H, Thomas NE. Image and statistical analysis of melanocytic histology. Histopathology. 2012;61(3):436\u201344.","journal-title":"Histopathology"},{"issue":"5 Suppl","key":"1988_CR7","doi-asserted-by":"publisher","first-page":"S2","DOI":"10.1016\/j.amjmed.2008.01.001","volume":"121","author":"ES Berner","year":"2008","unstructured":"Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med. 2008;121(5 Suppl):S2-23.","journal-title":"Am J Med"},{"issue":"25","key":"1988_CR8","doi-asserted-by":"publisher","first-page":"1792","DOI":"10.1056\/NEJM199406233302506","volume":"330","author":"ES Berner","year":"1994","unstructured":"Berner ES, Webster GD, Shugerman AA, Jackson JR, Algina J, Baker AL, Ball EV, Cobbs CG, Dennis VW, Frenkel EP, et al. Performance of four computer-based diagnostic systems. N Engl J Med. 1994;330(25):1792\u20136.","journal-title":"N Engl J Med"},{"issue":"10","key":"1988_CR9","doi-asserted-by":"publisher","first-page":"1223","DOI":"10.1001\/jama.293.10.1223","volume":"293","author":"AX Garg","year":"2005","unstructured":"Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223\u201338.","journal-title":"JAMA"},{"issue":"3","key":"1988_CR10","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1093\/fampra\/cmg301","volume":"20","author":"J Sandars","year":"2003","unstructured":"Sandars J, Esmail A. The frequency and nature of medical error in primary care: understanding the diversity across studies. Fam Pract. 2003;20(3):231\u20136.","journal-title":"Fam Pract"},{"issue":"4","key":"1988_CR11","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1111\/j.1525-1497.2005.30145.x","volume":"20","author":"CP Friedman","year":"2005","unstructured":"Friedman CP, Gatti GG, Franz TM, Murphy GC, Wolf FM, Heckerling PS, Fine PL, Miller TM, Elstein AS. Do physicians know when their diagnoses are correct? Implications for decision support and error reduction. J Gen Intern Med. 2005;20(4):334\u20139.","journal-title":"J Gen Intern Med"},{"issue":"2","key":"1988_CR12","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1007\/s11606-011-1804-8","volume":"27","author":"WF Bond","year":"2012","unstructured":"Bond WF, Schwartz LM, Weaver KR, Levick D, Giuliano M, Graber ML. Differential diagnosis generators: an evaluation of currently available computer programs. J Gen Intern Med. 2012;27(2):213\u20139.","journal-title":"J Gen Intern Med"},{"issue":"3","key":"1988_CR13","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0148991","volume":"11","author":"N Riches","year":"2016","unstructured":"Riches N, Panagioti M, Alam R, Cheraghi-Sohi S, Campbell S, Esmail A, Bower P. The effectiveness of electronic differential diagnoses (DDX) generators: a systematic review and meta-analysis. PLoS ONE. 2016;11(3): e0148991.","journal-title":"PLoS ONE"},{"issue":"8","key":"1988_CR14","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1056\/NEJM198208193070803","volume":"307","author":"RA Miller","year":"1982","unstructured":"Miller RA, Pople HE Jr, Myers JD. Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med. 1982;307(8):468\u201376.","journal-title":"N Engl J Med"},{"issue":"12","key":"1988_CR15","doi-asserted-by":"publisher","first-page":"1636","DOI":"10.1373\/clinchem.2012.182576","volume":"58","author":"PM Bossuyt","year":"2012","unstructured":"Bossuyt PM, Reitsma JB, Linnet K, Moons KG. Beyond diagnostic accuracy: the clinical utility of diagnostic tests. Clin Chem. 2012;58(12):1636\u201343.","journal-title":"Clin Chem"},{"key":"1988_CR16","doi-asserted-by":"crossref","unstructured":"Tcheng JE: Optimizing strategies for clinical decision support: summary of a meeting series: National Academy of Medicine. 2017.","DOI":"10.17226\/27122"},{"key":"1988_CR17","doi-asserted-by":"crossref","unstructured":"Balogh EP, Miller BT, Ball JR. Improving diagnosis in health care. 2015.","DOI":"10.17226\/21794"},{"issue":"5","key":"1988_CR18","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1007\/s00428-004-0980-x","volume":"444","author":"K Brauchli","year":"2004","unstructured":"Brauchli K, Oberli H, Hurwitz N, Kunze KD, Haroske G, Jundt G, Stauch G, Banach L, Wirdnam M, Mihatsch M, et al. Diagnostic telepathology: long-term experience of a single institution. Virchows Arch. 2004;444(5):403\u20139.","journal-title":"Virchows Arch"},{"issue":"Suppl 2","key":"1988_CR19","doi-asserted-by":"publisher","first-page":"S3","DOI":"10.1258\/135763305775124795","volume":"11","author":"K Brauchli","year":"2005","unstructured":"Brauchli K, Oberholzer M. The iPath telemedicine platform. J Telemed Telecare. 2005;11(Suppl 2):S3-7.","journal-title":"J Telemed Telecare"},{"key":"1988_CR20","unstructured":"\u00c4rztenetzwerk Coliquio Forum Patietentenf\u00e4lle [https:\/\/www.Coliquio.de]"},{"key":"1988_CR21","unstructured":"Isabel Health [https:\/\/www.isabelhealthcare.com\/]"},{"issue":"7\u20138","key":"1988_CR22","doi-asserted-by":"publisher","first-page":"488","DOI":"10.1177\/1357633X19840475","volume":"26","author":"P Fritz","year":"2020","unstructured":"Fritz P, Kleinhans A, Hubler M, Rokai R, Firooz H, Sediqi A, Khachatryan A, Sotoudeh K, Mamunts D, Desai M, et al. Experience with telepathology in combination with diagnostic assistance systems in countries with restricted resources. J Telemed Telecare. 2020;26(7\u20138):488\u201394.","journal-title":"J Telemed Telecare"},{"issue":"1","key":"1988_CR23","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1186\/s12911-017-0488-3","volume":"17","author":"P Fritz","year":"2017","unstructured":"Fritz P, Kleinhans A, Kuisle F, Albu P, Fritz-Kuisle C, Alscher MD. Medical diagnosis as a linguistic game. BMC Med Inform Decis Mak. 2017;17(1):103.","journal-title":"BMC Med Inform Decis Mak"},{"key":"1988_CR24","unstructured":"Memem7 [http:\/\/www.Memem7.de\/cloud]"},{"key":"1988_CR25","unstructured":"R Core Team: R: A language and environment for statistical computing. In. Vienna, Austria: R Foundation for Statistical Computing; 2020."},{"issue":"9","key":"1988_CR26","doi-asserted-by":"publisher","first-page":"619","DOI":"10.1136\/emj.2006.044107","volume":"24","author":"P Ramnarayan","year":"2007","unstructured":"Ramnarayan P, Cronje N, Brown R, Negus R, Coode B, Moss P, Hassan T, Hamer W, Britto J. Validation of a diagnostic reminder system in emergency medicine: a multi-centre study. Emerg Med J. 2007;24(9):619\u201324.","journal-title":"Emerg Med J"},{"key":"1988_CR27","first-page":"3244","volume":"2018","author":"M Reumann","year":"2018","unstructured":"Reumann M, Giovannini A, Nadworny B, Auer C, Girardi I, Marchiori C. Cognitive DDx assistant in rare diseases. Annu Int Conf IEEE Eng Med Biol Soc. 2018;2018:3244\u20137.","journal-title":"Annu Int Conf IEEE Eng Med Biol Soc"},{"issue":"1","key":"1988_CR28","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1515\/dx-2019-0033","volume":"8","author":"S Cheraghi-Sohi","year":"2021","unstructured":"Cheraghi-Sohi S, Alam R, Hann M, Esmail A, Campbell S, Riches N. Assessing the utility of a differential diagnostic generator in UK general practice: a feasibility study. Diagnosis (Berl). 2021;8(1):91\u20139.","journal-title":"Diagnosis (Berl)"},{"issue":"1","key":"1988_CR29","doi-asserted-by":"publisher","first-page":"1742967","DOI":"10.1080\/10872981.2020.1742967","volume":"25","author":"FS Leeds","year":"2020","unstructured":"Leeds FS, Atwa KM, Cook AM, Conway KA, Crawford TN. Teaching heuristics and mnemonics to improve generation of differential diagnoses. Med Educ Online. 2020;25(1):1742967.","journal-title":"Med Educ Online"},{"issue":"13","key":"1988_CR30","doi-asserted-by":"publisher","first-page":"1493","DOI":"10.1001\/archinte.165.13.1493","volume":"165","author":"ML Graber","year":"2005","unstructured":"Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med. 2005;165(13):1493\u20139.","journal-title":"Arch Intern Med"},{"issue":"Suppl 1","key":"1988_CR31","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1007\/s11606-007-0271-8","volume":"23","author":"ML Graber","year":"2008","unstructured":"Graber ML, Mathew A. Performance of a web-based clinical diagnosis support system for internists. J Gen Intern Med. 2008;23(Suppl 1):37\u201340.","journal-title":"J Gen Intern Med"},{"issue":"2","key":"1988_CR32","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1515\/dx-2014-0058","volume":"2","author":"JS Barbieri","year":"2015","unstructured":"Barbieri JS, French B, Umscheid CA. Uptake and impact of a clinical diagnostic decision support tool at an academic medical center. Diagnosis (Berl). 2015;2(2):123\u20137.","journal-title":"Diagnosis (Berl)"},{"key":"1988_CR33","unstructured":"TELOS criteria, Your guide to conducting a feasibility study in construction [https:\/\/gobridgit.com\/]"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01988-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-022-01988-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01988-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,27]],"date-time":"2023-11-27T17:37:09Z","timestamp":1701106629000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-022-01988-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,24]]},"references-count":33,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["1988"],"URL":"https:\/\/doi.org\/10.1186\/s12911-022-01988-2","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,24]]},"assertion":[{"value":"24 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 August 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 September 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This study did include only publicly available secondary and anonymized or published patient data and did not involve any real patient, hence no ethical approval was needed. All research was performed in accordance with the Declaration of Helsinki.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"Fritz P. and Kleinhans A. have developed the non-commercial Memem7. The other authors have no conflicts of interest to declare.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"254"}}