{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T07:23:09Z","timestamp":1787728989642,"version":"build-2784847793"},"reference-count":30,"publisher":"Wiley","issue":"8","license":[{"start":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T00:00:00Z","timestamp":1787011200000},"content-version":"vor","delay-in-days":17,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:00:00Z","timestamp":1785542400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Euro J of Neurology"],"published-print":{"date-parts":[[2026,8]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) are entering clinical workflows for information retrieval and decision support, but concerns persist about factual accuracy and traceability to evidence. Prof. Valmed is the first CE\u2010marked LLM\u2010based tool for medical information retrieval in Europe, raising the question whether certification aligns with reliable performance in neurology. This study aims to assess the answer accuracy of Prof. Valmed on a neurology benchmark and compare its performance with previously evaluated commercial LLM tools.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Prof. Valmed (V2.0.1_2.0.0) was tested on a 130\u2010item benchmark derived from American Academy of Neurology guidelines (65 case\u2010based, 65 knowledge\u2010based). Each question was prompted four times. Two raters scored responses as correct, inaccurate, wrong, or refused; disagreements were adjudicated. Modal ratings per question were used for analysis. Comparative performance against 16 LLMs or configurations previously assessed using the same protocol was evaluated.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Prof. Valmed achieved 76.2% correct answers, 14.6% inaccurate, 6.9% wrong, and 2.3% refused. In pairwise comparisons, Prof. Valmed performed significantly better than 8 models, comparable to several retrieval\u2010 or whitelist\u2010enabled tools, and significantly worse than one reasoning model with whitelisting of neurology sources. Overall, its performance was within the range of standard commercial tools.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The CE\u2010marked product demonstrated accuracy comparable to contemporary LLMs but did not outperform reasoning\u2010enabled systems. CE marking ensures regulatory conformity rather than inherent performance advantage, and the certification process may limit the integration of rapidly evolving model architectures into approved systems. Future evaluations should examine reasoning quality, source use, and update feasibility across medical domains.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1111\/ene.70736","type":"journal-article","created":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T13:31:32Z","timestamp":1787059892000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating the First\n                    <scp>CE<\/scp>\n                    \u2010Marked\n                    <scp>LLM<\/scp>\n                    \u2010Based Chatbot for Evidence\u2010Based Neurology"],"prefix":"10.1111","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-0835-0120","authenticated-orcid":false,"given":"Patricia","family":"Kirschner","sequence":"first","affiliation":[{"name":"Department of Neurology, Medical Faculty and University Hospital D\u00fcsseldorf Heinrich Heine University D\u00fcsseldorf  D\u00fcsseldorf Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-6219-988X","authenticated-orcid":false,"given":"Paula Z.","family":"Epping","sequence":"additional","affiliation":[{"name":"Department of Neurology, Medical Faculty and University Hospital D\u00fcsseldorf Heinrich Heine University D\u00fcsseldorf  D\u00fcsseldorf Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2571-3501","authenticated-orcid":false,"given":"Sven G.","family":"Meuth","sequence":"additional","affiliation":[{"name":"Department of Neurology, Medical Faculty and University Hospital D\u00fcsseldorf Heinrich Heine University D\u00fcsseldorf  D\u00fcsseldorf Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marc","family":"Pawlitzki","sequence":"additional","affiliation":[{"name":"Department of Neurology, Medical Faculty and University Hospital D\u00fcsseldorf Heinrich Heine University D\u00fcsseldorf  D\u00fcsseldorf Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2496-1415","authenticated-orcid":false,"given":"Lars","family":"Masanneck","sequence":"additional","affiliation":[{"name":"Department of Neurology, Medical Faculty and University Hospital D\u00fcsseldorf Heinrich Heine University D\u00fcsseldorf  D\u00fcsseldorf Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,8,18]]},"reference":[{"key":"e_1_2_11_2_1","doi-asserted-by":"publisher","DOI":"10.2196\/67383"},{"key":"e_1_2_11_3_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010025\u201001684\u20101"},{"key":"e_1_2_11_4_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2024.21700"},{"key":"e_1_2_11_5_1","doi-asserted-by":"publisher","DOI":"10.2196\/59823"},{"key":"e_1_2_11_6_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000877"},{"key":"e_1_2_11_7_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010025\u201001651\u2010w"},{"key":"e_1_2_11_8_1","doi-asserted-by":"publisher","DOI":"10.2196\/79379"},{"key":"e_1_2_11_9_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12911\u2010025\u201002954\u20104"},{"key":"e_1_2_11_10_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2023.14217"},{"key":"e_1_2_11_11_1","doi-asserted-by":"publisher","DOI":"10.3390\/brainsci15040347"},{"key":"e_1_2_11_12_1","doi-asserted-by":"publisher","DOI":"10.3390\/jcm14176333"},{"key":"e_1_2_11_13_1","doi-asserted-by":"publisher","DOI":"10.2196\/58158"},{"key":"e_1_2_11_14_1","doi-asserted-by":"publisher","DOI":"10.2196\/53297"},{"key":"e_1_2_11_15_1","unstructured":"Medical Co\u2010Pilot: First AI\u2010Supported Medical Device to Support Clinical Decisions accessed May 26 2025 https:\/\/www.vde.com\/en\/press\/press\u2010releases\/medical\u2010co\u2010pilot\u2010vde\u2010aiq\u2010valmed."},{"key":"e_1_2_11_16_1","unstructured":"Prof. Valmed Instructions for Use accessed October 6 2025 https:\/\/profvalmed.com\/instructions\u2010for\u2010use\/."},{"key":"e_1_2_11_17_1","doi-asserted-by":"publisher","DOI":"10.1212\/NE9.0000000000200260"},{"key":"e_1_2_11_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00296\u2010025\u201006068\u2010y"},{"key":"e_1_2_11_19_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010025\u201001536\u2010y"},{"key":"e_1_2_11_20_1","volume-title":"World's First Regulatory Clearance for a Large Language Model Medical Device","author":"Harvey H.","year":"2025"},{"key":"e_1_2_11_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10916\u2010025\u201002232\u2010w"},{"key":"e_1_2_11_22_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586\u2010023\u201006291\u20102"},{"key":"e_1_2_11_23_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010024\u201001258\u20107"},{"key":"e_1_2_11_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/s2589\u20107500(23)00083\u20103"},{"key":"e_1_2_11_25_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010025\u201001546\u2010w"},{"key":"e_1_2_11_26_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586\u2010025\u201008869\u20104"},{"key":"e_1_2_11_27_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010024\u201001263\u2010w"},{"key":"e_1_2_11_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpubeco.2016.11.010"},{"key":"e_1_2_11_29_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010024\u201001221\u20106"},{"key":"e_1_2_11_30_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41746\u2010022\u201000668\u20109"},{"key":"e_1_2_11_31_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41551\u2010019\u20100426\u2010z"}],"container-title":["European Journal of Neurology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/ene.70736","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/ene.70736","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/ene.70736","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T06:30:38Z","timestamp":1787725838000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1111\/ene.70736"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8]]},"references-count":30,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2026,8]]}},"alternative-id":["10.1111\/ene.70736"],"URL":"https:\/\/doi.org\/10.1111\/ene.70736","archive":["Portico"],"relation":{},"ISSN":["1351-5101","1468-1331"],"issn-type":[{"value":"1351-5101","type":"print"},{"value":"1468-1331","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8]]},"assertion":[{"value":"2026-03-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70736"}}