{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T22:07:38Z","timestamp":1790114858815,"version":"4.0.1"},"reference-count":26,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"\n Background<\/jats:title>\n Large language models (LLMs) are increasingly being considered for clinical decision support, yet their safety in risk-based cervical screening management remains insufficiently characterized.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study benchmarked the guideline concordance and safety-related performance of 3 LLMs in the initial American Society for Colposcopy and Cervical Pathology (ASCCP) risk-based management of abnormal cervical screening results, using a purposively constructed synthetic scenario set that oversamples complex and history-dependent decision nodes.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We developed 60 synthetic clinical scenarios reflecting initial abnormal screening management in immunocompetent, nonpregnant women aged 25 to 65 years using a predefined scenario coverage matrix. GPT-5.3, Gemini 3 Flash, and DeepSeek V3.2 were tested under 2 prompt conditions: Baseline Clinical Prompt and Guideline-Directed Prompt Package. Each scenario was run in 3 independent repetitions per model and prompt arm (1080 total observations). Responses were evaluated by 2 obstetrics and gynecology specialists, blinded to model and prompt-arm identity but not independent of gold-standard construction, using a prespecified rubric. The primary end point was the unsafe major error-free rate. Proportions are reported with CIs adjusted for within-scenario clustering, and generalized estimating equations were used for inferential comparisons.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Under the guideline-directed prompt package, the unsafe major error-free rate was 100% (95% CI 94%-100%) for GPT-5.3, 98.9% (95% CI 94%-99.8%) for Gemini 3 Flash, and 75% (95% CI 63.2%-84%) for DeepSeek V3.2. In the main-effects model, the guideline-directed prompt package was associated with higher odds of both unsafe major error-free performance (odds ratio [OR] 3.76, 95% CI 2.45-5.77; P<.001) and exact concordance (OR 8.27, 95% CI 4.14-16.52; P<.001). Error rates increased substantially with scenario complexity, rising from 3.1% in low-complexity to 29% in high-complexity scenarios. The most frequent error subtypes were undermanagement, genotype misinterpretation, and history neglect. Interrater agreement was almost perfect (weighted \u03ba=0.839, 95% CI 0.811-0.867).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Safety-oriented benchmark performance in initial ASCCP risk-based management varied markedly by model, prompt condition, and scenario complexity. The guideline-directed prompt package was associated with improvement in both safety and guideline concordance, but even the best-performing model remained vulnerable in complex, history-dependent scenarios. LLMs may have value as clinician-supervised decision support tools, but these benchmark findings should not be interpreted as supporting autonomous clinical use in cervical screening management.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/98131","type":"journal-article","created":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T21:45:06Z","timestamp":1790113506000},"page":"e98131","source":"Crossref","is-referenced-by-count":0,"title":["Safety-Oriented Benchmarking of Large Language Models in Risk-Based Management of Abnormal Cervical Screening Results: Scenario-Based Benchmark Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9861-6767","authenticated-orcid":false,"given":"\u00d6mer Osman","family":"Ero\u011flu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5049-1415","authenticated-orcid":false,"given":"Cans\u0131n","family":"Ero\u011flu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,22]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1097\/LGT.0000000000000525"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1097\/LGT.0000000000000529"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1097\/LGT.0000000000000528"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1097\/LGT.0000000000000788"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1097\/LGT.0000000000000804"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1097\/LGT.0000000000000802"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1097\/LGT.0000000000000614"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1093\/ehjdh\/ztaf028"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01684-1"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02277-8"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1186\/s12872-025-05431-y"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1016\/j.xcrm.2025.102323"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.3390\/ai7020077"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics16030424"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1002\/ijgo.15861"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.2196\/63626"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-03088-3"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1111\/1471-0528.70095"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/73.1.13"},{"issue":"1","key":"ref20","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"Landis, JR","year":"1977","journal-title":"Biometrics"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/80.1.27"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1002\/sim.1047"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1198\/000313001300339897"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2025.103222"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03425-5"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1089\/jwh.2022.0100"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e98131","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T21:45:07Z","timestamp":1790113507000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e98131"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,22]]},"references-count":26,"URL":"https:\/\/doi.org\/10.2196\/98131","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.98131","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,22]]}}}