{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T06:08:32Z","timestamp":1788847712853,"version":"build-2803163510"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T00:00:00Z","timestamp":1784592000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>STUDY QUESTION<\/jats:title>\n                    <jats:p>How do the three most widely accessible large language models perform in terms of accuracy, consistency, and reliability when answering clinically relevant questions derived from the 2022 ESHRE guideline on endometriosis?<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>SUMMARY ANSWER<\/jats:title>\n                    <jats:p>Model A achieved the highest accuracy scores, model C demonstrated significantly superior consistency across repeated queries, and all three models showed comparable but suboptimal reliability under free-tier access, while under official API access median accuracy converged across providers and reliability increased.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>WHAT IS KNOWN ALREADY<\/jats:title>\n                    <jats:p>Large language models are increasingly consulted by both clinicians and patients as readily accessible sources of medical information. In reproductive medicine, preliminary evidence suggests that individual platforms may retrieve endometriosis-related content with mixed fidelity. However, no study has simultaneously benchmarked multiple models against a single, internationally recognized endometriosis guideline. The extent to which these tools can be trusted to faithfully reproduce evidence-based recommendations on endometriosis diagnosis and management remains largely unexplored.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>STUDY DESIGN, SIZE, DURATION<\/jats:title>\n                    <jats:p>Cross-sectional, multi-platform comparative study. Fifty clinically relevant questions covering the diagnostic and therapeutic domains of the 2022 ESHRE endometriosis guideline were simultaneously submitted to all three models during December 2025. Each question was entered in duplicate using independent sessions to assess response consistency and reliability. Subgroup analyses were carried out by submitting the questions to the API version and to the free-tier version of the platforms available in May 2026.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>PARTICIPANTS\/MATERIALS, SETTING, METHODS<\/jats:title>\n                    <jats:p>The three models were accessed through their respective free web interfaces using new accounts, without prompt engineering, prior training, or retrieval-augmented generation. A zero-shot prompting approach was adopted. Accuracy was evaluated by two independent experts using the Global Quality Score (GQS); consistency was defined as identical responses across the three iterations; reliability was defined as the alignment of each response with the ESHRE guideline. Discrepancies were settled by a third reviewer.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>MAIN RESULTS AND THE ROLE OF CHANCE<\/jats:title>\n                    <jats:p>Significant differences in accuracy were observed across models (Kruskal\u2013Wallis H\u2009=\u200937.10, P\u2009&amp;lt;\u20090.001). Model A achieved the highest median GQS (5, interquartile range [IQR] 4\u20135), followed by model C (4, IQR 3\u20135) and model B (3, IQR 3\u20134). Post-hoc analysis confirmed that model A significantly outperformed model B (P\u2009&amp;lt;\u20090.001) but not model C (P\u2009=\u20090.123), while model C also scored significantly higher than model B (P\u2009&amp;lt; 0.001). For consistency, model C demonstrated a significantly higher rate of reproducible responses (92.0%) compared with model A (72.0%, P\u2009=\u20090.028) and model B (68.0%, P\u2009=\u20090.008). No significant between-model differences were found for reliability (\u03c72\u2009=\u20091.029, P\u2009=\u20090.598), with rates of 76.0% for model C, 68.0% for model A, and 68.0% for model B. In API and free-tier May 2026 subgroup analyses, median GQS converged to 4 across all three providers, and reliability rose to at least 76% for each provider.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>LIMITATIONS, REASONS FOR CAUTION<\/jats:title>\n                    <jats:p>Model outputs may change with subsequent updates. GQS retains a degree of subjectivity despite expert adjudication. The study tested factual recall rather than complex clinical reasoning, limiting generalizability to real-world decision-making scenarios.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>WIDER IMPLICATIONS OF THE FINDINGS<\/jats:title>\n                    <jats:p>These findings provide the first multi-platform benchmark of large language models against the latest endometriosis guideline. While models A and C retrieved guideline-concordant information with acceptable fidelity, none of the models achieved a level of reliability required for unsupervised clinical use. The dissociation between accuracy and consistency at the free tier, and its attenuation under API and more recent accesses, indicates that both model capability and commercial tier shape user-facing outputs. The dissociation between accuracy and consistency underscores that a model producing high-quality answers does not necessarily do so in a reproducible manner. Expert oversight remains crucial when interpreting their output regarding endometriosis care. Future research should extend this framework to additional endometriosis guidelines and to longitudinal monitoring of models\u2019 performance with their upgrades.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>FUNDING<\/jats:title>\n                    <jats:p>No external funding was received for this study.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>DISCLOSURES<\/jats:title>\n                    <jats:p>The authors declare no competing interests.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>TRIAL REGISTRATION NUMBER<\/jats:title>\n                    <jats:p>N\/A.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/humrep\/deag116","type":"journal-article","created":{"date-parts":[[2026,7,21]],"date-time":"2026-07-21T06:13:18Z","timestamp":1784614398000},"page":"1544-1551","source":"Crossref","is-referenced-by-count":0,"title":["Can AI speak endo? A multi-platform evaluation of large language models against ESHRE endometriosis guidelines"],"prefix":"10.1093","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3676-7716","authenticated-orcid":false,"given":"Gaetano","family":"Riemma","sequence":"first","affiliation":[{"name":"Obstetrics and Gynecology Unit, Department of Woman, Child and General and Specialized Surgery, University of Campania \u201cLuigi Vanvitelli\u201d , Naples,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Florindo Mario","family":"Caniglia","sequence":"additional","affiliation":[{"name":"Obstetrics and Gynecology Unit, Department of Woman, Child and General and Specialized Surgery, University of Campania \u201cLuigi Vanvitelli\u201d , Naples,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Camilla","family":"Casolari","sequence":"additional","affiliation":[{"name":"Obstetrics and Gynecology Unit, Department of Woman, Child and General and Specialized Surgery, University of Campania \u201cLuigi Vanvitelli\u201d , Naples,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Antonio","family":"Maiorana","sequence":"additional","affiliation":[{"name":"Gynaecologic and Obstetric Unit, ARNAS Civico Di Cristina Fatebenefratelli , Palermo,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5929-2357","authenticated-orcid":false,"given":"Mauro","family":"Cozzolino","sequence":"additional","affiliation":[{"name":"IVIRMA Global Research Alliance, IVI Bologna , Bologna,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5291-3693","authenticated-orcid":false,"given":"Vittorio","family":"Agrifoglio","sequence":"additional","affiliation":[{"name":"AMBRA Fertility Center , Palermo,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pasquale","family":"De Franciscis","sequence":"additional","affiliation":[{"name":"Obstetrics and Gynecology Unit, Department of Woman, Child and General and Specialized Surgery, University of Campania \u201cLuigi Vanvitelli\u201d , Naples,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luigi","family":"Cobellis","sequence":"additional","affiliation":[{"name":"Obstetrics and Gynecology Unit, Department of Woman, Child and General and Specialized Surgery, University of Campania \u201cLuigi Vanvitelli\u201d , Naples,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Raffaela Maria","family":"Carotenuto","sequence":"additional","affiliation":[{"name":"Obstetrics and Gynecology Unit, Department of Woman, Child and General and Specialized Surgery, University of Campania \u201cLuigi Vanvitelli\u201d , Naples,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7414-4224","authenticated-orcid":false,"given":"Andrea","family":"Etrusco","sequence":"additional","affiliation":[{"name":"AMBRA Fertility Center , Palermo,","place":["Italy"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,7,21]]},"reference":[{"key":"2026090801520694100_deag116-B1","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1002\/uog.70062","article-title":"Pelvic pain and endometriosis","volume":"66","author":"Abstracts of the 35th World Congress on Ultrasound in Obstetrics and Gynecology, 14\u201317 September, Cancun, Mexico","year":"2025","journal-title":"Ultrasound Obstet Gynecol"},{"key":"2026090801520694100_deag116-B2","doi-asserted-by":"crossref","first-page":"432","DOI":"10.1097\/AOG.0000000000006181","article-title":"Diagnosis of endometriosis: ACOG Clinical Practice Guideline No. 11","volume":"147","author":"American College of Obstetricians and Gynecologists","year":"2026","journal-title":"Obstet Gynecol"},{"key":"2026090801520694100_deag116-B3","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1016\/j.fertnstert.2012.05.031","article-title":"Endometriosis and infertility: a committee opinion","volume":"98","author":"American Society for Reproductive Medicine","year":"2012","journal-title":"Fertil Steril"},{"key":"2026090801520694100_deag116-B4","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/j.ajog.2019.02.033","article-title":"Assessing research gaps and unmet needs in endometriosis","volume":"221","author":"As-Sanie","year":"2019","journal-title":"Am J Obstet Gynecol"},{"key":"2026090801520694100_deag116-B5","doi-asserted-by":"crossref","first-page":"1477898","DOI":"10.3389\/fmed.2024.1477898","article-title":"Large language models in patient education: a scoping review of applications in medicine","volume":"11","author":"Aydin","year":"2024","journal-title":"Front Med (Lausanne)"},{"key":"2026090801520694100_deag116-B6","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1001\/jamainternmed.2023.1838","article-title":"Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum","volume":"183","author":"Ayers","year":"2023","journal-title":"JAMA Intern Med"},{"key":"2026090801520694100_deag116-B7","doi-asserted-by":"crossref","first-page":"106034","DOI":"10.1016\/j.ijmedinf.2025.106034","article-title":"Comparing ChatGPT and physicians\u2019 answers to endometriosis questions on Reddit: a blind expert evaluation","volume":"203","author":"Beaulieu","year":"2025","journal-title":"Int J Med Inform"},{"key":"2026090801520694100_deag116-B8","doi-asserted-by":"crossref","first-page":"hoac009","DOI":"10.1093\/hropen\/hoac009","article-title":"ESHRE guideline: endometriosis","volume":"2022","author":"Becker","year":"2022","journal-title":"Hum Reprod Open"},{"key":"2026090801520694100_deag116-B9","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1177\/1363459318817943","article-title":"I feel like I\u2019m being stabbed by a thousand tiny men\u201d: the challenges of communicating endometriosis pain","volume":"24","author":"Bullo","year":"2020","journal-title":"Health (London)"},{"key":"2026090801520694100_deag116-B10","doi-asserted-by":"crossref","first-page":"oyaf038","DOI":"10.1093\/oncolo\/oyaf038","article-title":"Medical accuracy of artificial intelligence chatbots in oncology: a scoping review","volume":"30","author":"Chen","year":"2025","journal-title":"Oncologist"},{"key":"2026090801520694100_deag116-B11","doi-asserted-by":"crossref","first-page":"1285","DOI":"10.1002\/ijgo.15959","article-title":"Evaluating the performance and clinical decision-making impact of ChatGPT-4 in reproductive medicine","volume":"168","author":"Chen","year":"2025","journal-title":"Int J Gynaecol Obstet"},{"key":"2026090801520694100_deag116-B12","doi-asserted-by":"crossref","first-page":"615","DOI":"10.5114\/biolsport.2023.125623","article-title":"From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing","volume":"40","author":"Dergaa","year":"2023","journal-title":"Biol Sport"},{"key":"2026090801520694100_deag116-B13","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1186\/s12871-025-03130-x","article-title":"Bridging the gap between scientists and clinicians: addressing collaboration challenges in clinical AI integration","volume":"25","author":"Gisselbaek","year":"2025","journal-title":"BMC Anesthesiol"},{"key":"2026090801520694100_deag116-B14","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1016\/j.fertnstert.2007.11.020","article-title":"Diagnostic experience among 4,334 women reporting surgically diagnosed endometriosis","volume":"91","author":"Greene","year":"2009","journal-title":"Fertil Steril"},{"key":"2026090801520694100_deag116-B15","doi-asserted-by":"crossref","first-page":"101730","DOI":"10.1016\/j.artd.2025.101730","article-title":"Comparative efficacy of ChatGPT and DeepSeek in addressing patient queries on gonarthrosis and total knee arthroplasty","volume":"33","author":"Gurbuz","year":"2025","journal-title":"Arthroplast Today"},{"key":"2026090801520694100_deag116-B16","doi-asserted-by":"crossref","first-page":"3412","DOI":"10.1093\/humrep\/des316","article-title":"Diagnostic delay for endometriosis in Austria and Germany: causes and possible consequences","volume":"27","author":"Hudelist","year":"2012","journal-title":"Hum Reprod"},{"key":"2026090801520694100_deag116-B17","doi-asserted-by":"crossref","first-page":"1552","DOI":"10.1093\/humrep\/det050","article-title":"Consensus on current management of endometriosis","volume":"28","author":"Johnson","year":"2013","journal-title":"Hum Reprod"},{"key":"2026090801520694100_deag116-B18","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1038\/s41746-024-01208-3","article-title":"Evaluating multimodal AI in medical diagnostics","volume":"7","author":"Kaczmarczyk","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2026090801520694100_deag116-B19","doi-asserted-by":"crossref","first-page":"58","DOI":"10.51866\/cm0006","article-title":"Use of ChatGPT in medical research and scientific writing","volume":"18","author":"Lee","year":"2023","journal-title":"Malays Fam Physician"},{"key":"2026090801520694100_deag116-B20","doi-asserted-by":"crossref","first-page":"176","DOI":"10.62347\/UIAP7979","article-title":"DeepSeek vs ChatGPT: a comparison study of their performance in answering prostate cancer radiotherapy questions in multiple languages","volume":"13","author":"Luo","year":"2025","journal-title":"Am J Clin Exp Urol"},{"key":"2026090801520694100_deag116-B21","volume-title":"D\u00e9m\u00ealer le vrai du faux sur l\u2019endom\u00e9triose","author":"Minist\u00e8re du Travail dlS, des Solidarit\u00e9s et des Familles (France)","year":"2024"},{"key":"2026090801520694100_deag116-B22","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1016\/j.pneurobio.2011.01.002","article-title":"The effect of pain on cognitive function: a review of clinical and preclinical research","volume":"93","author":"Moriarty","year":"2011","journal-title":"Prog Neurobiol"},{"key":"2026090801520694100_deag116-B23","author":"National Institute for Health and Care Excellence: Guidelines","year":"2017"},{"key":"2026090801520694100_deag116-B24","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1016\/j.fertnstert.2011.05.090","article-title":"Impact of endometriosis on quality of life and work productivity: a multicenter study across ten countries","volume":"96","author":"Nnoaham","year":"2011","journal-title":"Fertil Steril"},{"key":"2026090801520694100_deag116-B25","doi-asserted-by":"crossref","first-page":"e64593","DOI":"10.2196\/64593","article-title":"Understanding AI\u2019s role in endometriosis patient education and evaluating its information and accuracy: systematic review","volume":"3","author":"Oliveira","year":"2024","journal-title":"JMIR AI"},{"key":"2026090801520694100_deag116-B26","doi-asserted-by":"crossref","first-page":"e0318568","DOI":"10.1371\/journal.pone.0318568","article-title":"YouTube as an information source in deep margin elevation: reliability, accuracy and quality analysis","volume":"20","author":"Ozdemir","year":"2025","journal-title":"PLoS One"},{"key":"2026090801520694100_deag116-B27","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1002\/ijgo.15309","article-title":"Accuracy and reproducibility of ChatGPT\u2019s free version answers about endometriosis","volume":"165","author":"Ozgor","year":"2024","journal-title":"Int J Gynaecol Obstet"},{"key":"2026090801520694100_deag116-B28","doi-asserted-by":"crossref","first-page":"105178","DOI":"10.1016\/j.rbmo.2025.105178","article-title":"IVF\/ICSI or surgery as first approach for the treatment of infertility associated with ovarian and deep infiltrating endometriosis? A systematic review and meta-analysis","volume":"52","author":"Riemma","year":"2026","journal-title":"Reprod Biomed Online"},{"key":"2026090801520694100_deag116-B29","doi-asserted-by":"crossref","first-page":"887","DOI":"10.3390\/healthcare11060887","article-title":"ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns","volume":"11","author":"Sallam","year":"2023","journal-title":"Healthcare"},{"key":"2026090801520694100_deag116-B30","doi-asserted-by":"crossref","first-page":"6099","DOI":"10.1007\/s00405-024-08828-1","article-title":"Assessing the use of the novel tool Claude 3 in comparison to ChatGPT 4.0 as an artificial intelligence tool in the diagnosis and therapy of primary head and neck cancer cases","volume":"281","author":"Schmidl","year":"2024","journal-title":"Eur Arch Otorhinolaryngol"},{"key":"2026090801520694100_deag116-B31","doi-asserted-by":"crossref","first-page":"103758","DOI":"10.1016\/j.jsurg.2025.103758","article-title":"Guidelines and checklists for artificial intelligence (AI) in surgical education research: a comprehensive narrative review","volume":"83","author":"Silvestri","year":"2026","journal-title":"J Surg Educ"},{"key":"2026090801520694100_deag116-B32","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1001\/jamainternmed.2025.6078","article-title":"Scientific writing in the age of artificial intelligence","volume":"186","author":"Steiner","year":"2026","journal-title":"JAMA Intern Med"},{"key":"2026090801520694100_deag116-B33","doi-asserted-by":"crossref","first-page":"4056","DOI":"10.3390\/jcm12124056","article-title":"Understanding psychological symptoms of endometriosis from a research domain criteria perspective","volume":"12","author":"Van Stein","year":"2023","journal-title":"JCM"},{"key":"2026090801520694100_deag116-B34","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1002\/ijgo.15846","article-title":"TikTok as a platform for hysteroscopy information: an analytical video-based cross-sectional study to assess quality, reliability, and accuracy","volume":"168","author":"Vitale","year":"2025","journal-title":"Int J Gynaecol Obstet"},{"key":"2026090801520694100_deag116-B35","doi-asserted-by":"crossref","first-page":"4815","DOI":"10.1007\/s00405-025-09524-4","article-title":"ChatGPT versus DeepSeek in head and neck cancer staging and treatment planning: guideline-based study","volume":"282","author":"Vural Camalan","year":"2025","journal-title":"Eur Arch Otorhinolaryngol"},{"key":"2026090801520694100_deag116-B36","doi-asserted-by":"crossref","first-page":"23821205211024078","DOI":"10.1177\/23821205211024078","article-title":"Are we ready to integrate artificial intelligence literacy into medical school curriculum: students and faculty survey","volume":"8","author":"Wood","year":"2021","journal-title":"J Med Educ Curric Dev"},{"key":"2026090801520694100_deag116-B37","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1186\/s12911-025-03114-4","article-title":"Application of ChatGPT-based artificial intelligence in the diagnosis and management of polycystic ovary syndrome","volume":"25","author":"Zhu","year":"2025","journal-title":"BMC Med Inform Decis Mak"},{"key":"2026090801520694100_deag116-B38","doi-asserted-by":"crossref","first-page":"e66114","DOI":"10.2196\/66114","article-title":"Large language models in worldwide medical exams: platform development and comprehensive analysis","volume":"26","author":"Zong","year":"2024","journal-title":"J Med Internet Res"}],"container-title":["Human Reproduction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/humrep\/article-pdf\/41\/9\/1544\/69646180\/deag116.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/humrep\/article-pdf\/41\/9\/1544\/69646180\/deag116.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T05:52:18Z","timestamp":1788846738000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/humrep\/article\/41\/9\/1544\/8738631"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,21]]},"references-count":38,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,7,21]]},"published-print":{"date-parts":[[2026,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/humrep\/deag116","relation":{},"ISSN":["0268-1161","1460-2350"],"issn-type":[{"value":"0268-1161","type":"print"},{"value":"1460-2350","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,9]]},"published":{"date-parts":[[2026,7,21]]}}}