{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T06:25:31Z","timestamp":1787034331195,"version":"build-2736575974"},"reference-count":54,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T00:00:00Z","timestamp":1787011200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Immunol."],"abstract":"<jats:p>Large language models (LLMs) are increasingly used to mine biomedical literature into knowledge graphs (KGs), yet whether independent models agree when given the same text is largely untested. We passed 9,978 immune-checkpoint PubMed abstracts through four LLMs (DeepSeek, GLM, Kimi, and Qwen) under an identical prompt and an eight-entity-type schema. This yielded 140,462 triples and a 95,981-edge union graph. We benchmarked the four graphs along five axes: extraction, correctness against a human expert, topology, embedding-based link prediction, and external validation. Extraction volume varied more than twofold (20,122\u201345,957 triples), and pairwise edge overlap was low (Jaccard, 0.06\u20130.11). Against a 151-triple dual-annotated gold set, expert precision was 0.914 (95% CI = 0.858\u20130.949) with overlapping per-model intervals, while an LLM judge systematically under-rated precision on the 51 triples it scored (raw agreement, 0.65). External support rose with cross-model consensus, but naive majority agreement did not beat the best single model (DeepSeek, 0.377); only unanimous four-model agreement exceeded it (0.423,\u00a0+\u00a012% relative) at ~ 4% edge coverage. The unanimity interval (95% CI = 0.362\u20130.486) overlaps that of the best single model (0.358\u20130.397), so unanimity is best read as a conservative reliability filter rather than a general fix for graph quality. Consensus is a tunable precision\u2013coverage control, not a free-lunch gain; under the conditions tested here\u2014four models, one domain, one prompt, and schema\u2014the choice of extraction model mattered more than naive aggregation.<\/jats:p>","DOI":"10.3389\/fimmu.2026.1937268","type":"journal-article","created":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:38:21Z","timestamp":1787031501000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["LLMs build inconsistent knowledge graphs from one immune-checkpoint corpus: a five-axis benchmark and consensus precision\u2013coverage protocol"],"prefix":"10.3389","volume":"17","author":[{"given":"Qiang","family":"Xie","sequence":"first","affiliation":[{"name":"Department of Surgical Oncology, The First Affiliated Hospital of Bengbu Medical University","place":["Bengbu, Anhui, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Fang","sequence":"additional","affiliation":[{"name":"Department of Laboratory Medicine, The Third Affiliated Hospital of Zhengzhou University","place":["Zhengzhou, Henan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanan","family":"Hu","sequence":"additional","affiliation":[{"name":"Department of Infectious Diseases, The First Affiliated Hospital of Bengbu Medical University","place":["Bengbu, Anhui, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,8,18]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1038\/nrc3239","article-title":"The blockade of immune checkpoints in cancer immunotherapy","volume":"12","author":"Pardoll","year":"2012","journal-title":"Nat Rev Cancer"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1126\/science.271.5256.1734","article-title":"Enhancement of antitumor immunity by CTLA-4 blockade","volume":"271","author":"Leach","year":"1996","journal-title":"Sci (New York NY)"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1056\/nejmoa1003466","article-title":"Improved survival with ipilimumab in patients with metastatic melanoma","volume":"363","author":"Hodi","year":"2010","journal-title":"N Engl J Med"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1056\/nejmoa1200690","article-title":"Safety, activity, and immune correlates of anti-PD-1 antibody in cancer","volume":"366","author":"Topalian","year":"2012","journal-title":"N Engl J Med"},{"key":"B5","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1126\/science.aaa8172","article-title":"The future of immune checkpoint therapy","volume":"348","author":"Sharma","year":"2015","journal-title":"Sci (New York NY)"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.1126\/science.aar4060","article-title":"Cancer immunotherapy using checkpoint blockade","volume":"359","author":"Ribas","year":"2018","journal-title":"Sci (New York NY)"},{"key":"B7","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1056\/nejmoa1504030","article-title":"Combined nivolumab and ipilimumab or monotherapy in untreated melanoma","volume":"373","author":"Larkin","year":"2015","journal-title":"N\u00a0Engl J Med"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.1016\/j.annonc.2023.10.004","article-title":"Nivolumab versus docetaxel in advanced nonsquamous non-small-cell lung cancer","volume":"373","author":"Borghaei","year":"2015","journal-title":"N Engl J Med"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1056\/nejmoa1606774","article-title":"Pembrolizumab versus chemotherapy for PD-L1-positive non-small-cell lung cancer","volume":"375","author":"Reck","year":"2016","journal-title":"N Engl J Med"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1056\/nejmoa1503093","article-title":"Pembrolizumab versus ipilimumab in advanced melanoma","volume":"372","author":"Robert","year":"2015","journal-title":"N Engl J Med"},{"key":"B11","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1056\/nejmoa2109970","article-title":"Relatlimab and nivolumab versus nivolumab in untreated advanced melanoma","volume":"386","author":"Tawbi","year":"2022","journal-title":"N Engl J Med"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaa1348","article-title":"Cancer immunology. Mutational landscape determines sensitivity to PD-1 blockade in non-small cell lung cancer","volume":"348","author":"Rizvi","year":"2015","journal-title":"Sci (New York NY)"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2017.01.017","article-title":"Primary, adaptive, and acquired resistance to cancer immunotherapy","volume":"168","author":"Sharma","year":"2017","journal-title":"Cell"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.1016\/j.csbj.2020.05.017","article-title":"Constructing knowledge graphs and their biomedical applications","volume":"18","author":"Nicholson","year":"2020","journal-title":"Comput Struct Biotechnol J"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.7554\/elife.26726","article-title":"Systematic integration of biomedical knowledge prioritizes drugs for repurposing","volume":"6","author":"Himmelstein","year":"2017","journal-title":"eLife"},{"key":"B16","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1038\/s41597-023-01960-3","article-title":"Building a knowledge graph to enable precision medicine","volume":"10","author":"Chandak","year":"2023","journal-title":"Sci Data"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bty294","article-title":"Modeling polypharmacy side effects with graph convolutional networks","volume":"34","author":"Zitnik","year":"2018","journal-title":"Bioinf (Oxford England)"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/bty114","article-title":"A global network of biomedical relationships derived from text","volume":"34","author":"Percha","year":"2018","journal-title":"Bioinf (Oxford England)"},{"key":"B19","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1353\/pbm.1986.0087","article-title":"Fish oil, Raynaud's syndrome, and undiscovered public knowledge","volume":"30","author":"Swanson","year":"1986","journal-title":"Perspect Biol Med"},{"key":"B20","article-title":"Translating embeddings for modeling multi-relational data","author":"Bordes","year":"2013"},{"key":"B21","article-title":"RotatE: Knowledge graph embedding by relational rotation in complex space","author":"Sun","year":"2019"},{"key":"B22","article-title":"Complex embeddings for simple link prediction","author":"Trouillon","year":""},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1109\/tkde.2017.2754499","article-title":"Knowledge graph embedding: A survey of approaches and applications","volume":"29","author":"Wang","year":"2017","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3424672","article-title":"Knowledge graph embedding for link prediction: A comparative analysis","volume":"15","author":"Rossi","year":"2021","journal-title":"ACM Trans Knowl Discov Data"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1126\/science.286.5439.509","article-title":"Emergence of scaling in random networks","volume":"286","author":"Barabasi","year":"1999","journal-title":"Sci (New York NY)"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.1080\/07391102.2019.1676825","article-title":"Identification of potential leukocyte antigen-related protein (PTP-LAR) inhibitors through 3D QSAR pharmacophore-based virtual screening and molecular dynamics simulation","volume":"38","author":"Du","year":"2020","journal-title":"J Biomol Struct Dyn"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.3892\/br.2018.1051","article-title":"Computer-aided design of temozolomide derivatives based on alkylglycerone phosphate synthase structure with isothiocyanate and their pharmacokinetic\/toxicity prediction and anti-tumor activity","volume":"8","author":"Yang","year":"2018","journal-title":"Biomed Rep"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1016\/j.imbio.2022.152211","article-title":"Embryonic stem cells-derived exosomes enhance retrodifferentiation of retinal Muller cells by delivering BDNF protein to activate Wnt pathway","volume":"227","author":"Gao","year":"2022","journal-title":"Immunobiology"},{"key":"B29","doi-asserted-by":"publisher","DOI":"10.3390\/ijms15022157","article-title":"shRNA-mediated XRCC2 gene knockdown efficiently sensitizes colon tumor cells to X-ray irradiation in vitro and in vivo","volume":"15","author":"Wang","year":"2014","journal-title":"Int J Mol Sci"},{"key":"B30","article-title":"Language models are few-shot learners","author":"Brown","year":"2020"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.08774","article-title":"GPT-4 technical report","year":"2023","journal-title":"arXiv preprint"},{"key":"B32","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"B33","first-page":"1998","article-title":"Large language models are few-shot clinical information extractors","author":"Agrawal","year":"2022"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.10205","article-title":"ChatIE: Zero-shot information extraction via chatting with ChatGPT","author":"Wei","year":"2023","journal-title":"arXiv preprint"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.1109\/tkde.2024.3352100","article-title":"Unifying large language models and knowledge graphs: A roadmap","volume":"36","author":"Pan","year":"2024","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: A pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinf (Oxford England)"},{"key":"B37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2021","journal-title":"ACM Trans Comput Healthcare"},{"key":"B38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","author":"Ji","year":"2023","journal-title":"ACM Comput Surv"},{"key":"B39","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2412.19437","article-title":"DeepSeek-V3 technical report","year":"2024","journal-title":"arXiv preprint"},{"key":"B40","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.16609","article-title":"Qwen technical report","author":"Bai","year":"2023","journal-title":"arXiv preprint"},{"key":"B41","article-title":"GLM-130B: An open bilingual pre-trained model","author":"Zeng","year":"2023"},{"key":"B42","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ psychol Measurement"},{"key":"B43","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(90)90158-l","article-title":"High agreement but low kappa: I. The problems of two paradoxes","volume":"43","author":"Feinstein","year":"1990","journal-title":"J Clin Epidemiol"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(93)90018-v","article-title":"Bias, prevalence and kappa","volume":"46","author":"Byrt","year":"1993","journal-title":"J Clin Epidemiol"},{"key":"B45","doi-asserted-by":"crossref","DOI":"10.52202\/075280-2020","article-title":"Judging LLM-as-a-judge with MT-bench and chatbot arena","author":"Zheng","year":"2023"},{"key":"B46","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkac1000","article-title":"The STRING database in 2023: Protein-protein association networks and functional enrichment analyses for any sequenced genome of interest","volume":"51","author":"Szklarczyk","year":"2023","journal-title":"Nucleic Acids Res"},{"key":"B47","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1093\/nar\/28.1.27","article-title":"KEGG: Kyoto encyclopedia of genes and genomes","volume":"28","author":"Kanehisa","year":"2000","journal-title":"Nucleic Acids Res"},{"key":"B48","doi-asserted-by":"publisher","DOI":"10.1056\/nejmsa1012065","article-title":"The ClinicalTrials.gov results database--update and key issues","volume":"364","author":"Zarin","year":"2011","journal-title":"N Engl J Med"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.1038\/533452a","article-title":"1,500 scientists lift the lid on reproducibility","volume":"533","author":"Baker","year":"2016","journal-title":"Nature"},{"key":"B50","first-page":"1","article-title":"Ensemble methods in machine learning","author":"Dietterich","year":"2000"},{"key":"B51","article-title":"Self-consistency improves chain-of-thought reasoning in language models","author":"Wang","year":"2023"},{"key":"B52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jbi.2013.12.006","article-title":"NCBI disease corpus: a resource for disease name recognition and concept normalization","volume":"47","author":"Dogan","year":"2014","journal-title":"J BioMed Inf"},{"key":"B53","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkaa1084","article-title":"Integration of the Drug-Gene Interaction Database (DGIdb 4.0) with open crowdsource efforts","volume":"49","author":"Freshour","year":"2021","journal-title":"Nucleic Acids Res"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.1038\/ng.2764","article-title":"The Cancer Genome Atlas Pan-Cancer analysis project","volume":"45","author":"Weinstein","year":"2013","journal-title":"Nat Genet"}],"container-title":["Frontiers in Immunology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fimmu.2026.1937268\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:38:25Z","timestamp":1787031505000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fimmu.2026.1937268\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,18]]},"references-count":54,"alternative-id":["10.3389\/fimmu.2026.1937268"],"URL":"https:\/\/doi.org\/10.3389\/fimmu.2026.1937268","relation":{},"ISSN":["1664-3224"],"issn-type":[{"value":"1664-3224","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,18]]},"article-number":"1937268"}}