{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,13]],"date-time":"2026-09-13T15:04:12Z","timestamp":1789311852688,"version":"build-2803163510"},"reference-count":32,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T00:00:00Z","timestamp":1788998400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"Northwell Health Department of Medicine\/Medicine Service Line"},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["R01 AG072911"],"award-info":[{"award-number":["R01 AG072911"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["UG3\/UH3 AG060626"],"award-info":[{"award-number":["UG3\/UH3 AG060626"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Am J Hosp Palliat Care"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Evidence that advance care planning (ACP) improves goal-concordant care (GCC) remains limited. Challenges are lack of reproducible operational definition of GCC and labor-intensive nature of manual chart review.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Develop a structured approach to adjudicating GCC and compare large language model (LLM) and human assessments of concordance between goals of care (GOC) and end-of-life treatment.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Natural language processing-assisted chart review created longitudinal summaries of GOC and end-of-life treatment for decedents enrolled in two pragmatic ACP trials. First, human and LLM developed a shared adjudication framework. Second, two human adjudicators and two LLMs independently assigned GCC scores and rationales for 120 purposively selected cases. Pairwise agreement quantified with Gwet\u2019s AC2 after categorizing scores as low, intermediate, or high concordance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In phase 1, two LLMs produced near-identical scores across 11 complex cases, with human ratings within 1-2 points of the LLMs in 8 cases. In Phase 2, pairwise Gwet\u2019s AC2 coefficients ranged 0.43-0.75; agreement was highest for GPT-4o vs Human 1 (0.75) and GPT-4o vs Claude 3.5 Sonnet (0.71). Human 1 tended to assign higher and Human 2 lower scores relative to overall mean. Disagreement was greatest with changing goals, surrogate conflict, delayed code-status updates, or aggressive treatment despite documented treatment limitations.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>This proof-of-concept study demonstrates that LLMs generated GCC ratings and rationales broadly aligned with human assessments while revealing inherent subjectivity of adjudicating longitudinal goal-concordance. LLM-assisted adjudication may support scalable and standardized evaluation of documented GCC in pragmatic trials, with human review remaining essential for complex\/ambiguous cases.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/10499091261487682","type":"journal-article","created":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T12:56:07Z","timestamp":1789044967000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Models (LLMs) vs Manual Chart Review for Assessing Goal-Concordant Care: An Exploratory Study"],"prefix":"10.1177","author":[{"given":"Charlotta","family":"Lindvall","sequence":"first","affiliation":[{"name":"Dana-Farber Cancer Institute"},{"name":"Harvard Medical School"},{"name":"Brigham and Women\u2019s Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Liebowitz","sequence":"additional","affiliation":[{"name":"Brigham and Women\u2019s Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kaitlin","family":"Emmert-Tangredi","sequence":"additional","affiliation":[{"name":"Zucker School of Medicine at Hofstra\/Northwell"},{"name":"Feinstein Institutes for Medical Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Narda J.","family":"Martin","sequence":"additional","affiliation":[{"name":"Zucker School of Medicine at Hofstra\/Northwell"},{"name":"Feinstein Institutes for Medical Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3092-0265","authenticated-orcid":false,"given":"Shelby","family":"Brage","sequence":"additional","affiliation":[{"name":"Zucker School of Medicine at Hofstra\/Northwell"},{"name":"Feinstein Institutes for Medical Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael K.","family":"Paasche-Orlow","sequence":"additional","affiliation":[{"name":"Tufts University School of Medicine"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1093-3755","authenticated-orcid":false,"given":"Yuchiao","family":"Chang","sequence":"additional","affiliation":[{"name":"Massachusetts General Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua R.","family":"Lakin","sequence":"additional","affiliation":[{"name":"Dana-Farber Cancer Institute"},{"name":"Harvard Medical School"},{"name":"Brigham and Women\u2019s Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Itty","sequence":"additional","affiliation":[{"name":"Feinstein Institutes for Medical Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shreya","family":"Sanghani","sequence":"additional","affiliation":[{"name":"Feinstein Institutes for Medical Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Megan","family":"Carroll","sequence":"additional","affiliation":[{"name":"Zucker School of Medicine at Hofstra\/Northwell"},{"name":"Feinstein Institutes for Medical Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Trey","family":"Feldeisen","sequence":"additional","affiliation":[{"name":"Feinstein Institutes for Medical Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seth","family":"Randa","sequence":"additional","affiliation":[{"name":"University of California San Francisco"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aretha Delight","family":"Davis","sequence":"additional","affiliation":[{"name":"ACP Decisions"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James A.","family":"Tulsky","sequence":"additional","affiliation":[{"name":"Dana-Farber Cancer Institute"},{"name":"Harvard Medical School"},{"name":"Brigham and Women\u2019s Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angelo E.","family":"Volandes","sequence":"additional","affiliation":[{"name":"Massachusetts General Hospital"},{"name":"ACP Decisions"},{"name":"Geisel School of Medicine"},{"name":"Dartmouth Health"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8770-6377","authenticated-orcid":false,"given":"Edith A.","family":"Burns","sequence":"additional","affiliation":[{"name":"Zucker School of Medicine at Hofstra\/Northwell"},{"name":"Feinstein Institutes for Medical Research"},{"name":"Northwell Institute for Healthy Aging"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,9,10]]},"reference":[{"key":"e_1_3_6_2_2","doi-asserted-by":"publisher","DOI":"10.1136\/spcare-2025-005827"},{"key":"e_1_3_6_3_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2021-060201"},{"key":"e_1_3_6_4_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2021.16430"},{"key":"e_1_3_6_5_2","doi-asserted-by":"publisher","DOI":"10.1111\/jgs.70014"},{"key":"e_1_3_6_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2016.12.331"},{"key":"e_1_3_6_7_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2018.2317"},{"key":"e_1_3_6_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2022.06.004"},{"key":"e_1_3_6_9_2","doi-asserted-by":"publisher","DOI":"10.1111\/jgs.18000"},{"key":"e_1_3_6_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/s1470-2045(17)30582-x"},{"key":"e_1_3_6_11_2","doi-asserted-by":"publisher","DOI":"10.1177\/02692163251344428"},{"key":"e_1_3_6_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2021.02.030"},{"key":"e_1_3_6_13_2","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2017.0459"},{"key":"e_1_3_6_14_2","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMp1908153"},{"key":"e_1_3_6_15_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.41264"},{"key":"e_1_3_6_16_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2014.1183"},{"key":"e_1_3_6_17_2","doi-asserted-by":"publisher","DOI":"10.1097\/TA.0000000000003909"},{"key":"e_1_3_6_18_2","doi-asserted-by":"publisher","DOI":"10.1002\/cncr.35976"},{"key":"e_1_3_6_19_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.30370"},{"key":"e_1_3_6_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2021.06.025"},{"key":"e_1_3_6_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2020.08.024"},{"key":"e_1_3_6_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2024.11.016"},{"key":"e_1_3_6_23_2","doi-asserted-by":"publisher","DOI":"10.1200\/OP-24-00992"},{"key":"e_1_3_6_24_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.32556"},{"key":"e_1_3_6_25_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.9150"},{"key":"e_1_3_6_26_2","unstructured":"ClinicalRegex. Published online. 2026. https:\/\/lindvalllab.dana-farber.org\/clinicalregex.html"},{"key":"e_1_3_6_27_2","unstructured":"Veradigm (formerly Allscripts Sunrise Clinical Manager) EHR."},{"key":"e_1_3_6_28_2","unstructured":"OpenAI. (2024). Gpt-4o (May 13 version) [Large language model used via Northwell AI Hub]. https:\/\/ai.northwell.edu"},{"key":"e_1_3_6_29_2","unstructured":"Google. (2024). Gemini 2.0 [Large language model used via Northwell AI Hub]. https:\/\/ai.northwell.edu"},{"key":"e_1_3_6_30_2","unstructured":"Anthropic. (2024). Claude Sonnet 3.5 (june 20 version) [large language model used via northwell AI Hub]. [URL of Northwell AI Hub if applicable]."},{"key":"e_1_3_6_31_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmjqs-2015-004411"},{"key":"e_1_3_6_32_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.18967"},{"key":"e_1_3_6_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-024-08773-z"}],"container-title":["American Journal of Hospice and Palliative Medicine\u00ae"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10499091261487682","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/10499091261487682","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/10499091261487682","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,13]],"date-time":"2026-09-13T14:20:14Z","timestamp":1789309214000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/10499091261487682"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,10]]},"references-count":32,"alternative-id":["10.1177\/10499091261487682"],"URL":"https:\/\/doi.org\/10.1177\/10499091261487682","relation":{},"ISSN":["1049-9091","1938-2715"],"issn-type":[{"value":"1049-9091","type":"print"},{"value":"1938-2715","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,10]]},"article-number":"10499091261487682"}}