{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,18]],"date-time":"2026-01-18T07:29:17Z","timestamp":1768721357015,"version":"3.49.0"},"reference-count":11,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"4","content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:sec>\n            <jats:title>Context:<\/jats:title>\n            <jats:p>Public health practice involves the development of reports and plans, including funding progress reports, strategic plans, and community needs assessments. These documents are valuable data sources for program monitoring and evaluation. However, practitioners rarely have the bandwidth to thoroughly and rapidly review large amounts of primarily qualitative data to support real-time and continuous program improvement. Systematically examining and categorizing qualitative data through content analysis is labor-intensive. Large language models (LLMs), a type of generative artificial intelligence (AI) focused on language-based tasks, hold promise for expediting content analysis of public health documents, which, in turn, could facilitate continuous program improvement.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Objectives:<\/jats:title>\n            <jats:p>To explore the feasibility and potential of using LLMs to expedite content analysis of real-world public health documents. The focus was on comparing semiautomated outputs from GPT-4o with human outputs for abstracting and synthesizing information from health improvement plans.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Design:<\/jats:title>\n            <jats:p>Our study team conducted a content analysis of 4 publicly available community health improvement plans and compared the results with GPT-4o\u2019s performance on 20 data elements. We also assessed the resources required for both methods, including time spent on prompt engineering and error correction.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Main outcome measure(s):<\/jats:title>\n            <jats:p>Accuracy of data abstraction and time required.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results:<\/jats:title>\n            <jats:p>GPT-4o demonstrated abstraction accuracy of 79% (n\u00a0=\u00a017 errors) compared to 94% accuracy by the study team for individual plans, with 8 instances of falsified data. Out of the 18 synthesis data elements, GPT-4o made 9 errors, demonstrating an accuracy of 50%. On average, GPT-4o abstraction required fewer hours than study team abstraction, but resource savings diminished when accounting for time for developing prompts and identifying\/correcting errors.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusions:<\/jats:title>\n            <jats:p>Public health professionals who explore the use of generative AI tools should approach the method with cautious curiosity and consider the potential tradeoffs between resource savings and data accuracy.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1097\/phh.0000000000002148","type":"journal-article","created":{"date-parts":[[2025,2,26]],"date-time":"2025-02-26T11:01:56Z","timestamp":1740567716000},"page":"527-536","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":1,"title":["Content Analysis of Social Determinants of Health Accelerator Plans Using Artificial Intelligence: A Use Case for Public Health Practitioners"],"prefix":"10.1097","volume":"31","author":[{"given":"Kelli","family":"DePriest","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Feher III","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kailen","family":"Gore","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"LaShawn","family":"Glasgow","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Clint","family":"Grant","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter","family":"Holtgrave","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karen","family":"Hacker","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert","family":"Chew","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"276","published-online":{"date-parts":[[2025,2,26]]},"reference":[{"issue":"1","key":"R3-20250717","doi-asserted-by":"crossref","first-page":"E291","DOI":"10.1097\/PHH.0000000000001279","article-title":"Community health improvement plans: an analysis of approaches used by local health departments","volume":"28","author":"Carroll","year":"2022","journal-title":"J Public Health Manag Pract"},{"issue":"2","key":"R4-20250717","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1007\/s11764-019-00832-3","article-title":"Survivorship objectives in comprehensive cancer control plans: a systematic review","volume":"14","author":"Mollica","year":"2020","journal-title":"J Cancer Surviv"},{"issue":"Suppl 6","key":"R5-20250717","doi-asserted-by":"crossref","first-page":"S242","DOI":"10.1097\/PHH.0000000000001254","article-title":"Promoting health and well-being in healthy people 2030","volume":"27","author":"Pronk","year":"2021","journal-title":"J Public Health Manag Pract"},{"issue":"5","key":"R6-20250717","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1097\/NNR.0000000000000364","article-title":"Addressing health disparities in America: analysis of community health improvement plans","volume":"68","author":"Inderstrodt","year":"2019","journal-title":"Nurs Res"},{"issue":"4","key":"R7-20250717","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1177\/0033354918778034","article-title":"CDC\u2019s program evaluation journey: 1999 to present","volume":"133","author":"Kidder","year":"2018","journal-title":"Public Health Rep"},{"issue":"4","key":"R8-20250717","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1002\/jrsm.1710","article-title":"Data extraction for evidence synthesis using a large language model: a proof-of-concept study","volume":"15","author":"Gartlehner","year":"2024","journal-title":"Res Synth Methods"},{"issue":"6","key":"R9-20250717","doi-asserted-by":"crossref","first-page":"776","DOI":"10.1377\/hlthaff.2024.00050","article-title":"Transforming public health practice with generative artificial intelligence","volume":"43","author":"Bharel","year":"2024","journal-title":"Health Aff (Millwood)"},{"key":"R10-20250717","doi-asserted-by":"crossref","first-page":"1196397","DOI":"10.3389\/fpubh.2023.1196397","article-title":"Using artificial intelligence to improve public health: a narrative review","volume":"11","author":"Olawade","year":"2023","journal-title":"Front Public Health"},{"issue":"6","key":"R12-20250717","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1097\/PHH.0000000000001626","article-title":"Social determinants of health: an approach taken at CDC","volume":"28","author":"Hacker","year":"2022","journal-title":"J Public Health Manag Pract"},{"key":"R18-20250717","article-title":"LLM-assisted content analysis: using large language models to support deductive coding","author":"Chew","year":"2023","journal-title":"Comput Lang"},{"key":"R20-20250717","first-page":"157","article-title":"Lost in the middle: how language models use long contexts","volume":"2","author":"Liu","year":"2023","journal-title":"Comput Lang"}],"container-title":["Journal of Public Health Management &amp; Practice"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.lww.com\/10.1097\/PHH.0000000000002148","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,17]],"date-time":"2025-07-17T14:00:40Z","timestamp":1752760840000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.lww.com\/10.1097\/PHH.0000000000002148"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,26]]},"references-count":11,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025]]}},"URL":"https:\/\/doi.org\/10.1097\/phh.0000000000002148","relation":{},"ISSN":["1078-4659","1550-5022"],"issn-type":[{"value":"1078-4659","type":"print"},{"value":"1550-5022","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,26]]}}}