{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T12:36:16Z","timestamp":1793104576969,"version":"4.5.0"},"reference-count":33,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"9","license":[{"start":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T00:00:00Z","timestamp":1788220800000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0"}],"content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,9]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background:<\/jats:title>\n                    <jats:p>Scientific poster assessment lacks standardized and discipline-neutral rubrics. Assessment by human reviewers (HRs) is subject to inter-rater variability.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Aim:<\/jats:title>\n                    <jats:p>To assess PA\u00b2IRS (Poster Assessment via AI-Integrated Rubric System) framework for AI-assisted psychiatric poster evaluation, and conducted a reliability study comparing AI and HR agreement.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods:<\/jats:title>\n                    <jats:p>PA\u00b2IRS was developed through an AI-assisted iterative criterion refinement process modelled on Delphi principles. Sixty posters (30-case reports\/series [CR], 20 original research [OR], 10-systematic review-Meta-analysis [SRMA]) were randomly sampled. Three qualified mental health professionals served as independent reviewers. A custom-GPT (GPT-5.2, GO-subscription) provided AI assessments across three domains: Domain-A (content quality, poster-type specific), Domain-B (visual), and Domain-C (impact). PA\u00b2IRS is a 100-point instrument combining an AI-assessable poster component and an in-person interview component. This study concerns only the poster component. Intraclass correlation coefficients (ICCs), Passing\u2013Bablok regression, Bland\u2013Altman analysis, and variance component analysis were performed using appropriate statistical tools.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>\n                      AI\u2013human single-measure ICCs [Overall (0.62), Domain-A (0.63), Domain-B (0.44), Domain-C (0.55)] met or exceeded human-human ICCs (0.42, 0.40, 0.27, 0.48) across all domains. Four-rater ICC (with AI) reached 0.75. Variance ratios (AI\u2013human vs inter-human spread) were \u22641.0 across all domains for all posters combined. The SRMA subgroup showed variance ratios of 0.09\u20130.13 for Domains A and B (Bartlett\n                      <jats:italic toggle=\"yes\">P<\/jats:italic>\n                      \u2264 0.002). Overall score bias was 0.11 percentage points (pp); Domain-A showed a consistent maximum positive bias of 5.5 pp across subgroups.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion:<\/jats:title>\n                    <jats:p>AI\u2013human agreement was within or exceeded the inter-human reliability range across three domains. The domain-dependent agreement pattern is consistent with dual-process cognitive theory. PA\u00b2IRS supports use as a scalable, standardized, and cross-disciplinarily competent formative biomedical poster assessment tool.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.4103\/indianjpsychiatry_426_26","type":"journal-article","created":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T08:01:03Z","timestamp":1789632063000},"page":"805-814","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating psychiatric conference posters: Benchmarking a custom generative pre-trained transformer against human inter-rater variability using a structured assessment framework"],"prefix":"10.4103","volume":"68","author":[{"given":"Anusa Arunachalam","family":"Mohandoss","sequence":"first","affiliation":[{"name":"Department of Psychiatry, Shri Sathya Sai Medical College and Research Institute, Affiliated to Sri Balaji Vidyapeeth, Deemed to be University, SBV Nagar, Chennai Campus, Ammapettai, Chengalpet Taluk, Kancheepuram, Nellikuppam, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rooban","family":"Thavarajah","sequence":"additional","affiliation":[{"name":"Department of Oral and Maxillofacial Pathology, Marundeeswara Oral Pathology Services and Analytics, Shollinganallur, Chennai, Tamil Nadu, India"},{"name":"Department of Oral and Maxillofacial Pathology, Ragas Dental College and Hospital, Chennai, Affiliated to the Tamil Nadu Dr. MGR Medical University, Uthandi, Chennai, Tamil Nadu, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Raj Kiran","family":"Donthu","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, NRI Academy of Medical Sciences, Mangalagiri, Guntur, Andhra Pradesh, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Swapnajeet","family":"Sahoo","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, PGIMER, Chandigarh, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sai Krishna","family":"Tikka","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, All India Institute of Medical Sciences, Bibinagar, Hyderabad, Telangana, India"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shahul","family":"Ameen","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, St. Thomas Hospital, Kurisummoodu PO, Changanacherry, Kerala, India"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2026,9,17]]},"reference":[{"key":"R1-20261027","doi-asserted-by":"crossref","first-page":"453","DOI":"10.4103\/indianjpsychiatry.indianjpsychiatry_383_25","article-title":"Streamlining poster sessions: Tips for organisers","volume":"67","author":"Ameen","year":"2025","journal-title":"Indian J Psychiatry"},{"key":"R2-20261027","doi-asserted-by":"crossref","first-page":"101372","DOI":"10.1016\/j.ajpe.2025.101372","article-title":"Comparing analytic and mixed-approach rubrics for academic poster quality","volume":"89","author":"Peeters","year":"2025","journal-title":"Am J Pharm Educ"},{"key":"R3-20261027","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1007\/s13187-025-02622-1","article-title":"What makes a good poster? Evaluating# BetterPoster and classic formats at a scientific cancer conference","volume":"41","author":"Bentsen","year":"2026","journal-title":"J Cancer Educ"},{"key":"R4-20261027","doi-asserted-by":"crossref","first-page":"360","DOI":"10.1111\/tct.12584","article-title":"Academic poster design at a national conference: A need for standardized guidance?","volume":"14","author":"Gopal","year":"2017","journal-title":"Clin Teach"},{"key":"R5-20261027","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1111\/hir.12015","article-title":"What is the evidence that poster presentations are effective in promoting knowledge transfer? A state-of-the-art review","volume":"30","author":"Ilic","year":"2013","journal-title":"Health Info Libr J"},{"key":"R6-20261027","first-page":"100423","article-title":"Visual impression scoring of academic posters: Initial validity evidence using a mixed-approach rubric","volume":"13","author":"Khadka","year":"2024","journal-title":"Explor Res Clin Soc Pharm"},{"key":"R8-20261027","doi-asserted-by":"crossref","first-page":"207","DOI":"10.4103\/indianjpsychiatry_1181_25","article-title":"Paper polish: Development of a GPT to fine-tune scientific manuscripts","volume":"68","author":"Shahul","year":"2026","journal-title":"Indian J Psychiatry"},{"key":"R9-20261027","article-title":"Transforming science with large language models: A survey on AI-assisted scientific discovery, experimentation, content generation, and evaluation","author":"Eger","year":"2025","journal-title":"arXiv [Preprint]"},{"key":"R10-20261027","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1038\/s44277-024-00010-z","article-title":"Opportunities and risks of large language models in psychiatry","volume":"2","author":"Obradovich","year":"2024","journal-title":"NPP Digit Psychiatry Neurosci"},{"key":"R11-20261027","doi-asserted-by":"crossref","first-page":"1422807","DOI":"10.3389\/fpsyt.2024.1422807","article-title":"Applications of large language models in psychiatry: A systematic review","volume":"15","author":"Omar","year":"2024","journal-title":"Front Psychiatry"},{"key":"R12-20261027","doi-asserted-by":"crossref","first-page":"101167","DOI":"10.1016\/j.newideapsych.2025.101167","article-title":"Large language models (LLMs) as research Subjects: Status, opportunities and challenges","volume":"79","author":"Zhao","year":"2025","journal-title":"New Ideas in Psychology"},{"key":"R13-20261027","doi-asserted-by":"crossref","first-page":"102086","DOI":"10.1016\/j.ijlp.2025.102086","article-title":"Large language models and psychiatry","volume":"101","author":"Orr\u00f9","year":"2025","journal-title":"Int J Law Psychiatry"},{"key":"R14-20261027","doi-asserted-by":"crossref","first-page":"754","DOI":"10.1002\/jts.23082","article-title":"An overview of diagnostics and therapeutics using large language models","volume":"37","author":"Malgaroli","year":"2024","journal-title":"J Trauma Stress"},{"key":"R15-20261027","doi-asserted-by":"crossref","first-page":"1022","DOI":"10.1097\/ACM.0b013e3181ace703","article-title":"A universal model of diagnostic reasoning","volume":"84","author":"Croskerry","year":"2009","journal-title":"Acad Med"},{"issue":"(Suppl 1)","key":"R16-20261027","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/s10459-009-9182-2","article-title":"Clinical cognition and diagnostic error: Applications of a dual process model of reasoning","volume":"14","author":"Croskerry","year":"2009","journal-title":"Adv Health Sci Educ Theory Pract"},{"key":"R17-20261027","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1038\/nrn2787","article-title":"The free-energy principle: A unified brain theory?","volume":"11","author":"Friston","year":"2010","journal-title":"Nat Rev Neurosci"},{"key":"R18-20261027","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1016\/0010-0285(73)90004-2","article-title":"Perception in chess","volume":"4","author":"Chase","year":"1973","journal-title":"Cogn Psychol"},{"key":"R19-20261027","doi-asserted-by":"crossref","first-page":"e2218523120","DOI":"10.1073\/pnas.2218523120","article-title":"Using cognitive psychology to understand GPT-3","volume":"120","author":"Binz","year":"2023","journal-title":"Proc Natl Acad Sci U S A"},{"key":"R21-20261027","doi-asserted-by":"crossref","first-page":"100481","DOI":"10.1016\/j.caeai.2025.100481","article-title":"Evaluating large language models as raters in large-scale writing assessments","volume":"9","author":"Wang","year":"2025","journal-title":"Comput Educ Artif Intell"},{"key":"R22-20261027","article-title":"Assessing the quality of mental health support in LLM responses through multi-attribute human evaluation","author":"Badawi","journal-title":"arXiv [preprint] arXiv:2601.18630"},{"key":"R23-20261027","doi-asserted-by":"crossref","first-page":"1772","DOI":"10.1080\/0142159X.2025.2504106","article-title":"Using large language models (LLMs) to apply analytic rubrics to score post-encounter notes","volume":"47","author":"Runyon","year":"2025","journal-title":"Med Teach"},{"key":"R24-20261027","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/B978-0-444-53702-7.00007-5","article-title":"Effects of sleep deprivation on cognition","volume":"185","author":"Killgore","year":"2010","journal-title":"Prog Brain Res"},{"key":"R25-20261027","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1111\/j.1745-6916.2008.00067.x","article-title":"Affect and cognition","volume":"3","author":"Forgas","year":"2008","journal-title":"Perspect Psychol Sci"},{"key":"R26-20261027","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/(SICI)1099-0771(200001\/03)13:1<1::AID-BDM333>3.0.CO;2-S","article-title":"The affect heuristic in judgments of risks and benefits","volume":"13","author":"Finucane","year":"2000","journal-title":"J Behav Decis Mak"},{"key":"R27-20261027","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1186\/1472-6920-9-71","article-title":"What impact do posters have on academic knowledge transfer?","volume":"9","author":"Rowe","year":"2009","journal-title":"BMC Med Educ"},{"key":"R29-20261027","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"R30-20261027","doi-asserted-by":"crossref","first-page":"27730","DOI":"10.52202\/068431-2011","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"R31-20261027","doi-asserted-by":"crossref","first-page":"933","DOI":"10.1162\/tacl_a_00681","article-title":"State of what art? A call for multi-prompt LLM evaluation","volume":"12","author":"Mizrahi","year":"2024","journal-title":"Trans Assoc Comput Linguist"},{"key":"R32-20261027","doi-asserted-by":"crossref","first-page":"e2305016120","DOI":"10.1073\/pnas.2305016120","article-title":"ChatGPT outperforms crowd workers for text-annotation tasks","volume":"120","author":"Gilardi","year":"2023","journal-title":"Proc Natl Acad Sci U S A"},{"key":"R33-20261027","article-title":"Is a prompt and a few samples all you need?","author":"M\u00f8ller","year":"2023","journal-title":"arXiv [Preprint]"},{"key":"R34-20261027","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1037\/0893-3200.21.4.726","article-title":"Rethinking how family researchers model infrequent outcomes","volume":"21","author":"Atkins","year":"2007","journal-title":"J Fam Psychol"},{"key":"R37-20261027","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1162\/coli_a_00502","article-title":"Can large language models transform computational social science?","volume":"50","author":"Ziems","year":"2024","journal-title":"Comput Linguist"},{"key":"R38-20261027","article-title":"Automated annotation with generative AI requires validation","author":"Pangakis","year":"2023","journal-title":"arXiv [Preprint]"}],"container-title":["Indian Journal of Psychiatry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.ovid.com\/10.4103\/indianjpsychiatry_426_26","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T12:01:49Z","timestamp":1793102509000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ovid.com\/10.4103\/indianjpsychiatry_426_26"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9]]},"references-count":33,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2026]]}},"URL":"https:\/\/doi.org\/10.4103\/indianjpsychiatry_426_26","relation":{},"ISSN":["0019-5545","1998-3794"],"issn-type":[{"value":"0019-5545","type":"print"},{"value":"1998-3794","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9]]},"assertion":[{"value":"2026-04-15","name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-05","name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}}]}}