{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T21:44:02Z","timestamp":1792100642597,"version":"4.4.0"},"reference-count":47,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) are increasingly used to quantify psychological constructs from natural language, yet their outputs are typically evaluated only as predictions of an external label. When that label is itself imperfect, optimizing prediction conflates label agreement with construct measurement. No study has applied a multidomain measurement-oriented evaluation to LLM-derived psychological outputs.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We applied 5 psychometric evaluation domains to LLM-derived outputs to test whether measurement-oriented evaluation surfaces properties that prediction-oriented evaluation does not: extraction repeatability, convergent validity with a lexicon-based tool, ecological concurrent validity decomposed into within-person and between-person components, incremental value in a 4-arm comparison with that tool, and method-blinded expert evaluation with structured error attribution.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Thirteen linguistic features were extracted by GPT-5.2 (OpenAI) from 3556 person-days of daily voice narratives collected from 279 call center workers over 15 days via ecological momentary assessment. Features were evaluated across the 5 domains, with Linguistic Inquiry and Word Count, version 22, as the lexicon-based comparator. Multimodel robustness was assessed by re-extracting all features with Google Gemini 2.5 Flash.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Features showed high repeatability (mean intraclass correlation coefficient=0.987), construct-dependent convergent validity (Spearman \u03c1=0.08 to 0.79), and within-person tracking of same-day stress (negative emotion \u03b2=0.68). In a 4-arm comparison (baseline, baseline plus lexicon features, baseline plus LLM-derived features, and baseline plus both), LLM features increased explained variance in daily stress from 13.2% to 30.4%, versus 16.9% for the lexicon-based comparator. Among transcripts sampled for maximal method discordance, experts preferred LLM-derived profiles in 83.7% (261\/312) of blinded decisions. Error analysis showed that 53% (35\/66) of false-positive attributions reflected reference-label limitations, not model failures. Re-extraction with Google Gemini 2.5 Flash replicated key properties (person-level\n r<\/jats:italic>\n =0.94). Three phenomena were visible under measurement-oriented evaluation that prediction-oriented evaluation does not typically examine: divergence between within-person and between-person effects, reference-label limitations as a dominant error source, and moderation of feature meaning by individual differences.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Prediction metrics alone are insufficient for evaluating LLM-derived psychological measures. Measurement-oriented evaluation revealed properties, including level-dependent effects and asymmetric error attribution, that have direct implications for how LLM-based assessment studies interpret their results. For clinical use, the measure is a complement to existing screening instruments for longitudinal monitoring, not a stand-alone diagnostic.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/102768","type":"journal-article","created":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T20:47:29Z","timestamp":1792097249000},"page":"e102768-e102768","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating Large Language Model\u2013Derived Outputs as Psychological Measures: A Five-Domain Psychometric Framework With Application to Daily Stress"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-1504-0463","authenticated-orcid":false,"given":"Inyong","family":"Jeong","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0917-0789","authenticated-orcid":false,"given":"Soohyun","family":"Park","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0408-768X","authenticated-orcid":false,"given":"Min","family":"Jhon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-3455-1828","authenticated-orcid":false,"given":"Dae-Kwang","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8062-7696","authenticated-orcid":false,"given":"Jin-Hyun","family":"Park","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-3234-0101","authenticated-orcid":false,"given":"Yeongmin","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8812-4509","authenticated-orcid":false,"given":"MyeongGyun","family":"Jang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7409-6306","authenticated-orcid":false,"given":"Jae-Min","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9888-1090","authenticated-orcid":false,"given":"Ju-wan","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6482-3511","authenticated-orcid":false,"given":"Hwamin","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,15]]},"reference":[{"issue":"3","key":"R1","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1038\/s41562-025-02389-x","article-title":"Assessing personality using zero-shot generative AI scoring of brief open-ended text","volume":"10","author":"Wright","journal-title":"Nat Hum Behav"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-21289-8","article-title":"Rethinking psychometrics through LLMs: how item semantics shape measurement and prediction in psychological questionnaires","volume":"15","author":"Ravenda","journal-title":"Sci Rep"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/75347","article-title":"Psychometric evaluation of large language model embeddings for personality trait prediction","volume":"27","author":"Maharjan","journal-title":"J Med Internet Res"},{"issue":"5","key":"R4","doi-asserted-by":"publisher","first-page":"808","DOI":"10.1177\/17456916231214460","article-title":"AI psychometrics: assessing the psychological profiles of large language models through psychometric inventories","volume":"19","author":"Pellert","journal-title":"Perspect Psychol Sci"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","DOI":"10.2196\/83352","article-title":"Prediction of 12-week remission in patients with depressive disorder using reasoning-based large language models: model development and validation study","volume":"13","author":"Park","journal-title":"JMIR Ment Health"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2026.106577","article-title":"On-premises open-source large language models for privacy-preserving multimodal depression screening","volume":"220","author":"Kwon","journal-title":"Int J Med Inform"},{"issue":"9","key":"R7","doi-asserted-by":"publisher","first-page":"1426","DOI":"10.1017\/S0033291719000151","article-title":"Machine learning in mental health: a scoping review of methods and applications","volume":"49","author":"Shatte","journal-title":"Psychol Med"},{"issue":"4","key":"R8","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","journal-title":"JAMA"},{"issue":"4","key":"R9","doi-asserted-by":"crossref","first-page":"385","DOI":"10.2307\/2136404","volume":"24","author":"Cohen","journal-title":"J Health Soc Behav"},{"issue":"4","key":"R10","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/j.anr.2012.08.004","article-title":"Review of the psychometric evidence of the perceived stress scale","volume":"6","author":"Lee","journal-title":"Asian Nurs Res (Korean Soc Nurs Sci)"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1037\/a0038100","article-title":"Psychometric analysis of the Ten-Item Perceived Stress Scale","volume":"27","author":"Taylor","journal-title":"Psychol Assess"},{"issue":"4","key":"R12","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1037\/h0040957","article-title":"Construct validity in psychological tests","volume":"52","author":"Cronbach","journal-title":"Psychol Bull"},{"issue":"9","key":"R13","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1037\/0003-066X.50.9.741","article-title":"Validity of psychological assessment: validation of inferences from persons\u2019 responses and performances as scientific inquiry into score meaning","volume":"50","author":"Messick","journal-title":"Am Psychol"},{"key":"R14","unstructured":"Pennebaker JW Boyd RL Jordan K Blackburn K . The development and psychometric properties of LIWC2015. University of Texas at Austin. Sep 2015. [doi: 10.15781\/T29G6Z]"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Chung CK Pennebaker JW . What do we know when we LIWC a person? Text analysis as an assessment tool for traits, personal concerns and life stories. In: The SAGE Handbook of Personality and Individual Differences, Volume I: The Science of Personality and Individual Differences. SAGE Publication; 2018:341-360. [doi: 10.4135\/9781526470294.n16]","DOI":"10.4135\/9781526451163.n16"},{"key":"R16","unstructured":"Alaa A Hartvigsen T Golchini N et al. Medical large language model benchmarks should prioritize construct validity. arXiv. Preprint posted online on Mar 12, 2025. [doi: 10.48550\/arXiv.2503.10694]"},{"key":"R17","unstructured":"Li Y Huang Y Wang H et al. Evaluating large language models with psychometrics. arXiv. Preprint posted online on Jun 25, 2024. [doi: 10.48550\/arXiv.2406.17675]"},{"issue":"12","key":"R18","doi-asserted-by":"publisher","first-page":"1954","DOI":"10.1038\/s42256-025-01115-6","article-title":"A psychometric framework for evaluating and shaping personality traits in large language models","volume":"7","author":"Serapio-Garc\u00eda","journal-title":"Nat Mach Intell"},{"issue":"4","key":"R19","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1080\/13594320344000183","article-title":"What is typical for call centre jobs? Job characteristics, and service interactions in different call centres","volume":"12","author":"Zapf","journal-title":"Eur J Work Organ Psychol"},{"issue":"4","key":"R20","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1111\/1467-6486.00300","article-title":"Work relationships in telephone call centres: understanding emotional exhaustion and employee withdrawal","volume":"39","author":"Deery","journal-title":"J Manag Stud"},{"issue":"2","key":"R21","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1111\/nhs.12334","article-title":"Mental health status and its predictors among call center employees: a cross-sectional study","volume":"19","author":"Oh","journal-title":"Nurs Health Sci"},{"issue":"2","key":"R22","doi-asserted-by":"publisher","first-page":"95","DOI":"10.4103\/0019-557X.157508","article-title":"Stress, anxiety, and depression among call handlers employed in international call centers in the National Capital Region of Delhi","volume":"59","author":"Jeyapal","journal-title":"Indian J Public Health"},{"issue":"9","key":"R23","doi-asserted-by":"publisher","first-page":"1096","DOI":"10.30773\/pi.2026.0035","article-title":"Daily language as an objective indicator of depressive affective states: a two-week ecological momentary assessment study in emotional labor workers","volume":"23","author":"Kim","journal-title":"Psychiatry Investig"},{"issue":"9","key":"R24","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1046\/j.1525-1497.2001.016009606.x","article-title":"The PHQ-9: validity of a brief depression severity measure","volume":"16","author":"Kroenke","journal-title":"J Gen Intern Med"},{"issue":"2","key":"R25","first-page":"114","volume":"28","author":"Choi","journal-title":"J Korean Acad Fam Med"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1016\/j.jrp.2006.02.001","article-title":"Measuring personality in one minute or less: a 10-item short version of the Big Five Inventory in English and German","volume":"41","author":"Rammstedt","journal-title":"J Res Pers"},{"issue":"1","key":"R27","first-page":"15","volume":"17","author":"Kim","journal-title":"Korean J Biol Psychiatry"},{"issue":"2","key":"R28","first-page":"127","volume":"20","author":"Lee","journal-title":"Korean J Psychosom Med"},{"issue":"1","key":"R29","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","journal-title":"Nat Med"},{"key":"R30","unstructured":"Boyd RL Ashokkumar A Seraj S Pennebaker JW . The Development and Psychometric Properties of LIWC-22. University of Texas at Austin; 2022."},{"issue":"18","key":"R31","doi-asserted-by":"publisher","first-page":"17018","DOI":"10.1007\/s12144-023-04737-4","article-title":"Linguistic markers of processing the first months of the pandemic COVID-19: a psycholinguistic analysis of Italian university students\u2019 diaries","volume":"43","author":"Gandino","journal-title":"Curr Psychol"},{"issue":"4","key":"R32","doi-asserted-by":"publisher","first-page":"651","DOI":"10.5964\/ejop.v11i4.991","article-title":"Linguistic markers of processing trauma experience in women\u2019s written narratives during different breast cancer phases: implications for clinical interventions","volume":"11","author":"Martino","journal-title":"Eur J Psychol"},{"key":"R33","doi-asserted-by":"crossref","unstructured":"Troiano E Labat S Stranisci MA Damiano R Patti V Klinger R . Dealing with controversy: an emotion and coping strategy corpus based on role playing. In: Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics; 2024. [doi: 10.18653\/v1\/2024.findings-emnlp.89]","DOI":"10.18653\/v1\/2024.findings-emnlp.89"},{"issue":"9","key":"R34","doi-asserted-by":"publisher","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Hu","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R35","doi-asserted-by":"publisher","first-page":"2722","DOI":"10.1097\/JS9.0000000000002215","article-title":"AI-driven evidence synthesis: data extraction of randomized controlled trials with large language models","volume":"111","author":"Liu","journal-title":"Int J Surg"},{"key":"R36","doi-asserted-by":"crossref","unstructured":"Corso F Peppoloni V Mazzeo L et al. Combining clinician expertise with prompt engineering enhances small language models reliability for cancer entity recognition in electronic health records. medRxiv. Preprint posted online on Oct 21, 2025. [doi: 10.1101\/2025.10.16.25337917]","DOI":"10.1101\/2025.10.16.25337917"},{"issue":"4","key":"R37","doi-asserted-by":"publisher","first-page":"284","DOI":"10.1037\/1040-3590.6.4.284","article-title":"Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology","volume":"6","author":"Cicchetti","journal-title":"Psychol Assess"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1016\/j.paid.2016.06.069","article-title":"Effect size guidelines for individual differences researchers","volume":"102","author":"Gignac","journal-title":"Pers Individ Dif"},{"issue":"3","key":"R39","doi-asserted-by":"publisher","first-page":"485","DOI":"10.1002\/ejsp.2752","article-title":"Empirically derived guidelines for effect size interpretation in social psychology","volume":"51","author":"Lovakov","journal-title":"Euro J Social Psych"},{"issue":"3","key":"R40","doi-asserted-by":"publisher","first-page":"559","DOI":"10.1080\/10503307.2025.2494270","article-title":"How to interpret correlational process-outcome effect sizes in psychotherapy: a meta-analytic benchmark study","volume":"36","author":"G\u00f3mez Penedo","journal-title":"Psychother Res"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1093\/biomet\/73.1.13","article-title":"Longitudinal data analysis using generalized linear models","volume":"73","author":"Liang","journal-title":"Biometrika"},{"issue":"1","key":"R42","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1146\/annurev.psych.093008.100356","article-title":"The disaggregation of within-person and between-person effects in longitudinal models of change","volume":"62","author":"Curran","journal-title":"Annu Rev Psychol"},{"issue":"1","key":"R43","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","journal-title":"J R Stat Soc Ser B Methodol"},{"key":"R44","doi-asserted-by":"crossref","unstructured":"Ye H Xie Y Ren Y Fang H Zhang X Song G . Measuring human and AI values based on generative psychometrics with large language models. In: AAAI\u201925\/IAAI\u201925\/EAAI\u201925: Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence. AAAI Press; 2025:26400-26408. [doi: 10.1609\/aaai.v39i25.34839]","DOI":"10.1609\/aaai.v39i25.34839"},{"key":"R45","unstructured":"Sofroniew N Kauvar I Saunders W et al. Emotion concepts and their function in a large language model. arXiv. Preprint posted online on Apr 9, 2026. [doi: 10.48550\/arXiv.2604.07729]"},{"issue":"1","key":"R46","doi-asserted-by":"publisher","DOI":"10.1186\/s12955-025-02350-6","article-title":"Psychometric properties of the Chinese version of the SF-36v2 in patients with severe mental illnesses in China: factor structure, reliability, and validity","volume":"23","author":"Wang","journal-title":"Health Qual Life Outcomes"},{"key":"R47","unstructured":"Dataset. GitHub. URL: https:\/\/github.com\/5454dls\/pss-llm-psychometrics [Accessed 18-09-2026]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T20:47:31Z","timestamp":1792097251000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e102768"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,15]]},"references-count":47,"URL":"https:\/\/doi.org\/10.2196\/102768","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,15]]},"article-number":"v28i3e102768"}}