{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,5]],"date-time":"2026-08-05T08:51:37Z","timestamp":1785919897150,"version":"3.56.0"},"reference-count":31,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T00:00:00Z","timestamp":1785801600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Behav. Sci.","Behavioral Sciences"],"abstract":"<jats:p>The present study examines whether large language model (LLM) embeddings can approximate the psychometric validation process of a newly developed psychological scale. Using the Academic Psychological Distress Scale (APDS) as a case study, we replicated the full validation pipeline, traditionally performed on human response data, by applying exploratory and confirmatory factor analyses (EFA and CFA) to semantic similarity matrices derived from LLM embeddings. We first computed cosine similarity among all 58 APDS items and found a moderately strong correspondence with the participants\u2019 correlation matrix (\u03c1 = 0.57). An EFA performed on the embedding-based similarity matrix yielded a six-factor solution explaining 79.1% of the variance. The best-performing model (32 items, loading cutoff = 0.70) demonstrated acceptable model fit when tested on participants\u2019 responses (CFI = 0.88; TLI = 0.87; RMSEA = 0.07\u20130.08; SRMR = 0.06) numerically comparable to the fit obtained in the original human-based validation based on 367 participants\u2019 response. Reliability estimates were satisfactory for most factors (\u03c9\/\u03b1 = 0.82\u20130.95), although one factor showed lower internal consistency. The percentage of overlap between human-derived and embedding-derived structures, indicated an overall structural correspondence of 64%, with factor-level overlap ranging from 50% to 100%. While embeddings reproduced most dimensions of academic distress, they failed to recover the demoralization factor and instead introduced an additional behavioral cluster. These findings suggest that LLM embeddings can capture substantial aspects of the latent structure of psychological constructs and can support early-stage, low-data, embedding-first scale development. We propose a linguistic validity pipeline as a methodological framework for integrating semantic embeddings into psychometric validation procedures, highlighting their promise as a complementary tool for human-centered assessment.<\/jats:p>","DOI":"10.3390\/bs16081338","type":"journal-article","created":{"date-parts":[[2026,8,4]],"date-time":"2026-08-04T08:02:19Z","timestamp":1785830539000},"page":"1338","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Deep Semantics Analysis for Scale Development: Replicating Psychometrics Validation Process with Large Language Models"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1604-5161","authenticated-orcid":false,"given":"Nicola","family":"Milano","sequence":"first","affiliation":[{"name":"Department of Humanistic Studies, University of Naples Federico II, Via Porta di Massa 1, 80125 Naples, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rosa","family":"Pizzo","sequence":"additional","affiliation":[{"name":"Department of Humanistic Studies, University of Naples Federico II, Via Porta di Massa 1, 80125 Naples, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cristiano","family":"Scandurra","sequence":"additional","affiliation":[{"name":"Department of Humanistic Studies, University of Naples Federico II, Via Porta di Massa 1, 80125 Naples, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria Francesca","family":"Freda","sequence":"additional","affiliation":[{"name":"Department of Humanistic Studies, University of Naples Federico II, Via Porta di Massa 1, 80125 Naples, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5185-1313","authenticated-orcid":false,"given":"Davide","family":"Marocco","sequence":"additional","affiliation":[{"name":"Department of Humanistic Studies, University of Naples Federico II, Via Porta di Massa 1, 80125 Naples, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,8,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1037\/pspp0000480","article-title":"A deep learning approach to personality assessment: Generalizing across items and expanding the reach of survey-based research","volume":"126","author":"Abdurahman","year":"2024","journal-title":"Journal of Personality and Social Psychology"},{"key":"ref_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1162\/coli_a_00412","article-title":"Decoding word embeddings with brain-based semantic features","volume":"47","author":"Chersoni","year":"2021","journal-title":"Computational Linguistics"},{"key":"ref_4","unstructured":"Comrey, A. L., and Lee, H. B. (1992). A first course in factor analysis, Lawrence Erlbaum Associates. [2nd ed.]."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). Supervised learning of universal sentence representations from natural language inference data. arXiv.","DOI":"10.18653\/v1\/D17-1070"},{"key":"ref_6","first-page":"1","article-title":"Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis","volume":"10","author":"Costello","year":"2005","journal-title":"Practical Assessment, Research, and Evaluation"},{"key":"ref_7","unstructured":"Burstein, J., Doran, C., and Solorio, T. (2019). BERT: Pretraining of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, vol. 1 (long and short papers), Association for Computational Linguistics."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Feraco, T., and Toffalini, E. (2025). SEMbeddings: How to evaluate model misfit before data collection using large-language models. Frontiers in Psychology, 15.","DOI":"10.3389\/fpsyg.2024.1433339"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Guenole, N., D\u2019Urso, E. D., Samo, A., Sun, T., and Haslbeck, J. (2025). Enhancing scale development: Pseudo factor analysis of language embedding similarity matrices, OSF.","DOI":"10.31234\/osf.io\/vf3se_v2"},{"key":"ref_10","unstructured":"Hang, C. N., Tan, C. W., and Chiu, D. M. (3, January June). LLM teams: Harnessing large language models as multi-agent teammates for joint problem-solving. Thirteenth ACM Conference on Learning@ Scale, Seoul, Republic of Korea."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"8276","DOI":"10.1038\/s41598-024-59118-z","article-title":"Predicting attitudes toward ambiguity using natural language processing on free descriptions for open-ended question measurements","volume":"14","author":"Hitsuwari","year":"2024","journal-title":"Scientific Reports"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/10705519909540118","article-title":"Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives","volume":"6","author":"Hu","year":"1999","journal-title":"Structural Equation Modeling"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"8214","DOI":"10.3758\/s13428-024-02455-8","article-title":"A tutorial on open-source large language models for behavioral science","volume":"56","author":"Hussain","year":"2024","journal-title":"Behavior Research Methods"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"3918","DOI":"10.1038\/s41598-022-07520-w","article-title":"Natural language analyzed with AI-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy","volume":"12","author":"Kjell","year":"2022","journal-title":"Scientific Reports"},{"key":"ref_15","unstructured":"Li, Z., Shi, Y., Liu, Z., Yang, F., Payani, A., Liu, N., and Du, M. (2024). Quantifying multilingual performance of large language models across languages. arXiv."},{"key":"ref_16","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv."},{"key":"ref_17","unstructured":"Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. Q. (2013). Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26, Springer."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"100168","DOI":"10.1016\/j.crbeha.2025.100168","article-title":"Semantic analysis of test items through large language model embeddings predicts a-priori factorial structure of personality tests","volume":"8","author":"Milano","year":"2025a","journal-title":"Current Research in Behavioral Sciences"},{"key":"ref_19","first-page":"00131644251355485","article-title":"Human expertise and large language model embeddings in the content validity assessment of personality tests","volume":"86","author":"Milano","year":"2025b","journal-title":"Educational and Psychological Measurement"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Nilsson, A. H., Schwartz, H. A., Rosenthal, R. N., McKay, J. R., Vu, H., Cho, Y. M., Mahwish, S., Ganesan, A. V., and Ungar, L. (2024). Language-based EMA assessments help understand problematic alcohol consumption. PLoS ONE, 19, (3).","DOI":"10.1371\/journal.pone.0298300"},{"key":"ref_21","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Reimers, N., and Gurevych, I. (2020). Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv.","DOI":"10.18653\/v1\/2020.emnlp-main.365"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1201","DOI":"10.1177\/21676968251357346","article-title":"Academic psychological distress scale: Development and psychometric validation of a multidimensional measure for university students","volume":"13","author":"Scandurra","year":"2025","journal-title":"Emerging Adulthood"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"32931","DOI":"10.1007\/s12144-024-06802-y","article-title":"A concept analysis on academic psychological distress: Implications for clinical practice","volume":"43","author":"Scandurra","year":"2024","journal-title":"Current Psychology"},{"key":"ref_25","first-page":"16857","article-title":"Mpnet: Masked and permuted pre-training for language understanding","volume":"33","author":"Song","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_26","unstructured":"Song, W., Choi, D., Park, Y., Han, J., Lee, E.-J., and Jo, Y. (2025). Human psychometric questionnaires mischaracterize LLM psychology: Evidence from generation behavior. arXiv."},{"key":"ref_27","unstructured":"Sultimov, R., Volkov, A., Mitrovic, M., and Maximov, Y. (, January May). LLM-guided multi-agent evacuation coordination via episodic memory and cognitive task analysis. 25th International Conference on Autonomous Agents and Multiagent Systems, Paphos, Cyprus."},{"key":"ref_28","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Wu, S., and Dredze, M. (2020). Are all languages created equal in multilingual BERT?. arXiv.","DOI":"10.18653\/v1\/2020.repl4nlp-1.16"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"944","DOI":"10.1038\/s41562-024-02089-y","article-title":"Semantic embeddings reveal and address taxonomic incommensurability in psychological measurement","volume":"9","author":"Wulff","year":"2025","journal-title":"Nature Human Behaviour"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhelezniak, V., Savkov, A., Shen, A., and Hammerla, N. Y. (2019). Correlation coefficients and semantic textual similarity. arXiv.","DOI":"10.18653\/v1\/N19-1100"}],"container-title":["Behavioral Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-328X\/16\/8\/1338\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,5]],"date-time":"2026-08-05T08:39:04Z","timestamp":1785919144000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-328X\/16\/8\/1338"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,4]]},"references-count":31,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,8]]}},"alternative-id":["bs16081338"],"URL":"https:\/\/doi.org\/10.3390\/bs16081338","relation":{},"ISSN":["2076-328X"],"issn-type":[{"value":"2076-328X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,4]]}}}