{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T05:02:37Z","timestamp":1789534957355,"version":"build-2803163510"},"reference-count":64,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2026,9,13]],"date-time":"2026-09-13T00:00:00Z","timestamp":1789257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Behav. Sci.","Behavioral Sciences"],"abstract":"<jats:p>Researchers have long known that scores from Likert-style self-report measures of psychological traits are affected by both item wording (negative and positive) and multiple sources of measurement error (specific-factor, transient, and random-response), but the magnitude of such effects is rarely separated and individually estimated within research studies. When these effects are not distinguished, as is the case when reporting conventional reliability estimates (e.g., alpha, omega, split-half, test\u2013retest), they are confounded with trait variance and\/or each other, typically resulting in overestimation of score accuracy. We demonstrate how to quantify and separate trait, wording, and multiple measurement error effects within extended bifactor models using data from a large sample of respondents (n = 1796) who completed the Physical Appearance subscale from the Self-Description Questionnaire-III on two occasions. Results revealed that models including item wording effects for both single and multiple occasion designs provided noticeably superior fits to the data in comparison to models excluding such effects, with item wording accounting for 8.5% to 8.8% of total score variance across designs. We further demonstrate how to extend analyses to the individual item level to identify items that are most and least affected by wording and measurement error effects and provide code in R to enable readers to apply these techniques to their own data.<\/jats:p>","DOI":"10.3390\/bs16091636","type":"journal-article","created":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T08:10:23Z","timestamp":1789373423000},"page":"1636","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Quantifying Effects of Item Wording and Multiple Sources of Measurement Error Using Extended Bifactor Models"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9415-251X","authenticated-orcid":false,"given":"Walter P.","family":"Vispoel","sequence":"first","affiliation":[{"name":"Department of Psychological and Quantitative Foundations, University of Iowa, Iowa City, IA 52242, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7642-6161","authenticated-orcid":false,"given":"Hyeryung","family":"Lee","sequence":"additional","affiliation":[{"name":"School of Educational Foundations, Leadership, and Aviation, Oklahoma State University, Stillwater, OK 74076, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5855-0772","authenticated-orcid":false,"given":"Tingting","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Psychological and Quantitative Foundations, University of Iowa, Iowa City, IA 52242, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,13]]},"reference":[{"key":"ref_1","unstructured":"Bachman, J. G., Kahn, R. L., Mednick, M. T., Davidson, T. N., and Johnston, L. D. (2002). Youth in transition, volume I: Blueprint for a longitudinal study of adolescent boys (Standard No. ICPSR 3505)."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"101126","DOI":"10.1016\/j.stueduc.2022.101126","article-title":"Item wording effects in self-report measures and reading achievement: Does removing careless respondents help?","volume":"72","author":"Bulut","year":"2022","journal-title":"Studies in Educational Evaluation"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Byrne, B. M. (1996). Measuring self-concept across the life span: Methodological issues and selected instrumentation, American Psychological Association.","DOI":"10.1037\/10197-000"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1056","DOI":"10.1177\/0013164417727036","article-title":"On misconceptions and the limited usefulness of ordinal alpha","volume":"78","author":"Chalmers","year":"2018","journal-title":"Educational and Psychological Measurement"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1006\/jrpe.2000.2291","article-title":"The factor structure of global self-esteem among adolescents and adults","volume":"34","author":"Corwyn","year":"2000","journal-title":"Journal of Research in Personality"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02289289","article-title":"Test \u201creliability\u201d: Its meaning and determination","volume":"12","author":"Cronbach","year":"1947","journal-title":"Psychometrika"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1007\/BF02310555","article-title":"Coefficient alpha and the internal structure of tests","volume":"16","author":"Cronbach","year":"1951","journal-title":"Psychometrika"},{"key":"ref_8","unstructured":"Cronbach, L. J., Gleser, G. C., Nanda, H., and Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles, Wiley."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1111\/j.2044-8317.1963.tb00206.x","article-title":"Theory of generalizability: A liberalization of reliability theory","volume":"16","author":"Cronbach","year":"1963","journal-title":"British Journal of Statistical Psychology"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1037\/met0000083","article-title":"Anomalous results in G-factor models: Explanations and alternatives","volume":"22","author":"Eid","year":"2017","journal-title":"Psychological Methods"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Eid, M., Krumm, S., Koch, T., and Schulze, J. (2018). Bifactor models for predicting criteria by general and specific factors: Problems of nonidentifiability and alternative solutions. Journal of Intelligence, 6, (3).","DOI":"10.3390\/jintelligence6030042"},{"key":"ref_12","first-page":"2309","article-title":"Identifiability of bifactor models","volume":"31","author":"Fang","year":"2021","journal-title":"Statistica Sinica"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1177\/1073191117746503","article-title":"Cognitive abilities explain wording effects in the Rosenberg Self-Esteem Scale","volume":"27","author":"Gnambs","year":"2020","journal-title":"Assessment"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1007\/BF02287919","article-title":"Comparison of two factorial analyses","volume":"3","author":"Holzinger","year":"1938","journal-title":"Psychometrika"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1007\/BF02287965","article-title":"The bi-factor method","volume":"2","author":"Holzinger","year":"1937","journal-title":"Psychometrika"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v045.i07","article-title":"Amelia II: A program for missing data","volume":"45","author":"Honaker","year":"2011","journal-title":"Journal of Statistical Software"},{"key":"ref_17","unstructured":"Honaker, J., King, G., and Blackwell, M. (2026, August 19). Amelia: A program for missing data (R package version 1.8.1), Available online: https:\/\/cran.r-project.org\/web\/packages\/Amelia\/index.html."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Hong, H. (2022). Finding optimal factor analytic representations of the structure of personality. [Doctoral dissertation, University of Iowa].","DOI":"10.3102\/IP.23.2017891"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"111","DOI":"10.3390\/psych6010007","article-title":"Applying SEM, exploratory SEM, and Bayesian SEM to personality assessments","volume":"6","author":"Hong","year":"2024","journal-title":"Psych"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1037\/1082-989X.3.4.424","article-title":"Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification","volume":"3","author":"Hu","year":"1998","journal-title":"Psychological Methods"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/10705519909540118","article-title":"Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives","volume":"6","author":"Hu","year":"1999","journal-title":"Structural Equation Modeling"},{"key":"ref_22","unstructured":"Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., and Rosseel, Y. (2026, August 19). semTools: Useful tools for structural equation modeling (R package version 0.5-9), Available online: https:\/\/CRAN.R-project.org\/package=semTools."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1025","DOI":"10.1177\/0013164420906449","article-title":"Prior sensitivity in Bayesian structural equation modeling for sparse factor loading structures","volume":"80","author":"Liang","year":"2020","journal-title":"Educational and Psychological Measurement"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1504\/IJQRE.2014.060972","article-title":"An evaluation of WLSMV and Bayesian methods for confirmatory factor analysis with categorical indicators","volume":"2","author":"Liang","year":"2014","journal-title":"International Journal of Quantitative Research in Education"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1207\/s15327906mbr3901_4","article-title":"Confidence limits for the indirect effect: Distribution of the product and resampling methods","volume":"39","author":"MacKinnon","year":"2004","journal-title":"Multivariate Behavioral Research"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1037\/0012-1649.22.1.37","article-title":"Negative item bias in ratings scales for preadolescent children: A cognitive-developmental phenomenon","volume":"22","author":"Marsh","year":"1986","journal-title":"Developmental Psychology"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1111\/j.1745-3984.1987.tb00259.x","article-title":"The hierarchical structure of self-concept and the application of hierarchical confirmatory factor analysis","volume":"24","author":"Marsh","year":"1987","journal-title":"Journal of Educational Measurement"},{"key":"ref_28","unstructured":"Marsh, H. W. (1992). Self-Description Questionnaire (SDQ) III: A theoretical and empirical basis for the measurement of multiple dimensions of late adolescent self-concept. An interim test manual and research monograph, University of Western Sydney."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1177\/014662169101500106","article-title":"Confirmatory factor analysis of multitrait\u2013multimethod data: A comparison of the behavior of alternative models","volume":"15","author":"Marsh","year":"1991","journal-title":"Applied Psychological Measurement"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Maydeu-Olivares, A., and McArdle, J. J. (2005). Goodness of fit in structural equation models. Contemporary psychometrics, Erlbaum.","DOI":"10.4324\/9781410612977"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1207\/s15328007sem1103_2","article-title":"In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler\u2019s (1999) findings","volume":"11","author":"Marsh","year":"2004","journal-title":"Structural Equation Modeling"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1037\/a0019225","article-title":"Longitudinal tests of competing factor structures for the Rosenberg Self-Esteem Scale: Traits, ephemeral artifacts, and stable response styles","volume":"22","author":"Marsh","year":"2010","journal-title":"Psychological Assessment"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2044-8317.1970.tb00432.x","article-title":"The theoretical foundations of principal factor analysis, canonical factor analysis, and alpha factor analysis","volume":"23","author":"McDonald","year":"1970","journal-title":"British Journal of Mathematical and Statistical Psychology"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1177\/001316447803800111","article-title":"Generalizability in factorable domains: \u201cDomain validity and generalizability\u201d","volume":"38","author":"McDonald","year":"1978","journal-title":"Educational and Psychological Measurement"},{"key":"ref_35","unstructured":"McDonald, R. P. (1999). Test theory: A unified treatment, Erlbaum."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1037\/a0026802","article-title":"Bayesian structural equation modeling: A more flexible representation of substantive theory","volume":"17","author":"Asparouhov","year":"2012","journal-title":"Psychological Methods"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1080\/19312458.2012.679848","article-title":"Advantages of Monte Carlo confidence intervals for indirect effects","volume":"6","author":"Preacher","year":"2012","journal-title":"Communication Methods and Measures"},{"key":"ref_38","unstructured":"Revelle, W. (2026, August 19). psych: Procedures for psychological, psychometric, and personality research (R package version 2.5.6), Available online: https:\/\/cran.r-project.org\/web\/packages\/psych\/index.html."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1037\/a0029315","article-title":"When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions","volume":"17","author":"Rhemtulla","year":"2012","journal-title":"Psychological Methods"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1037\/met0000045","article-title":"Correction to: Evaluating bifactor models: Calculating and interpreting statistical indices","volume":"21","author":"Rodriguez","year":"2016","journal-title":"Psychological Methods"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Rosenberg, M. (1965). Society and the adolescent self-image, Princeton University Press.","DOI":"10.1515\/9781400876136"},{"key":"ref_42","unstructured":"Rosenberg, M. (1989). Society and the adolescent self-image, Wesleyan University Press. [Rev. ed.]."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v048.i02","article-title":"lavaan: An R package for structural equation modeling","volume":"48","author":"Rosseel","year":"2012","journal-title":"Journal of Statistical Software"},{"key":"ref_44","unstructured":"Rosseel, Y., Jorgensen, T. D., and De Wilde, L. (2025). lavaan: Latent variable analysis (R package version 0.6-21), Available online: https:\/\/cran.r-project.org\/web\/packages\/lavaan\/lavaan.pdf."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1037\/1082-989X.1.2.199","article-title":"Measurement error in psychological research: Lessons from 26 research scenarios","volume":"1","author":"Schmidt","year":"1996","journal-title":"Psychological Methods"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1037\/1082-989X.8.2.206","article-title":"Beyond alpha: An empirical investigation of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs","volume":"8","author":"Schmidt","year":"2003","journal-title":"Psychological Methods"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1037\/0022-3514.89.4.623","article-title":"Simultaneous administration of the Rosenberg Self-Esteem Scale in 53 nations: Exploring the universal and culture-specific features of global self-esteem","volume":"89","author":"Schmitt","year":"2005","journal-title":"Journal of Personality and Social Psychology"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"407","DOI":"10.3102\/00346543046003407","article-title":"Self-concept: Validation of construct interpretations","volume":"46","author":"Shavelson","year":"1976","journal-title":"Review of Educational Research"},{"key":"ref_49","unstructured":"Stekhoven, D. J. (2022). missForest: Nonparametric missing value imputation using random forest (R package version 1.5), Available online: https:\/\/cran.r-project.org\/web\/packages\/missForest\/index.html."},{"key":"ref_50","first-page":"79","article-title":"States and traits in psychological assessment","volume":"8","author":"Steyer","year":"1992","journal-title":"European Journal of Psychological Assessment"},{"key":"ref_51","unstructured":"Lindquist, E. F. (1951). Reliability. Educational measurement, American Council on Education."},{"key":"ref_52","unstructured":"van Buuren, S., and Groothuis-Oudshoorn, K. (2026, August 19). mice: Multivariate imputation by chained equations (R package version 3.15.0), Available online: https:\/\/cran.r-project.org\/web\/packages\/mice\/index.html."},{"key":"ref_53","first-page":"75","article-title":"Bayesian analyses: Where to start and what to report","volume":"16","author":"Depaoli","year":"2014","journal-title":"The European Health Psychologist"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1037\/0022-0663.87.1.134","article-title":"Self-concept in artistic domains: An extension of the Shavelson, Hubner, and Stanton (1976) model","volume":"87","author":"Vispoel","year":"1995","journal-title":"Journal of Educational Psychology"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1080\/07481756.2000.12069005","article-title":"Computerized versus paper-and-pencil assessment of self-concept: Score comparability and respondent preferences","volume":"33","author":"Vispoel","year":"2000","journal-title":"Measurement and Evaluation in Counseling and Development"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1080\/10705511.2023.2187734","article-title":"Benefits of doing generalizability theory analyses within structural equation modeling frameworks: Illustrations using the Rosenberg Self-Esteem Scale [Teacher\u2019s corner]","volume":"31","author":"Vispoel","year":"2024","journal-title":"Structural Equation Modeling"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"597","DOI":"10.1080\/00223891.2026.2628589","article-title":"Estimating item wording effects in self-report measures with generalizability theory-based SEMs: Illustrations using the Self-Description Questionnaire-III","volume":"108","author":"Vispoel","year":"2026","journal-title":"Journal of Personality Assessment"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Vispoel, W. P., Lee, H., and Chen, T. (2025). Structural equation modeling approaches to estimating score dependability within generalizability theory-based univariate, multivariate, and bifactor designs. Mathematics, 13, (6).","DOI":"10.3390\/math13061001"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"249","DOI":"10.3390\/psych5020019","article-title":"Using structural equation modeling to reproduce and extend ANOVA-based generalizability theory analyses for psychological assessments","volume":"5","author":"Vispoel","year":"2023","journal-title":"Psych"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"1093","DOI":"10.1037\/pas0001170","article-title":"Expanding bifactor models of psychological traits to account for multiple sources of measurement error","volume":"34","author":"Vispoel","year":"2022","journal-title":"Psychological Assessment"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1037\/met0000107","article-title":"Applications of generalizability theory and their relations to classical test theory and structural equation modeling","volume":"23","author":"Vispoel","year":"2018a","journal-title":"Psychological Methods"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1080\/00223891.2017.1296455","article-title":"Practical applications of generalizability theory for designing, evaluating, and improving psychological assessments","volume":"100","author":"Vispoel","year":"2018b","journal-title":"Journal of Personality Assessment"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1080\/00220973.2018.1448748","article-title":"Computerized and traditional administration of questionnaires: Psychometric quality and completion time for measures of self-concept","volume":"87","author":"Vispoel","year":"2019","journal-title":"Journal of Experimental Education"},{"key":"ref_64","unstructured":"Yu, C. Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. [Unpublished doctoral dissertation, University of California]."}],"container-title":["Behavioral Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-328X\/16\/9\/1636\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,16]],"date-time":"2026-09-16T04:12:56Z","timestamp":1789531976000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-328X\/16\/9\/1636"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,13]]},"references-count":64,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9]]}},"alternative-id":["bs16091636"],"URL":"https:\/\/doi.org\/10.3390\/bs16091636","relation":{},"ISSN":["2076-328X"],"issn-type":[{"value":"2076-328X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,13]]}}}