{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T10:30:27Z","timestamp":1790073027871,"version":"4.0.1"},"reference-count":43,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,12,27]],"date-time":"2023-12-27T00:00:00Z","timestamp":1703635200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Psychol."],"abstract":"<jats:p>Monte Carlo simulation is a common method of providing empirical evidence to verify statistics used in psychological studies. A representative set of conditions should be included in simulation studies. However, several recently published Monte Carlo simulation studies have not included the conditions of the null distribution of the statistic in their evaluations or comparisons of statistics and, therefore, have drawn incorrect conclusions. This present study proposes a design based on a common statistic evaluation procedure in psychology and machine learning, using a confusion matrix with four cells: true positive, true negative, false negative modified, and false positive modified. To illustrate this design, we employ an influential Monte Carlo simulation study by Trizano-Hermosilla and Alvarado (2016), which concluded that the Omega-indexed internal consistency should be preferred over other alternatives. Our results show that Omega can report an acceptable level of internal consistency (i.e., &amp;gt; 0.7) in a population with no relationship between every two items in some conditions, providing novel empirical evidence for comparing internal consistency indices.<\/jats:p>","DOI":"10.3389\/fpsyg.2023.1298534","type":"journal-article","created":{"date-parts":[[2023,12,27]],"date-time":"2023-12-27T18:26:42Z","timestamp":1703701602000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Monte Carlo simulation with confusion matrix paradigm \u2013 A sample of internal consistency indices"],"prefix":"10.3389","volume":"14","author":[{"given":"Yongtian","family":"Cheng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pablo A.","family":"P\u00e9rez-D\u00edaz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"K. V.","family":"Petrides","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johnson","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,12,27]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1177\/0734282917690302","article-title":"Simulating data for clinical research: a tutorial","volume":"36","author":"Beaujean","year":"2018","journal-title":"J. Psychoeduc. Assess."},{"key":"ref2","article-title":"Package \"GPArotation.\"","author":"Bernaards","year":"2015"},{"key":"ref3","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1177\/25152459198471","article-title":"Correcting for bias in psychology: a comparison of meta-analytic methods","volume":"2","author":"Carter","year":"2019","journal-title":"Adv. Methods Pract. Psychol. Sci."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"1351","DOI":"10.1037\/apl0000815","article-title":"From alpha to omega and beyond! A look at the past, present, and (possible) future of psychometric soundness in the journal of applied psychology","volume":"105","author":"Cortina","year":"2020","journal-title":"J. Appl. Psychol."},{"key":"ref7","first-page":"19","article-title":"Using likert-type scales in the social sciences","volume":"40","author":"Croasmun","year":"2011","journal-title":"J. Adult Educ."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"30","DOI":"10.20982\/tqmp.12.1.p030","article-title":"Why Welch's test is type I error robust. The quantitative methods","volume":"12","author":"Derrick","year":"2016","journal-title":"Psychology"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1037\/1082-989X.4.3.272","article-title":"Evaluating the use of exploratory factor analysis in psychological research","volume":"4","author":"Fabrigar","year":"1999","journal-title":"Psychol. Methods"},{"key":"ref10","first-page":"427","article-title":"Designing simulation studies","volume-title":"APA handbook of research methods in psychology","author":"Fan","year":"2012"},{"key":"ref11","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1080\/00273170701382864","article-title":"Sensitivity of fit indices to model misspecification and model types","volume":"42","author":"Fan","year":"2007","journal-title":"Multivar. Behav. Res."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1080\/00220973.2019.1582470","article-title":"Detecting selection bias in meta-analyses with multiple outcomes: a simulation study","volume":"89","author":"Fern\u00e1ndez-Castilla","year":"2021","journal-title":"J. Exp. Educ."},{"key":"ref9001","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1177\/00131644211073121","article-title":"Using simulated annealing to investigate sensitivity of SEM to external model misspecification","volume":"83","author":"Fisk","year":"2023","journal-title":"Educ Psychol Meas"},{"key":"ref14","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1080\/10705511.2019.1637741","article-title":"Estimating the maximum likelihood root mean square error of approximation (RMSEA) with nonnormal data: a Monte-Carlo study","volume":"27","author":"Gao","year":"2020","journal-title":"Struct. Equ. Model. Multidiscip. J."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"398","DOI":"10.1080\/01621459.1990.10476213","article-title":"Sampling-based approaches to calculating marginal densities","volume":"85","author":"Gelfand","year":"1990","journal-title":"J. Am. Stat. Assoc."},{"key":"ref16","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1037\/abn0000434","article-title":"Are fit indices used to test psychopathology structure biased? A simulation study","volume":"128","author":"Greene","year":"2019","journal-title":"J. Abnorm. Psychol."},{"key":"ref17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/19312458.2020.1718629","article-title":"Use omega rather than Cronbach's alpha for estimating reliability. But\u2026","volume":"14","author":"Hayes","year":"2020","journal-title":"Commun. Methods Meas."},{"key":"ref18","doi-asserted-by":"publisher","first-page":"685","DOI":"10.1016\/S0167-9473(02)00072-5","article-title":"Fast fifth-order polynomial transforms for generating univariate and multivariate nonnormal distributions","volume":"40","author":"Headrick","year":"2002","journal-title":"Comp. Statis. Data Anal."},{"key":"ref19","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1027\/1866-5888\/a000129","article-title":"Considering the effects of nonresponse on correlations between surveyed variables: a simulation study toprovide context to evaluate survey results","volume":"14","author":"Heggestad","year":"2015","journal-title":"J. Pers. Psychol."},{"key":"ref9002","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1037\/h0087424","article-title":"Statistical testing and null distributions: What to do when samples are not random","volume":"57","author":"Hunter","year":"2003","journal-title":"Can. J. Exp. Psychol."},{"key":"ref21","first-page":"000555","article-title":"Using the confusion matrix for improving ensemble classifiers","author":"Marom","year":"2010"},{"key":"ref22","volume-title":"Test theory: A unified treatment","author":"McDonald","year":"1999"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"412","DOI":"10.1037\/met0000144","article-title":"Thanks coefficient alpha, we'll take it from here","volume":"23","author":"McNeish","year":"2018","journal-title":"Psychol. Methods"},{"key":"ref24","article-title":"Find the greatest lower bound to reliability","author":"Moltner","year":"2015"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"885","DOI":"10.13031\/2013.23153","article-title":"Model evaluation guidelines for systematic quantification of accuracy in watershed simulations","volume":"50","author":"Moriasi","year":"2007","journal-title":"Trans. ASABE"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"129","DOI":"10.2333\/bhmk.40.129","article-title":"Is omega squared less biased? A comparison of three major effect size indices in one-way ANOVA","volume":"40","author":"Okada","year":"2013","journal-title":"Behaviormetrika"},{"key":"ref27","doi-asserted-by":"publisher","first-page":"979","DOI":"10.3758\/s13428-016-0760-y","article-title":"Negative estimate of variance-accounted-for effect size: how often it is obtained, and what happens if it is treated as zero","volume":"49","author":"Okada","year":"2017","journal-title":"Behav. Res. Methods"},{"key":"ref28","doi-asserted-by":"publisher","first-page":"146","DOI":"10.2307\/1165199","article-title":"A straightforward approach to Markov chain Monte Carlo methods for item response models","volume":"24","author":"Patz","year":"","journal-title":"J. Educ. Behav. Stat."},{"key":"ref29","doi-asserted-by":"publisher","first-page":"342","DOI":"10.2307\/1165367","article-title":"Applications and extensions of MCMC in IRT: multiple item types, missing data, and rated responses","volume":"24","author":"Patz","year":"","journal-title":"J. Educ. Behav. Stat."},{"key":"ref30","volume-title":"R: A language and environment for statistical computing","year":"2021"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1002\/jae.1278","article-title":"Rstudio: a plataform-independet IDE for R and sweave","volume":"27","author":"Racine","year":"2012","journal-title":"J. Appl. Econ."},{"key":"ref32","volume-title":"Psych: Procedures for personality and psychological research","author":"Revelle","year":"2015"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1016\/j.beproc.2018.01.004","article-title":"Evaluation of the confusion matrix method in the validation of an automated system for measuring feeding behaviour of cattle","volume":"148","author":"Ruuska","year":"2018","journal-title":"Behav. Process."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/s11336-008-9101-0","article-title":"On the use, the misuse, and the very limited usefulness of Cronbach's alpha","volume":"74","author":"Sijtsma","year":"2009","journal-title":"Psychometrika"},{"key":"ref35","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1177\/00131644211003261","article-title":"Sample size requirements for simple and complex mediation models","volume":"82","author":"Sim","year":"2022","journal-title":"Educ. Psychol. Meas."},{"key":"ref36","doi-asserted-by":"publisher","first-page":"685","DOI":"10.1080\/10485250310001634719","article-title":"On the null distribution of the Kruskal\u2013Wallis statistic","volume":"15","author":"Spurrier","year":"2003","journal-title":"Nonparamet. Stat."},{"key":"ref37","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1111\/j.1745-3984.2000.tb01076.x","article-title":"Monte Carlo based null distribution for an alternative goodness-of-fittest statistic in IRT models","volume":"37","author":"Stone","year":"2000","journal-title":"J. Educ. Meas."},{"key":"ref38","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1177\/070674370705200210","article-title":"What's under the ROC? An introduction to receiver operating characteristics curves","volume":"52","author":"Streiner","year":"2007","journal-title":"Can. J. Psychiatry"},{"key":"ref39","doi-asserted-by":"publisher","first-page":"1273","DOI":"10.1007\/s11165-016-9602-2","article-title":"The use of Cronbach's alpha when developing and reporting research instruments in science education","volume":"48","author":"Taber","year":"2018","journal-title":"Res. Sci. Educ."},{"key":"ref9004","first-page":"205","article-title":"Internal consistency: Do we really know what it is and how to assess it","volume":"2","author":"Tang","year":"2014","journal-title":"J. Psychol. Behav. Sci"},{"key":"ref40","doi-asserted-by":"publisher","first-page":"769","DOI":"10.3389\/fpsyg.2016.00769","article-title":"Best alternatives to Cronbach's alpha reliability in realistic conditions: congeneric and asymmetrical measurements","volume":"7","author":"Trizano-Hermosilla","year":"2016","journal-title":"Front. Psychol."},{"key":"ref41","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1080\/00221309.2022.2094310","article-title":"The effect of normality and outliers on bivariate correlation coefficients in psychology: a Monte Carlo simulation","volume":"150","author":"Ventura-Le\u00f3n","year":"2023","journal-title":"J. Gen. Psychol."},{"key":"ref42","doi-asserted-by":"crossref","DOI":"10.1088\/1757-899X\/324\/1\/012049","article-title":"Analysis of the mean absolute error (MAE) and the root mean square error (RMSE) in assessing rounding model","volume-title":"IOP Conference Series: Materials Science and Engineering","author":"Wang","year":"2018"},{"key":"ref9005","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1080\/13854046.2017.1317364","article-title":"The reliability of multidimensional neuropsychological measures: from alpha to omega","volume":"31","author":"Watkins","year":"2017","journal-title":"Clin. Neuropsychol."},{"key":"ref43","doi-asserted-by":"crossref","DOI":"10.1201\/9781466503236","volume-title":"Modern statistics for the social and behavioral sciences: a practical introduction","author":"Wilcox","year":"2011"}],"container-title":["Frontiers in Psychology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2023.1298534\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,27]],"date-time":"2023-12-27T18:26:48Z","timestamp":1703701608000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2023.1298534\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,27]]},"references-count":43,"alternative-id":["10.3389\/fpsyg.2023.1298534"],"URL":"https:\/\/doi.org\/10.3389\/fpsyg.2023.1298534","relation":{},"ISSN":["1664-1078"],"issn-type":[{"value":"1664-1078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,27]]},"article-number":"1298534"}}