{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T14:11:55Z","timestamp":1783779115451,"version":"3.55.0"},"reference-count":21,"publisher":"SAGE Publications","license":[{"start":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T00:00:00Z","timestamp":1783382400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T00:00:00Z","timestamp":1783382400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Educational and Psychological Measurement"],"abstract":"\n The purpose of this study is to examine the reliability and dependability of difference scores computed as the change between a pretest and a posttest administered to assess the effectiveness of an intervention. The data-collection design involved a nested structure, with persons (\n p<\/jats:italic>\n ) nested within groups (\n g<\/jats:italic>\n ), and groups nested within sites (\n s<\/jats:italic>\n ). Multivariate generalizability theory was employed to estimate the reliability and dependability of difference scores at the levels of persons, groups, and sites. The\n G<\/jats:italic>\n study designs included\n \n \n \n \n (<\/mml:mo>\n \n \n p<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n g<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n )<\/mml:mo>\n <\/mml:mrow>\n \n x<\/mml:mi>\n \n \n \n i<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n ,\n \n \n \n \n (<\/mml:mo>\n \n \n p<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n )<\/mml:mo>\n <\/mml:mrow>\n \n x<\/mml:mi>\n \n \n \n i<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n ,\n \n \n \n \n (<\/mml:mo>\n \n \n p<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n g<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n )<\/mml:mo>\n <\/mml:mrow>\n \n x<\/mml:mi>\n \n \n \n i<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n ,\n \n \n \n \n \n g<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n \n x<\/mml:mi>\n \n \n \n i<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n ,\n \n \n \n \n \n p<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n x<\/mml:mi>\n \n \n \n i<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n , and\n \n \n \n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n x<\/mml:mi>\n \n \n \n i<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n , with pretest and posttest serving as the two levels of the multivariate facet. In the designs with sites as the object of measurement,\n \n \n \n \n (<\/mml:mo>\n \n \n P<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n G<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n )<\/mml:mo>\n <\/mml:mrow>\n \n x<\/mml:mi>\n \n \n \n I<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n ,\n \n \n \n \n (<\/mml:mo>\n \n \n P<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n )<\/mml:mo>\n <\/mml:mrow>\n \n x<\/mml:mi>\n \n \n \n I<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n , and\n \n \n \n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n x<\/mml:mi>\n \n \n \n I<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n , omitting groups within sites or persons within groups led to an underestimation of error variances and inflated generalizability and dependability coefficients. The relative error correlations increased, and the absolute error and universe score correlations decreased across\n \n \n \n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n x<\/mml:mi>\n \n \n \n I<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n ,\n \n \n \n \n (<\/mml:mo>\n \n \n P<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n )<\/mml:mo>\n <\/mml:mrow>\n \n x<\/mml:mi>\n \n \n \n I<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n , and\n \n \n \n \n (<\/mml:mo>\n \n \n P<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n G<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n :<\/mml:mo>\n \n \n s<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n )<\/mml:mo>\n <\/mml:mrow>\n \n x<\/mml:mi>\n \n \n \n I<\/mml:mi>\n <\/mml:mrow>\n \n \u2022<\/mml:mo>\n <\/mml:mrow>\n <\/mml:msup>\n <\/mml:mrow>\n <\/mml:math>\n <\/jats:inline-formula>\n . Across all designs, generalizability and dependability coefficients were similar in magnitude, primarily due to the relatively small variance in items. Compared across different objects of measurement, both the generalizability and dependability coefficients were highest when the object of measurement was persons, and lowest when it was sites.\n <\/jats:p>","DOI":"10.1177\/00131644261451746","type":"journal-article","created":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T04:57:34Z","timestamp":1783486654000},"update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Reliability of Difference Scores Obtained From Nested Data Within a Multivariate Generalizability Theory Framework"],"prefix":"10.1177","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8617-1908","authenticated-orcid":false,"given":"Rabia","family":"Karatoprak Ersen","sequence":"first","affiliation":[{"name":"GESIS\u2014Leibniz Institute for the Social Sciences, Cologne, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5319-668X","authenticated-orcid":false,"given":"Won-Chan","family":"Lee","sequence":"additional","affiliation":[{"name":"The University of Iowa, Iowa City, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Donald B.","family":"Yarbrough","sequence":"additional","affiliation":[{"name":"The University of Iowa, Iowa City, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,7,7]]},"reference":[{"key":"e_1_3_3_2_1","author":"American Educational Research Association, American Psychological Association, & National Council on Measurement in Education","year":"2014","unstructured":"American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing.","journal-title":"Standards for educational and psychological testing"},{"key":"e_1_3_3_3_1","doi-asserted-by":"crossref","unstructured":"Brennan R. L. (1995). The conventional wisdom about group mean scores. Journal of Educational Measurement 32(4) 385\u2013396. https:\/\/doi.org\/10.1111\/j.1745-3984.1995.tb00473.x","DOI":"10.1111\/j.1745-3984.1995.tb00473.x"},{"key":"e_1_3_3_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3456-0"},{"key":"e_1_3_3_5_1","unstructured":"Brennan R. L. (2001b). mGENOVA [Computer software and manual]. Center for Advanced Studies in Measurement and Assessment The University of Iowa. https:\/\/education.uiowa.edu\/casma\/computer-programs#GENOVA"},{"key":"e_1_3_3_6_1","unstructured":"Brennan R. L. (2001c). urGENOVA [Computer Software and Manual]. Center for Advanced Studies in Measurement and Assessment the University of Iowa. https:\/\/education.uiowa.edu\/casma\/computer-programs#GENOVA"},{"key":"e_1_3_3_7_1","doi-asserted-by":"crossref","unstructured":"Brennan R. L. Kane M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement 14(3) 277\u2013289. http:\/\/www.jstor.org\/stable\/1434319","DOI":"10.1111\/j.1745-3984.1977.tb00045.x"},{"key":"e_1_3_3_8_1","doi-asserted-by":"crossref","unstructured":"Brennan R. L. Kim S. Y. Lee W. (2022). Extended multivariate generalizability theory with complex design structures. Educational and Psychological Measurement 82(4) 617\u2013642. https:\/\/doi.org\/10.1177\/001316442110497","DOI":"10.1177\/00131644211049746"},{"key":"e_1_3_3_9_1","doi-asserted-by":"crossref","unstructured":"Brennan R. L. Yin P. Kane M. T. (2003). Methodology for examining the reliability of group mean difference scores. Journal of Educational Measurement 40 207\u2013230. https:\/\/doi.org\/10.1111\/j.1745-3984.2003.tb01105.x","DOI":"10.1111\/j.1745-3984.2003.tb01105.x"},{"key":"e_1_3_3_10_1","doi-asserted-by":"crossref","unstructured":"Buchanan E. M. Miranda J. F. Stephens C. (2025). The reliability of student evaluations of teaching. Assessment & Evaluation in Higher Education 50(7) 997\u20131011. https:\/\/doi.org\/10.1080\/02602938.2025.2504618","DOI":"10.1080\/02602938.2025.2504618"},{"key":"e_1_3_3_11_1","doi-asserted-by":"crossref","unstructured":"Cronbach L. J. Linn R. L. Brennan R. L. Haertel E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement 57(3) 373\u2013399. https:\/\/doi.org\/10.1177\/0013164497057003001","DOI":"10.1177\/0013164497057003001"},{"key":"e_1_3_3_12_1","unstructured":"Fishbein B. Taneva M. Kowolik K. (2026). TIMSS 2023 longitudinal user guide for international database. Boston College TIMSS & PIRLS International Study Center. https:\/\/timss2023.org\/longitudinal\/data"},{"key":"e_1_3_3_13_1","doi-asserted-by":"crossref","unstructured":"Gu Z. Emons W. H. M. Sijtsma K. (2018). Review of issues about classical change scores: A multilevel modeling perspective on some enduring beliefs. Psychometrika 83 674\u2013695. https:\/\/doi.org\/10.1007\/s11336-018-9611-3","DOI":"10.1007\/s11336-018-9611-3"},{"key":"e_1_3_3_14_1","doi-asserted-by":"crossref","unstructured":"Jiang Z. Ouyang J. Shi D. Shi D. Zhang J. Xu L. Cai F. (2024). Customizing Bayesian multivariate generalizability theory to mixed-format tests. Behavior Research Methods 56(7) 8080\u20138090. https:\/\/doi.org\/10.3758\/s13428-024-02472-7","DOI":"10.3758\/s13428-024-02472-7"},{"key":"e_1_3_3_15_1","doi-asserted-by":"crossref","unstructured":"Kane M. (1996). The precision of measurements. Applied Measurement in Education 9(4) 355\u2013379. https:\/\/doi.org\/10.1207\/s15324818ame0904_4","DOI":"10.1207\/s15324818ame0904_4"},{"key":"e_1_3_3_16_1","doi-asserted-by":"crossref","unstructured":"Miller T. B. Kane M. (2001). The precision of change scores under absolute and relative interpretations. Applied Measurement in Education 14(4) 307\u2013327. https:\/\/doi.org\/10.1207\/S15324818AME1404_1","DOI":"10.1207\/S15324818AME1404_1"},{"key":"e_1_3_3_17_1","unstructured":"R Core Team. (2026). R: A language and environment for statistical computing. R Foundation for Statistical Computing Vienna Austria. https:\/\/www.R-project.org"},{"key":"e_1_3_3_18_1","doi-asserted-by":"crossref","unstructured":"Rubach C. von Keyserlingk L. Heckhausen J. Eccles J. S. (2025). The situatedness of instructional quality\u2014How situated are college students\u2019 ratings? British Journal of Educational Psychology 95(Suppl. 1) S316\u2013S336. https:\/\/doi.org\/10.1111\/bjep.70003","DOI":"10.1111\/bjep.70003"},{"key":"e_1_3_3_19_1","doi-asserted-by":"crossref","unstructured":"Thomas D. R. Zumbo B. D. (2011). Difference scores from the point of view of reliability and repeated-measures ANOVA. Educational and Psychological Measurement 72(1) 37\u201343. https:\/\/doi.org\/10.1177\/0013164411409929","DOI":"10.1177\/0013164411409929"},{"key":"e_1_3_3_20_1","doi-asserted-by":"crossref","unstructured":"Wade R. Yarbrough D. (2007). Service\u2013learning in the social studies: Civic outcomes of the 3rd\u201312th grade CiviConnections program. Theory & Research in Social Education 35(3) 366\u2013392. https:\/\/doi.org\/10.1080\/00933104.2007.10473341","DOI":"10.1080\/00933104.2007.10473341"},{"key":"e_1_3_3_21_1","doi-asserted-by":"crossref","unstructured":"Wei X. Haertel E. (2011). The effect of ignoring classroom-level variance in estimating the generalizability of school mean scores. Educational Measurement: Issues and Practice 30(1) 13\u201322. https:\/\/doi.org\/10.1111\/j.1745-3992.2010.00196.x","DOI":"10.1111\/j.1745-3992.2010.00196.x"},{"key":"e_1_3_3_22_1","doi-asserted-by":"crossref","unstructured":"Yin P. Brennan R. L. (2002). An investigation of difference scores for a grade-level testing program. International Journal of Testing 2(2) 83\u2013105. https:\/\/doi.org\/10.1207\/S15327574IJT0202_1","DOI":"10.1207\/S15327574IJT0202_1"}],"container-title":["Educational and Psychological Measurement"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00131644261451746","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/00131644261451746","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00131644261451746","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T13:20:12Z","timestamp":1783776012000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/00131644261451746"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,7]]},"references-count":21,"alternative-id":["10.1177\/00131644261451746"],"URL":"https:\/\/doi.org\/10.1177\/00131644261451746","relation":{},"ISSN":["0013-1644","1552-3888"],"issn-type":[{"value":"0013-1644","type":"print"},{"value":"1552-3888","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,7]]},"article-number":"00131644261451746"}}