{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T16:41:02Z","timestamp":1793378462180,"version":"4.5.0"},"reference-count":81,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T00:00:00Z","timestamp":1790726400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T00:00:00Z","timestamp":1790726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Behav Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Evaluating test\u2013retest reliability is of crucial importance for clinical decision-making and an integral component of studies in empirical sciences. In psychology, classical test theory focused on reliability coefficients, but research has shown that intraclass correlation coefficients and Cronbach\u2019s alpha have limited value in quantifying individual measurement error sources appropriately. Preliminary assumptions such as parallel readings and independency of error sources are violated by practice\/habituation effects, i.e., test\u2013retest systematic errors. Although awareness of the need for additional measurement quantifications to properly assess test\u2013retest agreement has increased, many such quantifications are insufficiently implemented. Furthermore, one of the most important agreement analyses introduced by Bland and Altman in 1986 has interpretation limitations, as the limits of agreement are descriptive and not linked to content-based clinical decision-making, oftentimes causing inappropriate usage. To counteract these limitations and improve the implementation of agreement analyses in addition to more commonly used reliability coefficients, we introduce a software package that allows predefining acceptable measurement limits (limits of acceptance) to evaluate the sufficiency of task habituation to gauge individual error sources. The package implements a comprehensive test\u2013retest evaluation in order to improve transparency of data quality reporting in psychological and cognitive research.<\/jats:p>","DOI":"10.3758\/s13428-026-03170-2","type":"journal-article","created":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T20:44:27Z","timestamp":1790801067000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["How much habituation is enough? An R application for ad hoc content-related limits of acceptance in behavioral research"],"prefix":"10.3758","volume":"58","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4964-2867","authenticated-orcid":false,"given":"Konstantin","family":"Warneke","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stanislav D.","family":"Siegel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2038-393X","authenticated-orcid":false,"given":"Jos\u00e9","family":"Afonso","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marco","family":"Herbsleb","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3626-3940","authenticated-orcid":false,"given":"Sebastian","family":"Wallot","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,9,30]]},"reference":[{"issue":"1","key":"3170_CR1","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1037\/amp0000191","volume":"73","author":"M Appelbaum","year":"2018","unstructured":"Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 3\u201325. https:\/\/doi.org\/10.1037\/amp0000191","journal-title":"American Psychologist"},{"issue":"4","key":"3170_CR2","doi-asserted-by":"publisher","first-page":"217","DOI":"10.2165\/00007256-199826040-00002","volume":"26","author":"G Atkinson","year":"1998","unstructured":"Atkinson, G., & Nevill, A. M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sports medicine. Sports Medicine, 26(4), 217\u2013238. https:\/\/doi.org\/10.2165\/00007256-199826040-00002","journal-title":"Sports Medicine"},{"issue":"4","key":"3170_CR3","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1080\/10543400701376480","volume":"17","author":"HX Barnhart","year":"2007","unstructured":"Barnhart, H. X., Haber, M. J., & Lin, L. I. (2007). An overview on assessing agreement with continuous measurements. Journal of Biopharmaceutical Statistics, 17(4), 529\u2013569. https:\/\/doi.org\/10.1080\/10543400701376480","journal-title":"Journal of Biopharmaceutical Statistics"},{"issue":"4","key":"3170_CR4","doi-asserted-by":"publisher","first-page":"466","DOI":"10.1002\/uog.5256","volume":"31","author":"JW Bartlett","year":"2008","unstructured":"Bartlett, J. W., & Frost, C. (2008). Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables. Ultrasound in Obstetrics & Gynecology, 31(4), 466\u2013475. https:\/\/doi.org\/10.1002\/uog.5256","journal-title":"Ultrasound in Obstetrics & Gynecology"},{"issue":"3","key":"3170_CR5","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1044\/2017_AJA-16-0100","volume":"26","author":"RJ Bennett","year":"2017","unstructured":"Bennett, R. J., Taljaard, D. S., Olaithe, M., Brennan-Jones, C., & Eikelboom, R. H. (2017). Evaluating random error in clinician-administered surveys: Theoretical considerations and clinical applications of interobserver reliability and agreement. American Journal of Audiology, 26(3), 191\u2013201. https:\/\/doi.org\/10.1044\/2017_AJA-16-0100","journal-title":"American Journal of Audiology"},{"issue":"8982","key":"3170_CR6","doi-asserted-by":"publisher","first-page":"1085","DOI":"10.1016\/S0140-6736(95)91748-9","volume":"346","author":"JM Bland","year":"1995","unstructured":"Bland, J. M., & Altman, D. G. (1995). Comparing methods of measurement: Why plotting difference against standard method is misleading. The Lancet, 346(8982), 1085\u20131087. https:\/\/doi.org\/10.1016\/S0140-6736(95)91748-9","journal-title":"The Lancet"},{"issue":"7080","key":"3170_CR7","doi-asserted-by":"publisher","first-page":"572","DOI":"10.1136\/bmj.314.7080.572","volume":"314","author":"JM Bland","year":"1997","unstructured":"Bland, J. M., & Altman, D. G. (1997). Statistics notes: Cronbach\u2019s alpha: Table 1. BMJ, 314(7080), 572. https:\/\/doi.org\/10.1136\/bmj.314.7080.572","journal-title":"BMJ"},{"issue":"2","key":"3170_CR8","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1177\/096228029900800204","volume":"8","author":"JM Bland","year":"1999","unstructured":"Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8(2), 135\u2013160. https:\/\/doi.org\/10.1177\/096228029900800204","journal-title":"Statistical Methods in Medical Research"},{"key":"3170_CR9","doi-asserted-by":"crossref","unstructured":"Bland, J. M., & Altmann, D. G. (1986). Statistical methods of assessing agreement between two methods of clinical measurement. Lancet, i, 307\u2013310.","DOI":"10.1016\/S0140-6736(86)90837-8"},{"issue":"5","key":"3170_CR10","doi-asserted-by":"publisher","first-page":"1330","DOI":"10.3758\/s13415-025-01336-7","volume":"25","author":"M Bogdanov","year":"2025","unstructured":"Bogdanov, M., Scott, J. N., Esfand, S. M., Boyle, B. W., Lees, T., Li, M., Woronko, S. E., Linton, S. R., Miller, C., Li, S., Bolton, P., Meisner, R. C., & Pizzagalli, D. A. (2025). Test-retest reliability and repeatability of behavioral and electrophysiological markers of cognitive control in an Eriksen Flanker Task. Cognitive, Affective, & Behavioral Neuroscience, 25(5), 1330\u20131358. https:\/\/doi.org\/10.3758\/s13415-025-01336-7","journal-title":"Cognitive, Affective, & Behavioral Neuroscience"},{"key":"3170_CR11","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.17122","volume":"12","author":"A Brandl","year":"2024","unstructured":"Brandl, A., Acikalin, E., Bartsch, K., Wilke, J., & Schleip, R. (2024a). Reliability and validity of an app-assisted tissue compliance meter in measuring tissue stiffness on a phantom model. PeerJ, 12, e17122. https:\/\/doi.org\/10.7717\/peerj.17122","journal-title":"PeerJ"},{"issue":"1","key":"3170_CR12","doi-asserted-by":"publisher","first-page":"20044","DOI":"10.1038\/s41598-024-70982-7","volume":"14","author":"A Brandl","year":"2024","unstructured":"Brandl, A., Wilke, J., Horstmann, T., Reer, R., Egner, C., Schmidt, T., & Schleip, R. (2024b). Quantifying thoracolumbar fascia deformation to discriminate acute low back pain patients and healthy individuals using ultrasound. Scientific Reports, 14(1), 20044. https:\/\/doi.org\/10.1038\/s41598-024-70982-7","journal-title":"Scientific Reports"},{"issue":"1","key":"3170_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/08957347.2011.532417","volume":"24","author":"RL Brennan","year":"2010","unstructured":"Brennan, R. L. (2010). Generalizability theory and classical test theory. Applied Measurement in Education, 24(1), 1\u201321. https:\/\/doi.org\/10.1080\/08957347.2011.532417","journal-title":"Applied Measurement in Education"},{"issue":"12","key":"3170_CR14","doi-asserted-by":"publisher","first-page":"1830","DOI":"10.3390\/life15121830","volume":"15","author":"D Cabarkapa","year":"2025","unstructured":"Cabarkapa, D., Smith, R., Chowning, L., Neltner, T., Johnson, Q. R., Yang, Y., & Munce, T. A. (2025). Reliability of vertical jump force-time metrics in collegiate athletes compared to recreationally active individuals. Life, 15(12), 1830. https:\/\/doi.org\/10.3390\/life15121830","journal-title":"Life"},{"issue":"9","key":"3170_CR15","doi-asserted-by":"publisher","first-page":"2048","DOI":"10.1093\/jn\/nxac116","volume":"152","author":"D Cataldi","year":"2022","unstructured":"Cataldi, D., Bennett, J. P., Quon, B. K., Liu, Y. E., Heymsfield, S. B., Kelly, T., & Shepherd, J. A. (2022). Agreement and precision of deuterium dilution for total body water and multicompartment body composition assessment in collegiate athletes. Journal of Nutrition, 152(9), 2048\u20132059. https:\/\/doi.org\/10.1093\/jn\/nxac116","journal-title":"Journal of Nutrition"},{"issue":"14","key":"3170_CR16","doi-asserted-by":"publisher","first-page":"2038","DOI":"10.1080\/09638288.2019.1688872","volume":"43","author":"GM Cesar","year":"2021","unstructured":"Cesar, G. M., Buster, T. W., & Burnfield, J. M. (2021). Test-retest reliability and minimal detectable change of the computerized dynamic posturography PROPRIO for adults with chronic traumatic brain injury. Disability and Rehabilitation, 43(14), 2038\u20132044. https:\/\/doi.org\/10.1080\/09638288.2019.1688872","journal-title":"Disability and Rehabilitation"},{"issue":"3","key":"3170_CR17","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1017\/S1355617703930074","volume":"9","author":"A Collie","year":"2003","unstructured":"Collie, A., Maruff, P., Darby, D. G., & McStephen, M. (2003). The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test\u2013retest intervals. Journal of the International Neuropsychological Society, 9(3), 419\u2013428. https:\/\/doi.org\/10.1017\/S1355617703930074","journal-title":"Journal of the International Neuropsychological Society"},{"key":"3170_CR18","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1016\/B0-12-370870-2\/00162-1","volume-title":"Encyclopedia of gerontology","author":"LM Collins","year":"2007","unstructured":"Collins, L. M. (2007). Research design and methods. In Encyclopedia of gerontology (pp. 433\u2013442). Elsevier. https:\/\/doi.org\/10.1016\/B0-12-370870-2\/00162-1"},{"issue":"1","key":"3170_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/BF02289289","volume":"12","author":"LJ Cronbach","year":"1947","unstructured":"Cronbach, L. J. (1947). Test \u201cReliability\u201d: Its meaning and determination. Psychometrika, 12(1), 1\u201316. https:\/\/doi.org\/10.1007\/BF02289289","journal-title":"Psychometrika"},{"issue":"3","key":"3170_CR20","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1007\/BF02310555","volume":"16","author":"LJ Cronbach","year":"1951","unstructured":"Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297\u2013334. https:\/\/doi.org\/10.1007\/BF02310555","journal-title":"Psychometrika"},{"key":"3170_CR21","volume-title":"the dependability of behavioral measurements: Theory of generalizability for scores and profiles","author":"LJ Cronbach","year":"1972","unstructured":"Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). the dependability of behavioral measurements: Theory of generalizability for scores and profiles. Wiley."},{"issue":"4","key":"3170_CR22","doi-asserted-by":"publisher","first-page":"297","DOI":"10.2165\/00007256-200838040-00003","volume":"38","author":"K Currell","year":"2008","unstructured":"Currell, K., & Jeukendrup, A. E. (2008). Validity, reliability and sensitivity of measures of sporting performance. Sports Medicine, 38(4), 297\u2013316. https:\/\/doi.org\/10.2165\/00007256-200838040-00003","journal-title":"Sports Medicine"},{"issue":"3","key":"3170_CR23","doi-asserted-by":"publisher","first-page":"356","DOI":"10.4085\/1062-6059-49.2.04","volume":"49","author":"G Del Rossi","year":"2014","unstructured":"Del Rossi, G., Malaguti, A., & Del Rossi, S. (2014). Practice effects associated with repeated assessment of a clinical test of reaction time. Journal of Athletic Training, 49(3), 356\u2013359. https:\/\/doi.org\/10.4085\/1062-6059-49.2.04","journal-title":"Journal of Athletic Training"},{"issue":"4","key":"3170_CR24","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1017\/S1355617799544056","volume":"5","author":"SS Dikmen","year":"1999","unstructured":"Dikmen, S. S., Heaton, R. K., Grant, I., & Temkin, N. R. (1999). Test-retest reliability and practice effects of expanded Halstead-Reitan neuropsychological test battery. Journal of the International Neuropsychological Society\u00a0: JINS, 5(4), 346\u2013356.","journal-title":"Journal of the International Neuropsychological Society\u00a0: JINS"},{"key":"3170_CR25","doi-asserted-by":"publisher","unstructured":"Do\u011fan, N. \u00d6. (2018). Bland-Altman analysis: A paradigm to understand correlation and agreement. https:\/\/doi.org\/10.1016\/j.tjem.2018.09.001","DOI":"10.1016\/j.tjem.2018.09.001"},{"issue":"1","key":"3170_CR26","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1186\/s12877-017-0634-x","volume":"17","author":"L Donath","year":"2017","unstructured":"Donath, L., Ludyga, S., Hammes, D., Rossmeissl, A., Andergassen, N., Zahner, L., & Faude, O. (2017). Absolute and relative reliability of acute effects of aerobic exercise on executive function in seniors. BMC Geriatrics, 17(1), 247. https:\/\/doi.org\/10.1186\/s12877-017-0634-x","journal-title":"BMC Geriatrics"},{"issue":"5","key":"3170_CR27","doi-asserted-by":"publisher","first-page":"624","DOI":"10.1080\/13803390490918589","volume":"27","author":"V Eileen De Monte","year":"2005","unstructured":"Eileen De Monte, V., Malke Geffen, G., & Kwapil, K. (2005). Test-retest reliability and practice effects of a rapid screen of mild traumatic brain injury. Journal of Clinical and Experimental Neuropsychology, 27(5), 624\u2013632. https:\/\/doi.org\/10.1080\/13803390490918589","journal-title":"Journal of Clinical and Experimental Neuropsychology"},{"issue":"2","key":"3170_CR28","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1080\/13854046.2012.659219","volume":"26","author":"E Estevis","year":"2012","unstructured":"Estevis, E., Basso, M. R., & Combs, D. (2012). Effects of practice on the Wechsler adult intelligence scale-IV across 3- and 6-month intervals. The Clinical Neuropsychologist, 26(2), 239\u2013254. https:\/\/doi.org\/10.1080\/13854046.2012.659219","journal-title":"The Clinical Neuropsychologist"},{"key":"3170_CR29","unstructured":"Food and Drug Administration (FDA). (2001). Guidance for industry: Bioanalytical method validation. http:\/\/Ww.Fda.Gov\/Cder\/Guidance\/Index.Htm"},{"issue":"6","key":"3170_CR30","doi-asserted-by":"publisher","first-page":"1901","DOI":"10.1519\/JSC.0b013e3181887e3f","volume":"22","author":"AJ Galpin","year":"2008","unstructured":"Galpin, A. J., Li, Y., Lohnes, C. A., & Schilling, B. K. (2008). A 4-week choice foot speed and choice reaction training program improves agility in previously non-agility trained, but active men and women. Journal of Strength and Conditioning Research, 22(6), 1901\u20131907. https:\/\/doi.org\/10.1519\/JSC.0b013e3181887e3f","journal-title":"Journal of Strength and Conditioning Research"},{"key":"3170_CR31","doi-asserted-by":"publisher","unstructured":"Gamer, M., & Lemon, J. (2019). Various coefficients of interrater reliability and agreement. https:\/\/doi.org\/10.32614\/CRAN.package.irr","DOI":"10.32614\/CRAN.package.irr"},{"issue":"2","key":"3170_CR32","doi-asserted-by":"publisher","first-page":"141","DOI":"10.11613\/BM.2015.015","volume":"25","author":"D Giavarina","year":"2015","unstructured":"Giavarina, D. (2015). Understanding Bland altman analysis. Biochemia Medica, 25(2), 141\u2013151. https:\/\/doi.org\/10.11613\/BM.2015.015","journal-title":"Biochemia Medica"},{"key":"3170_CR33","doi-asserted-by":"publisher","unstructured":"Gibbon, J. R., Parry, S. W., Witham, M. D., Yarnall, A., & Frith, J. (2022). Feasibility, reliability and safety of self-assessed orthostatic blood pressure at home. Age and Ageing, 51(7). https:\/\/doi.org\/10.1093\/ageing\/afac153","DOI":"10.1093\/ageing\/afac153"},{"issue":"1","key":"3170_CR34","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1186\/s40798-020-00260-z","volume":"6","author":"J Grgic","year":"2020","unstructured":"Grgic, J., Lazinica, B., Schoenfeld, B. J., & Pedisic, Z. (2020). Test\u2013retest reliability of the one-repetition maximum (1RM) strength assessment: A systematic review. Sports Medicine - Open, 6(1), 31. https:\/\/doi.org\/10.1186\/s40798-020-00260-z","journal-title":"Sports Medicine - Open"},{"issue":"5","key":"3170_CR35","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6579\/ab86d6","volume":"41","author":"S Haghayegh","year":"2020","unstructured":"Haghayegh, S., Kang, H.-A., Khoshnevis, S., Smolensky, M. H., & Diller, K. R. (2020). A comprehensive guideline for Bland\u2013Altman and intra class correlation calculations to properly compare two methods of measurement and interpret findings. Physiological Measurement, 41(5), 055012. https:\/\/doi.org\/10.1088\/1361-6579\/ab86d6","journal-title":"Physiological Measurement"},{"issue":"2","key":"3170_CR36","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1111\/j.1745-3992.1991.tb00195.x","volume":"10","author":"LM Harvill","year":"1991","unstructured":"Harvill, L. M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33\u201341. https:\/\/doi.org\/10.1111\/j.1745-3992.1991.tb00195.x","journal-title":"Educational Measurement: Issues and Practice"},{"issue":"1","key":"3170_CR37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2165\/00007256-200030010-00001","volume":"30","author":"WG Hopkins","year":"2000","unstructured":"Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), 1\u201315. https:\/\/doi.org\/10.2165\/00007256-200030010-00001","journal-title":"Sports Medicine"},{"issue":"1","key":"3170_CR38","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1177\/1357633X20914445","volume":"28","author":"K Hwang","year":"2022","unstructured":"Hwang, K., De Silva, A., Simpson, J. A., LoGiudice, D., Engel, L., Gilbert, A. S., Croy, S., & Haralambous, B. (2022). Video-interpreting for cognitive assessments: An intervention study and micro-costing analysis. Journal of Telemedicine and Telecare, 28(1), 58\u201367. https:\/\/doi.org\/10.1177\/1357633X20914445","journal-title":"Journal of Telemedicine and Telecare"},{"key":"3170_CR39","unstructured":"International Organization of Standardization. (2023). ISO 5725-1:2023(en) accuracy (trueness and precision) of measurement methods and results - Part 1: General principles and definitions. https:\/\/Www.Iso.Org\/Obp\/Ui\/#iso:Std:Iso:5725:-1:Ed-2:V1:En"},{"issue":"4","key":"3170_CR40","doi-asserted-by":"publisher","DOI":"10.1016\/j.arrct.2025.100501","volume":"7","author":"P Jogie","year":"2025","unstructured":"Jogie, P., Rahja, M., van den Berg, M., Salway, J., & Laver, K. (2025). Evaluating goal attainment scaling for individuals with mild cognitive impairment or dementia in rehabilitation settings. Archives of Rehabilitation Research and Clinical Translation, 7(4), 100501. https:\/\/doi.org\/10.1016\/j.arrct.2025.100501","journal-title":"Archives of Rehabilitation Research and Clinical Translation"},{"issue":"2","key":"3170_CR41","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","volume":"15","author":"TK Koo","year":"2016","unstructured":"Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155\u2013163. https:\/\/doi.org\/10.1016\/j.jcm.2016.02.012","journal-title":"Journal of Chiropractic Medicine"},{"issue":"6","key":"3170_CR42","doi-asserted-by":"publisher","first-page":"701","DOI":"10.1016\/j.jclinepi.2010.12.001","volume":"64","author":"J Kottner","year":"2011","unstructured":"Kottner, J., & Streiner, D. L. (2011). The difference between reliability and agreement. Journal of Clinical Epidemiology, 64(6), 701\u2013702. https:\/\/doi.org\/10.1016\/j.jclinepi.2010.12.001","journal-title":"Journal of Clinical Epidemiology"},{"key":"3170_CR43","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1016\/B978-0-12-801504-9.00009-X","volume-title":"The measurement of health and health status","author":"PFM Krabbe","year":"2017","unstructured":"Krabbe, P. F. M. (2017). Classical test theory. In The measurement of health and health status (pp. 153\u2013170). Elsevier. https:\/\/doi.org\/10.1016\/B978-0-12-801504-9.00009-X"},{"issue":"6","key":"3170_CR44","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1191\/1740774504cn057oa","volume":"1","author":"JM Lachin","year":"2004","unstructured":"Lachin, J. M. (2004). The role of measurement reliability in clinical trials. Clinical Trials, 1(6), 553\u2013566. https:\/\/doi.org\/10.1191\/1740774504cn057oa","journal-title":"Clinical Trials"},{"issue":"2","key":"3170_CR45","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1177\/1356336X9800400205","volume":"4","author":"K Lamb","year":"1998","unstructured":"Lamb, K. (1998). Test-retest reliability in quantitative physical education research: A commentary. European Physical Education Review, 4(2), 145\u2013152.","journal-title":"European Physical Education Review"},{"issue":"6","key":"3170_CR46","doi-asserted-by":"publisher","first-page":"579","DOI":"10.4085\/1062-6050-0391.22","volume":"58","author":"LB Lempke","year":"2023","unstructured":"Lempke, L. B., Shumski, E. J., Prato, T. A., & Lynall, R. C. (2023). Reliability and minimal detectable change of the standardized assessment of reaction time. Journal of Athletic Training, 58(6), 579\u2013587. https:\/\/doi.org\/10.4085\/1062-6050-0391.22","journal-title":"Journal of Athletic Training"},{"issue":"1","key":"3170_CR47","doi-asserted-by":"publisher","first-page":"255","DOI":"10.2307\/2532051","volume":"45","author":"LI-K Lin","year":"1989","unstructured":"Lin, L. I.-K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255\u2013268 https:\/\/www.jstor.org\/stable\/2532051","journal-title":"Biometrics"},{"issue":"2","key":"3170_CR48","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1111\/j.1748-6653.2011.02023.x","volume":"6","author":"AHY Lo","year":"2012","unstructured":"Lo, A. H. Y., Humphreys, M., Byrne, G. J., & Pachana, N. A. (2012). Test\u2013retest reliability and practice effects of the Wechsler Memory Scale-III. Journal of Neuropsychology, 6(2), 212\u2013231. https:\/\/doi.org\/10.1111\/j.1748-6653.2011.02023.x","journal-title":"Journal of Neuropsychology"},{"key":"3170_CR49","volume-title":"Statistical theories of mental test scores","author":"FM Lord","year":"1986","unstructured":"Lord, F. M., & Novick, M. R. (1986). Statistical theories of mental test scores. Addison-Wesley."},{"key":"3170_CR50","doi-asserted-by":"publisher","DOI":"10.1016\/j.nicl.2020.102168","volume":"26","author":"PA Luque Laguna","year":"2020","unstructured":"Luque Laguna, P. A., Combes, A. J. E., Streffer, J., Einstein, S., Timmers, M., Williams, S. C. R., & Dell\u2019Acqua, F. (2020). Reproducibility, reliability and variability of FA and MD in the older healthy population: A test-retest multiparametric analysis. NeuroImage: Clinical, 26, 102168. https:\/\/doi.org\/10.1016\/j.nicl.2020.102168","journal-title":"NeuroImage: Clinical"},{"issue":"8","key":"3170_CR51","doi-asserted-by":"publisher","first-page":"1237","DOI":"10.1123\/jsr.2021-0021","volume":"30","author":"RC Lynall","year":"2021","unstructured":"Lynall, R. C., Johnson, R. S., Lempke, L. B., & Schmidt, J. D. (2021). Test\u2013retest reliability of a functional reaction time assessment battery. Journal of Sport Rehabilitation, 30(8), 1237\u20131241. https:\/\/doi.org\/10.1123\/jsr.2021-0021","journal-title":"Journal of Sport Rehabilitation"},{"issue":"1","key":"3170_CR52","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/j.pscychresns.2006.01.008","volume":"147","author":"S Marenco","year":"2006","unstructured":"Marenco, S., Rawlings, R., Rohde, G. K., Barnett, A. S., Honea, R. A., Pierpaoli, C., & Weinberger, D. R. (2006). Regional distribution of measurement error in diffusion tensor imaging. Psychiatry Research: Neuroimaging, 147(1), 69\u201378. https:\/\/doi.org\/10.1016\/j.pscychresns.2006.01.008","journal-title":"Psychiatry Research: Neuroimaging"},{"issue":"20","key":"3170_CR53","doi-asserted-by":"publisher","first-page":"17020","DOI":"10.1007\/s12144-022-02886-6","volume":"42","author":"A Medvedeva","year":"2023","unstructured":"Medvedeva, A., Galli, G., Petrovskaya, A., & Medvedev, O. N. (2023). A novel application of generalizability theory to evaluate the reliability of the recognition memory test. Current Psychology, 42(20), 17020\u201317032. https:\/\/doi.org\/10.1007\/s12144-022-02886-6","journal-title":"Current Psychology"},{"key":"3170_CR54","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1016\/B978-0-08-044894-7.00235-9","volume-title":"International encyclopedia of education","author":"MD Miller","year":"2009","unstructured":"Miller, M. D. (2009). Classical test theory reliability. In International encyclopedia of education (3rd ed., pp. 27\u201330). https:\/\/doi.org\/10.1016\/B978-0-08-044894-7.00235-9","edition":"3"},{"issue":"4","key":"3170_CR55","doi-asserted-by":"publisher","first-page":"314","DOI":"10.1136\/bjsm.31.4.314","volume":"31","author":"AM Nevill","year":"1997","unstructured":"Nevill, A. M., & Atkinson, G. (1997). Assessing agreement between measurements recorded on a ratio scale in sports medicine and sports science. British Journal of Sports Medicine, 31(4), 314\u2013318. https:\/\/doi.org\/10.1136\/bjsm.31.4.314","journal-title":"British Journal of Sports Medicine"},{"issue":"6","key":"3170_CR56","doi-asserted-by":"publisher","first-page":"970","DOI":"10.1097\/HJH.0000000000003975","volume":"43","author":"T Palsma","year":"2025","unstructured":"Palsma, T., Claassen, J. A. H. R., Richard, E., & De Heus, R. A. A. (2025). Day-to-day blood pressure variability in older persons \u2013 Optimizing measurement. Journal of Hypertension, 43(6), 970\u2013975. https:\/\/doi.org\/10.1097\/HJH.0000000000003975","journal-title":"Journal of Hypertension"},{"issue":"2","key":"3170_CR57","doi-asserted-by":"publisher","first-page":"260","DOI":"10.1177\/0272989X10380925","volume":"31","author":"M Palta","year":"2011","unstructured":"Palta, M., Chen, H.-Y., Kaplan, R. M., Feeny, D., Cherepanov, D., & Fryback, D. G. (2011). Standard error of measurement of 5 health utility indexes across the range of health for use in estimating reliability and responsiveness. Medical Decision Making, 31(2), 260\u2013269. https:\/\/doi.org\/10.1177\/0272989X10380925","journal-title":"Medical Decision Making"},{"issue":"3","key":"3170_CR58","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1590\/s1806-37562018000000164","volume":"44","author":"CM Patino","year":"2018","unstructured":"Patino, C. M., & Ferreira, J. C. (2018). Internal and external validity: Can you apply research study results to your patients? Jornal Brasileiro de Pneumologia, 44(3), 183\u2013183. https:\/\/doi.org\/10.1590\/s1806-37562018000000164","journal-title":"Jornal Brasileiro de Pneumologia"},{"key":"3170_CR59","doi-asserted-by":"publisher","unstructured":"Piscicelli, C., Barra, J., Davoine, P., Chrispin, A., Nadeau, S., & P\u00e9rennou, D. (2015). Inter- and intra-rater reliability of the visual vertical in subacute stroke. Stroke, 46(7), 1979\u20131983. https:\/\/doi.org\/10.1161\/STROKEAHA.115.009610","DOI":"10.1161\/STROKEAHA.115.009610"},{"key":"3170_CR60","doi-asserted-by":"publisher","DOI":"10.1016\/j.biopsycho.2022.108370","volume":"172","author":"J Ramos-Loyo","year":"2022","unstructured":"Ramos-Loyo, J., Gonz\u00e1lez-Garrido, A. A., Llamas-Alonso, L. A., & Sequeira, H. (2022). Sex differences in cognitive processing: An integrative review of electrophysiological findings. Biological Psychology, 172, 108370. https:\/\/doi.org\/10.1016\/j.biopsycho.2022.108370","journal-title":"Biological Psychology"},{"issue":"3\u20134","key":"3170_CR61","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1080\/138255890969375","volume":"13","author":"S Reimers","year":"2006","unstructured":"Reimers, S., & Maylor, E. A. (2006). Gender effects on reaction time variability and trial-to-trial performance: Reply to Deary and Der (2005). Aging, Neuropsychology, and Cognition, 13(3\u20134), 479\u2013489. https:\/\/doi.org\/10.1080\/138255890969375","journal-title":"Aging, Neuropsychology, and Cognition"},{"issue":"3","key":"3170_CR62","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1159\/000493750","volume":"8","author":"LMJ Sanders","year":"2018","unstructured":"Sanders, L. M. J., Hortob\u00e1gyi, T., Balasingham, M., Van der Zee, E. A., & van Heuvelen, M. J. G. (2018). Psychometric properties of a flanker task in a sample of patients with dementia: A pilot study. Dementia and Geriatric Cognitive Disorders Extra, 8(3), 382\u2013392. https:\/\/doi.org\/10.1159\/000493750","journal-title":"Dementia and Geriatric Cognitive Disorders Extra"},{"key":"3170_CR63","doi-asserted-by":"publisher","DOI":"10.1016\/j.ridd.2022.104416","volume":"133","author":"EK Schworer","year":"2023","unstructured":"Schworer, E. K., Soltani, A., Altaye, M., Fidler, D. J., & Esbensen, A. J. (2023). Cognitive flexibility assessment in youth with down syndrome: Reliability, practice effects, and validity. Research in Developmental Disabilities, 133, 104416. https:\/\/doi.org\/10.1016\/j.ridd.2022.104416","journal-title":"Research in Developmental Disabilities"},{"key":"3170_CR64","doi-asserted-by":"publisher","unstructured":"Seamon, B. A., Kautz, S. A., Bowden, M. G., & Velozo, C. A. (2022). Revisiting the concept of minimal detectable change for patient-reported outcome measures. Physical Therapy, 102(8). https:\/\/doi.org\/10.1093\/ptj\/pzac068","DOI":"10.1093\/ptj\/pzac068"},{"issue":"2","key":"3170_CR65","doi-asserted-by":"publisher","first-page":"420","DOI":"10.1037\/0033-2909.86.2.420","volume":"86","author":"PE Shrout","year":"1979","unstructured":"Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420\u2013428. https:\/\/doi.org\/10.1037\/0033-2909.86.2.420","journal-title":"Psychological Bulletin"},{"issue":"5","key":"3170_CR66","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0285307","volume":"18","author":"T Taule","year":"2023","unstructured":"Taule, T., Eide, I. S., Fj\u00e6r, L., Myrberget, M.-A., Oseland, M. S., Rens\u00e5, M. A., Revheim, T., Tysnes, O.-B., A\u00dfmus, J., & Rekand, T. (2023). Norwegian version of the Edinburgh cognitive and behavioural ALS screen: Construct validity, internal consistency, inter-rater, and test-retest reliability. PLOS ONE, 18(5), e0285307. https:\/\/doi.org\/10.1371\/journal.pone.0285307","journal-title":"PLOS ONE"},{"issue":"1","key":"3170_CR67","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1177\/107319119800500110","volume":"5","author":"ME Theisen","year":"1998","unstructured":"Theisen, M. E., Rapport, L. J., Axelrod, B. N., & Brines, D. B. (1998). Effects of practice in repeated administrations of the Wechsler memory scale-revised in normal adults. Assessment, 5(1), 85\u201392. https:\/\/doi.org\/10.1177\/107319119800500110","journal-title":"Assessment"},{"issue":"3","key":"3170_CR68","doi-asserted-by":"publisher","first-page":"434","DOI":"10.1002\/j.1556-6676.1992.tb01631.x","volume":"70","author":"B Thompson","year":"1992","unstructured":"Thompson, B. (1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling & Development, 70(3), 434\u2013438. https:\/\/doi.org\/10.1002\/j.1556-6676.1992.tb01631.x","journal-title":"Journal of Counseling & Development"},{"issue":"6","key":"3170_CR69","doi-asserted-by":"publisher","first-page":"2123","DOI":"10.1213\/ANE.0000000000002924","volume":"126","author":"TR Vetter","year":"2018","unstructured":"Vetter, T. R., & Schober, P. (2018). Agreement analysis: What he said, she said versus you said. Anesthesia & Analgesia, 126(6), 2123\u20132128. https:\/\/doi.org\/10.1213\/ANE.0000000000002924","journal-title":"Anesthesia & Analgesia"},{"key":"3170_CR70","doi-asserted-by":"publisher","unstructured":"Warneke, K., Gronwald, T., Wallot, S., Magno, A., Hillebrecht, M., & Wirth, K. (2025a). discussion on the validity of commonly used reliability indices in sports medicine and exercise science - A critical review with data simulations. European Journal of Applied Physiology. https:\/\/doi.org\/10.1007\/s00421-025-05720-6","DOI":"10.1007\/s00421-025-05720-6"},{"key":"3170_CR71","doi-asserted-by":"publisher","unstructured":"Warneke, K., Lebelt, N., Wallot, S., Lohmann, L. H., Afonso, J., & Herbsleb, M. (2025b). (Mis-)understanding external validity \u2013 A narrative literature review with guideline synthesis on how to report reliability in sport and health science. Football Studies, 100011. https:\/\/doi.org\/10.1016\/j.footst.2025.100011","DOI":"10.1016\/j.footst.2025.100011"},{"key":"3170_CR72","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1016\/j.neuroscience.2025.12.007","volume":"593","author":"K Warneke","year":"2025","unstructured":"Warneke, K., Oraze, M., Herbsleb, M., Afonso, J., & Wallot, S. (2025c). Do Sex differences influence test habituation and internal data validity in neurocognitive testing? A blinded measurement error analysis. Neuroscience, 593, 106\u2013121.","journal-title":"Neuroscience"},{"issue":"8","key":"3170_CR73","doi-asserted-by":"publisher","first-page":"e70117","DOI":"10.1111\/ejn.70117","volume":"61","author":"K Warneke","year":"2025","unstructured":"Warneke, K., Oraze, M., Pl\u00f6schberger, G., Herbsleb, M., Afonso, J., & Wallot, S. (2025d). When testing becomes learning - Underscoring the relevance of habituation to improve internal validity of common neurocognitive tests. European Journal of Neuroscience, 61(8), e70117.","journal-title":"European Journal of Neuroscience"},{"key":"3170_CR74","doi-asserted-by":"publisher","unstructured":"Warneke, K., Siegel, S. D., Afonso, J., & Wallot, S. (2025e). What the mean absolute percentage error (MAPE) should adopt from Bland\u2013Altman analyses. German Journal of Exercise and Sport Research. https:\/\/doi.org\/10.1007\/s12662-025-01084-3","DOI":"10.1007\/s12662-025-01084-3"},{"issue":"1","key":"3170_CR75","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1519\/15184.1","volume":"19","author":"JP Weir","year":"2005","unstructured":"Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. The Journal of Strength and Conditioning Research, 19(1), 231. https:\/\/doi.org\/10.1519\/15184.1","journal-title":"The Journal of Strength and Conditioning Research"},{"key":"3170_CR76","doi-asserted-by":"publisher","unstructured":"Westheim, F., Gl\u00f8ersen, \u00d8., Harper, D., Laugsand, H., & Eriksrud, O. (2023). Reliability of phase-specific outcome measurements in change-of-direction tests using a motorized resistance device. Frontiers in Sports and Active Living, 5, 1212414. https:\/\/doi.org\/10.3389\/fspor.2023.1212414","DOI":"10.3389\/fspor.2023.1212414"},{"issue":"2","key":"3170_CR77","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1080\/13803395.2018.1526888","volume":"41","author":"N White","year":"2019","unstructured":"White, N., Flannery, L., McClintock, A., & Machado, L. (2019). Repeated computerized cognitive testing: Performance shifts and test\u2013retest reliability in healthy older adults. Journal of Clinical and Experimental Neuropsychology, 41(2), 179\u2013191. https:\/\/doi.org\/10.1080\/13803395.2018.1526888","journal-title":"Journal of Clinical and Experimental Neuropsychology"},{"issue":"3","key":"3170_CR78","first-page":"529","volume":"19","author":"J Wilke","year":"2020","unstructured":"Wilke, J., & Vogel, O. (2020). Computerized cognitive training with minimal motor component improves lower limb choice-reaction time. Journal of Sports Science & Medicine, 19(3), 529\u2013534.","journal-title":"Journal of Sports Science & Medicine"},{"key":"3170_CR79","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1016\/j.ptsp.2020.02.016","volume":"43","author":"J Wilke","year":"2020","unstructured":"Wilke, J., Vogel, O., & Ungricht, S. (2020). Can we measure perceptual-cognitive function during athletic movement? A framework for and reliability of a sports-related testing battery. Physical Therapy in Sport, 43, 120\u2013126. https:\/\/doi.org\/10.1016\/j.ptsp.2020.02.016","journal-title":"Physical Therapy in Sport"},{"key":"3170_CR80","doi-asserted-by":"publisher","first-page":"79","DOI":"10.3354\/cr030079","volume":"30","author":"C Willmott","year":"2005","unstructured":"Willmott, C., & Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30, 79\u201382. https:\/\/doi.org\/10.3354\/cr030079","journal-title":"Climate Research"},{"issue":"2","key":"3170_CR81","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1177\/096228029900800202","volume":"8","author":"EM Wright","year":"1999","unstructured":"Wright, E. M., & Royston, P. (1999). Calculating reference intervals for laboratory measurements. Statistical Methods in Medical Research, 8(2), 93\u2013112. https:\/\/doi.org\/10.1177\/096228029900800202","journal-title":"Statistical Methods in Medical Research"}],"container-title":["Behavior Research Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-026-03170-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13428-026-03170-2","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-026-03170-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T16:13:53Z","timestamp":1793376833000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13428-026-03170-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,30]]},"references-count":81,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2026,11]]}},"alternative-id":["3170"],"URL":"https:\/\/doi.org\/10.3758\/s13428-026-03170-2","relation":{},"ISSN":["1554-3528"],"issn-type":[{"value":"1554-3528","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,30]]},"assertion":[{"value":"8 January 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 August 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 September 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The study does not include any participants. Therefore, no ethical approval is available.","order":1,"name":"Ethics","label":"Ethical approval","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The study does not include any participants. Therefore, no consent to participate is available.","order":2,"name":"Ethics","label":"Consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The study does not include any participants. Therefore, no consent for publication is available.","order":3,"name":"Ethics","label":"Consent for publication","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"None of the included authors has a conflict of interest to declare.","order":4,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The code (and software) is accessible via the following link:\n                      \n                      as well as on the authors' Open Science Framework profile (osf.io\/gbcea). The study was not preregistered.","order":5,"name":"Ethics","label":"Open practices statement and code availability","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"307"}}