{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T06:29:28Z","timestamp":1775197768845,"version":"3.50.1"},"reference-count":60,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,2,7]],"date-time":"2023-02-07T00:00:00Z","timestamp":1675728000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Rapid guessing is an aberrant response behavior that commonly occurs in low-stakes assessments with little to no formal consequences for students. Recently, the availability of response time (RT) information in computer-based assessments has motivated researchers to develop various methods to detect rapidly guessed responses systematically. These methods often require researchers to identify an RT threshold subjectively for each item that could distinguish rapid guessing behavior from solution behavior. In this study, we propose a data-driven approach based on random search and genetic algorithm to search for the optimal RT threshold within a predefined search space. We used response data from a low-stakes math assessment administered to over 5000 students in 658 schools across the United States. As we demonstrated how to use our data-driven approach, we also compared its performance with those of the existing threshold-setting methods. The results show that the proposed method could produce viable RT thresholds for detecting rapid guessing in low-stakes assessments. Moreover, compared with the other threshold-setting methods, the proposed method yielded more liberal RT thresholds, flagging a larger number of responses. Implications for practice and directions for future research were discussed.<\/jats:p>","DOI":"10.3390\/a16020089","type":"journal-article","created":{"date-parts":[[2023,2,8]],"date-time":"2023-02-08T04:57:41Z","timestamp":1675832261000},"page":"89","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Rapid Guessing in Low-Stakes Assessments: Finding the Optimal Response Time Threshold with Random Search and Genetic Algorithm"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5853-1267","authenticated-orcid":false,"given":"Okan","family":"Bulut","sequence":"first","affiliation":[{"name":"Centre for Research in Applied Measurement and Evaluation, University of Alberta, Edmonton, AB T6G 2G5, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0861-9225","authenticated-orcid":false,"given":"Guher","family":"Gorgun","sequence":"additional","affiliation":[{"name":"Measurement, Evaluation, and Data Science, University of Alberta, Edmonton, AB T6G 2G5, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9622-3780","authenticated-orcid":false,"given":"Tarid","family":"Wongvorachan","sequence":"additional","affiliation":[{"name":"Measurement, Evaluation, and Data Science, University of Alberta, Edmonton, AB T6G 2G5, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6717-5620","authenticated-orcid":false,"given":"Bin","family":"Tan","sequence":"additional","affiliation":[{"name":"Measurement, Evaluation, and Data Science, University of Alberta, Edmonton, AB T6G 2G5, Canada"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"267","DOI":"10.3102\/0162373718759600","article-title":"The influence of rapidly guessed item responses on teacher value-added estimates: Implications for policy and practice","volume":"40","author":"Jensen","year":"2018","journal-title":"Educ. Eval. Policy Anal."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1037\/a0034716","article-title":"The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment","volume":"106","author":"Goldhammer","year":"2014","journal-title":"J. Educ. Psychol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"847","DOI":"10.1177\/0013164421991211","article-title":"A polytomous scoring approach to handle not-reached items in low-stakes assessments","volume":"81","author":"Gorgun","year":"2021","journal-title":"Educ. Psychol. Meas."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1080\/08957347.2016.1171766","article-title":"A new procedure for detection of students\u2019 rapid guessing responses using response time","volume":"29","author":"Guo","year":"2016","journal-title":"Appl. Meas. Educ."},{"key":"ref_5","unstructured":"Wise, S.L., and Ma, L. (2012, January 13\u201317). Setting response time thresholds for a cat item pool: The normative threshold method. Proceedings of the Annual Meeting of the National Council on Measurement in Education, Vancouver, BC, Canada."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1080\/08957347.2017.1353992","article-title":"A general approach to measuring test-taking effort on computer-based tests","volume":"30","author":"Wise","year":"2017","journal-title":"Appl. Meas. Educ."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1080\/08957347.2019.1660350","article-title":"An information-based approach to identifying rapid-guessing thresholds","volume":"32","author":"Wise","year":"2019","journal-title":"Appl. Meas. Educ."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1080\/08957347.2020.1789141","article-title":"Can culture be a salient predictor of test-taking engagement? An analysis of differential noneffortful responding on an international college-level assessment of critical thinking","volume":"33","author":"Rios","year":"2020","journal-title":"Appl. Meas. Educ."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1186\/s40536-021-00100-w","article-title":"Comparing different response time threshold setting methods to detect low effort on a large-scale assessment","volume":"9","author":"Soland","year":"2021","journal-title":"Large-Scale Assessments Educ."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1186\/s40536-021-00110-8","article-title":"Does the choice of response time threshold procedure substantially affect inferences concerning the identification and exclusion of rapid guessing responses? A meta-analysis","volume":"9","author":"Rios","year":"2021","journal-title":"Large-Scale Assessments Educ."},{"key":"ref_11","first-page":"147","article-title":"Rapid guessing rates across administration mode and test setting","volume":"62","author":"Kroehne","year":"2020","journal-title":"Psychol. Test Assess. Model."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1533","DOI":"10.3389\/fpsyg.2019.01533","article-title":"The onset of rapid-guessing behavior over the course of testing time: A matter of motivation and cognitive resources","volume":"10","author":"Lindner","year":"2019","journal-title":"Front. Psychol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"752","DOI":"10.2307\/3069616","article-title":"Limitations in the use of achievement tests as measures of educators\u2019 productivity","volume":"37","author":"Koretz","year":"2002","journal-title":"J. Hum. Resour."},{"key":"ref_14","unstructured":"AERA, APA, and NCME (2014). Chapter 1: Validity. In Standards for Educational and Psychological Testing, American Educational Research Association."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/ets2.12067","article-title":"Measuring motivation in low-stakes assessments","volume":"2015","author":"Finn","year":"2015","journal-title":"ETS Res. Rep. Ser."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1080\/08957347.2013.739453","article-title":"An investigation of examinee test-taking effort on a large-scale assessment","volume":"26","author":"Setzer","year":"2013","journal-title":"Appl. Meas. Educ."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1207\/s15326977ea1001_1","article-title":"Low examinee effort in low-stakes assessment: Problems and potential solutions","volume":"10","author":"Wise","year":"2005","journal-title":"Educ. Assess."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"2","DOI":"10.3389\/fams.2019.00002","article-title":"Invariance of the response processes between gender and modes in an assessment of reading","volume":"5","author":"Kroehne","year":"2019","journal-title":"Front. Appl. Math. Stat."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1080\/08957347.2011.555217","article-title":"Two approaches for identifying low-motivated students in a low-stakes assessment context","volume":"24","author":"Swerdzewski","year":"2011","journal-title":"Appl. Meas. Educ."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1207\/s15324818ame1802_2","article-title":"Response time effort: A new measure of examinee motivation in computer-based tests","volume":"18","author":"Wise","year":"2005","journal-title":"Appl. Meas. Educ."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Nagy, G., Ulitzsch, E., and Lindner, M.A. (2022). The role of rapid guessing and test-taking persistence in modelling test-taking engagement. J. Comput. Assist. Learn.","DOI":"10.1111\/jcal.12719"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1177\/00131644211003640","article-title":"Assessing the accuracy of parameter estimates in the presence of rapid guessing misclassifications","volume":"82","author":"Rios","year":"2022","journal-title":"Educ. Psychol. Meas."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"220","DOI":"10.3389\/fpsyg.2010.00220","article-title":"Random responding from participants is a threat to the validity of social science research results","volume":"1","author":"Osborne","year":"2011","journal-title":"Front. Psychol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1080\/10627191003673216","article-title":"Examinee noneffort and the validity of program assessment results","volume":"15","author":"Wise","year":"2010","journal-title":"Educ. Assess."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1080\/10627197.2022.2110465","article-title":"To What Degree Does Rapid Guessing Distort Aggregated Test Scores? A Meta-analytic Investigation","volume":"27","author":"Rios","year":"2022","journal-title":"Educ. Assess."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1177\/0013164421990429","article-title":"Is differential noneffortful responding associated with type I error in measurement invariance testing?","volume":"81","author":"Rios","year":"2021","journal-title":"Educ. Psychol. Meas."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1177\/01466216221108991","article-title":"Investigating the Effect of Differential Rapid Guessing on Population Invariance in Equating","volume":"46","author":"Deng","year":"2022","journal-title":"Appl. Psychol. Meas."},{"key":"ref_28","unstructured":"Schnipke, D.L. (1996). Assessing Speededness in Computer-Based Tests Using Item Response Times, The Johns Hopkins University."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1207\/s15324818ame1902_2","article-title":"An investigation of the differential effort received by items on a low-stakes computer-based test","volume":"19","author":"Wise","year":"2006","journal-title":"Appl. Meas. Educ."},{"key":"ref_30","unstructured":"Hadiana, D., Hayat, B., and Tola, B. (2021). International Conference on Educational Assessment and Policy (ICEAP 2020), Atlantis Press."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1177\/0013164406294779","article-title":"Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior","volume":"67","author":"Kong","year":"2007","journal-title":"Educ. Psychol. Meas."},{"key":"ref_32","unstructured":"De Ayala, R.J. (2009). The Theory and Practice of Item Response Theory, Guilford Press. Methodology in the Social Sciences."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1126","DOI":"10.1007\/s11336-016-9537-6","article-title":"Modelling conditional dependence between response time and accuracy","volume":"82","author":"Bolsinova","year":"2017","journal-title":"Psychometrika"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1082","DOI":"10.3102\/0034654317726529","article-title":"Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review","volume":"87","author":"Gierl","year":"2017","journal-title":"Rev. Educ. Res."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1080\/00220973.2019.1629577","article-title":"The Effect of the Most-Attractive-Distractor Location on Multiple-Choice Item Difficulty","volume":"88","author":"Shin","year":"2020","journal-title":"J. Exp. Educ."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1021\/ed085p572","article-title":"Does the answer order matter on multiple-choice exams?","volume":"85","author":"Tellinghuisen","year":"2008","journal-title":"J. Chem. Educ."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s40536-014-0008-1","article-title":"Using response time to investigate students\u2019 test-taking behaviors in a NAEP computer-based study","volume":"2","author":"Lee","year":"2014","journal-title":"Large-Scale Assess. Educ."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v053.i04","article-title":"GA: A Package for Genetic Algorithms in R","volume":"53","author":"Scrucca","year":"2013","journal-title":"J. Stat. Softw."},{"key":"ref_39","unstructured":"Richter, J. (2023, February 05). Randomsearch: Random Search for Expensive Functions. Available online: https:\/\/jakob-r.de\/randomsearch\/index.html."},{"key":"ref_40","unstructured":"Chalmers, P. (2023, February 05). Mirt: Multidimensional Item Response Theory. Available online: https:\/\/cran.r-project.org\/web\/packages\/mirt\/mirt.pdf."},{"key":"ref_41","unstructured":"R Core Team (2021). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_42","unstructured":"Zabinsky, Z.B. (2009). . Random Search Algorithms, Department of Industrial and Systems Engineering, University of Washington."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Floudas, C.A., and Pardalos, P.M. (2009). Encyclopedia of Optimization, Springer.","DOI":"10.1007\/978-0-387-74759-0"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Ramasubramanian, K., and Singh, A. (2018). Machine Learning Using R: With Time Series and Industry-Based Use Cases in R, Springer Science+Business Media.","DOI":"10.1007\/978-1-4842-4215-5"},{"key":"ref_45","first-page":"281","article-title":"Random search for hyper-parameter optimization","volume":"13","author":"Bergstra","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"66","DOI":"10.5243\/jsswr.2010.6","article-title":"An introduction to using multidimensional item response theory to assess latent factor structures","volume":"1","author":"Osteen","year":"2010","journal-title":"J. Soc. Soc. Work Res."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1038\/scientificamerican0792-66","article-title":"Genetic algorithms","volume":"267","author":"Holland","year":"1992","journal-title":"Sci. Am."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Popa, R. (2012). Genetic Algorithms in Applications, IntechOpen.","DOI":"10.5772\/2675"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"8091","DOI":"10.1007\/s11042-020-10139-6","article-title":"A review on genetic algorithm: Past, present, and future","volume":"80","author":"Katoch","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Mitchell, M. (1998). An Introduction to Genetic Algorithms, MIT Press.","DOI":"10.7551\/mitpress\/3927.001.0001"},{"key":"ref_51","first-page":"493","article-title":"Messy genetic algorithms: Motivation, analysis, and first results","volume":"3","author":"Goldberg","year":"1989","journal-title":"Complex Syst."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1080\/00273170802285743","article-title":"Item selection for the development of short forms of scales using an ant colony optimization algorithm","volume":"43","author":"Leite","year":"2008","journal-title":"Multivar. Behav. Res."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1111\/jedm.12275","article-title":"Using retest data to evaluate and improve effort-moderated scoring","volume":"58","author":"Wise","year":"2021","journal-title":"J. Educ. Meas."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1111\/jedm.12290","article-title":"Model-Based Treatment of Rapid Guessing","volume":"58","author":"Deribo","year":"2021","journal-title":"J. Educ. Meas."},{"key":"ref_55","unstructured":"Rodrigo, M.M., Matsuda, N., Cristea, A.I., and Dimitrova, V. (2022, January 27\u201331). Considering Disengaged Responses in Bayesian and Deep Knowledge Tracing. Proceedings of the Artificial Intelligence in Education, Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners\u2019 and Doctoral Consortium, Durham, UK."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Toton, S.L., and Maynes, D.D. (2019). Detecting Examinees With Pre-knowledge in Experimental Data Using Conditional Scaling of Response Times. Front. Educ., 4.","DOI":"10.3389\/feduc.2019.00049"},{"key":"ref_57","unstructured":"Liashchynskyi, P., and Liashchynskyi, P. (2019). Grid search, random search, genetic algorithm: A big comparison for NAS. arXiv."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1111\/j.1745-3984.2006.00002.x","article-title":"An application of item response time: The effort-moderated IRT model","volume":"43","author":"Wise","year":"2006","journal-title":"J. Educ. Meas."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1177\/01466216211051719","article-title":"Quantifying the distorting effect of rapid guessing on estimates of coefficient alpha","volume":"46","author":"Rios","year":"2022","journal-title":"Appl. Psychol. Meas."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Ulitzsch, E., Ulitzsch, V., He, Q., and L\u00fcdtke, O. (2022). A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks. Behav. Res. Methods, 1\u201321.","DOI":"10.3758\/s13428-022-01844-1"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/2\/89\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:26:33Z","timestamp":1760120793000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/2\/89"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,7]]},"references-count":60,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["a16020089"],"URL":"https:\/\/doi.org\/10.3390\/a16020089","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,7]]}}}