{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T15:24:59Z","timestamp":1773415499481,"version":"3.50.1"},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,5,24]],"date-time":"2024-05-24T00:00:00Z","timestamp":1716508800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100003392","name":"Natural Science Foundation of Fujian Province","doi-asserted-by":"crossref","award":["2022J01933"],"award-info":[{"award-number":["2022J01933"]}],"id":[{"id":"10.13039\/501100003392","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100020783","name":"Fujian Provincial Social Science Planning Project","doi-asserted-by":"crossref","award":["FJ2021BF050"],"award-info":[{"award-number":["FJ2021BF050"]}],"id":[{"id":"10.13039\/501100020783","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Open Fund of Key Laboratory of Hunan Province","award":["2015TP1002"],"award-info":[{"award-number":["2015TP1002"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Online Judge Systems (OJSs) play a crucial role in evaluating SQL programming skills. However, OJSs may not accurately evaluate students\u2019 queries as the error-detection capabilities of test sets are insufficient, resulting in false positives that can mislead students and hinder their learning. This study analyzes a large-scale OJS\u2019s evaluation dataset and identifies more than 110,000 (1.94%) false-positive queries. It also validates existing SQL error categorization and reveals a new type of logical error called deceptive error, which occurs when students construct queries that pass specific test cases but fail to solve the actual problem. This type of error has been overlooked in previous research and can provide new insights into how to improve OJSs by enhancing test cases and feedback. This study contributes to the understanding of assessment and evaluation practices and processes in programming education, particularly the contribution that OJSs make to student learning and to course, staff, and institutional development. It also suggests error prevention and detection techniques that can improve the effectiveness and fairness of OJSs in programming education and competitions.<\/jats:p>","DOI":"10.1145\/3654677","type":"journal-article","created":{"date-parts":[[2024,3,28]],"date-time":"2024-03-28T10:26:42Z","timestamp":1711621602000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["False Positives and Deceptive Errors in SQL Assessment: A Large-Scale Analysis of Online Judge Systems"],"prefix":"10.1145","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1733-8481","authenticated-orcid":false,"given":"Jinshui","family":"Wang","sequence":"first","affiliation":[{"name":"School of Computer Science and Mathematics, Fujian University of Technology, Fuzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9745-471X","authenticated-orcid":false,"given":"Shuguang","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Computer Science and Mathematics, Fujian University of Technology, Fuzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9306-6038","authenticated-orcid":false,"given":"Zhengyi","family":"Tang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Mathematics, Fujian University of Technology, Fuzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1212-1630","authenticated-orcid":false,"given":"Pengchen","family":"Lin","sequence":"additional","affiliation":[{"name":"School of Computer Science and Mathematics, Fujian University of Technology, Fuzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3835-6805","authenticated-orcid":false,"given":"Yupeng","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Mathematics, Fujian University of Technology, Fuzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,5,24]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/2839509.2844640"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2729094.2742620"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2899415.2899464"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2019.0100328"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1057\/ejis.1994.7"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00036"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113403"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2005.06.028"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00427157"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2020.08.079"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2021.100027"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-015-0395-0"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/1808266.1808276"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1177\/1049732305276687"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464825"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510187"},{"issue":"5","key":"e_1_3_4_18_2","first-page":"463","article-title":"A Review of Software Testing Techniques","volume":"7","author":"Kaur Manpreet","year":"2014","unstructured":"Manpreet Kaur and Rupinder Singh. 2014. A Review of Software Testing Techniques. International Journal of Electronic and Electrical Engineering 7, 5 (2014), 463\u2013474.","journal-title":"International Journal of Electronic and Electrical Engineering"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/HCC.2003.1260196"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(01)00018-5"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3514214"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551392"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366869"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2023.2199953"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00087"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.07.100"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/access.2021.3119145"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSCI54926.2021.00086"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10177-8"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767876"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2022.2076804"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.1995.1017"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE44824.2020.9274114"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3398377"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3231712"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21974"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1057\/palgrave.ejis.3000589"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.26599\/TST.2020.9010016"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3143560"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1142\/S0218194022500346"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0020-7373(85)80051-1"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2022.2072469"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3210713.3210721"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3654677","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3654677","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:49:04Z","timestamp":1750286944000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3654677"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,24]]},"references-count":42,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3654677"],"URL":"https:\/\/doi.org\/10.1145\/3654677","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,24]]},"assertion":[{"value":"2023-08-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}