{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,9]],"date-time":"2026-02-09T01:58:05Z","timestamp":1770602285933,"version":"3.49.0"},"reference-count":32,"publisher":"World Scientific Pub Co Pte Ltd","issue":"04","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61702140"],"award-info":[{"award-number":["61702140"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005046","name":"Natural Science Foundation of Heilongjiang Province","doi-asserted-by":"publisher","award":["F2018017"],"award-info":[{"award-number":["F2018017"]}],"id":[{"id":"10.13039\/501100005046","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Fundamental Research Foundation for Universities of Heilongjiang Province","award":["LGYC2018JC015"],"award-info":[{"award-number":["LGYC2018JC015"]}]},{"name":"Zhejiang A and F University Research Development Fund Talent Initiation Project","award":["2021LFR048"],"award-info":[{"award-number":["2021LFR048"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Soft. Eng. Knowl. Eng."],"published-print":{"date-parts":[[2023,4]]},"abstract":"<jats:p> Program online judge (POJ) systems allow students to view questions, submit solution code, and receive scores automatically via the web. Most POJs use test cases for scoring. When a POJ is scored by test case pass rate or a problem that has only one test case, students can usually score by providing the direct output of the test cases (direct output cheating). Currently, there is only one work on detecting such cheating. However, its precision is very low. To solve this problem, three novel approaches are proposed to detect direct output cheating: (i) Line Statistics, which computes the proportion of output calls against other statements; (ii) the control flow graph (CFG) Search computes the maximum similarity between the CFG of a program and that of known samples; (iii) abstract syntax tree (AST) Search identifies cheating by matching rules that are summarized from ASTs of previously detected cheating attempts. A student\u2019s code is marked as cheating if the similarity exceeds a predefined threshold; and a program is detected as cheating if the proportion exceeds a predefined threshold. The proposed approaches and three well-known code plagiarism detection tools (JPlag, Sherlock, and SIM) were evaluated using 100,000 submissions for 1153 problems from a POJ based on the C programming language. The F1 scores of these approaches were determined as 0.9752 (AST Search), 0.9440 (CFG Search), 0.7405 (Line Statistics), 0.6446 (JPlag), 0.1587 (Sherlock), and 0.0076 (SIM), respectively. The result indicates that (i) AST Search is most suitable for the detection of direct output cheating; (ii) traditional code search or plagiarism detection methods based on similarity calculations are not effective for complex cheat detection because these cheats are highly similar to normal code. <\/jats:p>","DOI":"10.1142\/s0218194023500043","type":"journal-article","created":{"date-parts":[[2023,2,17]],"date-time":"2023-02-17T02:44:28Z","timestamp":1676601868000},"page":"461-486","source":"Crossref","is-referenced-by-count":3,"title":["Three Approaches for Detecting Direct Output Cheating in Program Online Judge Systems"],"prefix":"10.1142","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3264-1681","authenticated-orcid":false,"given":"Jing","family":"Qiu","sequence":"first","affiliation":[{"name":"College of Mathematics and Computer Science, Zhejiang A&F University, Hangzhou 311300, P. R. China"}]},{"given":"Chunmei","family":"Shi","sequence":"additional","affiliation":[{"name":"College of Mathematics and Computer Science, Zhejiang A&F University, Hangzhou 311300, P. R. China"}]},{"given":"Yuehua","family":"Lv","sequence":"additional","affiliation":[{"name":"Institute of Scientific and Technical Information of Zhejiang Province, Hangzhou 311300, P. R. China"}]}],"member":"219","published-online":{"date-parts":[[2023,3,30]]},"reference":[{"key":"S0218194023500043BIB001","doi-asserted-by":"crossref","first-page":"4\u2013es","DOI":"10.1145\/1163405.1163409","volume":"5","author":"Douce C.","year":"2005","journal-title":"J. Educ. Resourc. Comput."},{"key":"S0218194023500043BIB002","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3143560","volume":"51","author":"Wasik S.","year":"2018","journal-title":"ACM Comput. Surv."},{"issue":"10","key":"S0218194023500043BIB003","first-page":"131","volume":"2","author":"Revilla M. A.","year":"2008","journal-title":"Olymp. Inform."},{"issue":"6","key":"S0218194023500043BIB004","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1142\/S0218194022500346","volume":"32","author":"Watanobe Y.","year":"2022","journal-title":"Int. J. Softw. Eng. Knowl. Eng."},{"key":"S0218194023500043BIB007","first-page":"1","volume-title":"2016 IEEE Frontiers in Education Conf.","author":"Su X.","year":"2016"},{"key":"S0218194023500043BIB008","first-page":"1016","volume":"8","author":"Prechelt L.","year":"2002","journal-title":"J. Univ. Comput. Sci."},{"key":"S0218194023500043BIB009","first-page":"13b3\/18","volume-title":"29th Annual Frontiers In Education Conf.","author":"Bowyer K. W.","year":"1999"},{"issue":"2","key":"S0218194023500043BIB010","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1109\/13.762946","volume":"42","author":"Joy M.","year":"1999","journal-title":"IEEE Trans. Educ."},{"key":"S0218194023500043BIB011","first-page":"141","volume-title":"Proc. 6th Baltic Sea Conf. Computing Education Research: Koli Calling","author":"Ahtiainen A.","year":"2006"},{"key":"S0218194023500043BIB012","first-page":"419","volume-title":"Proc. 11th Int. Conf. Computer Systems and Technologies and Workshop for PhD Students in Computing on International Conf. Computer Systems and Technologies","author":"Chud\u00e1 D.","year":"2010"},{"key":"S0218194023500043BIB013","first-page":"15","volume":"2020","author":"Zhang F.","year":"2020","journal-title":"Sci. Program."},{"issue":"8","key":"S0218194023500043BIB015","first-page":"707","volume":"10","author":"Levenshtein V. I.","year":"1966","journal-title":"Cybern. Control Theory"},{"key":"S0218194023500043BIB016","doi-asserted-by":"crossref","first-page":"865","DOI":"10.3233\/IDA-2012-00556","volume":"16","author":"Nikoli\u0107 M.","year":"2012","journal-title":"Intell. Data Anal."},{"issue":"2","key":"S0218194023500043BIB022","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1080\/08993400500150747","volume":"15","author":"Ala-Mutka K. M.","year":"2005","journal-title":"Comput. Sci. Educ."},{"key":"S0218194023500043BIB023","volume-title":"Proc. 10th Koli Calling Int. Conf. Computing Education Research","author":"Ihantola P.","year":"2010"},{"key":"S0218194023500043BIB024","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1109\/TE.2010.2098442","volume":"54","author":"Aleman J. L. F.","year":"2011","journal-title":"IEEE Trans. Educ."},{"key":"S0218194023500043BIB025","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1145\/1345443.1345441","volume-title":"2007 Working Group Reports ITiCSE Innovation and Technology in Computer Science Education, ACM SIGCSE Bulletin","author":"Pears A.","year":"2007"},{"key":"S0218194023500043BIB026","first-page":"23","volume-title":"2015 IEEE Int. Conf. Teaching, Assessment, and Learning for Engineering","author":"Staubitz T.","year":"2015"},{"key":"S0218194023500043BIB027","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2810239","volume":"22","author":"Corrigan-Gibbs H.","year":"2015","journal-title":"ACM Trans. Comput.\u2013Hum. Interact."},{"key":"S0218194023500043BIB028","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1080\/02602930600898569","volume":"32","author":"Badge J. L.","year":"2007","journal-title":"Assess. Eval. High Educ."},{"key":"S0218194023500043BIB029","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/S0360-1315(03)00030-7","volume":"41","author":"Cheang B.","year":"2003","journal-title":"Comput. Educ."},{"key":"S0218194023500043BIB031","first-page":"1129","volume-title":"Proc. 18th ACM Conf. Computer Supported Cooperative Work & Social Computing","author":"Li X.","year":"2015"},{"key":"S0218194023500043BIB032","doi-asserted-by":"crossref","first-page":"1609","DOI":"10.1109\/TMM.2017.2656064","volume":"19","author":"Atoum Y.","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"S0218194023500043BIB033","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.compedu.2016.04.008","volume":"100","author":"Northcutt C. G.","year":"2016","journal-title":"Comput. Educ."},{"key":"S0218194023500043BIB034","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.compedu.2017.08.008","volume":"115","author":"Top\u00eerceanu A.","year":"2017","journal-title":"Comput. Educ."},{"key":"S0218194023500043BIB035","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1145\/236452.236525","volume-title":"Proc. Twenty-Seventh SIGCSE Technical Symp. Computer Science Education","author":"Wise M. J.","year":"1996"},{"issue":"3","key":"S0218194023500043BIB036","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3313290","volume":"19","author":"Novak M.","year":"2019","journal-title":"ACM Trans. Comput. Educ."},{"issue":"3","key":"S0218194023500043BIB037","doi-asserted-by":"crossref","first-page":"41","DOI":"10.4018\/IJSI.2015070104","volume":"3","author":"Kikuchi H.","year":"2015","journal-title":"Int. J. Softw. Innov."},{"key":"S0218194023500043BIB038","first-page":"8","volume":"2017","author":"Fu D.","year":"2017","journal-title":"Sci. Program."},{"key":"S0218194023500043BIB039","first-page":"261","volume-title":"2020 IEEE 27th Int. Conf. Software Analysis, Evolution and Reengineering","author":"Wang W.","year":"2020"},{"key":"S0218194023500043BIB040","doi-asserted-by":"crossref","first-page":"1066","DOI":"10.1145\/3385412.3386001","volume-title":"Proc. 41st ACM SIGPLAN Conf. Programming Language Design and Implementation","author":"Premtoon V.","year":"2020"},{"key":"S0218194023500043BIB041","doi-asserted-by":"crossref","first-page":"50391","DOI":"10.1109\/ACCESS.2021.3069367","volume":"9","author":"Cheers H.","year":"2021","journal-title":"IEEE Access"}],"container-title":["International Journal of Software Engineering and Knowledge Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0218194023500043","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,8]],"date-time":"2023-05-08T02:16:40Z","timestamp":1683512200000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0218194023500043"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,30]]},"references-count":32,"journal-issue":{"issue":"04","published-print":{"date-parts":[[2023,4]]}},"alternative-id":["10.1142\/S0218194023500043"],"URL":"https:\/\/doi.org\/10.1142\/s0218194023500043","relation":{},"ISSN":["0218-1940","1793-6403"],"issn-type":[{"value":"0218-1940","type":"print"},{"value":"1793-6403","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,30]]}}}