{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T18:33:14Z","timestamp":1784399594790,"version":"3.55.0"},"reference-count":106,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,1,4]],"date-time":"2018-01-04T00:00:00Z","timestamp":1515024000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Center for Research and Development, Poland","award":["LIDER\/004\/103\/L-5\/13\/NCBR\/2014"],"award-info":[{"award-number":["LIDER\/004\/103\/L-5\/13\/NCBR\/2014"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2019,1,31]]},"abstract":"<jats:p>Online judges are systems designed for the reliable evaluation of algorithm source code submitted by users, which is next compiled and tested in a homogeneous environment. Online judges are becoming popular in various applications. Thus, we would like to review the state of the art for these systems. We classify them according to their principal objectives into systems supporting organization of competitive programming contests, enhancing education and recruitment processes, facilitating the solving of data mining challenges, online compilers and development platforms integrated as components of other custom systems. Moreover, we introduce a formal definition of an online judge system and summarize the common evaluation methodology supported by such systems. Finally, we briefly discuss an Optil.io platform as an example of an online judge system, which has been proposed for the solving of complex optimization problems. We also analyze the competition results conducted using this platform. The competition proved that online judge systems, strengthened by crowdsourcing concepts, can be successfully applied to accurately and efficiently solve complex industrial- and science-driven challenges.<\/jats:p>","DOI":"10.1145\/3143560","type":"journal-article","created":{"date-parts":[[2018,1,4]],"date-time":"2018-01-04T16:27:31Z","timestamp":1515083251000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":136,"title":["A Survey on Online Judge Systems and Their Applications"],"prefix":"10.1145","volume":"51","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2753-2862","authenticated-orcid":false,"given":"Szymon","family":"Wasik","sequence":"first","affiliation":[{"name":"Poznan University of Technology and Polish Academy of Sciences, Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maciej","family":"Antczak","sequence":"additional","affiliation":[{"name":"Poznan University of Technology, Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jan","family":"Badura","sequence":"additional","affiliation":[{"name":"Poznan University of Technology, Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Artur","family":"Laskowski","sequence":"additional","affiliation":[{"name":"Poznan University of Technology, Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tomasz","family":"Sternal","sequence":"additional","affiliation":[{"name":"Poznan University of Technology, Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,1,4]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1080\/08993400500150747"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-016-1237-9"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2729094.2742607"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1504\/EJIE.2012.051072"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.19173\/irrodl.v12i3.902"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00402268"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10951-014-0390-0"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-20009-0_24"},{"key":"e_1_2_2_9_1","volume-title":"Proceedings of the 7th International Technology, Education and Development Conference (INTED2013)","author":"Julio"},{"key":"e_1_2_2_10_1","unstructured":"Manuel M. Carral. 2013. Moodle Autograder Plugin. Master\u2019s thesis. Universidad de Cantabria.  Manuel M. Carral. 2013. Moodle Autograder Plugin. Master\u2019s thesis. Universidad de Cantabria."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(03)00030-7"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v33i1.2392"},{"key":"e_1_2_2_13_1","first-page":"21","article-title":"Programming trainings and informatics teaching through online contests","volume":"8","author":"Comb\u00e9fis S\u00e9bastien","year":"2014","journal-title":"Olympiads in Informatics"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/1149707.1149710"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1038\/clpt.2013.36"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-08918-8_10"},{"key":"e_1_2_2_17_1","first-page":"3","article-title":"Editorial","volume":"1","author":"Dagiene Valentina","year":"2007","journal-title":"Olympiads in Informatics"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/305786.305904"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.protcy.2013.12.207"},{"key":"e_1_2_2_20_1","volume-title":"Madadi","author":"Davuluri Prathibha","year":"2016"},{"key":"e_1_2_2_21_1","unstructured":"Ross Dawson and Steve Bynghall. 2012. Getting Results from Crowds. Advanced Human Technologies San Francisco CA.  Ross Dawson and Steve Bynghall. 2012. Getting Results from Crowds. Advanced Human Technologies San Francisco CA."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1038\/ng.1089"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2181037.2181040"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2500499"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMEIT.2011.6024016"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.4153\/CJM-1965-045-4"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1597849.1384371"},{"key":"e_1_2_2_28_1","unstructured":"Arpad E. Elo. 1978. The Rating of Chessplayers Past and Present. Arco Pub.  Arpad E. Elo. 1978. The Rating of Chessplayers Past and Present. Arco Pub."},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPASS.2015.7095802"},{"key":"e_1_2_2_30_1","volume-title":"Proceedings of the 2nd Symposium on Languages, Applications and Technologies (OpenAccess Series in Informatics (OASIcs)), Jos\u00e9 Paulo Leal, Ricardo Rocha, and Alberto Sim\u00f5es (Eds.)","volume":"29","author":"Fonte Daniela","year":"2013"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/1149707.1149713"},{"key":"e_1_2_2_32_1","volume-title":"Security of programming contest systems. Information Technologies at School","author":"Fori\u0161ek Michal","year":"2006"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/364914.364937"},{"key":"e_1_2_2_34_1","volume-title":"Johnson","author":"Garey Michael R.","year":"1979"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2010.56"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2795403.2795416"},{"key":"e_1_2_2_37_1","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe Jeff","year":"2006","journal-title":"Wired Magazine"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/CIE.2002.1186207"},{"key":"e_1_2_2_39_1","unstructured":"ICPC Foundation. 2016. ICPC Fact Sheet. Retrieved from https:\/\/icpc.baylor.edu\/worldfinals\/pdf\/Factsheet.pdf.  ICPC Foundation. 2016. ICPC Fact Sheet. Retrieved from https:\/\/icpc.baylor.edu\/worldfinals\/pdf\/Factsheet.pdf."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1930464.1930480"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858796.2858798"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-16012-2_6"},{"key":"e_1_2_2_43_1","volume-title":"The International Scientific Conference eLearning and Software for Education","volume":"1","author":"Ivanova Slavina","year":"2016"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1163405.1163407"},{"key":"e_1_2_2_45_1","volume-title":"Luck","author":"Joy Mike","year":"1995"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.bushor.2016.03.008"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICALT.2013.105"},{"key":"e_1_2_2_48_1","first-page":"715","article-title":"An online compiler module for grading programming assignments on Moodle distance education system","volume":"1","author":"Kaya M\u00fcmine","year":"2012","journal-title":"Global Journal of Technology"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21606"},{"key":"e_1_2_2_50_1","volume-title":"Keele University","author":"Software Engineering Group","year":"2007"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/169073.169105"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"e_1_2_2_53_1","first-page":"105","article-title":"USA Computing Olympiad (USACO)","volume":"1","author":"Kolstad Rob","year":"2007","journal-title":"Olympiads in Informatics"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.19173\/irrodl.v12i3.882"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78139-4_31"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(01)00018-5"},{"key":"e_1_2_2_57_1","volume-title":"Leal and Nelma Moreira","author":"Jos\u00e9","year":"1998"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.522"},{"key":"e_1_2_2_59_1","volume-title":"ISO 639 Macrolanguage, ISO 639: K, ISO 639: M, ISO 639: B, ISO 639: A, ISO 639: T, ISO 639-1","author":"Books LLC"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkv557"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1517632.1517643"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.5220\/0004842801760183"},{"key":"e_1_2_2_63_1","volume-title":"Proceedings of Perspectives on Computer Science Competitions for (High School) Students Workshop. https:\/\/www.bwinf.de\/competition-workshop\/papers.html.","author":"Manzoor Shahriar","year":"2006"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00158-003-0368-6"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818377"},{"key":"e_1_2_2_66_1","volume-title":"PDDL: The Planning Domain Definition Language. Technical Report. CVC TR-98-003\/DCS TR-1165","author":"Mcdermott Drew","year":"1998"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1016\/0167-6377(82)90039-6"},{"key":"e_1_2_2_68_1","article-title":"Docker: Lightweight Linux containers for consistent development and deployment","volume":"2014","author":"Merkel Dirk","year":"2014","journal-title":"Linux Journal."},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2964797.2964804"},{"key":"e_1_2_2_70_1","first-page":"273","article-title":"Online training and contests for informatics contestants of secondary school age","volume":"6","author":"N\u00e9meth \u00c1gnes E.","year":"2015","journal-title":"Edukacja-Technika-Informatyka"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2157136.2157267"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.5555\/2541917.2541921"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.5555\/1149707.1149718"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.2174\/1574893610666150928193510"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.5210\/fm.v20i12.6143"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/CINE.2015.31"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJTEL.2013.059495"},{"key":"e_1_2_2_78_1","first-page":"131","article-title":"Competitive learning in informatics: The UVa online judge experience","volume":"2","author":"Revilla Miguel A.","year":"2008","journal-title":"Olympiads in Informatics"},{"key":"e_1_2_2_79_1","volume-title":"Proceedings of the 2012 International Conference on e-Learning, e-Business, Enterprise Information Systems, 8 e-Government.","author":"del Pino Juan C.","year":"2012"},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITSIM.2010.5561488"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-7997-1_7"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1038\/nrg.2016.69"},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/872757.872770"},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2007.08.006"},{"key":"e_1_2_2_85_1","volume-title":"Revilla","author":"Skiena Steven S.","year":"2008"},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/2676723.2677297"},{"key":"e_1_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1109\/TALE.2015.7386010"},{"key":"e_1_2_2_88_1","volume-title":"MIC 2015: The XI Metaheuristics International Conference. http:\/\/www.cs.nott.ac.uk\/&sim;pszeo\/docs\/publications\/research-agenda-metaheuristic.pdf.","author":"Swan Jerry","year":"2015"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1004853"},{"key":"e_1_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40994-3_46"},{"key":"e_1_2_2_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/182107.182101"},{"key":"e_1_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1515\/fcds-2016-0004"},{"key":"e_1_2_2_93_1","doi-asserted-by":"publisher","DOI":"10.1186\/1752-0509-4-S2-S1"},{"key":"e_1_2_2_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2869098"},{"key":"e_1_2_2_95_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0145557"},{"key":"e_1_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1155\/2013\/320715"},{"key":"e_1_2_2_97_1","volume-title":"Peking university online judge and its applications","author":"Guo Wei Li","year":"2005"},{"key":"e_1_2_2_98_1","doi-asserted-by":"publisher","DOI":"10.1145\/2839509.2844616"},{"key":"e_1_2_2_99_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13529-3_4"},{"key":"e_1_2_2_100_1","doi-asserted-by":"publisher","DOI":"10.1109\/4235.585893"},{"key":"e_1_2_2_101_1","volume-title":"ICERI\u201911 Proceedings. IATED, 3738--3747","author":"Xavier Joao"},{"key":"e_1_2_2_102_1","doi-asserted-by":"publisher","DOI":"10.4028\/www.scientific.net\/AMM.490-491.1201"},{"key":"e_1_2_2_103_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20281"},{"key":"e_1_2_2_104_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSCI.2015.160"},{"key":"e_1_2_2_105_1","volume-title":"Proceedings of the 1st Moodle Research Conference (MRC\u201912)","author":"Zhigang Sun","year":"2012"},{"key":"e_1_2_2_106_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30126-1_106"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3143560","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3143560","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:13:21Z","timestamp":1750212801000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3143560"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,1,4]]},"references-count":106,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,1,31]]}},"alternative-id":["10.1145\/3143560"],"URL":"https:\/\/doi.org\/10.1145\/3143560","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,1,4]]},"assertion":[{"value":"2017-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-01-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}