{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T14:13:14Z","timestamp":1753884794124,"version":"3.41.2"},"reference-count":32,"publisher":"World Scientific Pub Co Pte Ltd","issue":"05","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Artif. Intell. Tools"],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p> Peer assessment, which requires students to evaluate their peers\u2019 submissions, has become the paradigm for solving the grading challenge of large-scale open-ended assignments teachers face in MOOCs. Since peer grades may be biased and unreliable, a group of probabilistic graph models are proposed to improve the estimation of true scores for assignments based on peer grades, by explicitly modeling the bias and reliability of each grader. However, these models assume that graders\u2019 reliability are only impacted by their knowledge\/ ability levels while ignoring their grading behaviors. In real life, graders\u2019 grading behaviors (e.g., the time consumed for reviewing an assignment) reflect the seriousness of the graders in the assessment and greatly affect their reliability. Following this intuition, we propose two novel probabilistic graph models, named BPG<jats:sub>6<\/jats:sub> and BPG<jats:sub>7<\/jats:sub>, for cardinal peer assessment, which optimize the modeling of the reliability of graders by incorporating various grading behaviors of them. Besides, we develop a peer assessment system, named BPA, which collects abundant grading behaviors of graders and thus facilitates the evaluation of the proposed models and our teaching practices. Experimental results on the collected datasets show the superiority of the proposed models in improving the estimation accuracy of the true scores of assignments by leveraging graders\u2019 grading behaviors. Finally, by analyzing the feedback from teaching practices of peer assessment using the BPA system, we not only prove the usability of the BPA system but also once again demonstrate the effectiveness of the proposed model in peer assessment. <\/jats:p>","DOI":"10.1142\/s0218213023600072","type":"journal-article","created":{"date-parts":[[2023,1,11]],"date-time":"2023-01-11T11:04:27Z","timestamp":1673435067000},"source":"Crossref","is-referenced-by-count":0,"title":["Improving Peer Assessment by Incorporating Grading Behaviors: Models and Practices"],"prefix":"10.1142","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4061-8262","authenticated-orcid":false,"given":"Jia","family":"Xu","sequence":"first","affiliation":[{"name":"College of Computer, Electronics and Information, Guangxi University, Nanning, Guangxi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jing","family":"Liu","sequence":"additional","affiliation":[{"name":"College of Computer, Electronics and Information, Guangxi University, Nanning, Guangxi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Panyuan","family":"Yang","sequence":"additional","affiliation":[{"name":"College of Computer, Electronics and Information, Guangxi University, Nanning, Guangxi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pin","family":"Lv","sequence":"additional","affiliation":[{"name":"College of Computer, Electronics and Information, Guangxi University, Nanning, Guangxi, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"219","published-online":{"date-parts":[[2023,8,16]]},"reference":[{"key":"S0218213023600072BIB001","first-page":"675","volume-title":"Proc. of the 2015 Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2015)","author":"Caragiannis I.","year":"2015"},{"issue":"6","key":"S0218213023600072BIB002","doi-asserted-by":"crossref","first-page":"526","DOI":"10.1111\/j.1365-2729.2008.00290.x","volume":"24","author":"Pare D. E.","year":"2008","journal-title":"Journal of Computer Assisted Learning"},{"issue":"3","key":"S0218213023600072BIB003","doi-asserted-by":"crossref","first-page":"249","DOI":"10.3102\/00346543068003249","volume":"68","author":"Topping K.","year":"1998","journal-title":"Review of Educational Research"},{"issue":"1","key":"S0218213023600072BIB004","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1207\/s15326977ea1101_1","volume":"11","author":"Sadler P. M.","year":"2006","journal-title":"Educational Assessment"},{"key":"S0218213023600072BIB006","first-page":"92","volume-title":"Lecture Notes in Computer Science","volume":"8699","author":"Gehringer E. F.","year":"2014"},{"key":"S0218213023600072BIB007","first-page":"833","volume-title":"Proc. of the Thirty-Second AAAI Conf. on Artificial Intelligence (AAAI 2018), the 30th Innovative Applications of Artificial Intelligence (IAAI 2018), and the 8th AAAI Symp. on Educational Advances in Artificial Intelligence (EAAI 2018)","author":"Wang W.","year":"2018"},{"issue":"4","key":"S0218213023600072BIB008","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1016\/j.learninstruc.2009.08.002","volume":"20","author":"Strijbos J. W.","year":"2010","journal-title":"Learning and Instruction"},{"issue":"5","key":"S0218213023600072BIB009","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1080\/02602930903541015","volume":"35","author":"Sadler D. R.","year":"2010","journal-title":"Assessment & Evaluation in Higher Education"},{"issue":"1","key":"S0218213023600072BIB010","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1080\/02602938.2013.795518","volume":"39","author":"Nicol D.","year":"2014","journal-title":"Assessment & Evaluation in Higher Education"},{"key":"S0218213023600072BIB011","doi-asserted-by":"crossref","first-page":"103894","DOI":"10.1016\/j.compedu.2020.103894","volume":"154","author":"Loro F. G.","year":"2020","journal-title":"Comput. Educ."},{"key":"S0218213023600072BIB012","first-page":"153","volume-title":"Proc. of the 6th Int. Conf. on Educational Data Mining (EDM 2013)","author":"Piech C.","year":"2013"},{"key":"S0218213023600072BIB013","first-page":"454","volume-title":"Proc. of the Twenty-Ninth AAAI Conf. on Artificial Intelligence (AAAI 2015)","author":"Mi F.","year":"2015"},{"key":"S0218213023600072BIB014","first-page":"341","volume-title":"Proc. of the 26th Int. Conf. on World Wide Web Companion (WWW 2017)","author":"Chan H. P.","year":"2017"},{"volume-title":"Proc. of the 12th Int. Conf. on Educational Data Mining (EDM, 2019)","year":"2019","author":"Wang T.","key":"S0218213023600072BIB015"},{"key":"S0218213023600072BIB016","doi-asserted-by":"crossref","first-page":"50466","DOI":"10.1109\/ACCESS.2021.3069055","volume":"9","author":"Xu J.","year":"2021","journal-title":"IEEE Access"},{"issue":"5","key":"S0218213023600072BIB017","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","volume":"29","author":"Friedman J. H.","year":"2001","journal-title":"Annals of Statistics"},{"key":"S0218213023600072BIB019","first-page":"1037","volume-title":"The 20th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD 2014)","author":"Raman K.","year":"2014"},{"key":"S0218213023600072BIB020","first-page":"327","volume-title":"L@S\u201921: Eighth ACM Conf. on Learning @ Scale (L@S 2021)","author":"Lin L.","year":"2021"},{"key":"S0218213023600072BIB021","first-page":"177","volume-title":"Proc. of the Second ACM Conf. on Learning @ Scale (L@S 2015)","author":"Waters A. E.","year":"2015"},{"key":"S0218213023600072BIB023","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1016\/j.knosys.2015.05.019","volume":"85","author":"Luaces O.","year":"2015","journal-title":"Knowl. Based Syst."},{"issue":"2","key":"S0218213023600072BIB024","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1109\/TLT.2016.2565476","volume":"10","author":"Capuano N.","year":"2017","journal-title":"IEEE Trans. Learn. Technol."},{"issue":"23","key":"S0218213023600072BIB025","doi-asserted-by":"crossref","first-page":"17679","DOI":"10.1007\/s00500-020-05155-5","volume":"24","author":"Capuano N.","year":"2020","journal-title":"Soft Comput."},{"issue":"3","key":"S0218213023600072BIB026","volume":"39","author":"Bradley R. A.","year":"1952","journal-title":"Biometrika"},{"issue":"293","key":"S0218213023600072BIB027","first-page":"1","volume":"67","author":"Luce D. R.","year":"2005","journal-title":"Journal of the American Statistical Association"},{"issue":"1","key":"S0218213023600072BIB028","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1093\/biomet\/44.1-2.114","volume":"44","author":"Mallows C. L.","year":"1957","journal-title":"Biometrika"},{"issue":"4","key":"S0218213023600072BIB029","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1037\/h0065439","volume":"21","author":"Thurstone L. L.","year":"1927","journal-title":"Journal of Abnormal & Social Psychology"},{"issue":"2","key":"S0218213023600072BIB030","first-page":"193","volume":"24","author":"Plackett R. L.","year":"1975","journal-title":"Journal of the Royal Statistical Society. Series C (Applied Statistics)"},{"key":"S0218213023600072BIB031","first-page":"415","volume-title":"The 45th ACM Technical Symp. on Computer Science Education (SIGCSE 2014)","author":"de Alfaro L.","year":"2014"},{"key":"S0218213023600072BIB032","first-page":"161","volume-title":"Proceedings of ASIS","volume":"98","author":"Brin S.","year":"1998"},{"key":"S0218213023600072BIB033","first-page":"909","volume-title":"ECAI 2014 \u2014 21st European Conf. on Artificial Intelligence, Including Prestigious Applications of Intelligent Systems (PAIS 2014)","author":"Walsh T.","year":"2014"},{"issue":"1","key":"S0218213023600072BIB034","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1111\/jcal.12316","volume":"35","author":"Garc\u00eda-Mart\u00ednez Carlos","year":"2019","journal-title":"J. Comput. Assist. Learn."},{"issue":"6","key":"S0218213023600072BIB035","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1109\/TPAMI.1984.4767596","volume":"6","author":"Geman S.","year":"1984","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."}],"container-title":["International Journal on Artificial Intelligence Tools"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0218213023600072","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,21]],"date-time":"2023-08-21T07:59:11Z","timestamp":1692604751000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S0218213023600072"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8]]},"references-count":32,"journal-issue":{"issue":"05","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.1142\/S0218213023600072"],"URL":"https:\/\/doi.org\/10.1142\/s0218213023600072","relation":{},"ISSN":["0218-2130","1793-6349"],"issn-type":[{"type":"print","value":"0218-2130"},{"type":"electronic","value":"1793-6349"}],"subject":[],"published":{"date-parts":[[2023,8]]},"article-number":"2360007"}}