{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T05:11:29Z","timestamp":1788757889471,"version":"build-2803163510"},"reference-count":37,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T00:00:00Z","timestamp":1788739200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Psychol."],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Before artificial intelligence-based judging systems can be applied to Taekwondo Poomsae, it is necessary to understand which performance components human judges can identify consistently and which remain difficult to evaluate reliably. This study examined reliability in freestyle Poomsae judging to identify evaluation components with stable and unstable human scoring.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Ten internationally certified Taekwondo Poomsae referees evaluated ten official competition videos twice, separated by a one-week washout period. Systematic session effects were evaluated with separate linear mixed-effects models estimated by restricted maximum likelihood using the SPSS MIXED procedure. Session was specified as a fixed effect, judge and video as crossed random intercepts, and the two observations within each judge-video pair as repeated measurements with an unstructured residual covariance matrix. Fixed-effect inference used Satterthwaite denominator degrees of freedom. Test\u2013retest stability was evaluated for identical judge-video pairs, with primary emphasis on the absolute-agreement ICC because systematic session effects were detected. Inter-rater agreement was evaluated separately by session using two-way random effects, single rating absolute-agreement and consistency ICCs with 95% confidence intervals and Kendall\u2019s W. Item-specific analyses were exploratory, and no multiplicity adjustment was applied.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      The total score increased by 0.304 points, 95% CI [0.187, 0.421], t (99)\u202f=\u202f5.177,\n                      <jats:italic>p<\/jats:italic>\n                      &amp;lt;\u202f0.001. At the nominal 0.05 level, exploratory item-specific increases were observed for jumping side kick (\n                      <jats:italic>\u03b2<\/jats:italic>\n                      =\u202f0.057,\n                      <jats:italic>p<\/jats:italic>\n                      &amp;lt;\u202f0.001), acrobatic kicking technique (\n                      <jats:italic>\u03b2<\/jats:italic>\n                      =\u202f0.021,\n                      <jats:italic>p<\/jats:italic>\n                      =\u202f0.003), and expression of energy (\n                      <jats:italic>\u03b2<\/jats:italic>\n                      =\u202f0.047,\n                      <jats:italic>p<\/jats:italic>\n                      =\u202f0.003); the increase for basic movements and practicability did not reach the 0.05 threshold (\n                      <jats:italic>p<\/jats:italic>\n                      =\u202f0.053). Absolute-agreement test\u2013retest ICCs ranged from 0.039 to 0.848 and were excellent for harmony and the total score. All single-rating inter-rater ICC(A,1) point estimates were below 0.40. For the total score, ICC(A,1) was \u22120.007, 95% CI [\u22120.018, 0.036], in the first session and 0.060, 95% CI [0.010, 0.224], in the second session.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Systematic session effects, absolute test\u2013retest agreement, relative consistency over time, and inter-rater agreement represent distinct aspects of judging reliability. Evaluation components with poor single-judge agreement require clearer operational definitions and further validation before computational or AI-assisted scoring is considered.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fpsyg.2026.1881296","type":"journal-article","created":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T04:23:19Z","timestamp":1788754999000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Revisiting judging reliability in taekwondo freestyle Poomsae: implications for AI-supported evaluation"],"prefix":"10.3389","volume":"17","author":[{"given":"Min-Woo","family":"Jeon","sequence":"first","affiliation":[{"name":"Department of Taekwondo, College of Physical Education, Kyung Hee University","place":["Yongin-si, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hong-Suk","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Sports Science, Hanyang University ERICA","place":["Ansan-si, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ji-Yong","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Sports Science, Hanyang University ERICA","place":["Ansan-si, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hye-Soo","family":"Cho","sequence":"additional","affiliation":[{"name":"Department of Sports Science, Hanyang University ERICA","place":["Ansan-si, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,7]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"95","DOI":"10.5550\/sgia.110702.en.095A","article-title":"Analysis results judging world championships in men\u2019s artistic gymnastics in the London 2009 year","volume":"7","author":"Atikovi\u0107","year":"2011","journal-title":"SportLogia"},{"key":"ref2","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1037\/0033-2909.106.2.265","article-title":"Exposure and affect: overview and meta-analysis of research, 1968-1987","volume":"106","author":"Bornstein","year":"1989","journal-title":"Psychol. Bull."},{"key":"ref3","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1016\/j.actpsy.2004.08.005","article-title":"Save the last dance for me: unwanted serial position effects in jury evaluations","volume":"118","author":"Bruine de Bruin","year":"2005","journal-title":"Acta Psychol."},{"key":"ref4","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1080\/17461391.2010.551416","article-title":"Reliability and validity of judging in women\u2019s artistic gymnastics at university games 2009","volume":"12","author":"Bu\u010dar","year":"2012","journal-title":"Eur. J. Sport Sci."},{"key":"ref5","doi-asserted-by":"publisher","first-page":"5","DOI":"10.52165\/sgj.6.3.5-12","article-title":"The judging of artistry components in female gymnastics: a cause for concern?","volume":"6","author":"Bu\u010dar Pajek","year":"2014","journal-title":"Sci. Gymnastics J."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1186\/s13102-021-00340-x","article-title":"Adequacy of setting standards for kick impact in the taekwondo electronic scoring system: comparison of a reference group model based on empirical data","volume":"13","author":"Choi","year":"2021","journal-title":"BMC Sports Sci. Med. Rehabil."},{"key":"ref7","doi-asserted-by":"publisher","first-page":"33","DOI":"10.14589\/ido.22.4.5","article-title":"A statistical analysis of the kata scoring system in sport karate","volume":"22","author":"Clark","year":"2022","journal-title":"Ido Mov. Cult. J. Martial Arts Anthropol."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1037\/0022-3514.59.1.61","article-title":"What do women want? Facialmetric assessment of multiple motives in the perception of male facial physical attractiveness","volume":"59","author":"Cunningham","year":"1990","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref9","doi-asserted-by":"publisher","first-page":"109","DOI":"10.2466\/05.22.24.27.PMS.112.1.109-121","article-title":"Influence of angle of view on judges\u2019 evaluations of inverted cross in men\u2019s rings","volume":"112","author":"Dallas","year":"2011","journal-title":"Percept. Mot. Skills"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1123\/jsep.26.1.154","article-title":"A reputation bias in figure skating judging","volume":"26","author":"Findlay","year":"2004","journal-title":"J. Sport Exerc. Psychol."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"640","DOI":"10.1249\/MSS.0000000000000425","article-title":"Judging the judges\u2019 performance in rhythmic gymnastics","volume":"47","author":"Flessas","year":"2015","journal-title":"Med. Sci. Sports Exerc."},{"key":"ref12","year":"2023"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"63","DOI":"10.52165\/sgj.4.1.63-72","article-title":"Judging performance in gymnastics: a matter of motor or visual experience?","volume":"4","author":"Heinen","year":"2012","journal-title":"Sci. Gymnastics J."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1515\/jqas-2019-0113","article-title":"Judging the judges: evaluating the accuracy and national bias of international gymnastics judges","volume":"17","author":"Heiniger","year":"2021","journal-title":"J. Quant. Anal. Sports"},{"key":"ref15","doi-asserted-by":"publisher","first-page":"4871","DOI":"10.3390\/s20174871","article-title":"TUHAD: taekwondo unit technique human action dataset with key frame-based CNN action recognition","volume":"20","author":"Lee","year":"2020","journal-title":"Sensors"},{"key":"ref16","doi-asserted-by":"publisher","first-page":"25","DOI":"10.52165\/sgj.2.1.25-34","article-title":"Reliability and validity of judging in men\u2019s artistic gymnastics at the 2009 university games","volume":"2","author":"Lesko\u0161ek","year":"2010","journal-title":"Sci. Gymnastics J."},{"key":"ref17","first-page":"13381","author":"Li","year":"2021"},{"key":"ref18","first-page":"1","article-title":"Evaluation of success in competitive figure skating: an analysis of interjudge reliability","volume":"11","author":"Lockwood","year":"2005","journal-title":"Avante"},{"key":"ref19","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1080\/1091367X.2012.639602","article-title":"Judging anomalies at the 2010 Olympics in men\u2019s figure skating","volume":"16","author":"Looney","year":"2012","journal-title":"Meas. Phys. Educ. Exerc. Sci."},{"key":"ref20","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1080\/10691898.2019.1632759","article-title":"Deep dive into visual representation and interrater agreement using data from a high-school diving competition","volume":"27","author":"McGee","year":"2019","journal-title":"J. Stat. Educ."},{"key":"ref21","doi-asserted-by":"publisher","first-page":"173","DOI":"10.2478\/hukin-2013-0038","article-title":"Is the quality of judging in women artistic gymnastics equivalent at major competitions of different levels?","volume":"37","author":"Pajek","year":"2013","journal-title":"J. Hum. Kinet."},{"key":"ref22","first-page":"304","author":"Parmar","year":"2019"},{"key":"ref23","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1038\/35003107","article-title":"Tall men have more reproductive success","volume":"403","author":"Pawlowski","year":"2000","journal-title":"Nature"},{"key":"ref24","doi-asserted-by":"crossref","first-page":"556","DOI":"10.1007\/978-3-319-10599-4_36","volume-title":"Computer Vision\u2014ECCV 2014","author":"Pirsiavash","year":"2014"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/j.ajsep.2023.01.002","article-title":"Sports performance judgments\u2014an update from a social cognitive perspective","volume":"3","author":"Plessner","year":"2023","journal-title":"Asian J. Sport Exerc. Psychol."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1016\/j.psychsport.2006.03.007","article-title":"Sports performance judgments from a social cognitive perspective","volume":"7","author":"Plessner","year":"2006","journal-title":"Psychol. Sport Exerc."},{"key":"ref27","doi-asserted-by":"publisher","first-page":"1001","DOI":"10.3389\/fpsyg.2019.01001","article-title":"Reliability of judging in DanceSport","volume":"10","author":"Premel\u010d","year":"2019","journal-title":"Front. Psychol."},{"key":"ref28","doi-asserted-by":"publisher","first-page":"934158","DOI":"10.3389\/fpsyg.2022.934158","article-title":"Improving reliability and validity in hip-hop dance assessment: judging standards that elevate the sport and competition","volume":"13","author":"Sato","year":"2022","journal-title":"Front. Psychol."},{"key":"ref29","doi-asserted-by":"publisher","first-page":"1593158","DOI":"10.3389\/fpsyg.2025.1593158","article-title":"Reliability of judging in Olympic breaking at the 2024 Paris games","volume":"16","author":"Sato","year":"2025","journal-title":"Front. Psychol."},{"key":"ref30","doi-asserted-by":"publisher","first-page":"420","DOI":"10.1080\/02701367.2001.10608979","article-title":"Memory-influenced biases in gymnastic judging occur across different prior processing conditions","volume":"72","author":"Ste-Marie","year":"2001","journal-title":"Res. Q. Exerc. Sport"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1098\/rspb.1999.0624","article-title":"Visual cues to female physical attractiveness","volume":"266","author":"Tov\u00e9e","year":"1999","journal-title":"Proc. Biol. Sci."},{"key":"ref32","year":"2024"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"4578","DOI":"10.1109\/TCSVT.2019.2927118","article-title":"Learning to score figure skating sport videos","volume":"30","author":"Xu","year":"2020","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1057\/s41599-025-04792-1","article-title":"Development and validation of standards for evaluating the quality of qualitative research on Olympics breakdance","volume":"12","author":"Yang","year":"2025","journal-title":"Humanit. Soc. Sci. Commun."},{"key":"ref35","doi-asserted-by":"publisher","first-page":"15","DOI":"10.4103\/2600-9404.324089","article-title":"Movement pattern recognition ability of Malaysian rhythmic gymnastics judges","volume":"2","author":"Yee","year":"2013","journal-title":"Malays. J. Mov. Health Exerc."},{"key":"ref36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1037\/h0025848","article-title":"Attitudinal effects of mere exposure","volume":"9","author":"Zajonc","year":"1968","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref37","doi-asserted-by":"publisher","first-page":"1632326","DOI":"10.3389\/fspor.2025.1632326","article-title":"Faster, more accurate? A feasibility study on replacing human judges with artificial intelligence in video review for the Paris Olympics taekwondo competition","volume":"7","author":"Zhang","year":"2025","journal-title":"Front. Sports Act. Living."}],"container-title":["Frontiers in Psychology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2026.1881296\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,7]],"date-time":"2026-09-07T04:23:21Z","timestamp":1788755001000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2026.1881296\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,7]]},"references-count":37,"alternative-id":["10.3389\/fpsyg.2026.1881296"],"URL":"https:\/\/doi.org\/10.3389\/fpsyg.2026.1881296","relation":{},"ISSN":["1664-1078"],"issn-type":[{"value":"1664-1078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,7]]},"article-number":"1881296"}}