{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T07:10:09Z","timestamp":1775027409755,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":24,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,3,13]],"date-time":"2020-03-13T00:00:00Z","timestamp":1584057600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DRL-1661036"],"award-info":[{"award-number":["DRL-1661036"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,3,23]]},"DOI":"10.1145\/3375462.3375508","type":"proceedings-article","created":{"date-parts":[[2020,3,13]],"date-time":"2020-03-13T17:48:06Z","timestamp":1584121686000},"page":"454-461","source":"Crossref","is-referenced-by-count":18,"title":["Testing the reliability of inter-rater reliability"],"prefix":"10.1145","author":[{"given":"Brendan","family":"Eagan","sequence":"first","affiliation":[{"name":"University of Wisconsin - Madison"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jais","family":"Brohinsky","sequence":"additional","affiliation":[{"name":"University of Wisconsin - Madison"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jingyi","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Wisconsin - Madison"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David Williamson","family":"Shaffer","sequence":"additional","affiliation":[{"name":"University of Wisconsin - Madison"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,3,13]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2018.52.1"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2018.52.6"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Robert L. Brennan. 2001. Generalizability theory. Springer-Verlag. Robert L. Brennan. 2001. Generalizability theory. Springer-Verlag.","DOI":"10.1007\/978-1-4757-3456-0"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2018.52.3"},{"key":"e_1_3_2_1_6_1","unstructured":"Jonathan Chang Sean Gerrish Chong Wang Jordan L. Boyd-Graber & David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in neural information processing system - Proceedings of the 2009 Conference 288--296. Jonathan Chang Sean Gerrish Chong Wang Jordan L. Boyd-Graber & David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in neural information processing system - Proceedings of the 2009 Conference 288--296."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2019.62.2"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Brendan Eagan Brad Rogers Rebecca Pozen Cody Marquart & David W. Shaffer. 2016. rhoR: Rho for inter rater reliability (Version 1.1.0). Retrieved from https:\/\/cran.r-project.org\/web\/packages\/rhoR\/index.html Brendan Eagan Brad Rogers Rebecca Pozen Cody Marquart & David W. Shaffer. 2016. rhoR: Rho for inter rater reliability (Version 1.1.0). Retrieved from https:\/\/cran.r-project.org\/web\/packages\/rhoR\/index.html","DOI":"10.32614\/CRAN.package.rhoR"},{"key":"e_1_3_2_1_10_1","unstructured":"Brendan Eagan Bradley Rogers Ronald Serlin Andrew R. Ruis Golnaz Arastoopour Irgens David W. Shaffer. 2017. Can We Rely on Reliability? Testing the Assumptions of Inter-Rater Reliability. Making a Difference: Prioritizing Equity and Access in CSCL: 12th International Conference on Computer-Supported Collaborative Learning eds. B. K. Smith M. Borge E. Mercier & K. Y. Lim (2017) 11:529--532. Brendan Eagan Bradley Rogers Ronald Serlin Andrew R. Ruis Golnaz Arastoopour Irgens David W. Shaffer. 2017. Can We Rely on Reliability? Testing the Assumptions of Inter-Rater Reliability. Making a Difference: Prioritizing Equity and Access in CSCL: 12th International Conference on Computer-Supported Collaborative Learning eds. B. K. Smith M. Borge E. Mercier & K. Y. Lim (2017) 11:529--532."},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of the 13th International Conference on Computer-Supported Collaborative Learning.","author":"Eagan Brendan","year":"2019"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/944919.944974"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"crossref","unstructured":"Joseph Fleiss Jacob Cohen & Brian Everitt. 1969. Large sample standard errors of kappa and weighted kappa. Psychological bulletin 72 5 323--327. Joseph Fleiss Jacob Cohen & Brian Everitt. 1969. Large sample standard errors of kappa and weighted kappa. Psychological bulletin 72 5 323--327.","DOI":"10.1037\/h0028106"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2018.52.7"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1080\/10508406.2013.802652"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2019.61.4"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.3102\/10769986017004297"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/10508406.2013.799475"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2018.52.4"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2018.51.2"},{"key":"e_1_3_2_1_21_1","first-page":"49","article-title":"Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence","volume":"17","author":"Papamitsiou Zacharoula","year":"2014","journal-title":"Journal of Educational Technology & Society"},{"key":"e_1_3_2_1_22_1","first-page":"8","article-title":"Methodological issues in the content analysis of computer conference transcripts","volume":"12","author":"Rourke Liam","year":"2001","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"e_1_3_2_1_23_1","unstructured":"David W. Shaffer. 2017. Quantitative ethnography. Madison Wisconsin: Cathcart Press. David W. Shaffer. 2017. Quantitative ethnography. Madison Wisconsin: Cathcart Press."},{"key":"e_1_3_2_1_24_1","first-page":"360","article-title":"Understanding interobserver agreement: The kappa statistic","volume":"37","author":"Viera Anthony J.","year":"2005","journal-title":"Family Medicine"}],"event":{"name":"LAK '20: 10th International Conference on Learning Analytics and Knowledge","location":"Frankfurt Germany","acronym":"LAK '20"},"container-title":["Proceedings of the Tenth International Conference on Learning Analytics & Knowledge"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3375462.3375508","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3375462.3375508","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3375462.3375508","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:31Z","timestamp":1750203871000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3375462.3375508"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,13]]},"references-count":24,"alternative-id":["10.1145\/3375462.3375508","10.1145\/3375462"],"URL":"https:\/\/doi.org\/10.1145\/3375462.3375508","relation":{},"subject":[],"published":{"date-parts":[[2020,3,13]]}}}