{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T19:45:02Z","timestamp":1784835902890,"version":"3.55.0"},"reference-count":89,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2013,12,1]],"date-time":"2013-12-01T00:00:00Z","timestamp":1385856000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000145","name":"Division of Information and Intelligent Systems","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000145","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Hasso Plattner Design Thinking Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2013,12]]},"abstract":"<jats:p>Peer and self-assessment offer an opportunity to scale both assessment and learning to global classrooms. This article reports our experiences with two iterations of the first large online class to use peer and self-assessment. In this class, peer grades correlated highly with staff-assigned grades. The second iteration had 42.9% of students\u2019 grades within 5% of the staff grade, and 65.5% within 10%. On average, students assessed their work 7% higher than staff did. Students also rated peers\u2019 work from their own country 3.6% higher than those from elsewhere. We performed three experiments to improve grading accuracy. We found that giving students feedback about their grading bias increased subsequent accuracy. We introduce short, customizable feedback snippets that cover common issues with assignments, providing students more qualitative peer feedback. Finally, we introduce a data-driven approach that highlights high-variance items for improvement. We find that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance. After revising rubrics, median grading error decreased from 12.4% to 9.9%.<\/jats:p>","DOI":"10.1145\/2505057","type":"journal-article","created":{"date-parts":[[2014,1,14]],"date-time":"2014-01-14T13:39:57Z","timestamp":1389706797000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":231,"title":["Peer and self assessment in massive online classes"],"prefix":"10.1145","volume":"20","author":[{"given":"Chinmay","family":"Kulkarni","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Koh Pang","family":"Wei","sequence":"additional","affiliation":[{"name":"Stanford University, and Coursera, Inc., Stanford, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huy","family":"Le","sequence":"additional","affiliation":[{"name":"Coursera, Inc., Mountain View, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Chia","sequence":"additional","affiliation":[{"name":"Stanford University, and Coursera, Inc., Stanford, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kathryn","family":"Papadopoulos","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Justin","family":"Cheng","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daphne","family":"Koller","sequence":"additional","affiliation":[{"name":"Stanford University, and Coursera, Inc., Stanford, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Scott R.","family":"Klemmer","sequence":"additional","affiliation":[{"name":"Stanford University, San Diego"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2013,12]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/235008.235010"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.43.5.997"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0033773"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.3200\/CTCH.53.1.27-31"},{"key":"e_1_2_1_5_1","volume-title":"Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice 17, 4","author":"Bennett R. E.","year":"1998","unstructured":"R. E. Bennett . 1998. Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice 17, 4 ( 1998 ). R. E. Bennett. 1998. Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice 17, 4 (1998)."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3984.1997.tb00512.x"},{"key":"e_1_2_1_7_1","volume-title":"Enhancing Learning through Self Assessment","author":"Boud D.","unstructured":"D. Boud . 1995. Enhancing Learning through Self Assessment . Routledge . D. Boud. 1995. Enhancing Learning through Self Assessment. Routledge."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1080\/713695728"},{"key":"e_1_2_1_9_1","first-page":"13","article-title":"Studying learning in the worldwide classroom: Research into edX's first MOOC","volume":"8","author":"Breslow L. B.","year":"2013","unstructured":"L. B. Breslow , D. E. Pritchard , J. DeBoer , G. S. Stump , A. D. Ho , and D. T. Seaton . 2013 . Studying learning in the worldwide classroom: Research into edX's first MOOC . Research & Practice in Assessment 8 (2013), 13 -- 25 . L. B. Breslow, D. E. Pritchard, J. DeBoer, G. S. Stump, A. D. Ho, and D. T. Seaton. 2013. Studying learning in the worldwide classroom: Research into edX's first MOOC. Research & Practice in Assessment 8 (2013), 13--25.","journal-title":"Research & Practice in Assessment"},{"key":"e_1_2_1_10_1","unstructured":"B. Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann.   B. Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/358916.358984"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the Frontiers in Education Conference","volume":"2","author":"Carlson P. A.","unstructured":"P. A. Carlson and F. C. Berry . 2003. Calibrated Peer Review and assessing learning outcomes . In Proceedings of the Frontiers in Education Conference , Vol. 2 . STIPES. P. A. Carlson and F. C. Berry. 2003. Calibrated Peer Review and assessing learning outcomes. In Proceedings of the Frontiers in Education Conference, Vol. 2. STIPES."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370020701851086"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1953163.1953344"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1083-6101.2008.00416.x"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02574-7_67"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1151954.1067468"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15326934crj0904_9"},{"key":"e_1_2_1_19_1","unstructured":"A. T. Corbett K. R. Koedinaer and W. Haaley. 2002. Cognitive tutors: From the research classroom to all classrooms. In P. S. Goodman Ed. Technology Enhanced Learning: Opportunities for Change. Lawrence Erlbaum Associates Mahwah NJ 235.  A. T. Corbett K. R. Koedinaer and W. Haaley. 2002. Cognitive tutors: From the research classroom to all classrooms. In P. S. Goodman Ed. Technology Enhanced Learning: Opportunities for Change. Lawrence Erlbaum Associates Mahwah NJ 235."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI'10)","author":"Dai P.","unstructured":"P. Dai , Mausam D., and D. S. Weld . 2010. Decision-theoretic control of crowd-sourced workflows . In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI'10) . P. Dai, Mausam D., and D. S. Weld. 2010. Decision-theoretic control of crowd-sourced workflows. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI'10)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1177\/1050651907311923"},{"key":"e_1_2_1_22_1","volume-title":"Assessment focus in studio: What is most prominent in architecture, art and design&quest","author":"Harpe B. De La","year":"2009","unstructured":"B. De La Harpe , J. F. Peterson , N. Frankham , R. Zehner , D. Neale , E. Musgrave , and R. McDermott . 2009. Assessment focus in studio: What is most prominent in architecture, art and design&quest ; International Journal of Art & Design Education 28, 1 ( 2009 ), 37--51. B. De La Harpe, J. F. Peterson, N. Frankham, R. Zehner, D. Neale, E. Musgrave, and R. McDermott. 2009. Assessment focus in studio: What is most prominent in architecture, art and design&quest; International Journal of Art & Design Education 28, 1 (2009), 37--51."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1879831.1879836"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_25_1","volume-title":"The Architecture of the Ecole des Beaux-Arts","author":"Drexler A.","unstructured":"A. Drexler , R. Chafee , and others. 1977. The Architecture of the Ecole des Beaux-Arts . MIT Press , Cambridge, MA . A. Drexler, R. Chafee, and others. 1977. The Architecture of the Ecole des Beaux-Arts. MIT Press, Cambridge, MA."},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"B. Efron and R. Tibshirani. 1993. An Introduction to the Bootstrap. Vol. 57. Chapman & Hall\/CRC Boca Raton FL.  B. Efron and R. Tibshirani. 1993. An Introduction to the Bootstrap. Vol. 57. Chapman & Hall\/CRC Boca Raton FL.","DOI":"10.1007\/978-1-4899-4541-9"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2007.05.002"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.3102\/00346543070003287"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642652"},{"key":"e_1_2_1_30_1","volume-title":"Practical art criticism","author":"Feldman E. B.","unstructured":"E. B. Feldman . 1994. Practical art criticism . Prentice Hall New York . E. B. Feldman. 1994. Practical art criticism. Prentice Hall New York."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1013115.1013152"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2160718.2160732"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1006\/jesp.1999.1409"},{"key":"e_1_2_1_34_1","unstructured":"T. Gallien and J. Oomen-Early. 2008. Personalized versus collective instructor feedback in the online courseroom: Does type of feedback affect student satisfaction academic performance and perceived connectedness with the instructor&quest; International Journal on E-Learning 7 3 (2008) 463--476.  T. Gallien and J. Oomen-Early. 2008. Personalized versus collective instructor feedback in the online courseroom: Does type of feedback affect student satisfaction academic performance and perceived connectedness with the instructor&quest; International Journal on E-Learning 7 3 (2008) 463--476."},{"key":"e_1_2_1_35_1","first-page":"46","article-title":"Web-Based student writing and reviewing in a large biology lecture course","volume":"36","author":"Gerdeman R. D.","year":"2007","unstructured":"R. D. Gerdeman , A. A. Russell , and K. J. Worden . 2007 . Web-Based student writing and reviewing in a large biology lecture course . Journal of College Science Teaching 36 , 5 (2007), 46 -- 52 . R. D. Gerdeman, A. A. Russell, and K. J. Worden. 2007. Web-Based student writing and reviewing in a large biology lecture course. Journal of College Science Teaching 36, 5 (2007), 46--52.","journal-title":"Journal of College Science Teaching"},{"key":"e_1_2_1_36_1","volume-title":"Creativity and HCI: From Experience to Design in Education","author":"Greenberg S.","unstructured":"S. Greenberg . 2009. Embedding a design studio course in a conventional computer science program . In Creativity and HCI: From Experience to Design in Education . Springer , 23--41. S. Greenberg. 2009. Embedding a design studio course in a conventional computer science program. In Creativity and HCI: From Experience to Design in Education. Springer, 23--41."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213880"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/5254.889104"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 1995 Frontiers in Education Conference.","volume":"2","author":"Hsi S.","unstructured":"S. Hsi and A. M. Agogino . 1995. Scaffolding knowledge integration through designing multimedia case studies of engineering design . In Proceedings of the 1995 Frontiers in Education Conference. Vol. 2 . IEEE, 4d1--1. S. Hsi and A. M. Agogino. 1995. Scaffolding knowledge integration through designing multimedia case studies of engineering design. In Proceedings of the 1995 Frontiers in Education Conference. Vol. 2. IEEE, 4d1--1."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441847"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1080\/10400410802059929"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1038\/nsmb.2119"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM.","author":"Kim H.","unstructured":"H. Kim and P. Hinds . 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East . In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM. H. Kim and P. Hinds. 2012. Harmony vs. disruption: The effect of iterative prototyping on teams creative processes and outcomes in the West and the East. In Proceedings of the ICIC: International Conference on Intercultural Collaboration. ACM."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460296.2460330"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1142405.1142429"},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","unstructured":"R. E. Kraut and P. Resnick. 2011. Evidence-Based Social Design: Mining the Social Sciences to Build Online Communities. MIT Press Cambridge MA.  R. E. Kraut and P. Resnick. 2011. Evidence-Based Social Design: Mining the Social Sciences to Build Online Communities. MIT Press Cambridge MA.","DOI":"10.7551\/mitpress\/8472.001.0001"},{"key":"e_1_2_1_49_1","volume-title":"Psychology: A Handbook of Best Practices","author":"Kuebli J. E.","year":"2008","unstructured":"J. E. Kuebli , R. D. Harvey , and J. H. Korn . 2008 . Critical thinking in critical courses: Principles and applications. In D. S. Dunn, J. S. Halonen, and R. A. Smith, Eds. Teaching Critical Thinking in Psychology: A Handbook of Best Practices . Wiley-Blackwell , New York , 137. J. E. Kuebli, R. D. Harvey, and J. H. Korn. 2008. Critical thinking in critical courses: Principles and applications. In D. S. Dunn, J. S. Halonen, and R. A. Smith, Eds. Teaching Critical Thinking in Psychology: A Handbook of Best Practices. Wiley-Blackwell, New York, 137."},{"key":"e_1_2_1_50_1","volume-title":"Retrieved","author":"Kurhila J.","year":"2012","unstructured":"J. Kurhila . 2012 . Human-Computer Interaction by Coursera opened for credit for the students of the Department . Retrieved December 13, 2013 from http:\/\/www.cs.helsinki.fi\/en\/uutiset\/72025. J. Kurhila. 2012. Human-Computer Interaction by Coursera opened for credit for the students of the Department. Retrieved December 13, 2013 from http:\/\/www.cs.helsinki.fi\/en\/uutiset\/72025."},{"key":"e_1_2_1_51_1","volume-title":"How Designers Think: The Design Process Demystified","author":"Lawson B.","unstructured":"B. Lawson . 2006. How Designers Think: The Design Process Demystified . Architectual Press . B. Lawson. 2006. How Designers Think: The Design Process Demystified. Architectual Press."},{"key":"e_1_2_1_52_1","first-page":"2012","article-title":"Education site expands slate of universities and courses","volume":"19","author":"Lewin T.","year":"2012","unstructured":"T. Lewin . 2012 a. Education site expands slate of universities and courses . The New York Times . September 19 , 2012 . T. Lewin. 2012a. Education site expands slate of universities and courses. The New York Times. September 19, 2012.","journal-title":"The New York Times"},{"key":"e_1_2_1_53_1","first-page":"2012","article-title":"One course, 150,000 students","volume":"18","author":"Lewin T.","year":"2012","unstructured":"T. Lewin . 2012 b. One course, 150,000 students . The New York Times . July 18 , 2012 . T. Lewin. 2012b. One course, 150,000 students. The New York Times. July 18, 2012.","journal-title":"The New York Times"},{"key":"e_1_2_1_54_1","first-page":"2012","article-title":"College of future could be come one, come all","volume":"19","author":"Lewin T.","year":"2013","unstructured":"T. Lewin . 2013 a. College of future could be come one, come all . The New York Times . November 19 , 2012 . T. Lewin. 2013a. College of future could be come one, come all. The New York Times. November 19, 2012.","journal-title":"The New York Times"},{"key":"e_1_2_1_55_1","first-page":"2013","article-title":"Five online courses are eligible for college credit","volume":"6","author":"Lewin T.","year":"2013","unstructured":"T. Lewin . 2013 b. Five online courses are eligible for college credit . The New York Times . February 6 , 2013 . T. Lewin. 2013b. Five online courses are eligible for college credit. The New York Times. February 6, 2013.","journal-title":"The New York Times"},{"key":"e_1_2_1_56_1","first-page":"2013","article-title":"Students rush to web classes, but profits may be much later","volume":"6","author":"Lewin T.","year":"2013","unstructured":"T. Lewin . 2013 c. Students rush to web classes, but profits may be much later . The New York Times . January 6 , 2013 . T. Lewin. 2013c. Students rush to web classes, but profits may be much later. The New York Times. January 6, 2013.","journal-title":"The New York Times"},{"key":"e_1_2_1_57_1","first-page":"2013","article-title":"Universities abroad join partnerships on the web","volume":"20","author":"Lewin T.","year":"2013","unstructured":"T. Lewin . 2013 d. Universities abroad join partnerships on the web . The New York Times . February 20 , 2013 . T. Lewin. 2013d. Universities abroad join partnerships on the web. The New York Times. February 20, 2013.","journal-title":"The New York Times"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society.","author":"Little J. L.","unstructured":"J. L. Little and E. L. Bjork . 2012. Pretesting with multiple-choice questions facilitates learning . In Proceedings of the Annual Meeting of the Cognitive Science Society. J. L. Little and E. L. Bjork. 2012. Pretesting with multiple-choice questions facilitates learning. In Proceedings of the Annual Meeting of the Cognitive Science Society."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1006\/jmla.1993.1027"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441792"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2240236.2240246"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1509\/jmkr.45.6.633"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1018755201899"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1080\/03075070600572090"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.241424"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/191666.191729"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/302979.302982"},{"key":"e_1_2_1_68_1","unstructured":"A. Pendleton-Jullian. 2010. Four (&plus;1) Studios. CreateSpace Independent Publishing.  A. Pendleton-Jullian. 2010. Four (&plus;1) Studios. CreateSpace Independent Publishing."},{"key":"e_1_2_1_69_1","volume-title":"Forms of Intellectual Development in the College Years","author":"Perry W. G.","unstructured":"W. G. Perry . 1970. Forms of Intellectual Development in the College Years . Holt, New York . W. G. Perry. 1970. Forms of Intellectual Development in the College Years. Holt, New York."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1002\/tl.37219956304"},{"key":"e_1_2_1_71_1","volume-title":"Higher Education: An Evidence-based Perspective","author":"Pintrich P.","year":"2007","unstructured":"P. Pintrich and A. Zusho . 2007 . Student motivation and self-regulated learning in the college classroom. In R. P. Perry and J. C. Smart, Eds. The Scholarship of Teaching and Learning in Higher Education: An Evidence-based Perspective . Springer , 731--810. P. Pintrich and A. Zusho. 2007. Student motivation and self-regulated learning in the college classroom. In R. P. Perry and J. C. Smart, Eds. The Scholarship of Teaching and Learning in Higher Education: An Evidence-based Perspective. Springer, 731--810."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1076\/csed.13.3.191.14945"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/199691.199716"},{"key":"e_1_2_1_74_1","volume-title":"The Design Studio: An exploration of its traditions and potential","author":"Sch\u00f6n D.","year":"1985","unstructured":"D. Sch\u00f6n . 1985. The Design Studio: An exploration of its traditions and potential . London : Royal Institute of British Architects ( 1985 ). D. Sch\u00f6n. 1985. The Design Studio: An exploration of its traditions and potential. London: Royal Institute of British Architects (1985)."},{"key":"e_1_2_1_75_1","unstructured":"A. Snodgrass and R. Coyne. 2006. Interpretation in architecture: Design as a Way of Thinking. Routledge.  A. Snodgrass and R. Coyne. 2006. Interpretation in architecture: Design as a Way of Thinking. Routledge."},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP'12)","author":"Socher R.","unstructured":"R. Socher , B. Huval , C. D. Manning , and A. Y. Ng . 2012. Semantic compositionality through recursive matrix-vector spaces . In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP'12) . R. Socher, B. Huval, C. D. Manning, and A. Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing (EMNLP'12)."},{"key":"e_1_2_1_77_1","unstructured":"C. A. Stanley and M. E. Porter. 2002. Engaging Large Classes: Strategies and Techniques for College Faculty. ERIC.  C. A. Stanley and M. E. Porter. 2002. Engaging Large Classes: Strategies and Techniques for College Faculty. ERIC."},{"key":"e_1_2_1_78_1","unstructured":"J. Surowiecki. 2005. The Wisdom of Crowds. Anchor.   J. Surowiecki. 2005. The Wisdom of Crowds. Anchor."},{"key":"e_1_2_1_79_1","volume-title":"Clickworkers on Mars. American Scientist 90, 3","author":"Szpir M.","year":"2002","unstructured":"M. Szpir . 2002. Clickworkers on Mars. American Scientist 90, 3 ( 2002 ). M. Szpir. 2002. Clickworkers on Mars. American Scientist 90, 3 (2002)."},{"key":"e_1_2_1_80_1","first-page":"29","article-title":"CritViz: Web-based software supporting peer critique in large creative classrooms","volume":"15","author":"Tinapple D.","year":"2013","unstructured":"D. Tinapple , L. Olson , and John Sadauskas . 2013 . CritViz: Web-based software supporting peer critique in large creative classrooms . Bulletin of the IEEE Technical Committee on Learning Technology 15 , 1 (2013), 29 . D. Tinapple, L. Olson, and John Sadauskas. 2013. CritViz: Web-based software supporting peer critique in large creative classrooms. Bulletin of the IEEE Technical Committee on Learning Technology 15, 1 (2013), 29.","journal-title":"Bulletin of the IEEE Technical Committee on Learning Technology"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124960"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/107005.107070"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.3102\/00346543068003249"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0142-694X(99)00002-2"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1080\/1470329032000103816"},{"key":"e_1_2_1_86_1","volume-title":"ACM SIGMOD Blog. Retreived","author":"Widom J.","year":"2012","unstructured":"J. Widom . 2012 . From 100 Students to 100,000 . ACM SIGMOD Blog. Retreived December 13, 2013 from http:\/\/wp.sigmod.org\/&quest;p=165. J. Widom. 2012. From 100 Students to 100,000. ACM SIGMOD Blog. Retreived December 13, 2013 from http:\/\/wp.sigmod.org\/&quest;p=165."},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97322"},{"key":"e_1_2_1_88_1","first-page":"1220","article-title":"Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics","volume":"1","author":"Zaidan O. F.","year":"2011","unstructured":"O. F. Zaidan and C. Callison-Burch . 2011 . Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics : Human Language Technologies , Vol. 1. 1220 -- 1229 . O. F. Zaidan and C. Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Vol. 1. 1220--1229.","journal-title":"Human Language Technologies"},{"key":"e_1_2_1_89_1","first-page":"289","article-title":"Reflections on theories of self-regulated learning and academic achievement","volume":"2","author":"Zimmerman B. J.","year":"2001","unstructured":"B. J. Zimmerman and D. H. Schunk . 2001 . Reflections on theories of self-regulated learning and academic achievement . Self-regulated Learning and Academic Achievement: Theoretical Perspectives 2 (2001), 289 -- 307 . B. J. Zimmerman and D. H. Schunk. 2001. Reflections on theories of self-regulated learning and academic achievement. Self-regulated Learning and Academic Achievement: Theoretical Perspectives 2 (2001), 289--307.","journal-title":"Self-regulated Learning and Academic Achievement: Theoretical Perspectives"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2505057","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2505057","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:28:57Z","timestamp":1750231737000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2505057"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,12]]},"references-count":89,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2013,12]]}},"alternative-id":["10.1145\/2505057"],"URL":"https:\/\/doi.org\/10.1145\/2505057","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,12]]},"assertion":[{"value":"2013-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-12-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}