{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T05:00:03Z","timestamp":1772773203741,"version":"3.50.1"},"reference-count":14,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2015,4,15]],"date-time":"2015-04-15T00:00:00Z","timestamp":1429056000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2015,4,15]]},"abstract":"<jats:p>In the fall of 2013, we offered an open online Introduction to Recommender Systems through Coursera, while simultaneously offering a for-credit version of the course on-campus using the Coursera platform and a flipped classroom instruction model. As the goal of offering this course was to experiment with this type of instruction, we performed extensive evaluation including surveys of demographics, self-assessed skills, and learning intent; we also designed a knowledge-assessment tool specifically for the subject matter in this course, administering it before and after the course to measure learning, and again 5 months later to measure retention. We also tracked students through the course, including separating out students enrolled for credit from those enrolled only for the free, open course.<\/jats:p>\n          <jats:p>Students had significant knowledge gains across all levels of prior knowledge and across all demographic categories. The main predictor of knowledge gain was effort expended in the course. Students also had significant knowledge retention after the course. Both of these results are limited to the sample of students who chose to complete our knowledge tests. Student completion of the course was hard to predict, with few factors contributing predictive power; the main predictor of completion was intent to complete. Students who chose a concepts-only track with hand exercises achieved the same level of knowledge of recommender systems concepts as those who chose a programming track and its added assignments, though the programming students gained additional programming knowledge. Based on the limited data we were able to gather, face-to-face students performed as well as the online-only students or better; they preferred this format to traditional lecture for reasons ranging from pure convenience to the desire to watch videos at a different pace (slower for English language learners; faster for some native English speakers). This article also includes our qualitative observations, lessons learned, and future directions.<\/jats:p>","DOI":"10.1145\/2728171","type":"journal-article","created":{"date-parts":[[2015,4,17]],"date-time":"2015-04-17T22:12:01Z","timestamp":1429308721000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["Teaching Recommender Systems at Large Scale"],"prefix":"10.1145","volume":"22","author":[{"given":"Joseph A.","family":"Konstan","sequence":"first","affiliation":[{"name":"University of Minnesota, Minneapolis, MN"}]},{"given":"J. D.","family":"Walker","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN"}]},{"given":"D. Christopher","family":"Brooks","sequence":"additional","affiliation":[{"name":"EDUCAUSE, Louisville, CO"}]},{"given":"Keith","family":"Brown","sequence":"additional","affiliation":[{"name":"University of Minnesota, Buford Ave,MN"}]},{"given":"Michael D.","family":"Ekstrand","sequence":"additional","affiliation":[{"name":"Texas State University, San Marcos, TX"}]}],"member":"320","published-online":{"date-parts":[[2015,4,15]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"5","article-title":"Predicting non-traditional student learning outcomes using data analytics--a pilot research study","volume":"28","author":"Buerck John P.","year":"2013","unstructured":"John P. Buerck , Srikanth P. Mudigonda , Stephanie E. Mooshegian , Kyle Collins , Nicholas Grimm , Kristen Bonney , and Hadley Kombrink . 2013 . Predicting non-traditional student learning outcomes using data analytics--a pilot research study . Journal of the Computer Science College 28 , 5 (May 2013), 260--265. John P. Buerck, Srikanth P. Mudigonda, Stephanie E. Mooshegian, Kyle Collins, Nicholas Grimm, Kristen Bonney, and Hadley Kombrink. 2013. Predicting non-traditional student learning outcomes using data analytics--a pilot research study. Journal of the Computer Science College 28, 5 (May 2013), 260--265.","journal-title":"Journal of the Computer Science College"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556325.2566250"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460296.2460332"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.19173\/irrodl.v15i4.1902"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.3102\/0013189X14523038"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043958"},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"E. J. Emanuel. 2013. Online education: MOOCs taken by educated few. Nature 503 7476 342--342.  E. J. Emanuel. 2013. Online education: MOOCs taken by educated few. Nature 503 7476 342--342.","DOI":"10.1038\/503342a"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460296.2460330"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556325.2566244"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5408\/McConnell_v53p462"},{"key":"e_1_2_1_11_1","volume-title":"Online Learning: A Meta-Analysis and Review of Online Learning Studies","author":"Means B.","year":"2010","unstructured":"B. Means , Y. Toyama , R. Murphy , M. Bakia , and K. Jones . 2010 . Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies . Center for Technology in Learning, U.S. Department of Education , Washington, DC. B. Means, Y. Toyama, R. Murphy, M. Bakia, and K. Jones. 2010. Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Center for Technology in Learning, U.S. Department of Education, Washington, DC."},{"key":"e_1_2_1_12_1","unstructured":"E. T. Pascarella and P. T. Terenzini. 2005. How College Affects Students: Volume 2 A Third Decade of Research. Jossey-Bass San Francisco CA.  E. T. Pascarella and P. T. Terenzini. 2005. How College Affects Students: Volume 2 A Third Decade of Research. Jossey-Bass San Francisco CA."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.85.2.305"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556325.2566240"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2728171","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2728171","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:16:32Z","timestamp":1750227392000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2728171"}},"subtitle":["Evaluation and Lessons Learned from a Hybrid MOOC"],"short-title":[],"issued":{"date-parts":[[2015,4,15]]},"references-count":14,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2015,4,15]]}},"alternative-id":["10.1145\/2728171"],"URL":"https:\/\/doi.org\/10.1145\/2728171","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,4,15]]},"assertion":[{"value":"2014-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-04-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}