{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T18:45:13Z","timestamp":1768416313781,"version":"3.49.0"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2024,11,7]],"date-time":"2024-11-07T00:00:00Z","timestamp":1730937600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,11,7]]},"abstract":"<jats:p>Measuring instructors' adoption of learning management system (LMS) tools is a critical first step in evaluating the efficacy of online teaching and learning at scale. Existing models for LMS adoption are often qualitative, learner-centered, and difficult to leverage towards institutional support. We propose depth-of-use (DOU): an intuitive measurement model for faculty's utilization of a university-wide LMS and their needs for institutional support. We hypothesis-test the relationship between DOU and course attributes like modality, participation, logistics, and outcomes. In a large-scale analysis of metadata from 30000+ courses offered at Virginia Tech over two years, we find that a pervasive need for scale, interoperability and ubiquitous access drives LMS adoption by university instructors. We then demonstrate how DOU can help faculty members identify the opportunity-cost of transition from legacy apps to LMS tools. We also describe how DOU can help instructional designers and IT organizational leadership evaluate the impact of their support allocation, faculty development and LMS evangelism initiatives.<\/jats:p>","DOI":"10.1145\/3687051","type":"journal-article","created":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T15:52:40Z","timestamp":1731081160000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Simplify, Consolidate, Intervene: Facilitating Institutional Support with Mental Models of Learning Management System Use"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8991-9827","authenticated-orcid":false,"given":"Taha","family":"Hassan","sequence":"first","affiliation":[{"name":"Department of Computer Science, Virginia Tech, Blacksburg, Virginia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5861-1029","authenticated-orcid":false,"given":"Bob","family":"Edmison","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Virginia Tech, Blacksburg, Virginia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3586-2924","authenticated-orcid":false,"given":"Daron","family":"Williams","sequence":"additional","affiliation":[{"name":"Technology-Enhanced Learning and Online Strategies, Virginia Tech, Blacksburg, Virginia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5487-814X","authenticated-orcid":false,"given":"Larry","family":"Cox II","sequence":"additional","affiliation":[{"name":"Technology-Enhanced Learning and Online Strategies, Virginia Tech, Blacksburg, Virginia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2389-285X","authenticated-orcid":false,"given":"Matthew","family":"Louvet","sequence":"additional","affiliation":[{"name":"Technology-Enhanced Learning and Online Strategies, Virginia Tech, Blacksburg, Virginia, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1341-0669","authenticated-orcid":false,"given":"Bart","family":"Knijnenburg","sequence":"additional","affiliation":[{"name":"School of Computing, Clemson University, Clemson, South Carolina, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9839-7146","authenticated-orcid":false,"given":"D. Scott","family":"McCrickard","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Virginia Tech, Blacksburg, Virginia, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,11,8]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.28945\/1368"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2010.07.007"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2330601.2330666"},{"key":"e_1_2_1_4_1","first-page":"2","article-title":"Practical and Pedagogical Issues for Teacher Adoption of IMS Learning Design Standards in Moodle LMS","volume":"2","author":"Berggren Anders","year":"2005","unstructured":"Anders Berggren, Daniel Burgos, Josep M Fontana, Don Hinkelman, Vu Hung, Anthony Hursh, and Ger Tielemans. 2005. Practical and Pedagogical Issues for Teacher Adoption of IMS Learning Design Standards in Moodle LMS. Journal of Interactive Media in Education, Vol. 2, 02 (2005), 2.","journal-title":"Journal of Interactive Media in Education"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1080\/02680513.2018.1548964"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.iheduc.2008.03.002"},{"key":"e_1_2_1_7_1","volume-title":"UBICOMM 2012: The Sixth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies. 35--40","author":"Casany Guerrero Mar\u00eda Jos\u00e9","year":"2012","unstructured":"Mar\u00eda Jos\u00e9 Casany Guerrero, Marc Alier Forment, Nikolaos Galanis, Enric Mayol Sarroca, and Jordi Piguillem Poch. 2012. Analyzing Moodle\/LMS logs to measure mobile access. In UBICOMM 2012: The Sixth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies. 35--40."},{"key":"e_1_2_1_8_1","volume-title":"A critical examination of the effects of learning management systems on university teaching and learning. Tertiary education and management","author":"Coates Hamish","year":"2005","unstructured":"Hamish Coates, Richard James, and Gabrielle Baldwin. 2005. A critical examination of the effects of learning management systems on university teaching and learning. Tertiary education and management, Vol. 11 (2005), 19--36."},{"key":"e_1_2_1_9_1","volume-title":"Can log files analysis estimate learners' level of motivation","author":"Cocea Mihaela","unstructured":"Mihaela Cocea and Stephan Weibelzahl. 2006. Can log files analysis estimate learners' level of motivation. In LWA. University of Hildesheim, Institute of Computer Science."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-75195-3_2"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8535.2009.00970.x"},{"key":"e_1_2_1_12_1","volume-title":"Information systems success: The quest for the dependent variable. Information systems research","author":"DeLone William H","year":"1992","unstructured":"William H DeLone and Ephraim R McLean. 1992. Information systems success: The quest for the dependent variable. Information systems research, Vol. 3, 1 (1992), 60--95."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1080\/07421222.2003.11045748"},{"key":"e_1_2_1_14_1","first-page":"307","article-title":"Towards an enhanced learning management system for blended learning in higher education incorporating distinct learners' profiles","volume":"17","author":"Dias Sofia Balula","year":"2014","unstructured":"Sofia Balula Dias and Jos\u00e9 Alves Diniz. 2014. Towards an enhanced learning management system for blended learning in higher education incorporating distinct learners' profiles. Journal of Educational Technology & Society, Vol. 17, 1 (2014), 307--319.","journal-title":"Journal of Educational Technology & Society"},{"key":"e_1_2_1_15_1","volume-title":"6th Annual LMS Data Update. https:\/\/edutechnica.com\/2018\/10\/06\/6th-annual-lms-data-update\/","unstructured":"edutechnica. 2018. 6th Annual LMS Data Update. https:\/\/edutechnica.com\/2018\/10\/06\/6th-annual-lms-data-update\/"},{"key":"e_1_2_1_16_1","volume-title":"SDM 2017 Tutorials: Opportunities, challenges and methods for higher education data mining. https:\/\/www-users.cs.umn.edu\/ elbad004\/sdm2017_tutorial\/higher_edm_SDM_2017","author":"Elbadrawy Asmaa","year":"2017","unstructured":"Asmaa Elbadrawy, Huzefa Rangwala, Aditya Johri, and George Karypis. 2017. SDM 2017 Tutorials: Opportunities, challenges and methods for higher education data mining. https:\/\/www-users.cs.umn.edu\/ elbad004\/sdm2017_tutorial\/higher_edm_SDM_2017.pdf"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723576.2723590"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jhlste.2021.100302"},{"key":"e_1_2_1_19_1","article-title":"Expanding the Technology Acceptance Model (TAM) to examine faculty use of Learning Management Systems (LMSs) in higher education institutions","volume":"11","author":"Fathema Nafsaniath","year":"2015","unstructured":"Nafsaniath Fathema, David Shannon, and Margaret Ross. 2015. Expanding the Technology Acceptance Model (TAM) to examine faculty use of Learning Management Systems (LMSs) in higher education institutions. Journal of Online Learning & Teaching, Vol. 11, 2 (2015).","journal-title":"Journal of Online Learning & Teaching"},{"key":"e_1_2_1_20_1","article-title":"A learning analytics tool for usability assessment in moodle environments","volume":"13","author":"Fenu Gianni","year":"2017","unstructured":"Gianni Fenu, Mirko Marras, and Massimiliano Meles. 2017. A learning analytics tool for usability assessment in moodle environments. Journal of e-Learning and Knowledge Society, Vol. 13, 3 (2017).","journal-title":"Journal of e-Learning and Knowledge Society"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJTEL.2012.051816"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.2168-9830.2008.tb00989.x"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.12867"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328413.3328416"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317596"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341161.3342954"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341525.3387375"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450613.3456811"},{"key":"e_1_2_1_29_1","volume-title":"Benjamin IP Rubinstein, and Rui Zhang","author":"He Jiazhen","year":"2015","unstructured":"Jiazhen He, James Bailey, Benjamin IP Rubinstein, and Rui Zhang. 2015. Identifying At-Risk Students in Massive Open Online Courses.. In AAAI. 1749--1755."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.19173\/irrodl.v16i3.2015"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/aaai.12058"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2019.02.012"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/08923640701713430"},{"key":"e_1_2_1_34_1","volume-title":"The adoption of a social learning system: Intrinsic value in the UTAUT model. British Journal of Educational Technology","author":"Khechine Hager","year":"2020","unstructured":"Hager Khechine, Benoit Raymond, and Marc Augier. 2020. The adoption of a social learning system: Intrinsic value in the UTAUT model. British Journal of Educational Technology (2020)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1952.10483441"},{"key":"e_1_2_1_36_1","volume-title":"Mining LMS data to develop an early warning system for educators: A proof of concept. Computers & education","author":"Macfadyen Leah P","year":"2010","unstructured":"Leah P Macfadyen and Shane Dawson. 2010. Mining LMS data to develop an early warning system for educators: A proof of concept. Computers & education, Vol. 54, 2 (2010), 588--599."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.1600193"},{"key":"e_1_2_1_38_1","volume-title":"Recommender systems handbook","author":"Manouselis Nikos","unstructured":"Nikos Manouselis, Hendrik Drachsler, Riina Vuorikari, Hans Hummel, and Rob Koper. 2011. Recommender systems in technology enhanced learning. In Recommender systems handbook. Springer, 387--415."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11528-023-00911-4"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1080\/13562517.2020.1739641"},{"key":"e_1_2_1_41_1","first-page":"1","article-title":"Global standards for enhancing quality in online learning","volume":"18","author":"Martin Florence","year":"2017","unstructured":"Florence Martin, Drew Polly, Annika Jokiaho, and Birgit May. 2017. Global standards for enhancing quality in online learning. Quarterly Review of Distance Education, Vol. 18, 2 (2017), 1--102.","journal-title":"Quarterly Review of Distance Education"},{"key":"e_1_2_1_42_1","volume-title":"A systematic review of research on online teaching and learning from 2009 to","author":"Martin Florence","year":"2018","unstructured":"Florence Martin, Ting Sun, and Carl D Westine. 2020. A systematic review of research on online teaching and learning from 2009 to 2018. Computers & education, Vol. 159 (2020), 104009."},{"key":"e_1_2_1_43_1","volume-title":"The 7th Europian Conference on e-Learning","author":"Martin Ludivine","year":"2008","unstructured":"Ludivine Martin, David Rold\u00e1n Mart\u00ednez, Olga Revilla, Maria J Aguilar, Olga C Santos, and Jes\u00fas G Boticario. 2008. Usability in e-Learning Platforms: heuristics comparison between Moodle, Sakai and dotLRN. In The 7th Europian Conference on e-Learning, Agia Napa, Cyprus. Citeseer."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159491"},{"key":"e_1_2_1_45_1","unstructured":"Margaret Mattes. 2017. The private side of public higher education. (2017)."},{"key":"e_1_2_1_46_1","volume-title":"International Conference on Technology Enhanced Learning, Milan. 1--8.","author":"Mazza Riccardo","year":"2004","unstructured":"Riccardo Mazza and Christian Milani. 2004. Gismo: a graphical interactive student monitoring tool for course management systems. In International Conference on Technology Enhanced Learning, Milan. 1--8."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2008.10.002"},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","unstructured":"Susan D Moisey and Judith A Hughes. 2008. Supporting the online learner. Theory and practice of online learning (2008) 419--439.","DOI":"10.15215\/aupress\/9781897425084.019"},{"key":"e_1_2_1_49_1","volume-title":"International Journal of Education and Development using ICT","author":"Mtebe Joel","year":"2015","unstructured":"Joel Mtebe. 2015. Learning management system success: Increasing learning management system usage in higher education in sub-Saharan Africa. International Journal of Education and Development using ICT, Vol. 11, 2 (2015)."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2004.11.007"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2009.06.011"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3231011"},{"key":"e_1_2_1_53_1","volume-title":"A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers & education","author":"Radianti Jaziar","year":"2020","unstructured":"Jaziar Radianti, Tim A Majchrzak, Jennifer Fromm, and Isabell Wohlgenannt. 2020. A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers & education, Vol. 147 (2020), 103778."},{"key":"e_1_2_1_54_1","volume-title":"Incentives and obstacles influencing higher education faculty and administrators to teach via distance","author":"Rockwell S Kay","year":"1999","unstructured":"S Kay Rockwell, Jolene Schauer, Susan Fritz, and David B Marx. 1999. Incentives and obstacles influencing higher education faculty and administrators to teach via distance. Faculty Publications: Agricultural Leadership, Education & Communication Department (1999), 53."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2010.2053532"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2007.05.016"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1080\/14703297.2018.1557068"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.14742\/ajet.3959"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0019968"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3586183.3606756"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637371"},{"key":"e_1_2_1_62_1","volume-title":"Teaching courses online: A review of the research. Review of educational research","author":"Tallent-Runnels Mary K","year":"2006","unstructured":"Mary K Tallent-Runnels, Julie A Thomas, William Y Lan, Sandi Cooper, Terence C Ahern, Shana M Shaw, and Xiaoming Liu. 2006. Teaching courses online: A review of the research. Review of educational research, Vol. 76, 1 (2006), 93--135."},{"key":"e_1_2_1_63_1","unstructured":"UNESCO. 2020. UNESCO's COVID-19 Education Response - Distance Learning Tools. https:\/\/en.unesco.org\/covid19\/educationresponse\/solutions\/"},{"key":"e_1_2_1_64_1","unstructured":"US Department of Homeland Security. 2016. STEM Designated Degree Program List. https:\/\/www.ice.gov\/sites\/default\/files\/documents\/Document\/2016\/stem-list.pdf"},{"key":"e_1_2_1_65_1","unstructured":"Virginia Tech AIE. 2021. Peer Institutions | Analytics and Institutional Effectiveness | Virginia Tech. https:\/\/aie.vt.edu\/analytics-and-ai\/peer-institutions.html\/"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637361"},{"key":"e_1_2_1_67_1","volume-title":"Imitation and education: A philosophical inquiry into learning by example","author":"Warnick Bryan R","unstructured":"Bryan R Warnick. 2009. Imitation and education: A philosophical inquiry into learning by example. SUNY Press."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11423-006-9018-1"},{"key":"e_1_2_1_69_1","volume-title":"Society for Information Technology & Teacher Education International Conference. Association for the Advancement of Computing in Education (AACE), 1163--1168","author":"Wilcox Diane","year":"2016","unstructured":"Diane Wilcox, Jane Thall, and Oris Griffin. 2016. One canvas, two audiences: How faculty and students use a newly adopted learning management system. In Society for Information Technology & Teacher Education International Conference. Association for the Advancement of Computing in Education (AACE), 1163--1168."},{"key":"e_1_2_1_70_1","doi-asserted-by":"crossref","unstructured":"Daron Williams Larry Cox Margaret Ellis Bob Edmison Taha Hassan M Aaron Bond Quinn Warnick Virginia Clark Daniel Yaffe Molly Domino et al. 2023. Data-Informed Learning Design in a Computer Science Course. In Learning Design and Technology: An International Compendium of Theory Research Practice and Policy. Springer 2323--2345.","DOI":"10.1007\/978-3-319-17461-7_176"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460296.2460324"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.19173\/irrodl.v10i6.741"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687051","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3687051","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T00:57:54Z","timestamp":1755737874000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687051"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,7]]},"references-count":72,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2024,11,7]]}},"alternative-id":["10.1145\/3687051"],"URL":"https:\/\/doi.org\/10.1145\/3687051","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,7]]},"assertion":[{"value":"2024-11-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}