{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T22:05:48Z","timestamp":1780351548662,"version":"3.54.1"},"reference-count":35,"publisher":"ASME International","issue":"3","license":[{"start":{"date-parts":[[2023,8,2]],"date-time":"2023-08-02T00:00:00Z","timestamp":1690934400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.asme.org\/publications-submissions\/publishing-information\/legal-policies"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1624727 I\/UCRC Pennsylvania State University Site: Center for Health Organization Transformation"],"award-info":[{"award-number":["1624727 I\/UCRC Pennsylvania State University Site: Center for Health Organization Transformation"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1738359 I\/UCRC University of Louisville: Center for Health Organization Transformation"],"award-info":[{"award-number":["1738359 I\/UCRC University of Louisville: Center for Health Organization Transformation"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["RIEF-1830741"],"award-info":[{"award-number":["RIEF-1830741"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["asmedigitalcollection.asme.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Engineering is an inherently creative and collaborative endeavor to solve real-world problems, in which collaborative problem solving (CPS) is considered one of the most critical professional skills. Hands-on practices and assessment methods are essential to promote deeper learning and foster the development of professional skills. However, most existing approaches are based on out-of-process procedures such as surveys, tests, or interviews that measure the effectiveness of learning activity in an aggregated way. It is desirable to quantify CPS dynamics during the learning process. Advancements in virtual reality (VR) provide great opportunities to realize digital learning environments to facilitate a learning-by-doing curriculum. In addition, sensors in VR systems allow us to collect in-process user behavioral data. This paper presents a multiplayer VR manufacturing simulation game for virtual hands-on learning experiences, as well as a behavioral modeling method for monitoring the CPS skills of participants. First, we developed the Virtual Learning Factory, where users play simulation games of various manufacturing paradigms. Second, we collected action logs from a sample of participants and used the same pattern to generate more data. Third, the behavioral data are modeled as dynamic networks for each player. Last, network features are calculated, and a CPS scoring method is driven from them. Experimental results show that the proposed behavioral modeling successfully captures different patterns of CPS dynamics according to manufacturing paradigms and individuals. This detailed assessment contributes to the development of appropriate student-specific interventions to improve learning outcomes.<\/jats:p>","DOI":"10.1115\/1.4063089","type":"journal-article","created":{"date-parts":[[2023,8,2]],"date-time":"2023-08-02T05:51:32Z","timestamp":1690955492000},"update-policy":"https:\/\/doi.org\/10.1115\/crossmarkpolicy-asme","source":"Crossref","is-referenced-by-count":5,"title":["Behavioral Modeling of Collaborative Problem Solving in Multiplayer Virtual Reality Manufacturing Simulation Games"],"prefix":"10.1115","volume":"24","author":[{"given":"Haedong","family":"Kim","sequence":"first","affiliation":[{"name":"The Pennsylvania State University Complex Systems Monitoring, Modeling, and Control Laboratory, , University Park, PA 16802"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tyler","family":"Hartleb","sequence":"additional","affiliation":[{"name":"University of Calgary Department of Computer Science, , Calgary, AB T2N 1N4 ,","place":["Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Khalid","family":"Bello","sequence":"additional","affiliation":[{"name":"The University of Louisville Department of Industrial Engineering, , Louisville, KY 40292"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faisal","family":"Aqlan","sequence":"additional","affiliation":[{"name":"The University of Louisville Department of Industrial Engineering, , Louisville, KY 40292"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Zhao","sequence":"additional","affiliation":[{"name":"University of Calgary Department of Computer Science, , Calgary, AB T2N 1N4 ,","place":["Canada"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hui","family":"Yang","sequence":"additional","affiliation":[{"name":"The Pennsylvania State University Complex Systems Monitoring, Modeling, and Control Laboratory, , University Park, PA 16802"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"33","published-online":{"date-parts":[[2023,10,10]]},"reference":[{"key":"2024101000020300400_CIT0001","doi-asserted-by":"publisher","first-page":"261","DOI":"10.3389\/fnhum.2018.00261","article-title":"Real World Problem-Solving","volume":"12","author":"Sarathy","year":"2018","journal-title":"Front. Hum. Neurosci."},{"issue":"6","key":"2024101000020300400_CIT0002","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1111\/jcal.12208","article-title":"An Analysis of Collaborative Problem-Solving Activities Mediated by Individual-Based and Collaborative Computer Simulations","volume":"33","author":"Chang","year":"2017","journal-title":"J. Comput. Assist. Learn."},{"issue":"4","key":"2024101000020300400_CIT0003","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1177\/1046878109334331","article-title":"Teaching Lean Manufacturing With Simulations and Games: A Survey and Future Directions","volume":"41","author":"Badurdeen","year":"2010","journal-title":"Simul. Gaming"},{"key":"2024101000020300400_CIT0004","doi-asserted-by":"crossref","first-page":"157","DOI":"10.4324\/9781315096773-8","volume-title":"Assessment of Problem Solving Using Simulations","author":"O\u2019Neil","year":"2017"},{"issue":"1","key":"2024101000020300400_CIT0005","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1002\/(SICI)1520-6386(199931)10:1\u00a134::AID-CIR6\u00bf3.0.CO;2-H","article-title":"Putting the Lesson Before the Test: Using Simulation to Analyze and Develop Competitive Strategies","volume":"10","author":"Reibstein","year":"1999","journal-title":"Competitive Intell. Rev."},{"issue":"2","key":"2024101000020300400_CIT0006","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1002\/j.2168-9830.1997.tb00272.x","article-title":"The Learning Factory\u2014A New Approach to Integrating Design and Manufacturing Into the Engineering Curriculum","volume":"86","author":"Lamancusa","year":"1997","journal-title":"J. Eng. Educ."},{"key":"2024101000020300400_CIT0007","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/j.promfg.2017.04.027","article-title":"Potentials and Limits of Learning Factories in Research, Innovation Transfer, Education, and Training","volume":"9","author":"Tisch","year":"2017","journal-title":"Procedia Manuf."},{"key":"2024101000020300400_CIT0008","doi-asserted-by":"crossref","DOI":"10.2139\/ssrn.3861488","article-title":"User-Oriented Design of Virtual Reality Supported Learning Factory Trainings: Methodology for the Generation of Suitable Design Elements","author":"Riemann","year":"2021"},{"issue":"4","key":"2024101000020300400_CIT0009","doi-asserted-by":"publisher","first-page":"e0250041","DOI":"10.1371\/journal.pone.0250041","article-title":"An Observational Study of Engineering Online Education During the COVID-19 Pandemic","volume":"16","author":"Asgari","year":"2021","journal-title":"PLoS One"},{"issue":"1","key":"2024101000020300400_CIT0010","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1109\/TE.2021.3079523","article-title":"Assessment of Collaborative Problem Solving in Engineering Students Through Hands-On Simulations","volume":"65","author":"Aqlan","year":"2021","journal-title":"IEEE Trans. Educ."},{"issue":"1","key":"2024101000020300400_CIT0011","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/s10639-022-11167-5","article-title":"Systematic Literature Review and Bibliometric Analysis on Virtual Reality and Education","volume":"28","author":"Rojas-S\u00e1nchez","year":"2023","journal-title":"Educ. Inf. Technol."},{"issue":"6","key":"2024101000020300400_CIT0012","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1016\/j.jsurg.2016.05.012","article-title":"A Virtual Reality Training Curriculum for Laparoscopic Colorectal Surgery","volume":"73","author":"Beyer-Berjot","year":"2016","journal-title":"J. Surg. Educ."},{"issue":"8","key":"2024101000020300400_CIT0013","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1016\/S2215-0366(18)30226-8","article-title":"Automated Psychological Therapy Using Immersive Virtual Reality for Treatment of Fear of Heights: A Single-Blind, Parallel-Group, Randomised Controlled Trial","volume":"5","author":"Freeman","year":"2018","journal-title":"Lancet Psychiatry"},{"issue":"3","key":"2024101000020300400_CIT0014","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1002\/cae.20323","article-title":"Simulation Games in Engineering Education: A State-of-the-Art Review","volume":"19","author":"Deshpande","year":"2011","journal-title":"Comput. Appl. Eng. Educ."},{"key":"2024101000020300400_CIT0015","first-page":"4046","article-title":"Games and Simulations in Industrial Engineering Education: A Review of the Cognitive and Affective Learning Outcomes","author":"Despeisse","year":"2018"},{"key":"2024101000020300400_CIT0016","doi-asserted-by":"crossref","first-page":"103910","DOI":"10.1016\/j.compedu.2020.103910","article-title":"Self-regulated Mobile Game-Based English Learning in a Virtual Reality Environment","volume":"154","author":"Chen","year":"2020","journal-title":"Comput. Educ."},{"key":"2024101000020300400_CIT0017","doi-asserted-by":"crossref","DOI":"10.5220\/0007718500790087","article-title":"Virtual Reality in Self-regulated Learning: Example in Art Domain","author":"Sakdavong","year":"2019"},{"issue":"1","key":"2024101000020300400_CIT0018","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1080\/08923647.2016.1119621","article-title":"Designing a Virtual-Reality-Based, Gamelike Math Learning Environment","volume":"30","author":"Xu","year":"2016","journal-title":"Am. J. Distance Educ."},{"issue":"1","key":"2024101000020300400_CIT0019","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s40692-020-00169-2","article-title":"Immersive Virtual Reality as a Pedagogical Tool in Education: A Systematic Literature Review of Quantitative Learning Outcomes and Experimental Design","volume":"8","author":"Hamilton","year":"2021","journal-title":"J. Comput. Educ."},{"key":"2024101000020300400_CIT0020","doi-asserted-by":"crossref","first-page":"103871","DOI":"10.1016\/j.compedu.2020.103871","article-title":"Learning Analytics for Student Modeling in Virtual Reality Training Systems: Lineworkers Case","volume":"151","author":"Santamar\u00eda-Bonfil","year":"2020","journal-title":"Comput. Educ."},{"issue":"11","key":"2024101000020300400_CIT0021","doi-asserted-by":"publisher","first-page":"1190","DOI":"10.1080\/24725854.2018.1555383","article-title":"The Internet of Things for Smart Manufacturing: A Review","volume":"51","author":"Yang","year":"2019","journal-title":"IISE Trans."},{"key":"2024101000020300400_CIT0022","doi-asserted-by":"crossref","DOI":"10.18260\/1-2--29067","article-title":"Use of Virtual Reality Tools in an Undergraduate Mechanical Engineering Manufacturing Course","author":"Syed","year":"2017"},{"key":"2024101000020300400_CIT0023","doi-asserted-by":"crossref","DOI":"10.18260\/1-2--34989","article-title":"Multiplayer Physical and Virtual Reality Games for Team-Based Manufacturing Simulation","author":"Zhao","year":"2020"},{"issue":"6","key":"2024101000020300400_CIT0024","doi-asserted-by":"publisher","first-page":"683","DOI":"10.1080\/13562517.2013.827653","article-title":"An Overview of Learning Analytics","volume":"18","author":"Clow","year":"2013","journal-title":"Teach. Higher Educ."},{"key":"2024101000020300400_CIT0025","doi-asserted-by":"publisher","first-page":"103778","DOI":"10.1016\/j.compedu.2019.103778","article-title":"A Systematic Review of Immersive Virtual Reality Applications for Higher Education: Design Elements, Lessons Learned, and Research Agenda","volume":"147","author":"Radianti","year":"2020","journal-title":"Comput. Educ."},{"key":"2024101000020300400_CIT0026","doi-asserted-by":"publisher","first-page":"117220","DOI":"10.1016\/j.eswa.2022.117220","article-title":"Sensor-Based Modeling of Problem-Solving in Virtual Reality Manufacturing Systems","volume":"201","author":"Zhu","year":"2022","journal-title":"Expert Syst. Appl."},{"key":"2024101000020300400_CIT0027","doi-asserted-by":"publisher","DOI":"10.1787\/9789264285521-en","volume-title":"PISA 2015 Results (Volume V): Collaborative Problem Solving","author":"Pe\u00f1a-L\u00f3pez","year":"2017"},{"issue":"4","key":"2024101000020300400_CIT0028","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1080\/08957347.2016.1209204","article-title":"Assessment of Collaborative Problem Solving in Education Environments","volume":"29","author":"Care","year":"2016","journal-title":"Appl. Meas. Educ."},{"issue":"2","key":"2024101000020300400_CIT0029","doi-asserted-by":"publisher","first-page":"10","DOI":"10.3390\/jintelligence5020010","article-title":"Complex Problem Solving in Assessments of Collaborative Problem Solving","volume":"5","author":"Graesser","year":"2017","journal-title":"J. Intell."},{"issue":"2","key":"2024101000020300400_CIT0030","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1007\/s10339-009-0345-0","article-title":"Complex Problem Solving: A Case for Complex Cognition","volume":"11","author":"Funke","year":"2010","journal-title":"Cogn. Process."},{"key":"2024101000020300400_CIT0031","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1007\/s11412-016-9234-6","article-title":"Quantitative Approach to Collaborative Learning: Performance Prediction, Individual Assessment, and Group Composition","volume":"11","author":"Cen","year":"2016","journal-title":"Int. J. Comput. Supported Collab. Learn."},{"key":"2024101000020300400_CIT0032","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/j.compedu.2018.04.011","article-title":"Evaluating a Tactile and a Tangible Multi-tablet Gamified Quiz System for Collaborative Learning in Primary Education","volume":"123","author":"Garcia-Sanjuan","year":"2018","journal-title":"Comput. Educ."},{"issue":"3","key":"2024101000020300400_CIT0033","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1108\/02621710210420282","article-title":"Team Effectiveness\u2014Development of an Audit Questionnaire","volume":"21","author":"Bateman","year":"2002","journal-title":"J. Manage. Develop."},{"issue":"4","key":"2024101000020300400_CIT0034","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1177\/0021886394304005","article-title":"The Impact of Group Interaction Styles on Problem-Solving Effectiveness","volume":"30","author":"Cooke","year":"1994","journal-title":"J. Appl. Behav. Sci."},{"issue":"13","key":"2024101000020300400_CIT0035","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s13278-018-0493-2","article-title":"Study on Centrality Measures in Social Networks: a Survey","volume":"8","author":"Das","year":"2018","journal-title":"Soc. Netw. Anal. Mining"}],"container-title":["Journal of Computing and Information Science in Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/24\/3\/031006\/7050074\/jcise_24_3_031006.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/24\/3\/031006\/7050074\/jcise_24_3_031006.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,10]],"date-time":"2024-10-10T11:52:54Z","timestamp":1728561174000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article\/24\/3\/031006\/1166123\/Behavioral-Modeling-of-Collaborative-Problem"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,10]]},"references-count":35,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,3,1]]}},"URL":"https:\/\/doi.org\/10.1115\/1.4063089","relation":{},"ISSN":["1530-9827","1944-7078"],"issn-type":[{"value":"1530-9827","type":"print"},{"value":"1944-7078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,10]]},"article-number":"031006"}}