{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T11:55:25Z","timestamp":1769946925634,"version":"3.49.0"},"reference-count":114,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2021,6,7]],"date-time":"2021-06-07T00:00:00Z","timestamp":1623024000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>This study utilized the application of authentic Virtual Reality (VR) to replicate the real-world complex system scenarios of a large retail supply chain. The proposed VR scenarios were developed based on an established systems thinking instrument that consists of seven dimensions: level of complexity, independence, interaction, change, uncertainty, systems\u2019 worldview, and flexibility. However, in this study, we only developed the VR scenarios for the first dimension, level of complexity, to assess an individual\u2019s Systems Thinking Skills (STS) when he or she engages in a turbulent virtual environment. The main objective of this study was to compare a student\u2019s STS when using traditional ST instruments versus VR scenarios for the complexity dimension. The secondary aim was to investigate the efficacy of VR scenarios utilizing three measurements: Simulation Sickness Questionnaire (SSQ), System Usability Scale (SUS), and Presence Questionnaire (PQ). In addition to the three measures, NASA TLX assessment was also performed to assess the perceived workload with regards to performing the tasks in VR scenarios. The results show students\u2019 preferences in the VR scenarios are not significantly different from their responses obtained using the traditional systems skills instrument. The efficacy measures confirmed that the developed VR scenarios are user friendly and lie in an acceptable region for users. Finally, the overall NASA TLX score suggests that users require 36% perceived work effort to perform the activities in VR scenarios.<\/jats:p>","DOI":"10.3390\/systems9020040","type":"journal-article","created":{"date-parts":[[2021,6,7]],"date-time":"2021-06-07T22:23:00Z","timestamp":1623104580000},"page":"40","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["An Assessment of Individuals\u2019 Systems Thinking Skills via Immersive Virtual Reality Complex System Scenarios"],"prefix":"10.3390","volume":"9","author":[{"given":"Vidanelage L.","family":"Dayarathna","sequence":"first","affiliation":[{"name":"Department of Industrial and Systems Engineering, Mississippi State University, Starkville, MS 39762, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sofia","family":"Karam","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, Mississippi State University, Starkville, MS 39762, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Raed","family":"Jaradat","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, Mississippi State University, Starkville, MS 39762, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael A.","family":"Hamilton","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, North Carolina A&T State University, 1601 E. Market Street, 402 McNair Hall Room 405, Greensboro, NC 27411, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Parker","family":"Jones","sequence":"additional","affiliation":[{"name":"Institute for Systems Engineering Research, Mississippi State University, ISER, 3909 Halls Ferry Road, Vicksburg, MS 39180, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emily S.","family":"Wall","sequence":"additional","affiliation":[{"name":"Center for Advanced Vehicular Systems Extension, Mississippi State University, 1 Research Blvd. Suite 200, Starkville, MS 39759, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Safae","family":"El Amrani","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, Mississippi State University, Starkville, MS 39762, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6775-585X","authenticated-orcid":false,"given":"Niamat Ullah","family":"Ibne Hossain","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, Mississippi State University, Starkville, MS 39762, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8640-4273","authenticated-orcid":false,"given":"Fatine","family":"Elakramine","sequence":"additional","affiliation":[{"name":"Department of Industrial and Systems Engineering, Mississippi State University, Starkville, MS 39762, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"113585","DOI":"10.1016\/j.eswa.2020.113585","article-title":"Integrating Systems Thinking Skills with Multi-Criteria Decision-Making Technology to Recruit Employee Candidates","volume":"60","author":"Karam","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1016\/j.cie.2015.11.018","article-title":"Competence-based performance model of multi-skilled workers","volume":"91","author":"Korytkowski","year":"2016","journal-title":"Comput. Ind. Eng."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1016\/j.outlook.2008.06.010","article-title":"Applications of complex systems theory in nursing education, research, and practice","volume":"56","author":"Clancy","year":"2008","journal-title":"Nurs. Outlook"},{"key":"ref_4","first-page":"1","article-title":"Assessment of Workforce Systems Preferences\/Skills Based on Employment Domain","volume":"32","author":"Jaradat","year":"2019","journal-title":"EMJ Eng. Manag. J."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1203","DOI":"10.1109\/JSYST.2017.2652218","article-title":"Individual capacity and organizational competency for systems thinking","volume":"12","author":"Jaradat","year":"2017","journal-title":"IEEE Syst. J."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1002\/sdr.281","article-title":"Links between systems thinking and complex decision making","volume":"20","author":"Maani","year":"2004","journal-title":"Syst. Dyn. Rev. J. Syst. Dyn. Soc."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Sterman, J. (2002, January 29\u201330). System Dynamics: Systems thinking and modeling for a complex world. Proceedings of the ESD Internal Symposium, MIT, Cambridge, MA, USA.","DOI":"10.1109\/EMR.2002.1022404"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/09500690601101672","article-title":"An evolutionary approach to harnessing complex systems thinking in the science and technology classroom","volume":"30","author":"Yoon","year":"2008","journal-title":"Int. J. Sci. Educ."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.cie.2019.04.048","article-title":"A Modified Firefly Algorithm for optimizing a multi stage supply chain network with stochastic demand and fuzzy production capacity","volume":"133","author":"Khalifehzadeh","year":"2019","journal-title":"Comput. Ind. Eng."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1535","DOI":"10.1016\/j.cie.2008.09.038","article-title":"Supply chain modeling in uncertain environment with bi-objective approach","volume":"56","author":"Mahnam","year":"2009","journal-title":"Comput. Ind. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Boardman, J., and Sauser, B. (2008). Systems Thinking: Coping with 21st Century Problems, CRC Press.","DOI":"10.1201\/9781420054927"},{"key":"ref_12","unstructured":"Cook, S.C., and Sproles, N. (2000, January 15\u201317). Synoptic views of defense Systems development. Proceedings of the SETE 2000, Brisbane, Australia."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.51594\/estj.v1i1.36","article-title":"A discrete event simulation (des) based approach to maximize the patient throughput in outpatient clinic","volume":"1","author":"Dayarathna","year":"2019","journal-title":"Eng. Sci. Tech. J."},{"key":"ref_14","unstructured":"Hossain, N.U.I. (2018). A synthesis of definitions for systems engineering. Proceedings of the International Annual Conference of the American Society for Engineering Management, Coeur d\u2019Alene, ID, USA, 17\u201320 October 2018, American Society for Engineering Management (ASEM)."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1504\/IJSSE.2015.068813","article-title":"Complex system governance requires systems thinking-how to find systems thinkers","volume":"6","author":"Jaradat","year":"2015","journal-title":"Int. J. Syst. Syst. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Jamshidi, M. (2008). Emergence in system of systems. System of Systems Engineering-Innovation for the 21st Century, John Wiley & Sons.","DOI":"10.1002\/9780470403501"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1002\/sres.3850120306","article-title":"Only connect! An annotated bibliography reflecting the breadth and diversity of systems thinking","volume":"12","author":"Lane","year":"1995","journal-title":"Syst. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1002\/(SICI)1520-6858(1998)1:4<267::AID-SYS3>3.0.CO;2-D","article-title":"Architecting principles for systems-of-systems","volume":"1","author":"Maier","year":"1998","journal-title":"Syst. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Checkland, P. (1999). Systems thinking. Rethinking Management Information Systems, Oxford University Press.","DOI":"10.1093\/oso\/9780198775331.003.0004"},{"key":"ref_20","first-page":"1","article-title":"A Literature Review of the Universal and Atomic Elements of Complex Cognition","volume":"Volume 20","author":"Cabrera","year":"2021","journal-title":"The Routledge Handbook of Systems Thinking"},{"key":"ref_21","unstructured":"Cabrera, D., and Cabrera, L. (2015). Systems Thinking Made Simple: New Hope for Solving Wicked Problems, Odyssean Press. [1st ed.]."},{"key":"ref_22","unstructured":"Cabrera, D., Cabrera, L., and Midgley, G. (2021). The Four Waves of Systems Thinking. The Routledge Handbook of Systems Thinking, Routledge."},{"key":"ref_23","unstructured":"(2020, May 21). The Home of the Beergame. Available online: https:\/\/beergame.org\/."},{"key":"ref_24","unstructured":"(2020, May 21). Learning Technologies. Available online: http:\/\/responsive.net\/littlefield.html."},{"key":"ref_25","unstructured":"(2020, May 25). Capsim. Available online: https:\/\/www.capsim.com\/."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1002\/cplx.1027","article-title":"Problem solving, cognition, and complex systems: Differences between experts and novices","volume":"6","author":"Jacobson","year":"2001","journal-title":"Complexity"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Bruni, R., Carrubbo, L., Cavacece, Y., and Sarno, D. (2018). An Overview of the Contribution of Systems Thinking Within Management and Marketing. Social Dynamics in a Systems Perspective, Springer.","DOI":"10.1007\/978-3-319-61967-5_13"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1325","DOI":"10.1016\/j.ijproman.2015.02.005","article-title":"Implementing systems thinking to manage risk in public private partnership projects","volume":"33","author":"Loosemore","year":"2015","journal-title":"Int. J. Proj. Manag."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Hossain, N.U.I., Dayarathna, V.L., Nagahi, M., and Jaradat, R. (2020). Systems Thinking: A Review and Bibliometric Analysis. Systems, 8.","DOI":"10.3390\/systems8030023"},{"key":"ref_30","unstructured":"Spector, M.J., Lockee, B.B., and Childress, M.D. (2019). Complexity and systems thinking models in education: Applications for leaders. Learning, Design, and Technology: An International Compendium of Theory, Research, Practice, and Policy, Springer."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1007\/s11528-020-00538-9","article-title":"Analyzing the Complexities of Online Education Systems: A Systems Thinking Perspective","volume":"64","author":"Tamim","year":"2020","journal-title":"TechTrends"},{"key":"ref_32","first-page":"1190","article-title":"High school programme in electro-optics: A case study on interdisciplinary learning and systems thinking","volume":"30","author":"Gero","year":"2014","journal-title":"Int. J. Eng. Edu."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1016\/j.apergo.2009.12.011","article-title":"Human factors in patient safety as an innovation","volume":"41","author":"Carayon","year":"2010","journal-title":"Appl. Ergon."},{"key":"ref_34","first-page":"100","article-title":"High-school course on engineering design: Enhancement of students\u2019 motivation and development of systems thinking skills","volume":"32","author":"Gero","year":"2016","journal-title":"Int. J. Eng. Educ."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1858","DOI":"10.1111\/1556-4029.13758","article-title":"A systems approach to biometrics in the military domain","volume":"63","author":"Wilson","year":"2018","journal-title":"J. Forensic Sci."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1016\/j.ssci.2018.12.020","article-title":"Systems Theoretic Accident Model and Process (STAMP) applied to a Royal Navy Hawk jet missile simulation exercise","volume":"113","author":"Stanton","year":"2019","journal-title":"Saf. Sci."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2362","DOI":"10.3168\/jds.S0022-0302(91)78410-5","article-title":"Systems thinking and practice in agriculture","volume":"74","author":"Bawden","year":"1991","journal-title":"J. Dairy Sci."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/j.agsy.2018.12.008","article-title":"Contribution of systems thinking and complex adaptive system attributes to sustainable food production: Example from a climate-smart village","volume":"171","author":"Kessler","year":"2019","journal-title":"Agric. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"53","DOI":"10.3733\/ca.v070n02p53","article-title":"Conservation agriculture: Systems thinking for sustainable farming","volume":"70","author":"Mitchell","year":"2016","journal-title":"Calif. Agric."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1016\/j.pce.2011.06.006","article-title":"Volcanic ash impacts on critical infrastructure","volume":"45","author":"Wilson","year":"2012","journal-title":"Phys. Chem. Earth"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"15S","DOI":"10.1177\/1010539515595694","article-title":"Urbanization, extreme events, and health: The case for systems approaches in mitigation, management, and response","volume":"28","author":"Siri","year":"2016","journal-title":"Asia Pac. J. Public Health"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"S196","DOI":"10.1016\/j.amepre.2008.05.014","article-title":"Systems thinking to improve the public\u2019s health","volume":"35","author":"Leischow","year":"2008","journal-title":"Am. J. Prev. Med."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"538","DOI":"10.2105\/AJPH.2005.066001","article-title":"Practical challenges of systems thinking and modeling in public health","volume":"96","author":"Trochim","year":"2006","journal-title":"Am. J. Public Health"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Madni, A.M., Boehm, B., Ghanem, R., Erwin, D., and Wheaton, M.J. (2018). Evaluating how system health assessment can trigger anticipatory action for resilience. Disciplinary Convergence in Systems Engineering Research, Springer.","DOI":"10.1007\/978-3-319-62217-0"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1002\/pfi.4170300510","article-title":"The fifth discipline, the art and practice of the learning organization","volume":"30","author":"Senge","year":"1991","journal-title":"Perform. Instr."},{"key":"ref_46","unstructured":"Senge, P.M. (1990). The Art and Practice of the Learning Organization, Doubleday."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/j.ijdrr.2014.05.001","article-title":"Preparing for complex interdependent risks: A system of systems approach to building disaster resilience","volume":"9","author":"Cavallo","year":"2014","journal-title":"Int. J. Disaster Risk Reduct."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.ress.2014.10.008","article-title":"A systems approach to risk management through leading safety indicators","volume":"136","author":"Leveson","year":"2015","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1829","DOI":"10.1016\/j.ssci.2012.04.011","article-title":"Road transport in drift? Applying contemporary systems thinking to road safety","volume":"50","author":"Salmon","year":"2012","journal-title":"Saf. Sci."},{"key":"ref_50","first-page":"28","article-title":"Using systems thinking in patient safety: A case study on medicine management","volume":"24","author":"Brimble","year":"2017","journal-title":"Nurs. Manag."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1080\/00140139.2017.1329939","article-title":"A participatory systems approach to design for safer integrated medicine management","volume":"61","author":"Jun","year":"2018","journal-title":"Ergonomics"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1111\/jep.12856","article-title":"Thinking about complexity in health: A systematic review of the key systems thinking and complexity ideas in health","volume":"24","author":"Rusoja","year":"2018","journal-title":"J. Eval. Clin. Pract."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1177\/875697281704800207","article-title":"Toward a systemic view to cost overrun causation in infrastructure projects: A review and implications for research","volume":"48","author":"Love","year":"2017","journal-title":"Proj. Manag. J."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1108\/17538371211192937","article-title":"Managing project problem-solving patterns","volume":"5","author":"Cavaleri","year":"2012","journal-title":"Int. J. Manag. Proj. Bus"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"288","DOI":"10.1177\/8756972819829641","article-title":"Systemic view to understanding design change causation and exploitation of communications and knowledge","volume":"50","author":"Yap","year":"2019","journal-title":"Proj. Manag. J."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"e06161","DOI":"10.1016\/j.heliyon.2021.e06161","article-title":"Globalization, quality and systems thinking: Integrating global quality Management and a systems view","volume":"7","author":"Bashan","year":"2021","journal-title":"Heliyon"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1108\/17542731011053280","article-title":"Systems thinking in quality management","volume":"22","author":"Conti","year":"2010","journal-title":"TQM J."},{"key":"ref_58","unstructured":"Dorani, K., Mortazavi, A., Dehdarian, A., Mahmoudi, H., Khandan, M., and Mashayekhi, A.N. (2015, January 19\u201323). Developing question sets to assess systems thinking skills. Proceedings of the 33rd International Conference of the System Dynamics Society, Cambridge, MA, USA."},{"key":"ref_59","first-page":"518","article-title":"Development of system thinking skills in the context of earth system education","volume":"42","author":"Assaraf","year":"2005","journal-title":"J. Res. Sci. Teach. Off. J. Natl. Assoc. Res. Sci. Teach."},{"key":"ref_60","unstructured":"Stave, K., and Hopper, M. (August, January 29). What constitutes systems thinking? A proposed taxonomy. Proceedings of the 25th International Conference of the System Dynamics Society, Boston, MA, USA."},{"key":"ref_61","unstructured":"Richmond, B. (2000). The \u201cThinking\u201d in Systems Thinking: Seven Essential Skills, Pegasus Communications."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1016\/j.evalprogplan.2008.04.001","article-title":"Distinctions, systems, relationships, and perspectives (DSRP): A theory of thinking and of things","volume":"31","author":"Cabrera","year":"2008","journal-title":"Eval. Program Plann."},{"key":"ref_63","unstructured":"Cabrera, D., Cabrera, L., and Midgley, G. (2021). Developing Personal Mastery of Systems Thinking. The Routledge Handbook of Systems Thinking, Routledge."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"534","DOI":"10.1002\/sres.2351","article-title":"A unifying theory of systems thinking with psychosocial applications","volume":"32","author":"Cabrera","year":"2015","journal-title":"Syst. Res. Behav. Sci."},{"key":"ref_65","unstructured":"Cabrera, D., Cabrera, L., and Midgley, G. (2021). Developing and Validating a Measurement of Systems Thinking: The Systems Thinking and Metacognitive Inventory (STMI). The Routledge Handbook of Systems Thinking, Routledge."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1002\/sdr.432","article-title":"Assessing individuals\u2019 understanding of nonlinear causal structures in complex systems","volume":"26","author":"Plate","year":"2010","journal-title":"Syst. Dyn. Rev."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1177\/0193945917697219","article-title":"Development of the systems thinking scale for adolescent behavior change","volume":"40","author":"Moore","year":"2018","journal-title":"West. J. Nurs. Res."},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1002\/sdr.198","article-title":"Bathtub dynamics: Initial results of a systems thinking inventory","volume":"16","author":"Sweeney","year":"2000","journal-title":"Syst. Dyn. Rev."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"125","DOI":"10.5617\/nordina.255","article-title":"System competence\u2013Are elementary students able to deal with a biological system?","volume":"6","author":"Sommer","year":"2010","journal-title":"Nord. Stud. Sci. Educ."},{"key":"ref_70","first-page":"63","article-title":"Computers in education: A brief history","volume":"24","author":"Molnar","year":"1997","journal-title":"Journal"},{"key":"ref_71","unstructured":"Sutherland, I.E. (1965). The ultimate display. Information Processing 1965, Proceedings of the IFIP Congress, New York, NY, USA, 24\u201329 May 1965, Macmillan and Co."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1115\/1.4041428","article-title":"Efficacy Investigation of Virtual Reality Teaching Module in Manufacturing System Design Course","volume":"141","author":"Ma","year":"2019","journal-title":"J. Mech. Des."},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Bricken, M., and Byrne, C.M. (1993). Summer students in virtual reality: A pilot study on educational applications of virtual reality technology. Virtual Real., 199\u2013217.","DOI":"10.1016\/B978-0-12-745045-2.50019-2"},{"key":"ref_74","first-page":"23","article-title":"Virtual reality in the classroom","volume":"33","author":"Pantelidis","year":"1993","journal-title":"Educ. Technol."},{"key":"ref_75","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1023\/A:1012009725532","article-title":"Experimental comparison of virtual reality with traditional teaching methods for teaching radioactivity","volume":"5","author":"Crosier","year":"2000","journal-title":"Educ. Inform. Tech."},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1007\/s10055-006-0039-1","article-title":"Presence: A unique characteristic in educational virtual environments","volume":"10","author":"Mikropoulos","year":"2006","journal-title":"Virtual Real."},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1080\/10494820500173714","article-title":"Brave new (interactive) worlds: A review of the design affordances and constraints of two 3D virtual worlds as interactive learning environments","volume":"13","author":"Dickey","year":"2005","journal-title":"Interact. Learn. Environ."},{"key":"ref_78","doi-asserted-by":"crossref","unstructured":"Dawley, L., and Dede, C. (2014). Situated Learning in Virtual Worlds and Immersive Simulations. Handbook of Research on Educational Communications and Technology, Springer.","DOI":"10.1007\/978-1-4614-3185-5_58"},{"key":"ref_79","unstructured":"Hamilton, M., Jaradat, R., Jones, P., Wall, E., Dayarathna, V., Ray, D., and Hsu, G. (2018, January 24\u201327). Immersive Virtual Training Environment for Teaching Single and Multi-Queuing Theory: Industrial Engineering Queuing Theory Concepts. Proceedings of the ASEE Annual Conference and Exposition, Salt Lake City, UT, USA."},{"key":"ref_80","first-page":"181","article-title":"Virtual Reality and education","volume":"58","author":"Byrne","year":"1994","journal-title":"IFIP Trans. A Comput. Sci. Tech."},{"key":"ref_81","unstructured":"Furness, T.A., and Winn, W. (2020, May 25). The Impact of Three-Dimensional Immersive Virtual Environments on Modern Pedagogy. Available online: http:\/\/vcell.ndsu.nodak.edu\/~ganesh\/seminar\/1997_The%20Impact%20of%20Three%20Dimensional%20Immersive%20Virtual%20Environments%20on%20Modern%20Pedagogy.htm."},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"626","DOI":"10.1109\/TVCG.2012.43","article-title":"Evaluating display fidelity and interaction fidelity in a virtual reality game","volume":"18","author":"McMahan","year":"2012","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_83","first-page":"626","article-title":"Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia","volume":"1","author":"Opdyke","year":"1995","journal-title":"Am. J. Psychiatry"},{"key":"ref_84","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1016\/j.gie.2011.07.063","article-title":"Strategies for training in diagnostic upper endoscopy: A prospective, randomized trial","volume":"75","author":"Ende","year":"2012","journal-title":"Gastrointest. Endosc."},{"key":"ref_85","doi-asserted-by":"crossref","first-page":"6","DOI":"10.4253\/wjge.v6.i1.6","article-title":"Virtual reality simulators for gastrointestinal endoscopy training","volume":"6","author":"Triantafyllou","year":"2014","journal-title":"World J. Gastrointest. Endosc."},{"key":"ref_86","unstructured":"Durlach, N.I., and Mavor, A.S. (1995). Virtual Reality Scientific and Technological Challenges, National Academy Press."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3912\/OJIN.Vol18No03Man01","article-title":"Quality and safety education for nurses (QSEN): The key is systems thinking","volume":"18","author":"Dolansky","year":"2013","journal-title":"Online J. Issues Nurs."},{"key":"ref_88","unstructured":"Hooper, M., and Stave, K.A. (2008, January 20\u201324). Assessing the Effectiveness of Systems Thinking Interventions in the Classroom. Proceedings of the 26th International Conference of the System Dynamics Society, Athens, Greece."},{"key":"ref_89","first-page":"1","article-title":"Attracting institutional support through better assessment of systems thinking","volume":"17","author":"Plate","year":"2008","journal-title":"Creat. Learn. Exch. Newsl."},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1109\/TSMC.2016.2607224","article-title":"Validation studies of a questionnaire developed to measure students\u2019 engagement with systems thinking","volume":"48","author":"Camelia","year":"2016","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1109\/TSMCC.2002.804450","article-title":"Characteristics of engineering systems thinking-a 3D approach for curriculum content","volume":"32","author":"Frank","year":"2002","journal-title":"IEEE Trans. Syst. Man Cybern. Part C"},{"key":"ref_92","doi-asserted-by":"crossref","unstructured":"Cogan, P.B. (2012). Assessing the Capacity for Engineering Systems Thinking (Cest) and Other Competencies of Systems Engineers. Systems Engineering-Practice and Theory, InTech.","DOI":"10.5772\/2121"},{"key":"ref_93","doi-asserted-by":"crossref","unstructured":"Deterding, S., Dixon, D., Khaled, R., and Nacke, L. (2011, January 28\u201330). From game design elements to gamefulness: Defining gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, Tampere, Finland.","DOI":"10.1145\/2181037.2181040"},{"key":"ref_94","unstructured":"Juul, J. (2003, January 4\u20136). The game, the player, the world: Looking for a heart of gameness. Proceedings of the DiGRA International Conference, Utrecht, The Netherlands."},{"key":"ref_95","doi-asserted-by":"crossref","unstructured":"Bogost, I. (2007). Persuasive Games, MIT Press.","DOI":"10.7551\/mitpress\/5334.001.0001"},{"key":"ref_96","unstructured":"McGonigal, J. (2011). Reality Is Broken: Why Games Make Us Better and How They Can Change the World, Penguin Books."},{"key":"ref_97","first-page":"14","article-title":"Game design as marketing: How game mechanics create demand for virtual goods","volume":"5","author":"Hamari","year":"2010","journal-title":"Int. J. Bus. Sci. Appl. Manag."},{"key":"ref_98","unstructured":"Hamari, J., and J\u00e4rvinen, A. (2011). Building customer relationship through game mechanics in social games. Business, Technological, and Social Dimensions of Computer Games: Multidisciplinary Developments, IGI Global."},{"key":"ref_99","unstructured":"Stenros, J., and Sotamaa, O. (2009, January 1\u20134). Commoditization of Helping Players Play: Rise of the Service Paradigm. Proceedings of the DiGRA: Breaking New Ground: Innovation in Games, Play, Practice and Theory, Brunel University, London, UK."},{"key":"ref_100","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1287\/ited.8.1.3","article-title":"Online games to teach operations","volume":"8","author":"Wood","year":"2007","journal-title":"INFORMS Trans. Educ."},{"key":"ref_101","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.tsc.2018.03.003","article-title":"Assessing systems thinking: A tool to measure complex reasoning through ill-structured problems","volume":"28","author":"Grohs","year":"2018","journal-title":"Think. Ski. Creat."},{"key":"ref_102","first-page":"1938","article-title":"Assessment of the Efficacy and Effectiveness of Virtual Reality Teaching Module: A Gender-based Comparison","volume":"36","author":"Dayarathna","year":"2020","journal-title":"Int. J. Eng. Educ."},{"key":"ref_103","doi-asserted-by":"crossref","first-page":"622","DOI":"10.1080\/09638280110111397","article-title":"An evaluation of the efficacy of training people with learning disabilities in a virtual environment","volume":"24","author":"Brooks","year":"2002","journal-title":"Disabil. Rehabil."},{"key":"ref_104","first-page":"123","article-title":"Virtual reality and cognitive assessment and Rehabilitation: The State of the Art","volume":"Volume 44","author":"Rivs","year":"1997","journal-title":"Virtual Reality in Neuro-Psycho-Physiology: Cognitive, Clinical and Methodological Issues in Assessment and Rehabilitation"},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1207\/s15327108ijap0303_3","article-title":"Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness","volume":"3","author":"Kennedy","year":"1993","journal-title":"Int. J. Aviat. Psychol."},{"key":"ref_106","first-page":"4","article-title":"SUS-A quick and dirty usability scale","volume":"189","author":"Brooke","year":"1996","journal-title":"Usability Eval. Ind."},{"key":"ref_107","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1162\/105474698565686","article-title":"Measuring presence in virtual environments: A presence questionnaire","volume":"7","author":"Witmer","year":"1998","journal-title":"Presence"},{"key":"ref_108","unstructured":"Hart, S.G. (1986). NASA Task Load Index (TLX). Volume 1.0; Paper and Pencil Package."},{"key":"ref_109","doi-asserted-by":"crossref","unstructured":"Bevan, N., Carter, J., Earthy, J., Geis, T., and Harker, S. (2016). New ISO standards for usability, usability reports and usability measures. Proceedings of the 2016 International Conference on Human-Computer Interaction (HCI\u20192016), Toronto, ON, Canada, 17\u201322 July 2016, Springer.","DOI":"10.1007\/978-3-319-39510-4_25"},{"key":"ref_110","doi-asserted-by":"crossref","unstructured":"Hart, S.G. (2006). NASA-Task Load Index (NASA-TLX) 20 Years Later. Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, Sage Publications.","DOI":"10.1037\/e577632012-009"},{"key":"ref_111","unstructured":"Stirgus, E., Nagahi, M., Ma, J., Jaradat, R., Strawderman, L., and Eakin, D. (2019, January 16\u201319). Determinants of systems thinking in college engineering students: Research initiation. Proceedings of the ASEE Annual Conference and Exposition, Tampa, FL, USA."},{"key":"ref_112","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1080\/10382046.2017.1386413","article-title":"Systems thinking in geography: Can high school students do it?","volume":"28","author":"Cox","year":"2019","journal-title":"Int. Res. Geogr. Environ. Educ."},{"key":"ref_113","first-page":"510","article-title":"The effect of an individual\u2019s education level on their systems skills in the system of systems domain","volume":"7","author":"Nagahi","year":"2020","journal-title":"J. Manag. Anal."},{"key":"ref_114","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1504\/IJSSE.2020.112312","article-title":"Holistic and reductionist thinker: A comparison study based on individuals\u2019 skillset and personality types","volume":"10","author":"Nagahi","year":"2020","journal-title":"Int. J. Syst. Syst. Eng."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/9\/2\/40\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:11:49Z","timestamp":1760163109000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/9\/2\/40"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,7]]},"references-count":114,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["systems9020040"],"URL":"https:\/\/doi.org\/10.3390\/systems9020040","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,7]]}}}