{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T23:52:42Z","timestamp":1783122762610,"version":"3.54.6"},"reference-count":50,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2020,4,17]],"date-time":"2020-04-17T00:00:00Z","timestamp":1587081600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>The myriad of problems facing the world today are increasingly complex, dynamic, and transcend multiple domains. This necessitates the need for trans-disciplinary approaches capable of providing a framework to help solve these problems. Systems thinking provides the skills necessary for people to approach these types of problems. However, a lack of awareness and understanding of systems thinking hinders a potential systems-literate and systems-capable society. Systems thinking is comprised of four underlying concepts or skills: distinction-making, organizing systems, inter-relating, and perspective-taking. The path to becoming a systems thinker follows a process comprised of three levels\u2014sensibility (awareness of systems), literacy (knowledge of systems), and capability (understanding of systems)\u2014repeated across multiple learning phases. During this research study, a method was defined to measure whether non-experts learned the underlying systems thinking concepts according to this learning process. An experiment was conducted with 97 middle and high school students who were asked to draw a fish-tank system before and after being taught to apply the systems thinking concepts as skills for identifying elements, interactions, and roles\/purposes. The results provide evidence to conclude that student learning of systems thinking significantly increased from the first drawing to the second drawing.<\/jats:p>","DOI":"10.3390\/systems8020011","type":"journal-article","created":{"date-parts":[[2020,4,21]],"date-time":"2020-04-21T04:49:38Z","timestamp":1587444578000},"page":"11","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["A Method for Measuring Systems Thinking Learning"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1984-5882","authenticated-orcid":false,"given":"Seth","family":"Taylor","sequence":"first","affiliation":[{"name":"School of Mechanical, Industrial, and Manufacturing Engineering, Oregon State University, 204 Rogers Hall, Corvallis, OR 97331, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0332-8752","authenticated-orcid":false,"given":"Javier","family":"Calvo-Amodio","sequence":"additional","affiliation":[{"name":"School of Mechanical, Industrial, and Manufacturing Engineering, Oregon State University, 204 Rogers Hall, Corvallis, OR 97331, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jay","family":"Well","sequence":"additional","affiliation":[{"name":"Science and Math Investigative Learning Experiences (SMILE) Program, Oregon State University, 18 Gladys Valley Center, Corvallis, OR 97331, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,4,17]]},"reference":[{"key":"ref_1","unstructured":"Suppe, F. (1977). The Structure of Scientific Theories, University of Illinois Press. [2nd ed.]."},{"key":"ref_2","unstructured":"Zalta, E.N. Vienna Circle. The Stanford Encyclopedia of Philosophy, Available online: https:\/\/plato.stanford.edu\/archives\/spr2019\/entries\/vienna-circle\/."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1037\/h0059151","article-title":"Reductionism","volume":"52","author":"Sloane","year":"1945","journal-title":"Psychol. Rev."},{"key":"ref_4","unstructured":"Jackson, M.C. (2003). Systems Thinking: Creative Holism for Managers, John Wiley & Sons Ltd."},{"key":"ref_5","unstructured":"Dori, D., Sillitto, H., Griego, R.M., McKinney, D., Arnold, E.P., Godfrey, P., Martin, J., Jackson, S., and Krob, D. (2019). System definition, system worldviews, and systemness characteristics. IEEE Syst. J., 1\u201311."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/S1053-5357(00)00058-5","article-title":"Systemism: The alternative to individualism and holism","volume":"29","author":"Bunge","year":"2000","journal-title":"J. Socio-Econ."},{"key":"ref_7","unstructured":"Madni, A.M., Boehm, B., Erwin, D.A., Ghanem, R., and Wheaton, M.J. (2017). Three general systems principles and their derivation: Insights from the philosophy of science applied to systems concepts. Disciplinary Convergence: Implications for Systems Engineering Research, Proceedings of the 15th Annual Conference on Systems Engineering Research, Redondo Beach, CA, USA, 23\u201325 March 2017, Systems Engineering Research Center (SERC)."},{"key":"ref_8","unstructured":"Checkland, P.B. (1981). Systems Thinking, Systems Practice, John Wiley & Sons Ltd."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/0362-3319(91)90040-B","article-title":"Advancing the social sciences through the interdisciplinary enterprise","volume":"28","author":"Stember","year":"1991","journal-title":"Soc. Sci. J."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Hofkirchner, W. (2017). Transdisciplinarity needs systemism. Systems, 5.","DOI":"10.3390\/systems5010015"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1002\/sres.2171","article-title":"Managing complexity issues through evolutionary learning laboratories","volume":"30","author":"Bosch","year":"2013","journal-title":"Syst. Res. Behav. Sci."},{"key":"ref_12","unstructured":"Kahneman, D. (2011). Thinking Fast and Slow, Farrar, Straus, and Giroux."},{"key":"ref_13","unstructured":"Crowell, F.A. (1992, January 12\u201317). Systems Literacy and the Literate Design of Educational Systems. Proceedings of the Thirty-Sixth Annual Meeting for the International Society for the System Sciences (ISSS), Denver, CO, USA."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1002\/sres.2491","article-title":"Observations on systems literacy at the international society for systems sciences (ISSS) 2016 conference","volume":"34","author":"Tuddenham","year":"2017","journal-title":"Syst. Res. Behav. Sci."},{"key":"ref_15","unstructured":"Cabrera, D., and Cabrera, L. (2015). Systems Thinking Made Simple: New Hope for Solving Wicked Problems, Odyssean Press."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1002\/sres.2436","article-title":"Governing in the anthropocene: Contributions from systems thinking in practice?","volume":"33","author":"Ison","year":"2016","journal-title":"Syst. Res. Behav. Sci."},{"key":"ref_17","unstructured":"NGSS Lead States (2013). Next Generation Science Standards: For States, By States, The National Academies Press."},{"key":"ref_18","first-page":"405","article-title":"Technology-supported inquiry for learning about aquatic ecosystems","volume":"10","author":"Eberbach","year":"2014","journal-title":"Eurasia J. Math. Sci. Technol. Educ."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1002\/tea.21187","article-title":"Using representational tools to learn about complex systems: A tale of two classrooms","volume":"52","author":"Liu","year":"2015","journal-title":"J. Res. Sci. Teach."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1023","DOI":"10.1002\/tea.20297","article-title":"Promoting complex systems learning through the use of conceptual representations in hypermedia","volume":"46","author":"Liu","year":"2009","journal-title":"J. Res. Sci. Teach."},{"key":"ref_21","unstructured":"Cabrera, D.A. (2006). Systems Thinking. [Ph.D. Thesis, Cornell University]."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1016\/j.evalprogplan.2007.12.001","article-title":"Systems thinking","volume":"31","author":"Cabrera","year":"2008","journal-title":"Eval. Program Plan."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Fran\u00e7ois, C. (2004). International Encyclopedia of Systems and Cybernetics, K.G. Saur Verlag GmbH. [2nd ed.].","DOI":"10.1515\/9783110968019"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Rousseau, D., Billingham, J., and Calvo-Amodio, J. (2018). Systemic semantics: A systems approach to building ontologies and concept maps. Systems, 6.","DOI":"10.3390\/systems6030032"},{"key":"ref_25","unstructured":"Deming, W.E. (1994). The New Economics: For Industry, Government, Education, Massachusetts Institute of Technology, Center for Advanced Educational Services. [2nd ed.]."},{"key":"ref_26","first-page":"14","article-title":"On learning and the systems that facilitate it","volume":"1","author":"Ackoff","year":"1999","journal-title":"Reflect. Soc. Organ. Learn. J."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1002\/sres.2155","article-title":"Systems thinking: Common ground or untapped territory?","volume":"28","author":"Buckle","year":"2012","journal-title":"Syst. Res. Behav. Sci."},{"key":"ref_28","unstructured":"Creative Learning Exchange (CLE). Creative Learning Exchange Brochure, Available online: http:\/\/static.clexchange.org\/ftp\/CLEBrochure_2013.pdf."},{"key":"ref_29","unstructured":"HeHfafOpen University (OU). MSc in Systems Thinking in Practice, Available online: http:\/\/www.openuniversity.edu\/courses\/postgraduate\/qualifications\/f47."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Buckle, P. (2018). Maturity models for systems thinking. Systems, 6.","DOI":"10.3390\/systems6020023"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1207\/S15327809JLS0903_2","article-title":"Designing to learn about complex systems","volume":"9","author":"Hmelo","year":"2000","journal-title":"J. Learn. Sci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1080\/1533015X.2013.797860","article-title":"Fostering reasoning about complex systems: Using the aquarium to teach systems thinking","volume":"12","author":"Jordan","year":"2013","journal-title":"Appl. Environ. Educ. Commun."},{"key":"ref_33","unstructured":"Sensibility. Merriam Webster, Available online: https:\/\/www.merriam-webster.com\/dictionary\/sensibility."},{"key":"ref_34","unstructured":"Literacy. Merriam Webster, Available online: https:\/\/www.merriam-webster.com\/dictionary\/literacy."},{"key":"ref_35","unstructured":"Literate. Merriam Webster, Available online: https:\/\/www.merriam-webster.com\/dictionary\/literate#h1."},{"key":"ref_36","unstructured":"Capability. Merriam Webster, Available online: https:\/\/www.merriam-webster.com\/dictionary\/capability."},{"key":"ref_37","unstructured":"Capable. Merriam Webster, Available online: https:\/\/www.merriam-webster.com\/dictionary\/capable."},{"key":"ref_38","unstructured":"Comprehensive. Merriam Webster, Available online: https:\/\/www.merriam-webster.com\/dictionary\/comprehensive."},{"key":"ref_39","unstructured":"Comprehension. Merriam Webster, Available online: https:\/\/www.merriam-webster.com\/dictionary\/comprehension."},{"key":"ref_40","unstructured":"Ng, E.-H., Nepal, B., Schott, E., and Keathley, H. (2018, January 17\u201320). A Proposed Methodology for Developing Systems Thinking Lessons By and For Non-Experts. Proceedings of the American Society for Engineering Management (ASEM) 2018 International Annual Conference, Coeur d\u2019Alene, ID, USA."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Baca\u00ebr, N. (2011). Verhulst and the logistic equation (1838). A Short History of Mathematical Population Dynamics, Springer.","DOI":"10.1007\/978-0-85729-115-8_6"},{"key":"ref_42","unstructured":"Leedy, P.D., and Ormrod, J.E. (2016). Practical Research Planning and Design, Pearson Education, Inc.. [11th ed.]."},{"key":"ref_43","unstructured":"Mills, D. (1987). The Encyclopedia of the Marine Aquarium, Crescent Books."},{"key":"ref_44","unstructured":"Sharpe, S. (2019, July 27). Causes and cures for green aquarium water. The Spruce Pets. Available online: https:\/\/www.thesprucepets.com\/causes-and-cures-for-green-aquarium-water-1378633."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1207\/s15516709cog2801_7","article-title":"Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions","volume":"28","author":"Pfeffer","year":"2004","journal-title":"Cogn. Sci."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"587","DOI":"10.1111\/j.1468-2958.2002.tb00826.x","article-title":"Content analysis in mass communication: Assessment and reporting of intercoder reliability","volume":"28","author":"Lombard","year":"2002","journal-title":"Hum. Commun. Res."},{"key":"ref_47","unstructured":"Krippendorff, K. Computing Krippendorff\u2019s Alpha-Reliability. University of Pennsylvania Scholarly Commons: Annenberg School of Communication, Available online: https:\/\/repository.upenn.edu\/asc_papers\/43\/."},{"key":"ref_48","unstructured":"Neuendorf, K.A. (2002). The Content Analysis Guidebook, Sage."},{"key":"ref_49","unstructured":"Montgomery, D.C. (2013). Design and Analysis of Experiments, John Wiley & Sons, Inc.. [8th ed.]."},{"key":"ref_50","unstructured":"Conover, W.J. (1999). Practical Nonparametric Statistics, John Wiley & Sons, Inc.. [3rd ed.]."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/8\/2\/11\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:21:20Z","timestamp":1760361680000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/8\/2\/11"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,17]]},"references-count":50,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["systems8020011"],"URL":"https:\/\/doi.org\/10.3390\/systems8020011","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,17]]}}}