{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:06:45Z","timestamp":1755839205196,"version":"3.41.0"},"reference-count":73,"publisher":"Association for Computing Machinery (ACM)","issue":"CHI PLAY","license":[{"start":{"date-parts":[[2021,10,5]],"date-time":"2021-10-05T00:00:00Z","timestamp":1633392000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,5]]},"abstract":"<jats:p>For citizen science games (CSGs) to be successful in advancing scientific research, they must effectively train players. Designing tutorials for training can be aided through developing a skill chain of required skills and their dependencies, but skill chain development is an intensive process. In this work, we hypothesized that free recall may be a simpler yet effective method of directly eliciting skill chains. We elicited 23 skill chains from players and developers and augmented our reflexive thematic analysis with 11 semi-structured interviews in order to determine how players and developers conceptualize skill trees and whether free recall can be used as an alternative to more resource-intensive cognitive task analyses. We provide three main contributions: (1) a comparison of skill chain conceptualizations between players and developers and across prior literature; (2) insights to the process of free recall in eliciting CSG skill chains; and (3) a preliminary toolkit of CSG skill-based design recommendations based on our findings. We conclude CSG developers should: give the big picture up front; embrace social learning and paratext use; reinforce the intended structure of knowledge; situate learning within applicable, meaningful contexts; design for discovery and self-reflection; and encourage practice and learning beyond the tutorial. Free recall was ineffective for determining a traditional skill chain but was able to elicit the core gameplay loops, tutorial overviews, and some expert insights.<\/jats:p>","DOI":"10.1145\/3474671","type":"journal-article","created":{"date-parts":[[2021,10,6]],"date-time":"2021-10-06T22:59:48Z","timestamp":1633561188000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["How do Players and Developers of Citizen Science Games Conceptualize Skill Chains?"],"prefix":"10.1145","volume":"5","author":[{"given":"Josh Aaron","family":"Miller","sequence":"first","affiliation":[{"name":"Northeastern University, Boston, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Britton","family":"Horn","sequence":"additional","affiliation":[{"name":"Trinity University, San Antonio, TX, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"Guthrie","sequence":"additional","affiliation":[{"name":"Northeastern University, Boston, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonathan","family":"Romano","sequence":"additional","affiliation":[{"name":"Eterna Massive Open Laboratory; State University of New York at Buffalo; &amp; Stanford University, Buffalo, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guy","family":"Geva","sequence":"additional","affiliation":[{"name":"Eterna Massive Open Laboratory, , Israel"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Celia","family":"David","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amy Robinson","family":"Sterling","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Seth","family":"Cooper","sequence":"additional","affiliation":[{"name":"Northeastern University, Boston, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,10,6]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"power-laws, and pareto-a ranking tutorial","author":"Adamic Lada A","year":"2000","unstructured":"Lada A Adamic. 2000. Zipf, power-laws, and pareto-a ranking tutorial. Xerox Palo Alto Research Center, Palo Alto, CA (2000)."},{"key":"e_1_2_1_2_1","first-page":"165","article-title":"Knowledge component (KC) approaches to learner modeling","volume":"1","author":"Aleven Vincent","year":"2013","unstructured":"Vincent Aleven and Kenneth R Koedinger. 2013. Knowledge component (KC) approaches to learner modeling. Design Recommendations for Intelligent Tutoring Systems 1 (2013), 165--182.","journal-title":"Design Recommendations for Intelligent Tutoring Systems"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207687"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.12113"},{"key":"e_1_2_1_5_1","volume-title":"Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference.","author":"Aytemiz Batu","year":"2018","unstructured":"Batu Aytemiz, Isaac Karth, Jesse Harder, Adam M Smith, and Jim Whitehead. 2018. Talin: A Framework for Dynamic Tutorials based on the Skill Atoms Theory. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcom.2011.12.001"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134657"},{"key":"e_1_2_1_8_1","volume-title":"Using thematic analysis in psychology. Qualitative research in psychology 3, 2","author":"Braun Virginia","year":"2006","unstructured":"Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/2159676X.2019.1628806"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/capr.12360"},{"key":"e_1_2_1_11_1","volume-title":"Simon","author":"Chase William G.","year":"1973","unstructured":"William G. Chase and Herbert A. Simon. 1973. Perception in chess. Cognitive psychology 4, 1 (1973), 55--81."},{"key":"e_1_2_1_12_1","volume-title":"Categorization and representation of physics problems by experts and novices. Cognitive science 5, 2","author":"Chi Michelene TH","year":"1981","unstructured":"Michelene TH Chi, Paul J Feltovich, and Robert Glaser. 1981. Categorization and representation of physics problems by experts and novices. Cognitive science 5, 2 (1981), 121--152."},{"volume-title":"Handbook of research on educational communications and technology","author":"Clark Richard","key":"e_1_2_1_13_1","unstructured":"Richard Clark. 2014. Cognitive task analysis for expert-based instruction in healthcare. In Handbook of research on educational communications and technology. Springer, 541--551."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2011.09.003"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5840\/thinking19888129"},{"key":"e_1_2_1_16_1","unstructured":"Daniel Cook. 2007. The chemistry of game design. Gamasutra. http:\/\/www.gamasutra.com\/view\/feature\/1524\/the_chemistry_of_game_design.php. (2007)."},{"key":"e_1_2_1_17_1","volume-title":"Knowledge elicitation. Handbook of applied cognition","author":"Cooke Nancy J","year":"1999","unstructured":"Nancy J Cooke. 1999. Knowledge elicitation. Handbook of applied cognition (1999), 479--509."},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Seth Cooper. 2014. A framework for scientific discovery through video games. Morgan & Claypool.","DOI":"10.1145\/2625848"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature09304"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-016-9170-1"},{"volume-title":"Working minds: A practitioner's guide to cognitive task analysis","author":"Crandall Beth","key":"e_1_2_1_21_1","unstructured":"Beth Crandall, Gary Klein, Gary A Klein, Robert R Hoffman, and others. 2006. Working minds: A practitioner's guide to cognitive task analysis. Mit Press."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15430421tip3903_2"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1177\/1075547015609322"},{"volume-title":"Who Takes Part in Online Citizen Science? In Online Citizen Science and the Widening of Academia","author":"Curtis Vickie","key":"e_1_2_1_24_1","unstructured":"Vickie Curtis. 2018. Who Takes Part in Online Citizen Science? In Online Citizen Science and the Widening of Academia. Springer, 45--68."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-011-9106-8"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370024.2014.993471"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1177\/1555412015590063"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41233-019-0030--8"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.4018\/IJGCMS.2018010101"},{"key":"e_1_2_1_30_1","volume-title":"Designing for Problem-Solvers. PRACTICE: Game Design in Detail. (2014).","author":"Gage Zach","year":"2021","unstructured":"Zach Gage. 2014. Designing for Problem-Solvers. PRACTICE: Game Design in Detail. (2014). Retrieved from http:\/\/stfj.net\/DesigningForProblem-Solvers\/ on May 13, 2021."},{"key":"e_1_2_1_31_1","volume-title":"Problem Solving: Moving from Routine to Nonroutine and Beyond. Notices of the AMs 60, 10","author":"Garelick Barry","year":"2013","unstructured":"Barry Garelick. 2013. Problem Solving: Moving from Routine to Nonroutine and Beyond. Notices of the AMs 60, 10 (2013)."},{"key":"e_1_2_1_32_1","volume-title":"Phi Kappa Phi Forum","volume":"85","author":"Gee James Paul","year":"2005","unstructured":"James Paul Gee. 2005. Good video games and good learning. In Phi Kappa Phi Forum, Vol. 85. The Honor Society of Phi Kappa Phi, 33."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2006.322537"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1525\/aa.1994.96.3.02a00100"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1080\/10508406.2010.508029"},{"key":"e_1_2_1_36_1","volume-title":"DiGRA'20 Abstract-Proceedings of the 2020 DiGRA International Conference. York.","author":"Hesketh Joseph","year":"2020","unstructured":"Joseph Hesketh, Christoph Sebastian Deterding, and Jeremy Gow. 2020. How Players Learn Team-versus-Team Esports: First Results from A Grounded Theory Study. In DiGRA'20 Abstract-Proceedings of the 2020 DiGRA International Conference. York."},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting","volume":"46","author":"Hoffman Robert R.","unstructured":"Robert R. Hoffman, John W. Coffey, Mary Jo Carnot, and Joseph D. Novak. 2002. An empirical comparison of methods for eliciting and modeling expert knowledge. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 46. SAGE Publications Sage CA: Los Angeles, CA, 482--486."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3116595.3116640"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/3505378.3505402"},{"key":"e_1_2_1_41_1","volume-title":"Master maker: Understanding gaming skill through practice and habit from gameplay behavior. Topics in cognitive science 9, 2","author":"Huang Jeff","year":"2017","unstructured":"Jeff Huang, Eddie Yan, Gifford Cheung, Nachiappan Nagappan, and Thomas Zimmermann. 2017. Master maker: Understanding gaming skill through practice and habit from gameplay behavior. Topics in cognitive science 9, 2 (2017), 437--466."},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the AAAI Workshop on Challenges in Game AI","volume":"4","author":"Hunicke Robin","year":"2004","unstructured":"Robin Hunicke, Marc LeBlanc, and Robert Zubek. 2004. MDA: A formal approach to game design and game research. In Proceedings of the AAAI Workshop on Challenges in Game AI, Vol. 4. 1722."},{"key":"e_1_2_1_43_1","volume-title":"Game-play breakdowns and breakthroughs: exploring the relationship between action, understanding, and involvement. Human-computer interaction 30, 3--4","author":"Iacovides Ioanna","year":"2015","unstructured":"Ioanna Iacovides, Anna L. Cox, Patrick McAndrew, James Aczel, and Eileen Scanlon. 2015. Game-play breakdowns and breakthroughs: exploring the relationship between action, understanding, and involvement. Human-computer interaction 30, 3--4 (2015), 202--231."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2468356.2468553"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.22323\/2.15030205"},{"volume-title":"fast and slow","author":"Kahneman Daniel","key":"e_1_2_1_46_1","unstructured":"Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.5555\/3505378.3505403"},{"volume-title":"Teaching as a design science: Building pedagogical patterns for learning and technology","author":"Laurillard Diana","key":"e_1_2_1_49_1","unstructured":"Diana Laurillard. 2013. Teaching as a design science: Building pedagogical patterns for learning and technology. Routledge."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1313039111"},{"key":"e_1_2_1_51_1","volume-title":"Crowd Intelligence in EyeWire. Technical report. (2015).","author":"Lee Kisuk","year":"2021","unstructured":"Kisuk Lee. 2015. Crowd Intelligence in EyeWire. Technical report. (2015). Retrieved from http:\/\/kisuklee.wikidot.com\/research#toc2 on May 6, 2021."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society","volume":"35","author":"Lindstedt John","year":"2013","unstructured":"John Lindstedt and Wayne Gray. 2013. Extreme expertise: Exploring expert behavior in Tetris. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 35."},{"key":"e_1_2_1_53_1","volume-title":"Slow serious games, interactions and play: Designing for positive and serious experience and reflection. Entertainment computing 14","author":"Marsh Tim","year":"2016","unstructured":"Tim Marsh. 2016. Slow serious games, interactions and play: Designing for positive and serious experience and reflection. Entertainment computing 14 (2016), 45--53."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341215.3356274"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3337722.3337735"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of The 13th International Conference on Educational Data Mining (EDM","author":"Nguyen Huy Anh","year":"2020","unstructured":"Huy Anh Nguyen, Xinying Hou, John Stamper, and Bruce M McLaren. 2020. Moving beyond test scores: analyzing the effectiveness of a digital learning game through learning analytics. In Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020). 487--495."},{"key":"e_1_2_1_57_1","unstructured":"David Osborne. 2016. Tales From a Crowdsourced Game Design. (2016). https:\/\/www.gdcvault.com\/browse\/gdc-16\/play\/1023216"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.15346\/hc.v5i1.1"},{"key":"e_1_2_1_59_1","volume-title":"Exploring data quality in games with a purpose. iConference 2014 Proceedings","author":"Prestopnik Nathan","year":"2014","unstructured":"Nathan Prestopnik, Kevin Crowston, and Jun Wang. 2014. Exploring data quality in games with a purpose. iConference 2014 Proceedings (2014)."},{"key":"e_1_2_1_60_1","first-page":"1442","article-title":"A comparative study of conceptual graph and concept map","volume":"9","author":"Mohamad Rasli Ruziana Binti","year":"2014","unstructured":"Ruziana Binti Mohamad Rasli, Faudziah Ahmad, and Siti Sakira Kamaruddin. 2014. A comparative study of conceptual graph and concept map. Journal of Engineering and Applied Sciences 9, 9 (2014), 1442--1446.","journal-title":"Journal of Engineering and Applied Sciences"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341215.3356310"},{"volume-title":"The Art of Game Design: A book of lenses","author":"Schell Jesse","key":"e_1_2_1_62_1","unstructured":"Jesse Schell. 2008. The Art of Game Design: A book of lenses. CRC press."},{"volume-title":"Learning theories an educational perspective","author":"Schunk Dale H","key":"e_1_2_1_63_1","unstructured":"Dale H Schunk. 2012. Learning theories an educational perspective sixth edition. Pearson."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1037\/0278-7393.4.6.592"},{"key":"e_1_2_1_65_1","unstructured":"Bethesda Softworks. 1994--2020. The Elder Scrolls. (1994--2020)."},{"volume-title":"Games, learning, and society: Learning and meaning in the digital age","author":"Squire Kurt","key":"e_1_2_1_66_1","unstructured":"Kurt Squire. 2012. Designed Cultures. In Games, learning, and society: Learning and meaning in the digital age, Constance Steinkuehler, Kurt Squire, and Sasha Barab (Eds.). Cambridge University Press, Chapter 2."},{"key":"e_1_2_1_67_1","volume-title":"Daniel Cuadrado S\u00e1nchez, Miguel A Luengo-Oroz, and Marisa Ponti.","author":"Stankovic Igor","year":"2017","unstructured":"Igor Stankovic, Mar'ia Postigo Camps, Daniel Cuadrado S\u00e1nchez, Miguel A Luengo-Oroz, and Marisa Ponti. 2017. Performance of Players and Data Quality in a Citizen Science Casual Game. (2017). https:\/\/osf.io\/preprints\/socarxiv\/vztxn\/"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.12.074"},{"key":"e_1_2_1_69_1","unstructured":"Gustavo F Tondello and Lennart E Nacke. 2019. A Pilot Study of a Digital Skill Tree in Gameful Education. (2019)."},{"key":"e_1_2_1_70_1","unstructured":"Valve. 2007. Portal. (2007)."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02297047"},{"volume-title":"Ten steps to complex learning: A systematic approach to four-component instructional design","author":"Van Merri\u00ebnboer Jeroen JG","key":"e_1_2_1_72_1","unstructured":"Jeroen JG Van Merri\u00ebnboer and Paul A Kirschner. 2017. Ten steps to complex learning: A systematic approach to four-component instructional design. Routledge."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2008.4497403"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025841"},{"key":"e_1_2_1_75_1","volume-title":"Active Learning: The Pedagogy of the Game Tutorial.","author":"Williams G. Christopher","year":"2009","unstructured":"G. Christopher Williams. 2009. Active Learning: The Pedagogy of the Game Tutorial. (2009). https:\/\/www.popmatters.com\/111485-active-learning-the-pedagogy-of-the-game-tutorial-2496084217.html"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3474671","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3474671","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:42Z","timestamp":1750191522000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3474671"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,5]]},"references-count":73,"journal-issue":{"issue":"CHI PLAY","published-print":{"date-parts":[[2021,10,5]]}},"alternative-id":["10.1145\/3474671"],"URL":"https:\/\/doi.org\/10.1145\/3474671","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2021,10,5]]},"assertion":[{"value":"2021-10-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}