{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T20:13:36Z","timestamp":1775506416713,"version":"3.50.1"},"reference-count":84,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,10,18]],"date-time":"2021-10-18T00:00:00Z","timestamp":1634515200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["#2038406, #1504634 and #1502564"],"award-info":[{"award-number":["#2038406, #1504634 and #1502564"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2022,3,31]]},"abstract":"<jats:p>\n            In this article, we leverage ideas from the theory of coevolutionary computation to analyze interactions of students with problems. We introduce the idea of\n            <jats:italic>informatively<\/jats:italic>\n            easy or hard concepts. Our approach is different from more traditional analyses of problem difficulty such as item analysis in the sense that we consider Pareto dominance relationships within the multidimensional structure of student\u2013problem performance data rather than average performance measures. This method allows us to uncover not just the problems on which students are struggling but also the variety of difficulties different students face. Our approach is to apply methods from the Dimension Extraction Coevolutionary Algorithm\u00a0to analyze problem-solving logs of students generated when they use an online software tutoring suite for introductory computer programming called\n            <jats:italic>problets<\/jats:italic>\n            . The results of our analysis not only have implications for how to scale up and improve adaptive tutoring software but also have the promise of contributing to the identification of common misconceptions held by students and thus, eventually, to the construction of a concept inventory for introductory programming.\n          <\/jats:p>","DOI":"10.1145\/3477968","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T00:57:14Z","timestamp":1634605034000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Identifying Informatively Easy and Informatively Hard Concepts"],"prefix":"10.1145","volume":"22","author":[{"given":"R. Paul","family":"Wiegand","sequence":"first","affiliation":[{"name":"Winthrop University, Rock Hill, SC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anthony","family":"Bucci","sequence":"additional","affiliation":[{"name":"Independent, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amruth N.","family":"Kumar","sequence":"additional","affiliation":[{"name":"Ramapo College of New Jersey, Mahwah, NJ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer","family":"Albert","sequence":"additional","affiliation":[{"name":"The Citadel, Charleston, SC, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alessio","family":"Gaspar","sequence":"additional","affiliation":[{"name":"University of South Florida, Tampa, FL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1189215.1189182"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2676723.2677258"},{"key":"e_1_3_1_4_2","unstructured":"L. W. Anderson D. R. Krathwohl P. W. Airasian K. A. Cruikshank R. E. Mayer P. R. Pintrich J. Raths and M. C. Wittrock. 2001. A Taxonomy for Learning Teaching and Assessing: A Revision of Bloom\u2019 Taxonomy of Educational Objectives . Longman New York."},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1097\/01.mlr.0000103528.48582.7c"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.5555\/536791"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/1539024.1508923"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428029.3428039"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/1272848.1272879"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/1354249"},{"key":"e_1_3_1_11_2","first-page":"501","volume-title":"Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation","author":"Bucci A.","year":"2004","unstructured":"A. Bucci, J. B. Pollack, and E. D. de Jong. 2004. Automated extraction of problem structure. In Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation (GECCO\u201904). ACM, 501\u2013512."},{"key":"e_1_3_1_12_2","volume-title":"Proceedings of the 29th International Conference of the Florida Artificial Intelligence Research Society","author":"Bucci Anthony","year":"2016","unstructured":"Anthony Bucci, R. Paul Wiegand, Amruth N. Kumar, Jennifer L. Albert, and Alessio Gaspar. 2016. Dimension extraction analysis of student performance on problems. In Proceedings of the 29th International Conference of the Florida Artificial Intelligence Research Society (FLAIRS\u201916). AAAI Press, 6."},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3304221.3319771"},{"key":"e_1_3_1_14_2","doi-asserted-by":"crossref","unstructured":"R. Caceffo P. Frank-Bolton R. Souza and R. Azevedo. 2019b. Identifying and Validating Java Misconceptions\u2014Complementary Material . Technical Report 19-05. Institute of Computing University of Campinas SP Brasil.","DOI":"10.1145\/3304221.3319771"},{"key":"e_1_3_1_15_2","volume-title":"Technical Report 18-06, Institute of Computing","author":"Caceffo Ricardo","year":"2018","unstructured":"Ricardo Caceffo, Raysa Benatti Guilherme Gama, and Rodolfo Azevedo Tales Aparecida, Tania Caldas. 2018. A concept inventory for CS1 introductory programming courses in C. In Technical Report 18-06, Institute of Computing, University of Campinas, SP, Brasil."},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2839509.2844559"},{"key":"e_1_3_1_17_2","first-page":"193","volume-title":"Proceedings of the 8th International Conference in Software Engineering Research and Innovation","author":"Carlos Enr\u00edquez Ram\u00edrez","year":"2020","unstructured":"Enr\u00edquez Ram\u00edrez Carlos, Raluy Herrero Mariza, and Luz Mar\u00eda Vega Sosa. 2020. Proposed framework for employing cognitive process for resolving programming problems for novice programmers. In Proceedings of the 8th International Conference in Software Engineering Research and Innovation (CONISOFT\u201920). IEEE, 193\u2013197."},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1002\/tea.3660301007"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01099821"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1119\/1.1374249"},{"issue":"1","key":"e_1_3_1_21_2","first-page":"40","article-title":"Peer instruction: Engaging students one-on-one, all at once","volume":"1","author":"Crouch Catherine H.","year":"2007","unstructured":"Catherine H. Crouch, Jessica Watkins, Adam P. Fagen, and Eric Mazur. 2007. Peer instruction: Engaging students one-on-one, all at once. Res.-bas. Reform Univ. Phys. 1, 1 (2007), 40\u201395.","journal-title":"Res.-bas. Reform Univ. Phys."},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2157136.2157148"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/1143997.1144056"},{"key":"e_1_3_1_24_2","doi-asserted-by":"crossref","unstructured":"E. D. de Jong and A. Bucci. 2008. Objective set compression: Test-based problems and multi-objective optimization. In Multi-Objective Problem Solving from Nature: From Concepts to Applications . Springer New York.","DOI":"10.1007\/978-3-540-72964-8_17"},{"key":"e_1_3_1_25_2","unstructured":"Christinr being MIA e DeMars. 2010. Item Response Theory . Oxford University Press New York NY."},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-4541-9"},{"key":"e_1_3_1_27_2","doi-asserted-by":"crossref","unstructured":"Susan E. Embretson and Steven P. Reise. 2013. Item Response Theory . Psychology Press Rotterdman The Netherlands.","DOI":"10.4324\/9781410605269"},{"key":"e_1_3_1_28_2","unstructured":"H. L. Erickson. 2002. Concept-Based Curriculum and Instruction: Teaching Beyond the Facts . SAGE Publications."},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3017680.3017756"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.5555\/286139.286166"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/1352322.1352226"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/1789934.1789935"},{"issue":"11","key":"e_1_3_1_33_2","first-page":"1043","article-title":"Common sense concepts about motion","volume":"53","author":"Hallouin I. A.","year":"1985","unstructured":"I. A. Hallouin and D. Hestenes. 1985. Common sense concepts about motion. Am. J. Phys. 53, 11 (1985), 1043\u20131055.","journal-title":"Am. J. Phys."},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3992.1989.tb00313.x"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404527"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/2133797.2133800"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734298"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2014.970781"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2445196.2445254"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1119\/1.2343497"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/0167-2789(90)90076-2"},{"issue":"3","key":"e_1_3_1_42_2","article-title":"An overview of recent developments in cognitive diagnostic computer adaptive assessments","volume":"15","author":"Huebner Alan","year":"2010","unstructured":"Alan Huebner. 2010. An overview of recent developments in cognitive diagnostic computer adaptive assessments. Pract. Assess. Res. Eval. 15, 3 (2010).","journal-title":"Pract. Assess. Res. Eval."},{"key":"e_1_3_1_43_2","volume-title":"Proceedings of the 3rd Annual Genetic Programming Conference","author":"Juill\u00e9 H.","year":"1998","unstructured":"H. Juill\u00e9 and Jordan B. Pollack. 1998. Coevolving the \u201cideal\u201d trainer: Application to the discovery of cellular automata rules. In Proceedings of the 3rd Annual Genetic Programming Conference. IEEE Press."},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734299"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10648-010-9132-9"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/2538862.2538902"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/1089733.1089734"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2015.7344046"},{"issue":"1","key":"e_1_3_1_49_2","first-page":"256","article-title":"Explanation of step-by-step execution as feedback for problems on program analysis, and its generation in model-based problem-solving tutors","volume":"4","author":"Kumar A. N.","year":"2006","unstructured":"A. N. Kumar. 2006a. Explanation of step-by-step execution as feedback for problems on program analysis, and its generation in model-based problem-solving tutors. Technol. Instr. Cogn. Learn. J. 4, 1 (2006), 256\u2013260.","journal-title":"Technol. Instr. Cogn. Learn. J."},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/11768012_18"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30950-2_68"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07221-0_30"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2676723.2677301"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/2729094.2742587"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/T4E.2014.27"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2899415.2899432"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293881.3295779"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2883851.2883901"},{"key":"e_1_3_1_59_2","unstructured":"Jan H. F. Meyer and Ray Land. 2003. Threshold concepts and troublesome knowledge:linkages to ways of thinking and practising. In Improving Student Learning\u2014Theory and Practice Ten Years on Oxford: Oxford Centre for Staff and Learning Development . Oxford Press Brussels Belgium 412\u2013424."},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10734-004-6779-5"},{"key":"e_1_3_1_61_2","unstructured":"Jan H. F. Meyer and Ray Land. 2005b. Threshold concepts and troublesome knowledge:linkages to ways of thinking and practising. In Improving Student Learning\u2014Diversity and Inclusivity: Proceedings of the 12th Improving Student Learning Conference . Oxford Press Birmingham UK 53\u201364."},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/2723576.2723622"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/2960310.2960316"},{"key":"e_1_3_1_64_2","doi-asserted-by":"crossref","unstructured":"E. Popovici A. Bucci R. P. Wiegand and E. D. de Jong. 2012. Coevolutionary principles. In Handbook of Natural Computing . Springer 987\u20131033.","DOI":"10.1007\/978-3-540-92910-9_31"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/2462476.2462490"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/2591708.2591722"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/2591708.2591722"},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077618"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.5555\/1696468"},{"key":"e_1_3_1_70_2","doi-asserted-by":"crossref","unstructured":"P. Rosenbaum. 1984. Testing the Local Independence Assumption in Item Response Theory . Technical Report. ETS.","DOI":"10.1002\/j.2330-8516.1984.tb00049.x"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1098-2736(199704)34:4<377::AID-TEA7>3.0.CO;2-O"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2020.106349"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3344429.3372502"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/2960310.2960337"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/2490822"},{"issue":"1","key":"e_1_3_1_76_2","first-page":"1","article-title":"Empirical estimation errors in item response theory as a function of test properties","volume":"1989","author":"Stocking M.","year":"1989","unstructured":"M. Stocking. 1989. Empirical estimation errors in item response theory as a function of test properties. ETS Res. Rep. Ser. 1989, 1 (1989), 1\u201376.","journal-title":"ETS Res. Rep. Ser."},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/2724660.2728682"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2014.970779"},{"key":"e_1_3_1_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734297"},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/1953163.1953200"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2014.970782"},{"key":"e_1_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/2591708.2591749"},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.5555\/2955239.2955343"},{"key":"e_1_3_1_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/2538862.2538886"},{"key":"e_1_3_1_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/2839509.2844629"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477968","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477968","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477968","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:10:38Z","timestamp":1750183838000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477968"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,18]]},"references-count":84,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,3,31]]}},"alternative-id":["10.1145\/3477968"],"URL":"https:\/\/doi.org\/10.1145\/3477968","relation":{},"ISSN":["1946-6226","1946-6226"],"issn-type":[{"value":"1946-6226","type":"print"},{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,18]]},"assertion":[{"value":"2021-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}