{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T08:37:44Z","timestamp":1780735064777,"version":"3.54.1"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,8,19]],"date-time":"2024-08-19T00:00:00Z","timestamp":1724025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Currently, there is little evidence about how non-English-speaking students learn computer programming. For example, there are few validated assessment instruments to measure the development of programming skills, especially for the Spanish-speaking population. Having valid assessment instruments is essential to identify the difficulties of the students, make formative decisions, and evaluate the impact of pedagogical interventions. This article presents the outcomes of the early steps toward a validation in Spanish of the SCS1 assessment, derived from the first conceptual inventory to measure programming knowledge decoupled from a particular language. The goal of this initial step is twofold: (1) identifying the questions that are too difficult in this version and might need to be adapted based on empirical evidence of how students answer them and (2) identifying students\u2019 common difficulties with the most challenging questions. This study started with a \u201cthink-aloud\u201d protocol to validate whether the Spanish version was similar to the original instrument and if the students understood the translation as intended. Then, the instrument was applied to a sample of 71 university students who had previously taken an introductory programming course using Python. We computed the difficulty index for each question and identified students\u2019 difficulties in answering them. We also propose potential improvements for these items. The main contributions of this study include (1) progress toward the validation of an instrument for evaluating programming skills in Spanish, (2) noting the advantage that experienced students in Python have to complete the instrument, (3) identifying the tendency of students to translate the pseudo-code from Spanish into English, and (4) the identification of common mistakes, like misunderstanding the scope of local variables or the confusion between returning and printing.<\/jats:p>","DOI":"10.1145\/3665921","type":"journal-article","created":{"date-parts":[[2024,6,15]],"date-time":"2024-06-15T05:41:17Z","timestamp":1718430077000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Identifying Difficult Questions and Student Difficulties in a Spanish Version of a Programming Assessment Instrument (SCS1)"],"prefix":"10.1145","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8720-0002","authenticated-orcid":false,"given":"Camilo","family":"Vieira","sequence":"first","affiliation":[{"name":"Universidad del Norte, Barranquilla, Colombia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9868-5831","authenticated-orcid":false,"given":"Andrea","family":"V\u00e1squez","sequence":"additional","affiliation":[{"name":"Universidad T\u00e9cnica Federico Santa Mar\u00eda, Valpara\u00edso, Chile"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7495-6273","authenticated-orcid":false,"given":"Federico","family":"Meza","sequence":"additional","affiliation":[{"name":"Universidad T\u00e9cnica Federico Santa Mar\u00eda, Santiago, Chile"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1205-1126","authenticated-orcid":false,"given":"Roxana","family":"Quintero-Manes","sequence":"additional","affiliation":[{"name":"Universidad del Norte, Barranquilla, Colombia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6935-9212","authenticated-orcid":false,"given":"Pedro","family":"Godoy","sequence":"additional","affiliation":[{"name":"Universidad T\u00e9cnica Federico Santa Mar\u00eda, Valpara\u00edso, Chile"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,8,19]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1929887.1929905"},{"issue":"5","key":"e_1_3_2_3_2","first-page":"1656","article-title":"Item analysis using a derived science achievement test data","volume":"4","author":"Bichi Ado Abdu","year":"2015","unstructured":"Ado Abdu Bichi. 2015. Item analysis using a derived science achievement test data. International Journal of Science and Research (IJSR) 4, 5 (2015), 1656\u20131662.","journal-title":"International Journal of Science and Research (IJSR)"},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1145\/3304221.3319771","volume-title":"Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE \u201919)","author":"Caceffo Ricardo","year":"2019","unstructured":"Ricardo Caceffo, Pablo Frank-Bolton, Renan Souza, and Rodolfo Azevedo. 2019. Identifying and validating java misconceptions toward a cs1 concept inventory. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE \u201919). ACM, New York, NY, 23\u201329. DOI: 10.1145\/3304221.3319771"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2023.2255058"},{"issue":"1","key":"e_1_3_2_6_2","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1037\/0021-9010.78.1.98","article-title":"What is coefficient alpha? An examination of theory and applications","volume":"78","author":"Cortina Jose M.","year":"1993","unstructured":"Jose M. Cortina. 1993. What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology 78, 1 (1993), 98.","journal-title":"Journal of Applied Psychology"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3051457.3051464"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339407"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2022.2079867"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/107005.107066"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cola.2022.101158"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/1352322.1352226"},{"key":"e_1_3_2_13_2","first-page":"375","article-title":"Rob\u00f3tica educativa en contextos inclusivos: El caso de las aulas hospitalarias [Educational robotics in inclusive contexts: The case of hospital classrooms]","volume":"24","author":"Gonz\u00e1lez Carina Gonz\u00e1lez","year":"2021","unstructured":"Carina Gonz\u00e1lez Gonz\u00e1lez, Ver\u00f3nica Violant, Alfonso Infante, Lorena C\u00e1ceres Garc\u00eda, and M. D. Guzm\u00e1n Franco. 2021. Rob\u00f3tica educativa en contextos inclusivos: El caso de las aulas hospitalarias [Educational robotics in inclusive contexts: The case of hospital classrooms]. Educaci\u00f3n XXI [Education XXI] 24, 375\u2013403.","journal-title":"Educaci\u00f3n XXI [Education XXI]"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173970"},{"key":"e_1_3_2_15_2","unstructured":"Mark Guzdial. 2019. We should stop saying \u201clanguage independent.\u201d We don\u2019t know how to do that. Retrieved from https:\/\/cacm.acm.org\/blogs\/blog-cacm\/238782-we-should-stop-saying-language-independent-we-dont-know-how-to-do-that\/fulltext"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3992.1993.tb00543.x"},{"key":"e_1_3_2_17_2","first-page":"587","article-title":"Sobre el uso adecuado del coeficiente de correlaci\u00f3n de Pearson: definici\u00f3n, propiedades y suposiciones [On the proper use of Pearson's correlation coefficient: definition, properties and assumptions]","volume":"37","author":"Hern\u00e1ndez-Lalinde Juan","year":"2018","unstructured":"Juan Hern\u00e1ndez-Lalinde, Jhon-Franklin Espinosa-Castro, Mariana Pe\u00f1aloza Tarazona, Johel Rodr\u00edguez, Jos\u00e9 Chac\u00f3n, Sandra Carrillo Sierra, Valmore Pirela, and Cristian Toloza. 2018. Sobre el uso adecuado del coeficiente de correlaci\u00f3n de Pearson: definici\u00f3n, propiedades y suposiciones [On the proper use of Pearson's correlation coefficient: definition, properties and assumptions]. Archivos Venezolanos de Farmacolog\u00eda y Terap\u00e9utica [Venezuelan Archives of Pharmacology and Therapeutics] 37, 587\u2013595.","journal-title":"Archivos Venezolanos de Farmacolog\u00eda y Terap\u00e9utica [Venezuelan Archives of Pharmacology and Therapeutics]"},{"key":"e_1_3_2_18_2","volume-title":"The Relationship Between Item Difficulty and Discrimination Indices in Multiple-Choice Tests in a Physical Science Course","author":"Hotiu Angelica","year":"2006","unstructured":"Angelica Hotiu. 2006. The Relationship Between Item Difficulty and Discrimination Indices in Multiple-Choice Tests in a Physical Science Course. Master's thesis. Florida Atlantic University, Boca Raton, FL."},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818314.2818331"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/1929887.1929902"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3478431.3499299"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/1595496.1562930"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.03.025"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/T4E.2012.30"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/LaTiCE.2015.38"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1145\/2960310.2960316","volume-title":"Proceedings of the 2016 ACM Conference on International Computing Education Research (ICER \u201916)","author":"Parker Miranda C.","year":"2016","unstructured":"Miranda C. Parker, Mark Guzdial, and Shelly Engleman. 2016. Replication, validation, and use of a language independent CS1 knowledge assessment. In Proceedings of the 2016 ACM Conference on International Computing Education Research (ICER \u201916). ACM, New York, NY, 93\u2013101. DOI: 10.1145\/2960310.2960316"},{"key":"e_1_3_2_27_2","first-page":"60","volume-title":"Proceedings of the 17th ACM Conference on International Computing Education Research (ICER \u201921)","author":"Parker Miranda C.","year":"2021","unstructured":"Miranda C. Parker, Mark Guzdial, and Allison Elliott Tew. 2021. Uses, revisions, and the future of validated assessments in computing education: A case study of the FCS1 and SCS1. In Proceedings of the 17th ACM Conference on International Computing Education Research (ICER \u201921). ACM, New York, NY, 60\u201368. DOI: 10.1145\/3446871.3469744"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077618"},{"key":"e_1_3_2_29_2","volume-title":"Shut Down or Restart? The Way Forward for Computing in UK Schools","author":"Society The Royal","year":"2012","unstructured":"The Royal Society. 2012. Shut Down or Restart? The Way Forward for Computing in UK Schools. The Royal Society. Retrieved from https:\/\/www.stem.org.uk\/rx326t"},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1145\/3374135.3385321","volume-title":"Proceedings of the 2020 ACM Southeast Conference(ACM SE \u201920)","author":"Song Yang","year":"2020","unstructured":"Yang Song, Yunkai Xiao, Jonathan Stephens, Emma Ruesch, Sean Roginski, and Lucas Layman. 2020. Suitability of SCS1 as a pre-CS2 assessment instrument: A comparison with short deliberate-practice questions. In Proceedings of the 2020 ACM Southeast Conference(ACM SE \u201920). ACM, New York, NY, 313\u2013314. DOI: 10.1145\/3374135.3385321"},{"key":"e_1_3_2_31_2","volume-title":"Visual Program Simulation in Introductory Programming Education","author":"Sorva Juha","year":"2012","unstructured":"Juha Sorva. 2012. Visual Program Simulation in Introductory Programming Education. Doctoral thesis. School of Science. Retrieved from http:\/\/urn.fi\/URN:ISBN:978-952-60-4626-6"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/2534973"},{"issue":"4","key":"e_1_3_2_33_2","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1002\/j.2168-9830.2005.tb00864.x","article-title":"A statics concept inventory: Development and psychometric analysis","volume":"94","author":"Steif Paul S.","year":"2005","unstructured":"Paul S. Steif and John A. Dantzler. 2005. A statics concept inventory: Development and psychometric analysis. Journal of Engineering Education 94, 4 (2005), 363\u2013371.","journal-title":"Journal of Engineering Education"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-8126-4_12"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/1734263.1734297"},{"key":"e_1_3_2_36_2","first-page":"1","volume-title":"Proceedings of the IEEE Frontiers in Education Conference (FIE \u201916)","author":"Timmermann Dion","year":"2016","unstructured":"Dion Timmermann, Christian Kautz, and Volker Skwarek. 2016. Evidence-based re-design of an introductory course \u201cprogramming in C\u201d. In Proceedings of the IEEE Frontiers in Education Conference (FIE \u201916). 1\u20135. DOI: 10.1109\/FIE.2016.7757492"},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1145\/3372782.3406270","volume-title":"Proceedings of the 2020 ACM Conference on International Computing Education Research (ICER \u201920)","author":"Tshukudu Ethel","year":"2020","unstructured":"Ethel Tshukudu and Quintin Cutts. 2020. Understanding conceptual transfer for students learning new programming languages. In Proceedings of the 2020 ACM Conference on International Computing Education Research (ICER \u201920). ACM, New York, NY, 227\u2013237. DOI: 10.1145\/3372782.3406270"},{"key":"e_1_3_2_38_2","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1007\/s12528-020-09267-7","article-title":"Providing students with agency to self-scaffold in a computational science and engineering course","volume":"33","author":"Vieira Camilo","year":"2021","unstructured":"Camilo Vieira, Alejandra J. Magana, Anindya Roy, and Michael Falk. 2021. Providing students with agency to self-scaffold in a computational science and engineering course. Journal of Computing in Higher Education 33, 328\u2013366.","journal-title":"Journal of Computing in Higher Education"},{"key":"e_1_3_2_39_2","volume-title":"Probability & Statistics for Engineers and Scientists","author":"Walpole Ronald E.","year":"2016","unstructured":"Ronald E. Walpole, Raymond H. Myers, Sharon L. Myers, and Keying Ye. 2016. Probability & Statistics for Engineers and Scientists (9th ed.). Pearson Education, Boston.","edition":"9"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/1118178.1118215"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287370"},{"issue":"1","key":"e_1_3_2_42_2","article-title":"Why computer occupations are behind strong STEM employment growth in the 2019\u201329 decade","volume":"10","author":"Zilberman Alan","year":"2021","unstructured":"Alan Zilberman and Lindsey Ice. 2021. Why computer occupations are behind strong STEM employment growth in the 2019\u201329 decade. Beyond the Numbers: Employment & Unemployment 10, 1 (Jan. 2021).","journal-title":"Beyond the Numbers: Employment & Unemployment"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3665921","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3665921","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T23:44:27Z","timestamp":1750290267000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3665921"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,19]]},"references-count":41,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3665921"],"URL":"https:\/\/doi.org\/10.1145\/3665921","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,19]]},"assertion":[{"value":"2022-10-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}