{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T15:42:27Z","timestamp":1774366947214,"version":"3.50.1"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2023,5,25]],"date-time":"2023-05-25T00:00:00Z","timestamp":1684972800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,25]],"date-time":"2023-05-25T00:00:00Z","timestamp":1684972800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005222","name":"Jyv\u00e4\u00e4kyl\u00e4n Yliopisto","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100005222","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005222","name":"University of Jyv\u00e4skyl\u00e4","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005222","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Software Qual J"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In this article, we present a replication of an empirical experiment that evaluates intuitiveness and comprehensibility of keywords relating to different concepts in programming languages, originally conducted by Stefik and Gellenbeck. Novice programmers face many barriers when learning programming. One of these barriers is syntax, which for many languages is not designed based on empirical evidence. The purpose of the experiment was to provide more empirical evidence on the subject, to find out if the results of the original experiment can be replicated and if conducting the experiment in an environment where English is not the native language affects the results. The results of our experiment replicated most of the findings of the original study and provided further evidence that some syntactic choices in many popular programming languages are unintuitive for novice programmers. Our results suggest that the native language of participants who otherwise had good English skills had little effect when compared to the original study. These results may support programming language designers in making evidence-based design decisions and teachers of introductory programming courses in identifying some of the barriers novice programmers face.<\/jats:p>","DOI":"10.1007\/s11219-023-09631-7","type":"journal-article","created":{"date-parts":[[2023,5,25]],"date-time":"2023-05-25T15:02:24Z","timestamp":1685026944000},"page":"1211-1240","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["A replication study on the intuitiveness of programming language syntax"],"prefix":"10.1007","volume":"31","author":[{"given":"Vilma","family":"Lappi","sequence":"first","affiliation":[]},{"given":"Ville","family":"Tirronen","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8140-7368","authenticated-orcid":false,"given":"Jonne","family":"Itkonen","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,5,25]]},"reference":[{"key":"9631_CR1","doi-asserted-by":"publisher","unstructured":"Aarts, A. A., Anderson, J. E., Anderson, C. J., et al. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https:\/\/doi.org\/10.1126\/science.aac4716","DOI":"10.1126\/science.aac4716"},{"key":"9631_CR2","doi-asserted-by":"publisher","unstructured":"Becker, B. A. (2016). An effective approach to enhancing compiler error messages. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (pp. 126\u2013131). Association for Computing Machinery, New York, NY, USA, SIGCSE \u201916. https:\/\/doi.org\/10.1145\/2839509.2844584","DOI":"10.1145\/2839509.2844584"},{"issue":"7391","key":"9631_CR3","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1038\/483531a","volume":"483","author":"CG Begley","year":"2012","unstructured":"Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483(7391), 531\u2013533. https:\/\/doi.org\/10.1038\/483531a","journal-title":"Nature"},{"issue":"6","key":"9631_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3011286.3011301","volume":"41","author":"Y Bosse","year":"2017","unstructured":"Bosse, Y., & Gerosa, M. A. (2017). Why is programming so difficult to learn? Patterns of difficulties related to programming learning mid-stage. SIGSOFT Software Engineering Notes, 41(6), 1\u20136. https:\/\/doi.org\/10.1145\/3011286.3011301","journal-title":"SIGSOFT Software Engineering Notes"},{"key":"9631_CR5","doi-asserted-by":"publisher","unstructured":"Brown, N. C. C., & Altadmri, A. (2017). Novice Java programming mistakes: Large-scale data vs. educator beliefs. ACM Transactions on Computing Education, 17(2). https:\/\/doi.org\/10.1145\/2994154","DOI":"10.1145\/2994154"},{"key":"9631_CR6","doi-asserted-by":"publisher","unstructured":"Brown, N. C. C., Altadmri, A., Sentance, S., et al. (2018). Blackbox, five years on: An evaluation of a large-scale programming data collection project. In Proceedings of the 2018 ACM Conference on International Computing Education Research (pp. 196\u2013204). Association for Computing Machinery, New York, NY, USA, ICER \u201918. https:\/\/doi.org\/10.1145\/3230977.3230991","DOI":"10.1145\/3230977.3230991"},{"key":"9631_CR7","unstructured":"Council of Europe. (2020). Common European framework of reference for languages. Retrieved June 11, 2020, from https:\/\/www.coe.int\/en\/web\/common-european-framework-reference-languages\/home"},{"key":"9631_CR8","doi-asserted-by":"publisher","unstructured":"Dasgupta, S., & Hill, B. M. (2017). Learning to code in localized programming languages. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (pp. 33\u201339). Association for Computing Machinery, New York, NY, USA, L@S \u20191. https:\/\/doi.org\/10.1145\/3051457.3051464","DOI":"10.1145\/3051457.3051464"},{"key":"9631_CR9","doi-asserted-by":"publisher","unstructured":"Denny, P., Luxton-Reilly, A., & Carpenter, D. (2014). Enhancing syntax error messages appears ineffectual. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education (pp. 273\u2013278). Association for Computing Machinery, New York, NY, USA, ITiCSE \u201914. https:\/\/doi.org\/10.1145\/2591708.2591748","DOI":"10.1145\/2591708.2591748"},{"key":"9631_CR10","doi-asserted-by":"publisher","unstructured":"Denny, P., Luxton-Reilly, A., Tempero, E., et al. (2011). Understanding the syntax barrier for novices. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education (pp. 208\u2013212). Association for Computing Machinery, New York, NY, USA, ITiCSE \u201911. https:\/\/doi.org\/10.1145\/1999747.1999807","DOI":"10.1145\/1999747.1999807"},{"key":"9631_CR11","doi-asserted-by":"publisher","unstructured":"Denny, P., Luxton-Reilly, A., & Tempero, E. (2012). All syntax errors are not equal. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (pp. 75\u201380). Association for Computing Machinery, New York, NY, USA, ITiCSE \u201912. https:\/\/doi.org\/10.1145\/2325296.2325318","DOI":"10.1145\/2325296.2325318"},{"key":"9631_CR12","unstructured":"Department for Education (UK). (2013). National curriculum in England: Computing programmes of study. Retrieved January 8, 2021, from https:\/\/www.gov.uk\/government\/publications\/national-curriculum-in-england-computing-programmes-of-study\/national-curriculum-in-england-computing-programmes-of-study"},{"key":"9631_CR13","doi-asserted-by":"publisher","unstructured":"Devanbu, P., Zimmermann, T., & Bird, C. (2016). Belief & evidence in empirical software engineering. In Proceedings of the 38th International Conference on Software Engineering (pp. 108\u2013119). Association for Computing Machinery, New York, NY, USA, ICSE \u201916. https:\/\/doi.org\/10.1145\/2884781.2884812","DOI":"10.1145\/2884781.2884812"},{"key":"9631_CR14","doi-asserted-by":"publisher","unstructured":"Feij\u00f3o-Garc\u00eda, P. G., McNamara, K., & Stuart, J. (2020). The effects of native language on block-based programming introduction: A work in progress with hispanic population. In 2020 Research on Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT) (pp 1\u20132). https:\/\/doi.org\/10.1109\/RESPECT49803.2020.9272513","DOI":"10.1109\/RESPECT49803.2020.9272513"},{"key":"9631_CR15","unstructured":"Finnish National Agency for Education. (2014). Perusopetuksen opetussuunnitelman perusteet. https:\/\/eperusteet.opintopolku.fi\/beta\/#\/fi\/perusopetus\/419550\/tiedot"},{"key":"9631_CR16","doi-asserted-by":"publisher","unstructured":"Guo, P. J. (2018). Non-native English speakers learning computer programming: Barriers, desires, and design opportunities. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1\u201314). Association for Computing Machinery, New York, NY, USA, CHI \u201918. https:\/\/doi.org\/10.1145\/3173574.3173970","DOI":"10.1145\/3173574.3173970"},{"key":"9631_CR17","doi-asserted-by":"publisher","unstructured":"Hao, Q., Smith, I. V. D. H., Iriumi, N., et al. (2019). A systematic investigation of replications in computing education research. ACM Transactions on Computing Education, 19(4). https:\/\/doi.org\/10.1145\/3345328","DOI":"10.1145\/3345328"},{"key":"9631_CR18","doi-asserted-by":"crossref","unstructured":"Hermans, F., Swidan, A., & Aivaloglou, E. (2018). Code phonology: An exploration into the vocalization of code. In Proceedings of the 26th Conference on Program Comprehension (pp. 308\u2013311).","DOI":"10.1145\/3196321.3196355"},{"key":"9631_CR19","doi-asserted-by":"publisher","unstructured":"Hornb\u00e6k, K., Sander, S. S., Bargas-Avila, J. A., et al. (2014). Is once enough? On the extent and content of replications in human-computer interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3523\u20133532). Association for Computing Machinery, New York, NY, USA, CHI \u201914. https:\/\/doi.org\/10.1145\/2556288.2557004","DOI":"10.1145\/2556288.2557004"},{"key":"9631_CR20","unstructured":"Kaijanaho, A. J. (2015). Evidence-based programming language design: A philosophical and methodological exploration. University of Jyv\u00e4skyl\u00e4, Jyv\u00e4skyl\u00e4. http:\/\/urn.fi\/URN:ISBN:978-951-39-6388-0"},{"issue":"3","key":"9631_CR21","doi-asserted-by":"publisher","first-page":"239","DOI":"10.2307\/2332303","volume":"33","author":"MG Kendall","year":"1945","unstructured":"Kendall, M. G. (1945). The treatment of ties in ranking problems. Biometrika, 33(3), 239\u2013251. https:\/\/doi.org\/10.2307\/2332303","journal-title":"Biometrika"},{"issue":"1","key":"9631_CR22","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1007\/s10664-013-9279-3","volume":"20","author":"A Ko","year":"2015","unstructured":"Ko, A., LaToza, T., & Burnett, M. (2015). A practical guide to controlled experiments of software engineering tools with human participants. Empirical Software Engineering, 20(1), 110\u2013141. https:\/\/doi.org\/10.1007\/s10664-013-9279-3","journal-title":"Empirical Software Engineering"},{"key":"9631_CR23","doi-asserted-by":"crossref","unstructured":"Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70(3, Pt.1), 151 \u2013 159.","DOI":"10.1037\/h0026141"},{"issue":"6","key":"9631_CR24","doi-asserted-by":"publisher","first-page":"304","DOI":"10.3102\/0013189X14545513","volume":"43","author":"MC Makel","year":"2014","unstructured":"Makel, M. C., & Plucker, J. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304\u2013316. https:\/\/doi.org\/10.3102\/0013189X14545513","journal-title":"Educational Researcher"},{"key":"9631_CR25","doi-asserted-by":"publisher","unstructured":"Marceau, G., Fisler, K., & Krishnamurthi, S. (2011a). Measuring the effectiveness of error messages designed for novice programmers. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (pp. 499\u2013504). Association for Computing Machinery, New York, NY, USA, SIGCSE \u201911. https:\/\/doi.org\/10.1145\/1953163.1953308","DOI":"10.1145\/1953163.1953308"},{"key":"9631_CR26","doi-asserted-by":"publisher","unstructured":"Marceau, G., Fisler, K., & Krishnamurthi, S. (2011b). Mind your language: On novices\u2019 interactions with error messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (pp. 3\u201318). Association for Computing Machinery, New York, NY, USA, Onward! 2011. https:\/\/doi.org\/10.1145\/2048237.2048241","DOI":"10.1145\/2048237.2048241"},{"key":"9631_CR27","doi-asserted-by":"publisher","unstructured":"Pears, A., Seidman, S., Malmi, L., et al. (2007). A survey of literature on the teaching of introductory programming. In Working Group Reports on ITiCSE on Innovation and Technology in Computer Science Education (pp. 204\u2013223). Association for Computing Machinery, New York, NY, USA, ITiCSE-WGR \u201907. https:\/\/doi.org\/10.1145\/1345443.1345441","DOI":"10.1145\/1345443.1345441"},{"key":"9631_CR28","doi-asserted-by":"publisher","unstructured":"Pettit, R. S., Homer, J., & Gee, R. (2017). Do enhanced compiler error messages help students? Results inconclusive. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (pp 465\u2013470). Association for Computing Machinery, New York, NY, USA, SIGCSE \u201917. https:\/\/doi.org\/10.1145\/3017680.3017768","DOI":"10.1145\/3017680.3017768"},{"key":"9631_CR29","doi-asserted-by":"publisher","unstructured":"Piech, C., & Abu-El-Haija, S. (2020). Human languages in source code: Auto-translation for localized instruction. In Proceedings of the Seventh ACM Conference on Learning @ Scale (pp. 167\u2013174). Association for Computing Machinery, New York, NY, USA, L@S \u201920. https:\/\/doi.org\/10.1145\/3386527.3405916","DOI":"10.1145\/3386527.3405916"},{"key":"9631_CR30","doi-asserted-by":"publisher","unstructured":"Reestman, K., & Dorn, B. (2019). Native language\u2019s effect on Java compiler errors. In Proceedings of the 2019 ACM Conference on International Computing Education Research (pp. 249\u2013257). Association for Computing Machinery, New York, NY, USA, ICER \u201919. https:\/\/doi.org\/10.1145\/3291279.3339423","DOI":"10.1145\/3291279.3339423"},{"issue":"2","key":"9631_CR31","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1037\/a0015108","volume":"13","author":"S Schmidt","year":"2009","unstructured":"Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90\u2013100. https:\/\/doi.org\/10.1037\/a0015108","journal-title":"Review of General Psychology"},{"key":"9631_CR32","unstructured":"Siegfried, R. M., Chays, D., & Herbert, K. (2008). Will there ever be consensus on CS1? In FECS (pp. 18\u201323)."},{"issue":"5","key":"9631_CR33","doi-asserted-by":"publisher","first-page":"1299","DOI":"10.1007\/s10664-013-9286-4","volume":"19","author":"J Siegmund","year":"2014","unstructured":"Siegmund, J., K\u00e4stner, C., Liebig, J., et al. (2014). Measuring and modeling programming experience. Empirical Software Engineering, 19(5), 1299\u20131334. https:\/\/doi.org\/10.1007\/s10664-013-9286-4","journal-title":"Empirical Software Engineering"},{"issue":"1","key":"9631_CR34","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1007\/s11219-010-9106-7","volume":"19","author":"A Stefik","year":"2011","unstructured":"Stefik, A., & Gellenbeck, E. (2011). Empirical studies on programming language stimuli. Software Quality Journal, 19(1), 65\u201399. https:\/\/doi.org\/10.1007\/s11219-010-9106-7","journal-title":"Software Quality Journal"},{"key":"9631_CR35","doi-asserted-by":"publisher","unstructured":"Stefik, A., & Hanenberg, S. (2014). The programming language wars: Questions and responsibilities for the programming language community. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (pp. 283\u2013299). Association for Computing Machinery, New York, NY, USA, Onward! 2014. https:\/\/doi.org\/10.1145\/2661136.2661156","DOI":"10.1145\/2661136.2661156"},{"issue":"4","key":"9631_CR36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2534973","volume":"13","author":"A Stefik","year":"2013","unstructured":"Stefik, A., & Siebert, S. (2013). An empirical investigation into programming language syntax. ACM Transactions on Computing Education (TOCE), 13(4), 1\u201340. https:\/\/doi.org\/10.1145\/2534973","journal-title":"ACM Transactions on Computing Education (TOCE)"},{"key":"9631_CR37","doi-asserted-by":"publisher","unstructured":"Stefik, A., Hanenberg, S., McKenney, M., et al. (2014). What is the foundation of evidence of human factors decisions in language design? An empirical study on programming language workshops. In Proceedings of the 22nd International Conference on Program Comprehension (pp. 223\u2013231). Association for Computing Machinery, New York, NY, USA, ICPC 2014. https:\/\/doi.org\/10.1145\/2597008.2597154","DOI":"10.1145\/2597008.2597154"},{"key":"9631_CR38","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1017\/S0956796815000179","volume":"25","author":"V Tirronen","year":"2015","unstructured":"Tirronen, V., Uusi-M\u00e4kel\u00e4, S., & Isom\u00f6tt\u00f6nen, V. (2015). Understanding beginners\u2019 mistakes with Haskell. Journal of Functional Programming, 25, 30. https:\/\/doi.org\/10.1017\/S0956796815000179","journal-title":"Journal of Functional Programming"},{"key":"9631_CR39","unstructured":"Webropol. (2020). Webropol survey tool website. Retrieved June 11, 2020, from https:\/\/webropol.com\/"},{"issue":"3","key":"9631_CR40","first-page":"321","volume":"18","author":"W Wiersma","year":"2013","unstructured":"Wiersma, W. (2013). The validity of surveys: Online and offline. Oxford Internet Institute, 18(3), 321\u2013340.","journal-title":"Oxford Internet Institute"}],"container-title":["Software Quality Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-023-09631-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11219-023-09631-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-023-09631-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,10]],"date-time":"2023-11-10T15:12:53Z","timestamp":1699629173000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11219-023-09631-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,25]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["9631"],"URL":"https:\/\/doi.org\/10.1007\/s11219-023-09631-7","relation":{},"ISSN":["0963-9314","1573-1367"],"issn-type":[{"value":"0963-9314","type":"print"},{"value":"1573-1367","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,25]]},"assertion":[{"value":"17 April 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 May 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}