{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T15:03:50Z","timestamp":1781708630351,"version":"3.54.5"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"name":"JSPS KAKENHI","award":["JP20H05706, JP24H00692"],"award-info":[{"award-number":["JP20H05706, JP24H00692"]}]},{"name":"Spanish Ministry of Science, Innovation, and Universities under the Excellence Network AI4Software","award":["Red2022-134647-T"],"award-info":[{"award-number":["Red2022-134647-T"]}]},{"name":"Dependentium project","award":["PID2022-139551NB-I00"],"award-info":[{"award-number":["PID2022-139551NB-I00"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:p>\n                    The ability to measure developer proficiency is crucial, as it reflects an individual\u2019s capability to understand and interpret efficient, effective, and well-structured code. It is an essential aspect of ensuring software quality. Although prior work has proposed different approaches for measuring proficiency through code, the process of learning various coding concepts remains non-trivial and highly debated. This article proposes a framework for determining code proficiency through learning aids that serve as ground-truth\u2014namely, textbooks\u2014and two automated methods: \u00dcbersequence and Clustering. We conducted an empirical study to identify Python code proficiency levels. We then assessed the effectiveness of the framework by addressing two key research questions. Using a dataset of 22 introductory Python textbooks and code constructs referenced from Python\n                    <jats:italic toggle=\"yes\">AST<\/jats:italic>\n                    , we achieved a high coverage of 85.51% for Python code constructs. Our analysis reveals a significantly high similarity in the sequence of code construct introductions across the textbooks, supporting our methodology of using textbooks to assess code proficiency. The resulting \u00dcbersequence demonstrates the feasibility of assigning proficiency levels to individual code constructs, while clustering further enables a structured grouping perspective. To demonstrate practical applicability, we present examples and initiate discussions on future research directions, particularly software maintenance tasks such as bug assignment and code reviews.\n                  <\/jats:p>","DOI":"10.1145\/3769864","type":"journal-article","created":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T08:38:26Z","timestamp":1760517506000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Determining Code Proficiency Levels from Python Textbooks"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1263-5903","authenticated-orcid":false,"given":"Ruksit","family":"Rojpaisarnkit","sequence":"first","affiliation":[{"name":"Nara Institute of Science and Technology, Ikoma, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1442-6761","authenticated-orcid":false,"given":"Gregorio","family":"Robles","sequence":"additional","affiliation":[{"name":"Universidad Rey Juan Carlos, Fuenlabrada, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9682-460X","authenticated-orcid":false,"given":"Jesus M.","family":"Gonzalez-Barahona","sequence":"additional","affiliation":[{"name":"Universidad Rey Juan Carlos, Fuenlabrada, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7418-9323","authenticated-orcid":false,"given":"Kenichi","family":"Matsumoto","sequence":"additional","affiliation":[{"name":"Nara Institute of Science and Technology, Ikoma, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2324-0608","authenticated-orcid":false,"given":"Raula Gaikovina","family":"Kula","sequence":"additional","affiliation":[{"name":"The University of Osaka, Suita, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,17]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3276954.3276960"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM55253.2022.00021"},{"key":"e_1_3_2_4_2","volume-title":"Head First Python: A Brain-Friendly Guide","author":"Barry Paul","year":"2010","unstructured":"Paul Barry. 2010. Head First Python: A Brain-Friendly Guide (1st ed.). O\u2019Reilly Media."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1088\/1742-5468\/2008\/10\/p10008"},{"key":"e_1_3_2_6_2","volume-title":"Python for Kids: A Playful Introduction to Programming","author":"Briggs J. R.","year":"2012","unstructured":"J. R. Briggs. 2012. Python for Kids: A Playful Introduction to Programming. No Starch Press. Retrieved from https:\/\/books.google.es\/books?id=Wd_PJ2y68uIC"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.5555\/2664446.2664484"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786854"},{"key":"e_1_3_2_9_2","volume-title":"Core Python Programming","author":"Chun Wesley","year":"2001","unstructured":"Wesley Chun. 2001. Core Python Programming, Vol. 1, Prentice Hall Professional."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/2544322.2544337"},{"key":"e_1_3_2_11_2","volume-title":"Coding Projects in Python (DK Help Your Kids)","author":"DK","year":"2017","unstructured":"DK. 2017. Coding Projects in Python (DK Help Your Kids). D.K. Publishing."},{"key":"e_1_3_2_12_2","volume-title":"Coding Games in Python (DK Help Your Kids)","author":"DK","year":"2018","unstructured":"DK. 2018. Coding Games in Python (DK Help Your Kids). D.K. Publishing."},{"key":"e_1_3_2_13_2","volume-title":"Think Python","author":"Downey Allen. B.","year":"2015","unstructured":"Allen. B. Downey. 2015. Think Python. O\u2019Reilly Media. Retrieved from https:\/\/books.google.es\/books?id=mZwbCw AAQBAJ"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2019.8668024"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC57359.2022.00065"},{"key":"e_1_3_2_16_2","volume-title":"Making Use of Python","author":"Gupta Rashi","year":"2003","unstructured":"Rashi Gupta. 2003. Making Use of Python. Wiley. Retrieved from https:\/\/books.google.es\/books?id=zkDdhUDbrCIC"},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","unstructured":"Aric Hagberg Pieter J. Swart and Daniel A. Schult. 2008. Exploring Network Structure Dynamics and Function Using NetworkX. Retrieved from https:\/\/www.osti.gov\/biblio\/960616","DOI":"10.25080\/TCWV9851"},{"key":"e_1_3_2_18_2","volume-title":"Python 3 for Absolute Beginners","author":"Hall T.","year":"2010","unstructured":"T. Hall and J. P. Stacey. 2010. Python 3 for Absolute Beginners. Apress. Retrieved from https:\/\/books.google.es\/books?id=fD1QMt-irpkC"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-20290-3"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510454.3522684"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.5555\/3033231"},{"key":"e_1_3_2_22_2","volume-title":"A Python Book: Beginning Python, Advanced Python, and Python Exercises","author":"Kuhlman Dave","year":"2009","unstructured":"Dave Kuhlman. 2009. A Python Book: Beginning Python, Advanced Python, and Python Exercises. Dave Kuhlman Lutz."},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524610.3527879"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608128"},{"key":"e_1_3_2_25_2","volume-title":"Introducing Python: Modern Computing in Simple Packages","author":"Lubanovic Bill","year":"2014","unstructured":"Bill Lubanovic. 2014. Introducing Python: Modern Computing in Simple Packages (1st ed.). O\u2019Reilly Media.","edition":"1"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.5555\/2543548"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2622669"},{"key":"e_1_3_2_28_2","volume-title":"Python Crash Course: A Hands-On, Project-Based Introduction to Programming","author":"Matthes E.","year":"2015","unstructured":"E. Matthes. 2015. Python Crash Course: A Hands-On, Project-Based Introduction to Programming. No Starch Press. Retrieved from https:\/\/books.google.es\/books?id=igYvDwAAQBAJ"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639227"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639187"},{"key":"e_1_3_2_31_2","volume-title":"Python: For Beginners A Crash Course Guide To Learn Python in 1 Week","author":"Needham Timothy C.","year":"2020","unstructured":"Timothy C. Needham. 2020. Python: For Beginners A Crash Course Guide To Learn Python in 1 Week. White Flower Publishing."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1103\/physreve.69.026113"},{"key":"e_1_3_2_33_2","volume-title":"The CEFR in Practice","author":"North Brian","year":"2014","unstructured":"Brian North. 2014. The CEFR in Practice, Vol. 4, Cambridge University Press."},{"key":"e_1_3_2_34_2","volume-title":"Python QuickStart Guide: The Simplified Beginner\u2019s Guide to Python Programming Using Hands-On Projects and Real-World Applications","author":"Oliver Robert","year":"2023","unstructured":"Robert Oliver. 2023. Python QuickStart Guide: The Simplified Beginner\u2019s Guide to Python Programming Using Hands-On Projects and Real-World Applications. ClydeBank Media LLC."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00098"},{"key":"e_1_3_2_36_2","doi-asserted-by":"crossref","unstructured":"Chaiyong Ragkhitwetsagul Komsan Kongwongsupak Thanakrit Maneesawas Natpichsinee Puttiwarodom Ruksit Rojpaisarnkit Morakot Choetkiertikul Raula Gaikovina Kula and Thanwadee Sunetnanta. 2024. jscefr: A framework to evaluate the code proficiency for JavaScript. arXiv:2408.16452. Retrieved from https:\/\/arxiv.org\/abs\/2408.16452","DOI":"10.1109\/ICSME58944.2024.00092"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524610.3527878"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME58944.2024.00035"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/IWESEP49350.2019.00016"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","unstructured":"Ina. Schieferdecker. 2020. Responsible software engineering. In The Future of Software Quality Assurance. Springer 137\u2013146. DOI: 10.1007\/978-3-030-29509-7_11","DOI":"10.1007\/978-3-030-29509-7_11"},{"key":"e_1_3_2_41_2","unstructured":"Yusuke Shinyama Philippe Guglielmetti and Pieter Marsman. 2023. pdminer.six. Retrieved from April 12 2023 from https:\/\/pdfminersix.readthedocs.io\/en\/latest\/"},{"key":"e_1_3_2_42_2","volume-title":"Programming in Python 3: A Complete Introduction to the Python Language","author":"Summerfield M.","year":"2010","unstructured":"M. Summerfield. 2010. Programming in Python 3: A Complete Introduction to the Python Language. Addison-Wesley. Retrieved from https:\/\/books.google.es\/books?id=H9emM_LGFDEC"},{"key":"e_1_3_2_43_2","volume-title":"Invent Your Own Computer Games with Python","author":"Sweigart Al","year":"2016","unstructured":"Al Sweigart. 2016. Invent Your Own Computer Games with Python. No Starch Press."},{"key":"e_1_3_2_44_2","volume-title":"Automate the Boring Stuff with Python: practical Programming for Total Beginners","author":"Sweigart Al","year":"2019","unstructured":"Al Sweigart. 2019. Automate the Boring Stuff with Python: practical Programming for Total Beginners. No Starch Press."},{"key":"e_1_3_2_45_2","volume-title":"Coding for Kids: Python: Learn to Code with 50 Awesome Games and Activities","author":"Tacke Adrienne B.","year":"2019","unstructured":"Adrienne B. Tacke. 2019. Coding for Kids: Python: Learn to Code with 50 Awesome Games and Activities. Sourcebooks, Inc."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/2393596.2393656"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-39940-9_606"},{"key":"e_1_3_2_48_2","volume-title":"Python Programming for Beginners: Learn the Basics of Python Programming (Python Crash Course, Programming for Dummies)","author":"Tudor James","year":"2019","unstructured":"James Tudor. 2019. Python Programming for Beginners: Learn the Basics of Python Programming (Python Crash Course, Programming for Dummies). Independently published."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00038"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/2487788.2487832"},{"issue":"5","key":"e_1_3_2_51_2","first-page":"360","article-title":"Understanding interobserver agreement: The kappa statistic","volume":"37","author":"Viera A.","year":"2005","unstructured":"A. Viera and J. Garrett. 2005. Understanding interobserver agreement: The kappa statistic. Family Medicine 37, 5 (2005), 360\u2013363.","journal-title":"Family Medicine"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1111\/lang.12223"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.5555\/2349018"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1002\/0470011815.b2a15150"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534864"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3769864","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T14:44:15Z","timestamp":1781707455000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3769864"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,17]]},"references-count":54,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7,31]]}},"alternative-id":["10.1145\/3769864"],"URL":"https:\/\/doi.org\/10.1145\/3769864","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,17]]},"assertion":[{"value":"2024-12-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-09","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}