{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T07:16:30Z","timestamp":1776323790092,"version":"3.50.1"},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA1","license":[{"start":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T00:00:00Z","timestamp":1714348800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science Foundation","award":["2319014"],"award-info":[{"award-number":["2319014"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2024,4,29]]},"abstract":"<jats:p>This paper documents a year-long experiment to \u201cprofile\u201d the process of learning a programming language: gathering data to understand what makes a language hard to learn, and using that data to improve the learning process. We added interactive quizzes to The Rust Programming Language, the official textbook for learning Rust. Over 13 months, 62,526 readers answered questions 1,140,202 times. First, we analyze the trajectories of readers. We find that many readers drop-out of the book early when faced with difficult language concepts like Rust\u2019s ownership types. Second, we use classical test theory and item response theory to analyze the characteristics of quiz questions. We find that better questions are more conceptual in nature, such as asking why a program does not compile vs. whether a program compiles. Third, we performed 12 interventions into the book to help readers with difficult questions. We find that on average, interventions improved quiz scores on the targeted questions by +20<\/jats:p>","DOI":"10.1145\/3649812","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T17:53:50Z","timestamp":1714413230000},"page":"29-54","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Profiling Programming Language Learning"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8639-6541","authenticated-orcid":false,"given":"Will","family":"Crichton","sequence":"first","affiliation":[{"name":"Brown University, Providence, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5184-1975","authenticated-orcid":false,"given":"Shriram","family":"Krishnamurthi","sequence":"additional","affiliation":[{"name":"Brown University, Providence, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,4,29]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.3102\/00346543049002280"},{"key":"e_1_2_1_2_1","unstructured":"Jeremy Avigad Leonardo de Moura Soonho Kong and Sebastian Ullrich. 2023. Theorem Proving in Lean 4. https:\/\/lean-lang.org\/theorem_proving_in_lean4\/"},{"key":"e_1_2_1_3_1","volume-title":"The Basics of Item Response Theory (2 ed.)","author":"Baker Frank B.","unstructured":"Frank B. Baker. 2001. The Basics of Item Response Theory (2 ed.). ERIC Clearinghouse on Assessment and Evaluation, College Park, MD."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_2_1_5_1","first-page":"1","article-title":"Pyro: Deep Universal Probabilistic Programming","volume":"20","author":"Bingham Eli","year":"2019","unstructured":"Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research, 20, 28 (2019), 1\u20136. http:\/\/jmlr.org\/papers\/v20\/18-403.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_6_1","volume-title":"APA Handbook of Research Methods in Psychology. 2","author":"Braun Virginia","unstructured":"Virginia Braun and Victoria Clarke. 2012. Thematic Analysis. In APA Handbook of Research Methods in Psychology. 2, American Psychological Association."},{"key":"e_1_2_1_7_1","volume-title":"Proc. ACM Program. Lang., 7, OOPSLA2","author":"Crichton Will","year":"2023","unstructured":"Will Crichton, Gavin Gray, and Shriram Krishnamurthi. 2023. A Grounded Conceptual Model for Ownership Types in Rust. Proc. ACM Program. Lang., 7, OOPSLA2 (2023), Article 265, oct, 29 pages. https:\/\/doi.org\/10.1145\/3622841 10.1145\/3622841"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","unstructured":"Will Crichton and Shriram Krishnamurthi. 2024. Artifact for \"Profiling Programming Language Learning\". https:\/\/doi.org\/10.5281\/zenodo.10798571 10.5281\/zenodo.10798571","DOI":"10.5281\/zenodo.10798571"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 53rd ACM Technical Symposium on Computer Science Education -","volume":"1","author":"Ericson Barbara J.","year":"2022","unstructured":"Barbara J. Ericson, Hisamitsu Maeda, and Paramveer S. Dhillon. 2022. Detecting Struggling Students from Interactive Ebook Data: A Case Study Using CSAwesome. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1 (SIGCSE 2022). Association for Computing Machinery, New York, NY, USA. 418\u2013424. isbn:9781450390705 https:\/\/doi.org\/10.1145\/3478431.3499354 10.1145\/3478431.3499354"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 51st ACM Technical Symposium on Computer Science Education (SIGCSE \u201920)","author":"Barbara","unstructured":"Barbara J. Ericson and Bradley N. Miller. 2020. Runestone: A Platform for Free, On-Line, and Interactive Ebooks. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (SIGCSE \u201920). Association for Computing Machinery, New York, NY, USA. 1012\u20131018. isbn:9781450367936 https:\/\/doi.org\/10.1145\/3328778.3366950 10.1145\/3328778.3366950"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Taylor Fausak. 2022. 2022 State of Haskell Survey Results. https:\/\/taylor.fausak.me\/2022\/11\/18\/haskell-survey-results","DOI":"10.1079\/searchRxiv.2022.00005"},{"key":"e_1_2_1_12_1","volume-title":"Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021","author":"Fulton Kelsey R.","year":"2021","unstructured":"Kelsey R. Fulton, Anna Chan, Daniel Votipka, Michael Hicks, and Michelle L. Mazurek. 2021. Benefits and Drawbacks of Adopting a Secure Programming Language: Rust as a Case Study. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021). USENIX Association, Berkeley, CA. 597\u2013616. isbn:978-1-939133-25-0 https:\/\/www.usenix.org\/conference\/soups2021\/presentation\/fulton"},{"key":"e_1_2_1_13_1","unstructured":"Google. 2023. Comprehensive Rust. https:\/\/google.github.io\/comprehensive-rust\/"},{"key":"e_1_2_1_14_1","volume-title":"The State of Developer Ecosystem","year":"2022","unstructured":"JetBrains. 2022. The State of Developer Ecosystem 2022. https:\/\/www.jetbrains.com\/lp\/devecosystem-2022\/"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.19173\/irrodl.v15i1.1651"},{"key":"e_1_2_1_16_1","unstructured":"Steve Klabnik and Carol Nichols. 2022. The Rust Programming Language. https:\/\/doc.rust-lang.org\/book"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2022.104562"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3984.1988.tb00288.x"},{"key":"e_1_2_1_20_1","unstructured":"Niko Matsakis. 2017. Non-lexical lifetimes. https:\/\/rust-lang.github.io\/rfcs\/2094-nll.html"},{"key":"e_1_2_1_21_1","unstructured":"Alice Merrick. 2023. Go Developer Survey 2023 Q1 Results. https:\/\/go.dev\/blog\/survey2023-q1-results"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages &t Applications (OOPSLA \u201913)","author":"Leo","unstructured":"Leo A. Meyerovich and Ariel S. Rabkin. 2013. Empirical Analysis of Programming Language Adoption. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages &t Applications (OOPSLA \u201913). Association for Computing Machinery, New York, NY, USA. 1\u201318. isbn:9781450323741 https:\/\/doi.org\/10.1145\/2509136.2509515 10.1145\/2509136.2509515"},{"key":"e_1_2_1_23_1","volume-title":"OCaml Users Survey","author":"Foundation Caml Software","year":"2022","unstructured":"OCaml Software Foundation. 2022. OCaml Users Survey 2022. https:\/\/ocaml-sf.org\/docs\/2022\/ocaml-user-survey-2022.pdf"},{"key":"e_1_2_1_24_1","volume-title":"Perkins and Gavriel Salomon","author":"David","year":"1994","unstructured":"David N. Perkins and Gavriel Salomon. 1994. Transfer of Learning. In International Encyclopedia of Education (2 ed.). Elsevier Science, Amsterdam, Netherlands."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER \u201919)","author":"Porter Leo","year":"2019","unstructured":"Leo Porter, Daniel Zingaro, Soohyun Nam Liao, Cynthia Taylor, Kevin C. Webb, Cynthia Lee, and Michael Clancy. 2019. BDSI: A Validated Concept Inventory for Basic Data Structures. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER \u201919). Association for Computing Machinery, New York, NY, USA. 111\u2013119. isbn:9781450361859 https:\/\/doi.org\/10.1145\/3291279.3339404 10.1145\/3291279.3339404"},{"key":"e_1_2_1_26_1","unstructured":"Robert Randolph. 2022. State of Clojure 2022 Results. https:\/\/clojure.org\/news\/2022\/06\/02\/state-of-clojure-2022"},{"key":"e_1_2_1_27_1","volume-title":"Introduction to psychometric theory","author":"Raykov Tenko","unstructured":"Tenko Raykov and George A Marcoulides. 2011. Introduction to psychometric theory. Routledge."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0024117"},{"key":"e_1_2_1_29_1","unstructured":"Scala Center Team. 2022. Scala Developer Survey 2022 Results. https:\/\/www.scala-lang.org\/blog\/2022\/12\/14\/scala-developer-survey-results-2022.html"},{"key":"e_1_2_1_30_1","volume-title":"Stack Overflow Developer Survey","author":"Overflow Stack","year":"2023","unstructured":"Stack Overflow. 2023. Stack Overflow Developer Survey 2023. https:\/\/survey.stackoverflow.co\/2023\/"},{"key":"e_1_2_1_31_1","unstructured":"The Rust Survey Team. 2020. Rust Survey 2020 Results. https:\/\/blog.rust-lang.org\/2020\/12\/16\/rust-survey-2020.html"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 2017 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!","author":"Wilson Preston Tunnell","year":"2017","unstructured":"Preston Tunnell Wilson, Justin Pombrio, and Shriram Krishnamurthi. 2017. Can We Crowdsource Language Design? In Proceedings of the 2017 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2017). Association for Computing Machinery, New York, NY, USA. 1\u201317. isbn:9781450355308 https:\/\/doi.org\/10.1145\/3133850.3133863 10.1145\/3133850.3133863"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1207\/S1532690XCI2103_01"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE \u201919)","author":"Xie Benjamin","unstructured":"Benjamin Xie, Matthew J. Davidson, Min Li, and Amy J. Ko. 2019. An Item Response Theory Evaluation of a Language-Independent CS1 Knowledge Assessment. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE \u201919). Association for Computing Machinery, New York, NY, USA. 699\u2013705. isbn:9781450358903 https:\/\/doi.org\/10.1145\/3287324.3287370 10.1145\/3287324.3287370"},{"key":"e_1_2_1_35_1","volume-title":"9th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU 2018","volume":"6","author":"Zeng Anna","year":"2019","unstructured":"Anna Zeng and Will Crichton. 2019. Identifying Barriers to Adoption for Rust through Online Discourse. In 9th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU 2018), Titus Barik, Joshua Sunshine, and Sarah Chasins (Eds.) (OpenAccess Series in Informatics (OASIcs), Vol. 67). Schloss Dagstuhl\u2013Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany. 5:1\u20135:6. isbn:978-3-95977-091-0 issn:2190-6807 https:\/\/doi.org\/10.4230\/OASIcs.PLATEAU.2018.5 10.4230\/OASIcs.PLATEAU.2018.5"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 44th International Conference on Software Engineering (ICSE \u201922)","author":"Zhu Shuofei","year":"2022","unstructured":"Shuofei Zhu, Ziyi Zhang, Boqin Qin, Aiping Xiong, and Linhai Song. 2022. Learning and Programming Challenges of Rust: A Mixed-Methods Study. In Proceedings of the 44th International Conference on Software Engineering (ICSE \u201922). Association for Computing Machinery, New York, NY, USA. 1269\u20131281. isbn:9781450392211 https:\/\/doi.org\/10.1145\/3510003.3510164 10.1145\/3510003.3510164"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649812","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3649812","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:06Z","timestamp":1750287246000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649812"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,29]]},"references-count":36,"journal-issue":{"issue":"OOPSLA1","published-print":{"date-parts":[[2024,4,29]]}},"alternative-id":["10.1145\/3649812"],"URL":"https:\/\/doi.org\/10.1145\/3649812","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,29]]},"assertion":[{"value":"2024-04-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}