{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T05:40:13Z","timestamp":1773207613451,"version":"3.50.1"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2024,8,12]],"date-time":"2024-08-12T00:00:00Z","timestamp":1723420800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,12]],"date-time":"2024-08-12T00:00:00Z","timestamp":1723420800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100008007","name":"Universit\u00e4t Paderborn","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008007","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2024,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Jupyter notebooks have emerged as the predominant tool for data scientists to develop and share machine learning solutions, primarily using Python as the programming language. Despite their widespread adoption, a significant fraction of these notebooks, when shared on public repositories, suffer from insufficient documentation and a lack of coherent narrative. Such shortcomings compromise the readability and understandability of the notebook. Addressing this shortcoming, this paper introduces <jats:sc>HeaderGen<\/jats:sc>, a tool-based approach that automatically augments code cells in these notebooks with descriptive markdown headers, derived from a predefined taxonomy of machine learning operations. Additionally, it systematically classifies and displays function calls in line with this taxonomy. The mechanism that powers <jats:sc>HeaderGen<\/jats:sc> is an enhanced call graph analysis technique, building upon the foundational analysis available in <jats:italic>PyCG<\/jats:italic>. To improve precision, <jats:sc>HeaderGen<\/jats:sc> extends PyCG\u2019s analysis with return-type resolution of external function calls, type inference, and flow-sensitivity. Furthermore, leveraging type information, <jats:sc>HeaderGen<\/jats:sc> employs pattern matching techniques on the code syntax to annotate code cells. We conducted an empirical evaluation on 15 real-world Jupyter notebooks sourced from Kaggle. The results indicate a high accuracy in call graph analysis, with precision at 95.6% and recall at 95.3%. The header generation has a precision of 85.7% and a recall rate of 92.8% with regard to headers created manually by experts. A user study corroborated the practical utility of <jats:sc>HeaderGen<\/jats:sc>, revealing that users found <jats:sc>HeaderGen<\/jats:sc> useful in tasks related to comprehension and navigation. To further evaluate the type inference capability of static analysis tools, we introduce <jats:sc>TypeEvalPy<\/jats:sc>, a framework for evaluating type inference tools for Python with an in-built micro-benchmark containing 154 code snippets and 845 type annotations in the ground truth. Our comparative analysis on four tools revealed that <jats:sc>HeaderGen<\/jats:sc> outperforms other tools in exact matches with the ground truth.<\/jats:p>","DOI":"10.1007\/s10664-024-10525-w","type":"journal-article","created":{"date-parts":[[2024,8,12]],"date-time":"2024-08-12T11:03:12Z","timestamp":1723460592000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Static analysis driven enhancements for comprehension in machine learning notebooks"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6724-7962","authenticated-orcid":false,"given":"Ashwin Prasad Shivarpatna","family":"Venkatesh","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Samkutty","family":"Sabu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mouli","family":"Chekkapalli","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiawei","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Li","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eric","family":"Bodden","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,8,12]]},"reference":[{"key":"10525_CR1","unstructured":"Pyright (2022) static type checker for Python. https:\/\/github.com\/microsoft\/pyright"},{"key":"10525_CR2","unstructured":"Pytype (2022) Google, https:\/\/github.com\/google\/pytype"},{"key":"10525_CR3","unstructured":"MOPSA\/MOPSA (2024) analyzer $$\\cdot $$ GitLab.https:\/\/gitlab.com\/mopsa\/mopsa-analyzer"},{"key":"10525_CR4","doi-asserted-by":"publisher","unstructured":"Adeli M, Nelson N, Chattopadhyay S, Coffey H, Henley A, Sarma A (2020) Supporting Code Comprehension via Annotations: Right Information at the Right Time and Place. In: 2020 IEEE symposium on visual languages and human-centric computing (VL\/HCC), pp 1\u201310. https:\/\/doi.org\/10.1109\/VL\/HCC50065.2020.9127264","DOI":"10.1109\/VL\/HCC50065.2020.9127264"},{"key":"10525_CR5","doi-asserted-by":"publisher","unstructured":"Allamanis M, Barr ET, Ducousso S, Gao Z (2020) Typilus: Neural type hints. In: Proceedings of the 41st ACM SIGPLAN conference on programming language design and implementation, ACM, London UK, pp 91\u2013105. https:\/\/doi.org\/10.1145\/3385412.3385997","DOI":"10.1145\/3385412.3385997"},{"key":"10525_CR6","unstructured":"Barstad V, Goodwin M, Gj\u00f8s\u00e6ter T (2014) Predicting source code quality with static analysis and machine learning. In: Norsk IKT-konferanse for forskning og utdanning"},{"issue":"1","key":"10525_CR7","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen J (1960) A Coefficient of Agreement for Nominal Scales. Educ Psychol Measure 20(1):37\u201346. https:\/\/doi.org\/10.1177\/001316446002000104","journal-title":"Educ Psychol Measure"},{"key":"10525_CR8","doi-asserted-by":"publisher","unstructured":"Di\u00a0Grazia L, Pradel M (2022a) The evolution of type annotations in python: An empirical study. In: Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering, ACM, Singapore Singapore, pp 209\u2013220, https:\/\/doi.org\/10.1145\/3540250.3549114","DOI":"10.1145\/3540250.3549114"},{"key":"10525_CR9","doi-asserted-by":"crossref","unstructured":"Di\u00a0Grazia L, Pradel M (2022b) The evolution of type annotations in python: An empirical study. In: Proceedings of the 30th Acm joint meeting on european software engineering conference and symposium on the foundations of software engineering","DOI":"10.1145\/3540250.3549114"},{"key":"10525_CR10","doi-asserted-by":"crossref","unstructured":"Epperson W, Wang A, DeLIne R, Drucker S (2022) Strategies for reuse and sharing among data scientists in software teams. In: ICSE 2022","DOI":"10.1109\/ICSE-SEIP55303.2022.9793945"},{"key":"10525_CR11","doi-asserted-by":"publisher","unstructured":"Grotov K, Titov S, Sotnikov V, Golubev Y, Bryksin T (2022) A large-scale comparison of Python code in Jupyter notebooks and scripts. In: Proceedings of the 19th international conference on mining software repositories, association for computing machinery, New York, NY, USA, MSR \u201922, pp 353\u201336https:\/\/doi.org\/10.1145\/3524842.3528447","DOI":"10.1145\/3524842.3528447"},{"key":"10525_CR12","doi-asserted-by":"publisher","DOI":"10.1145\/3652153","author":"Y Guo","year":"2024","unstructured":"Guo Y, Chen Z, Chen L, Xu W, Li Y, Zhou Y, Xu B (2024) Generating Python Type Annotations from Type Inference: How Far Are We? ACM Trans Softw Eng Methodolo. https:\/\/doi.org\/10.1145\/3652153","journal-title":"ACM Trans Softw Eng Methodolo"},{"key":"10525_CR13","unstructured":"Halter D (2022) Jedi - an awesome autocompletion, static analysis and refactoring library for Python"},{"issue":"3","key":"10525_CR14","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1016\/0096-0551(78)90009-7","volume":"3","author":"K Kennedy","year":"1978","unstructured":"Kennedy K (1978) Use-definition chains with applications. Comput Lang 3(3):163\u201317. https:\/\/doi.org\/10.1016\/0096-0551(78)90009-7","journal-title":"Comput Lang"},{"key":"10525_CR15","doi-asserted-by":"publisher","unstructured":"Kery MB, Radensky M, Arya M, John BE, Myers BA (2018) The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool. In: Proceedings of the 2018 CHI conference on human factors in computing systems, association for computing machinery, New York, NY, USA, CHI \u201918, pp 1\u20131. https:\/\/doi.org\/10.1145\/3173574.3173748","DOI":"10.1145\/3173574.3173748"},{"issue":"2","key":"10525_CR16","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1093\/comjnl\/27.2.97","volume":"27","author":"DE Knuth","year":"1984","unstructured":"Knuth DE (1984) Literate Programming. Comput J 27(2):97\u2013111. https:\/\/doi.org\/10.1093\/comjnl\/27.2.97","journal-title":"Comput J"},{"key":"10525_CR17","doi-asserted-by":"publisher","unstructured":"Koenzen AP, Ernst NA, Storey MAD (2020) Code Duplication and Reuse in Jupyter Notebooks. In: 2020 IEEE symposium on visual languages and human-centric computing (VL\/HCC), pp 1\u20139. https:\/\/doi.org\/10.1109\/VL\/HCC50065.2020.9127202","DOI":"10.1109\/VL\/HCC50065.2020.9127202"},{"key":"10525_CR18","doi-asserted-by":"publisher","unstructured":"Kummita S, Piskachev G, Sp\u00e4th J, Bodden E (2021) Qualitative and Quantitative Analysis of Callgraph Algorithms for Python. In: 2021 International conference on code quality (ICCQ), pp 1\u201315. https:\/\/doi.org\/10.1109\/ICCQ51190.2021.9392986","DOI":"10.1109\/ICCQ51190.2021.9392986"},{"key":"10525_CR19","doi-asserted-by":"publisher","unstructured":"Li L, Wang J, Quan H (2022) Scalpel: The python static analysis framework. https:\/\/doi.org\/10.48550\/ARXIV.2202.11840arXiv:2202.11840","DOI":"10.48550\/ARXIV.2202.11840"},{"key":"10525_CR20","doi-asserted-by":"publisher","unstructured":"Mir AM, Lato\u0161kinas E, Proksch S, Gousios G (2022) Type4Py: Practical deep similarity learning-based type inference for python. In: Proceedings of the 44th international conference on software engineering, association for computing machinery, New York, NY, USA, ICSE \u201922, pp 2241\u2013225. https:\/\/doi.org\/10.1145\/3510003.3510124","DOI":"10.1145\/3510003.3510124"},{"key":"10525_CR21","unstructured":"Monat R, Ouadjaout A, Min\u00e9 A (2020a) Static type analysis by abstract interpretation of python programs. In: 34th European conference on object-oriented programming (ECOOP 2020), Schloss Dagstuhl-Leibniz-Zentrum f\u00fcr Informatik"},{"key":"10525_CR22","doi-asserted-by":"crossref","unstructured":"Monat R, Ouadjaout A, Min\u00e9 A (2020b) Value and allocation sensitivity in static python analyses. In: Proceedings of the 9th ACM SIGPLAN international workshop on the state of the art in program analysis, pp 8\u201313","DOI":"10.1145\/3394451.3397205"},{"key":"10525_CR23","doi-asserted-by":"publisher","unstructured":"Peng Y, Gao C, Li Z, Gao B, Lo D, Zhang Q, Lyu M (2022) Static inference meets deep learning: A hybrid type inference approach for python. In: Proceedings of the 44th international conference on software engineering, association for computing machinery, New York, NY, USA, ICSE \u201922, pp 2019\u2013203. https:\/\/doi.org\/10.1145\/3510003.3510038","DOI":"10.1145\/3510003.3510038"},{"key":"10525_CR24","doi-asserted-by":"publisher","unstructured":"Pimentel JF, Murta L, Braganholo V, Freire J (2019) A Large-Scale Study About Quality and Reproducibility of Jupyter Notebooks. In: 2019 IEEE\/ACM 16th international conference on mining software repositories (MSR), IEEE, Montreal, QC, Canada, pp 507\u2013517. https:\/\/doi.org\/10.1109\/MSR.2019.00077","DOI":"10.1109\/MSR.2019.00077"},{"key":"10525_CR25","doi-asserted-by":"publisher","unstructured":"Quaranta L, Calefato F, Lanubile F (2022) Eliciting Best Practices for Collaboration with Computational Notebooks. Proc ACM Human-Comput Inter 6(CSCW1):87:1\u201387:4. https:\/\/doi.org\/10.1145\/3512934","DOI":"10.1145\/3512934"},{"key":"10525_CR26","doi-asserted-by":"publisher","unstructured":"Ramasamy D, Sarasua C, Bacchelli A, Bernstein A (2022) Workflow analysis of data science code in public GitHub repositories. Empir Softw Eng 28(1). https:\/\/doi.org\/10.1007\/s10664-022-10229-z","DOI":"10.1007\/s10664-022-10229-z"},{"key":"10525_CR27","doi-asserted-by":"publisher","unstructured":"Rule A, Tabard A, Hollan JD (2018) Exploration and Explanation in Computational Notebooks. In: Proceedings of the 2018 CHI conference on Human factors in computing systems, Association for Computing Machinery, New York, NY, USA, CHI \u201918, pp 1\u201312. https:\/\/doi.org\/10.1145\/3173574.3173606","DOI":"10.1145\/3173574.3173606"},{"key":"10525_CR28","doi-asserted-by":"publisher","unstructured":"Salis V, Sotiropoulos T, Louridas P, Spinellis D, Mitropoulos D (2021) PyCG: Practical Call Graph Generation in Python. In: 2021 IEEE\/ACM 43rd international conference on software engineering (ICSE), IEEE, Madrid, Spain, pp 1646\u20131657,https:\/\/doi.org\/10.1109\/ICSE43902.2021.00146","DOI":"10.1109\/ICSE43902.2021.00146"},{"key":"10525_CR29","unstructured":"Samuel S, Mietchen D (2022) Computational reproducibility of Jupyter notebooks from biomedical publications. arXiv:2209.04308"},{"key":"10525_CR30","unstructured":"serge-sans-paille (2022) Gast, Beniget!"},{"key":"10525_CR31","doi-asserted-by":"crossref","unstructured":"Ugurel S, Krovetz R, Giles CL (2002) What\u2019s the code? automatic classification of source code archives. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 632\u2013638","DOI":"10.1145\/775047.775141"},{"key":"10525_CR32","doi-asserted-by":"publisher","unstructured":"Venkatesh APS, Wang J, Li L, Bodden E (2023a) Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis. In: 2023 IEEE international conference on software analysis, evolution and reengineering (SANER), IEEE Computer Society, pp 391\u201340. https:\/\/doi.org\/10.1109\/SANER56733.2023.00044","DOI":"10.1109\/SANER56733.2023.00044"},{"key":"10525_CR33","doi-asserted-by":"publisher","unstructured":"Venkatesh APS, Wang J, Li L, Bodden E (2023b) Enhancing Comprehension and Navigation in Jupyter Notebooks with Static Analysis. In: 2023 IEEE international conference on software analysis, evolution and reengineering (SANER), IEEE Computer Society, pp 391\u201340. https:\/\/doi.org\/10.1109\/SANER56733.2023.00044","DOI":"10.1109\/SANER56733.2023.00044"},{"issue":"14","key":"10525_CR34","doi-asserted-by":"publisher","first-page":"3359","DOI":"10.3390\/rs14143359","volume":"14","author":"J Wagemann","year":"2022","unstructured":"Wagemann J, Fierli F, Mantovani S, Siemen S, Seeger B, Bendix J (2022) Five guiding principles to make jupyter notebooks fit for earth observation data education. Remote Sensing 14(14):3359","journal-title":"Remote Sensing"},{"issue":"2","key":"10525_CR35","doi-asserted-by":"publisher","first-page":"17:1","DOI":"10.1145\/3489465","volume":"29","author":"AY Wang","year":"2022","unstructured":"Wang AY, Wang D, Drozdal J, Muller M, Park S, Weisz JD, Liu X, Wu L, Dugan C (2022) Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks. ACM Trans Comput Human Inter 29(2):17:1-17:33. https:\/\/doi.org\/10.1145\/3489465","journal-title":"ACM Trans Comput Human Inter"},{"key":"10525_CR36","doi-asserted-by":"publisher","unstructured":"Wang J, Kuo Ty, Li L, Zeller A (2020a) Assessing and restoring reproducibility of Jupyter notebooks. In: Proceedings of the 35th IEEE\/ACM international conference on automated software engineering, ACM, Virtual Event Australia, pp 138\u2013149. https:\/\/doi.org\/10.1145\/3324884.3416585","DOI":"10.1145\/3324884.3416585"},{"key":"10525_CR37","doi-asserted-by":"publisher","unstructured":"Wang J, Li L, Zeller A (2020b) Better code, better sharing: On the need of analyzing jupyter notebooks. In: Proceedings of the ACM\/IEEE 42nd international conference on software engineering: new ideas and emerging results, Association for Computing Machinery, New York, NY, USA, ICSE-NIER \u201920, pp 53\u201356. https:\/\/doi.org\/10.1145\/3377816.3381724","DOI":"10.1145\/3377816.3381724"},{"key":"10525_CR38","doi-asserted-by":"crossref","unstructured":"Yang C, Brower-Sinning RA, Lewis GA, K\u00e4stner C (2022a) Data leakage in notebooks: Static detection and better processes. In: Proceedings of the 37th IEEE\/ACM international conference on automated software engineering","DOI":"10.1145\/3551349.3556918"},{"key":"10525_CR39","doi-asserted-by":"crossref","unstructured":"Yang Y, Milanova A, Hirzel M (2022b) Complex Python Features in the Wild","DOI":"10.1145\/3524842.3528467"},{"key":"10525_CR40","unstructured":"Zevin S, Holzem C (2017) Machine learning based source code classification using syntax oriented features. arXiv:1703.07638"},{"issue":"1","key":"10525_CR41","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1140\/epjds\/s13688-022-00327-9","volume":"11","author":"G Zhang","year":"2022","unstructured":"Zhang G, Merrill MA, Liu Y, Heer J, Althoff T (2022) Coral: Code representation learning with weakly-supervised transformers for analyzing data analysis. EPJ Data Sci 11(1):14","journal-title":"EPJ Data Sci"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10525-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-024-10525-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-024-10525-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,6]],"date-time":"2024-09-06T04:00:39Z","timestamp":1725595239000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-024-10525-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,12]]},"references-count":41,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,9]]}},"alternative-id":["10525"],"URL":"https:\/\/doi.org\/10.1007\/s10664-024-10525-w","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,12]]},"assertion":[{"value":"2 July 2024","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 August 2024","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"136"}}