{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:08:45Z","timestamp":1776110925101,"version":"3.50.1"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,4,18]],"date-time":"2024-04-18T00:00:00Z","timestamp":1713398400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"INES, CNPq","award":["465614\/2014-0"],"award-info":[{"award-number":["465614\/2014-0"]}]},{"DOI":"10.13039\/501100002322","name":"CAPES","doi-asserted-by":"crossref","award":["88887.136410\/2017-00"],"award-info":[{"award-number":["88887.136410\/2017-00"]}],"id":[{"id":"10.13039\/501100002322","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100006162","name":"FACEPE","doi-asserted-by":"crossref","award":["APQ-0399-1.03\/17"],"award-info":[{"award-number":["APQ-0399-1.03\/17"]}],"id":[{"id":"10.13039\/501100006162","id-type":"DOI","asserted-by":"crossref"}]},{"name":"PRONEX","award":["APQ\/0388-1.03\/14"],"award-info":[{"award-number":["APQ\/0388-1.03\/14"]}]},{"name":"FAPESB INCITE","award":["PIE0002\/2022"],"award-info":[{"award-number":["PIE0002\/2022"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>Computational notebooks, such as Jupyter, have been widely adopted by data scientists to write code for analyzing and visualizing data. Despite their growing adoption and popularity, few studies have been found to understand Jupyter development challenges from the practitioners\u2019 point of view. This article presents a systematic study of bugs and challenges that Jupyter practitioners face through a large-scale empirical investigation. We mined 14,740 commits from 105 GitHub open source projects with Jupyter Notebook code. Next, we analyzed 30,416 StackOverflow posts, which gave us insights into bugs that practitioners face when developing Jupyter Notebook projects. Next, we conducted 19 interviews with data scientists to uncover more details about Jupyter bugs and to gain insight into Jupyter developers\u2019 challenges. Finally, to validate the study results and proposed taxonomy, we conducted a survey with 91 data scientists. We highlight bug categories, their root causes, and the challenges that Jupyter practitioners face.<\/jats:p>","DOI":"10.1145\/3641539","type":"journal-article","created":{"date-parts":[[2024,1,22]],"date-time":"2024-01-22T12:30:57Z","timestamp":1705926657000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Bug Analysis in Jupyter Notebook Projects: An Empirical Study"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-7716-3244","authenticated-orcid":false,"given":"Taijara Loiola","family":"De Santana","sequence":"first","affiliation":[{"name":"Federal University of Bahia, Institute of Computing (IC-UFBA), Salvador, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0197-8249","authenticated-orcid":false,"given":"Paulo Anselmo Da Mota Silveira","family":"Neto","sequence":"additional","affiliation":[{"name":"Federal University Rural of Pernambuco (UFRPE), Recife, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9312-6715","authenticated-orcid":false,"given":"Eduardo Santana","family":"De Almeida","sequence":"additional","affiliation":[{"name":"Federal University of Bahia, Institute of Computing (IC-UFBA), Salvador, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8221-5352","authenticated-orcid":false,"given":"Iftekhar","family":"Ahmed","sequence":"additional","affiliation":[{"name":"University of California, Irvine, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,4,18]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183519.3183549"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.5555\/645920.672836"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568233"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3076253"},{"key":"e_1_3_3_6_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI\u201920)","author":"Chattopadhyay Souti","year":"2020","unstructured":"Souti Chattopadhyay, Ishita Prasad, Austin Z. Henley, Anita Sarma, and Titus Barik. 2020. What\u2019s wrong with computational notebooks? Pain points, needs, and design opportunities. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI\u201920). ACM, 1\u201312."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/2500499"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.46743\/2160-3715\/2015.2281"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380397"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300500"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338955"},{"key":"e_1_3_3_12_2","volume-title":"Project Jupyter:Computational Narratives as the Engine of Collaborative Data Science","author":"Jupyter Project","year":"2015","unstructured":"Project Jupyter. 2015. Project Jupyter:Computational Narratives as the Engine of Collaborative Data Science. Retrieved January 26, 2024 from https:\/\/blog.jupyter.org\/project-jupyter-computational-narratives-as-the-engine-of-collaborative-data-science-2b5fb94c3c58"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2012.219"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173748"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379337.3415842"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884783"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5_3"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/VL\/HCC50065.2020.9127202"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183519.3183548"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00051"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.7287\/peerj.preprints.2617v1"},{"key":"e_1_3_3_23_2","volume-title":"Proceedings of the 44rd IEEE\/ACM International Conference on Software Engineering (ICSE\u201922)","author":"Patra Jibesh","year":"2021","unstructured":"Jibesh Patra and Michael Pradel. 2021. Nalin:Learning from runtime behavior to find name-value inconsistencies in Jupyter Notebooks. In Proceedings of the 44rd IEEE\/ACM International Conference on Software Engineering (ICSE\u201922). ACM."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00077"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-09961-9"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3278142.3278149"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380409"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3274419"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173606"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173606"},{"key":"e_1_3_3_31_2","volume-title":"The Coding Manual for Qualitative Researchers","author":"Salda\u00f1a J.","year":"2009","unstructured":"J. Salda\u00f1a. 2009. The Coding Manual for Qualitative Researchers. SAGE. 01475499https:\/\/books.google.com.br\/books?id=OE7LngEACAAJ"},{"key":"e_1_3_3_32_2","volume-title":"The Coding Manual for Qualitative Researchers","author":"Saldana J.","year":"2015","unstructured":"J. Saldana. 2015. The Coding Manual for Qualitative Researchers. SAGE."},{"key":"e_1_3_3_33_2","volume-title":"Interviewing as Qualitative Research:A Guide for Researchers in Education and the Social Sciences (3rd ed.)","author":"Seidman I.","year":"2006","unstructured":"I. Seidman. 2006. Interviewing as Qualitative Research:A Guide for Researchers in Education and the Social Sciences (3rd ed.). Teachers College Press."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-10-0983-9_32"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2013.38"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416543"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2012.22"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359141"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468559"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416585"},{"key":"e_1_3_3_41_2","first-page":"53","volume-title":"Proceedings of the 42nd International Conference on Software Engineering, New Ideas, and Emerging Results (ICSE-NIER\u201920)","author":"Wang Jiawei","year":"2020","unstructured":"Jiawei Wang, Li Li, and Andreas Zeller. 2020b. Better code, better sharing:On the need of analyzing Jupyter Notebooks. In Proceedings of the 42nd International Conference on Software Engineering, New Ideas, and Emerging Results (ICSE-NIER\u201920). ACM, 53\u201356."},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377816.3381724"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00144"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678520"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3641539","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3641539","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:05:51Z","timestamp":1750291551000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3641539"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,18]]},"references-count":44,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3641539"],"URL":"https:\/\/doi.org\/10.1145\/3641539","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,18]]},"assertion":[{"value":"2022-10-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}