{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T21:07:58Z","timestamp":1768338478299,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":55,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,12,21]],"date-time":"2020-12-21T00:00:00Z","timestamp":1608508800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,12,21]]},"DOI":"10.1145\/3324884.3416585","type":"proceedings-article","created":{"date-parts":[[2021,1,27]],"date-time":"2021-01-27T23:39:02Z","timestamp":1611790742000},"page":"138-149","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":47,"title":["Assessing and restoring reproducibility of Jupyter notebooks"],"prefix":"10.1145","author":[{"given":"Jiawei","family":"Wang","sequence":"first","affiliation":[{"name":"Monash University, Melbourne, Australia"}]},{"given":"Tzu-yang","family":"Kuo","sequence":"additional","affiliation":[{"name":"Hong Kong University of Science and Technology, Hong Kong, China"}]},{"given":"Li","family":"Li","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}]},{"given":"Andreas","family":"Zeller","sequence":"additional","affiliation":[{"name":"CISPA Helmholtz Center for Information Security, Saarbr\u00fccken, Germany"}]}],"member":"320","published-online":{"date-parts":[[2021,1,27]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629575.1629594"},{"key":"e_1_3_2_1_2_1","volume-title":"Retrieved","year":"2019","unstructured":"Anaconda. 2019. Anaconda Enterprise 4 Repository: Open Data Science Hub. (2019). Retrieved August 23, 2019 from https:\/\/docs.continuum.io\/anaconda-repository\/"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2008.89"},{"key":"e_1_3_2_1_4_1","volume-title":"Sampling in Software Engineering Research: A Critical Review and Guidelines. arXiv preprint arXiv:2002.07764","author":"Baltes Sebastian","year":"2020","unstructured":"Sebastian Baltes and Paul Ralph. 2020. Sampling in Software Engineering Research: A Critical Review and Guidelines. arXiv preprint arXiv:2002.07764 (2020)."},{"key":"e_1_3_2_1_5_1","volume-title":"2018 IEEE\/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 433--444","author":"Bell Jonathan","year":"2018","unstructured":"Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. 2018. DeFlaker: Automatically detecting flaky tests. In 2018 IEEE\/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 433--444."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2017.12.029"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2016.05.513"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330564"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2014.30"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-38791-8_58"},{"key":"e_1_3_2_1_11_1","volume-title":"Retrieved","year":"2019","unstructured":"Conda. 2019. Conda: Package, dependency and environment management for any language. (2019). Retrieved August 23, 2019 from https:\/\/docs.conda.io\/en\/latest\/"},{"key":"e_1_3_2_1_12_1","volume-title":"Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics surveys 4","author":"Fay Michael P","year":"2010","unstructured":"Michael P Fay and Michael A Proschan. 2010. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics surveys 4 (2010), 1."},{"key":"e_1_3_2_1_13_1","volume-title":"Retrieved","year":"2019","unstructured":"freezegun. 2019. Let your Python tests travel through time. (2019). Retrieved August 23, 2019 from https:\/\/pypi.org\/project\/freezegun\/0.1.11\/"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-011-9181-9"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00047"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2018.8506576"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173748"},{"key":"e_1_3_2_1_18_1","unstructured":"Thomas Kluyver Benjamin Ragan-Kelley Fernando P\u00e9rez Brian E Granger Matthias Bussonnier Jonathan Frederic Kyle Kelley Jessica B Hamrick Jason Grout Sylvain Corlay et al. 2016. Jupyter Notebooks---a publishing format for reproducible computational workflows. In ELPUB. 87--90."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/27.2.97"},{"key":"e_1_3_2_1_20_1","volume-title":"Mining Android Crash Fixes in the Absence of Issue- and Change-Tracking Systems. In The 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA","author":"Kong Pingfan","year":"2019","unstructured":"Pingfan Kong, Li Li, Jun Gao, Tegawend\u00e9 F Bissyand\u00e9, and Jacques Klein. 2019. Mining Android Crash Fixes in the Absence of Issue- and Change-Tracking Systems. In The 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019)."},{"key":"e_1_3_2_1_21_1","volume-title":"Automated Testing of Android Apps: A Systematic Literature Review","author":"Kong Pingfan","year":"2018","unstructured":"Pingfan Kong, Li Li, Jun Gao, Kui Liu, Tegawend\u00e9 F Bissyand\u00e9, and Jacques Klein. 2018. Automated Testing of Android Apps: A Systematic Literature Review. IEEE Transactions on Reliability (2018)."},{"key":"e_1_3_2_1_22_1","volume-title":"9th {USENIX} Workshop on the Theory and Practice of Provenance (TaPP","author":"Koop David","year":"2017","unstructured":"David Koop and Jay Patel. 2017. Dataflow notebooks: encoding and tracking dependencies of cells. In 9th {USENIX} Workshop on the Theory and Practice of Provenance (TaPP 2017)."},{"key":"e_1_3_2_1_23_1","volume-title":"CiD: Automating the Detection of API-related Compatibility Issues in Android Apps. In The ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA","author":"Li Li","year":"2018","unstructured":"Li Li, Tegawend\u00e9 F Bissyand\u00e9, Haoyu Wang, and Jacques Klein. 2018. CiD: Automating the Detection of API-related Compatibility Issues in Android Apps. In The ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2018)."},{"key":"e_1_3_2_1_24_1","volume-title":"Retrieved","author":"Standard Library The Python","year":"2019","unstructured":"The Python Standard Library. 2019. Build-in Functions. (2019). Retrieved August 23, 2019 from https:\/\/docs.python.org\/3\/library\/functions.html"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635920"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.11144\/Javeriana.upsy10-2.cdcp"},{"key":"e_1_3_2_1_27_1","volume-title":"Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research.","author":"Mockus Audris","year":"2010","unstructured":"Audris Mockus, Bente Anda, and Dag IK Sj\u00f8berg. 2010. Experiences from replicating a case study to investigate reproducibility of software development. In Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research."},{"key":"e_1_3_2_1_28_1","volume-title":"2017 USENIX Annual Technical Conference (USENIX ATC 17)","author":"O'Callahan Robert","year":"2017","unstructured":"Robert O'Callahan, Chris Jones, Nathan Froyd, Kyle Huey, Albert Noll, and Nimrod Partush. 2017. Engineering record and replay for deployability. In 2017 USENIX Annual Technical Conference (USENIX ATC 17). 377--389."},{"key":"e_1_3_2_1_29_1","volume-title":"The Twenty-Eighth International Flairs Conference.","author":"O'Hara Keith","year":"2015","unstructured":"Keith O'Hara, Douglas Blank, and James Marshall. 2015. Computational notebooks for AI education. In The Twenty-Eighth International Flairs Conference."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09683-z"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.286418"},{"key":"e_1_3_2_1_32_1","volume-title":"Why Jupyter is data scientists' computational notebook of choice. Nature news 563","author":"Perkel Jeffrey M.","year":"2018","unstructured":"Jeffrey M. Perkel. 2018. Why Jupyter is data scientists' computational notebook of choice. Nature news 563 (2018), 145--146."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00077"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338959"},{"key":"e_1_3_2_1_35_1","volume-title":"Retrieved","year":"2019","unstructured":"PythonClock. 2019. Python 2.7 Countdown. (2019). Retrieved August 23, 2019 from https:\/\/pythonclock.org"},{"key":"e_1_3_2_1_36_1","volume-title":"AGU Fall Meeting Abstracts.","author":"Ragan-Kelley Min","year":"2014","unstructured":"Min Ragan-Kelley, F Perez, B Granger, T Kluyver, P Ivanov, J Frederic, and M Bussonnier. 2014. The Jupyter\/IPython architecture: a unified view of computational research, from interactive exploration to communication and publication.. In AGU Fall Meeting Abstracts."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/JCDL.2017.7991618"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300107"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180224"},{"key":"e_1_3_2_1_40_1","unstructured":"Zhilei Ren Changlin Liu Xusheng Xiao He Jiang and Tao Xie. [n. d.]. Root Cause Localization for Unreproducible Builds via Causality Analysis over System Call Tracing. ([n. d.])."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.03.009"},{"key":"e_1_3_2_1_42_1","volume-title":"NanoDJ: a Dockerized Jupyter notebook for interactive Oxford Nanopore MinION sequence manipulation and genome assembly. BMC bioinformatics 20, 1","author":"Rodr\u00edguez-P\u00e9rez H\u00e9ctor","year":"2019","unstructured":"H\u00e9ctor Rodr\u00edguez-P\u00e9rez, Tamara Hern\u00e1ndez-Beeftink, Jos\u00e9 M Lorenzo-Salazar, Jos\u00e9 L Roda-Garc\u00eda, Carlos J P\u00e9rez-Gonz\u00e1lez, Marcos Colebrook, and Carlos Flores. 2019. NanoDJ: a Dockerized Jupyter notebook for interactive Oxford Nanopore MinION sequence manipulation and genome assembly. BMC bioinformatics 20, 1 (2019), 234."},{"key":"e_1_3_2_1_43_1","volume-title":"Fernando P\u00e9rez, et al.","author":"Rule Adam","year":"2018","unstructured":"Adam Rule, Amanda Birmingham, Cristal Zuniga, Ilkay Altintas, Shih-Cheng Huang, Rob Knight, Niema Moshiri, Mai H Nguyen, Sara Brin Rosenthal, Fernando P\u00e9rez, et al. 2018. Ten simple rules for reproducible research in Jupyter notebooks. arXiv preprint arXiv:1810.08055 (2018)."},{"key":"e_1_3_2_1_44_1","volume-title":"Fernando P\u00e9rez, et al.","author":"Rule Adam","year":"2019","unstructured":"Adam Rule, Amanda Birmingham, Cristal Zuniga, Ilkay Altintas, Shih-Cheng Huang, Rob Knight, Niema Moshiri, Mai H Nguyen, Sara Brin Rosenthal, Fernando P\u00e9rez, et al. 2019. Ten simple rules for writing and sharing computational analyses in Jupyter Notebooks. (2019)."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274419"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173606"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00022"},{"key":"e_1_3_2_1_48_1","volume-title":"International Semantic Web Conference (P&D\/Industry\/BlueSky).","author":"Samuel Sheeba","year":"2018","unstructured":"Sheeba Samuel and Birgitta Konig-Ries. 2018. ProvBook: Provenance-based Semantic Enrichment of Interactive Notebooks for Reproducibility.. In International Semantic Web Conference (P&D\/Industry\/BlueSky)."},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1038\/515151ax"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/602421.602422"},{"key":"e_1_3_2_1_51_1","volume-title":"Jupyter for data science: Exploratory analysis, statistical modeling, machine learning, and data visualization with Jupyter","author":"Toomey Dan","unstructured":"Dan Toomey. 2017. Jupyter for data science: Exploratory analysis, statistical modeling, machine learning, and data visualization with Jupyter. Packt Publishing Ltd."},{"key":"e_1_3_2_1_52_1","volume-title":"Better Sharing: On the Need of Analyzing Jupyter Notebooks. In The 42nd International Conference on Software Engineering, NIER Track (ICSE","author":"Wang Jiawei","year":"2020","unstructured":"Jiawei Wang, Li Li, and Andreas Zeller. 2020. Better Code, Better Sharing: On the Need of Analyzing Jupyter Notebooks. In The 42nd International Conference on Software Engineering, NIER Track (ICSE 2020)."},{"key":"e_1_3_2_1_53_1","volume-title":"EDBT\/ICDT Workshops.","author":"Watson Alex","year":"2019","unstructured":"Alex Watson, Scott Bateman, and Suprio Ray. 2019. PySnippet: Accelerating Exploratory Data Analysis in Jupyter Notebook through Facilitated Access to Example Code.. In EDBT\/ICDT Workshops."},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1736020.1736039"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-017-0037-8"}],"event":{"name":"ASE '20: 35th IEEE\/ACM International Conference on Automated Software Engineering","location":"Virtual Event Australia","acronym":"ASE '20","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS"]},"container-title":["Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3324884.3416585","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3324884.3416585","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:38Z","timestamp":1750197698000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3324884.3416585"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12,21]]},"references-count":55,"alternative-id":["10.1145\/3324884.3416585","10.1145\/3324884"],"URL":"https:\/\/doi.org\/10.1145\/3324884.3416585","relation":{},"subject":[],"published":{"date-parts":[[2020,12,21]]},"assertion":[{"value":"2021-01-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}