{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T12:21:49Z","timestamp":1780575709412,"version":"3.54.1"},"reference-count":72,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,3,23]],"date-time":"2023-03-23T00:00:00Z","timestamp":1679529600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,3,23]],"date-time":"2023-03-23T00:00:00Z","timestamp":1679529600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"publisher","award":["200020\u02d9184994"],"award-info":[{"award-number":["200020\u02d9184994"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100006447","name":"University of Zurich","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100006447","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2023,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Data science is an exploratory and iterative process that often leads to complex and unstructured code. This code is usually poorly documented and, consequently, hard to understand by a third party. In this paper, we first collect empirical evidence for the non-linearity of data science code from real-world Jupyter notebooks, confirming the need for new approaches that aid in data science code interaction and comprehension. Second, we propose a visualisation method that elucidates implicit workflow information in data science code and assists data scientists in navigating the so-called<jats:italic>garden of forking paths<\/jats:italic>in non-linear code. The visualisation also provides information such as the rationale and the identification of the data science pipeline step based on cell annotations. We conducted a user experiment with data scientists to evaluate the proposed method, assessing the influence of (i) different workflow visualisations and (ii) cell annotations on code comprehension. Our results show that visualising the exploration helps the users obtain an overview of the notebook, significantly improving code comprehension. Furthermore, our qualitative analysis provides more insights into the difficulties faced during data science code comprehension.<\/jats:p>","DOI":"10.1007\/s10664-023-10289-9","type":"journal-article","created":{"date-parts":[[2023,3,23]],"date-time":"2023-03-23T09:03:13Z","timestamp":1679562193000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Visualising data science workflows to support third-party notebook comprehension: an empirical study"],"prefix":"10.1007","volume":"28","author":[{"given":"Dhivyabharathi","family":"Ramasamy","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cristina","family":"Sarasua","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alberto","family":"Bacchelli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abraham","family":"Bernstein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,3,23]]},"reference":[{"issue":"4","key":"10289_CR1","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1109\/2.488299","volume":"29","author":"T Ball","year":"1996","unstructured":"Ball T, Eick SG (1996) Software visualization in the large. Computer 29(4):33\u201343. https:\/\/doi.org\/10.1109\/2.488299","journal-title":"Computer"},{"issue":"6","key":"10289_CR2","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1080\/10447310802205776","volume":"24","author":"A Bangor","year":"2008","unstructured":"Bangor A, Kortum PT, Miller JT (2008) An empirical evaluation of the system usability scale. Intl J Hum\u2013Comput Interact 24(6):574\u2013594","journal-title":"Intl J Hum\u2013Comput Interact"},{"issue":"OOPSLA","key":"10289_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3360594","volume":"3","author":"R Bavishi","year":"2019","unstructured":"Bavishi R, Lemieux C, Fox R, Sen K, Stoica I (2019) AutoPandas: neural-backed generators for program synthesis. Proc ACM on Programm Lang 3(OOPSLA):1\u201327","journal-title":"Proc ACM on Programm Lang"},{"key":"10289_CR4","doi-asserted-by":"publisher","unstructured":"Begel A, Nagappan N (2008) Pair programming: What\u2019s in it for me?. In: Proceedings of the 2nd ACM-IEEE international symposium on empirical software engineering and measurement, ESEM \u201908. https:\/\/doi.org\/10.1145\/1414004.1414026. Association for Computing Machinery, New York, pp 120\u2013128","DOI":"10.1145\/1414004.1414026"},{"key":"10289_CR5","doi-asserted-by":"publisher","unstructured":"Brandt J, Guo PJ, Lewenstein J, Klemmer SR (2008) Opportunistic programming: How rapid ideation and prototyping occur in practice. In: Proceedings of the 4th international workshop on end-user software engineering, WEUSE \u201908. https:\/\/doi.org\/10.1145\/1370847.1370848. Association for Computing Machinery, New York, pp 1\u20135","DOI":"10.1145\/1370847.1370848"},{"key":"10289_CR6","unstructured":"Brooke J (1996) Sus: a \u201cquick and dirty\u2019usability. Usability evaluation in industry, p 189"},{"key":"10289_CR7","volume-title":"Statistical analysis and power for the behavioral sciences","author":"J Cohen","year":"1988","unstructured":"Cohen J (1988) Statistical analysis and power for the behavioral sciences, 2nd edn. Erinbaum, Hillsdale","edition":"2nd edn."},{"key":"10289_CR8","doi-asserted-by":"crossref","unstructured":"Collberg C, Kobourov S, Nagra J, Pitts J, Wampler K (2003) A system for graph-based visualization of the evolution of software. In: Proceedings of the 2003 ACM symposium on Software visualization, pp 77\u2013ff","DOI":"10.1145\/774833.774844"},{"issue":"2","key":"10289_CR9","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1147\/sj.282.0294","volume":"28","author":"TA Corbi","year":"1989","unstructured":"Corbi TA (1989) Program understanding: Challenge for the 1990\u2019s. IBM Syst J 28(2):294\u2013306. https:\/\/doi.org\/10.1147\/sj.282.0294","journal-title":"IBM Syst J"},{"key":"10289_CR10","doi-asserted-by":"crossref","unstructured":"Cornelissen B, Zaidman A, van Deursen A, van Rompaey B (2009) Trace visualization for program comprehension: A controlled experiment. In: 2009 IEEE 17th international conference on program comprehension, pp 100\u2013109","DOI":"10.1109\/ICPC.2009.5090033"},{"key":"10289_CR11","doi-asserted-by":"crossref","unstructured":"DeLine R, Czerwinski M, Robertson G (2005) Easing program comprehension by sharing navigation data. In: 2005 IEEE symposium on visual languages and human-centric computing (VL\/HCC\u201905). IEEE, pp 241\u2013248","DOI":"10.1109\/VLHCC.2005.32"},{"key":"10289_CR12","unstructured":"Dictionary OL (2020) Marg - Oxford learner\u2019s dictionary. https:\/\/www.oxfordlearnersdictionaries.com\/definition\/english\/marg?q=marg. Accessed 15 Sept 2020"},{"key":"10289_CR13","unstructured":"Fjelstad R, Hamlen W (1979) Application program maintenance study: report to our respondents. Proceedings of GUIDE 48"},{"key":"10289_CR14","doi-asserted-by":"crossref","unstructured":"Francese R, Risi M, Scanniello G, Tortora G (2017) Users\u2019 perception on the use of metricattitude to perform source code comprehension tasks: A focus group study. In: 2017 21st international conference information visualisation (IV). IEEE, pp 8\u201313","DOI":"10.1109\/iV.2017.26"},{"key":"10289_CR15","unstructured":"Gelman A, Loken E (2013) The garden of forking paths: Why multiple comparisons can be a problem even when there is no \u201cfishing expedition\u201d or \u201cp-hacking\u201d and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University"},{"issue":"1","key":"10289_CR16","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1109\/MIS.2010.9","volume":"26","author":"Y Gil","year":"2010","unstructured":"Gil Y, Ratnakar V, Kim J, Gonzalez-Calero P, Groth P, Moody J, Deelman E (2010) Wings: Intelligent workflow-based design of computational experiments. IEEE Intell Syst 26(1):62\u201372","journal-title":"IEEE Intell Syst"},{"key":"10289_CR17","doi-asserted-by":"crossref","unstructured":"Granger B, P\u00e9rez F (2021) Jupyter: Thinking and storytelling with code and data. Authorea Preprints","DOI":"10.22541\/au.161298309.98344404\/v1"},{"key":"10289_CR18","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1006\/jvlc.1996.0009","volume":"7","author":"T Green","year":"1996","unstructured":"Green T, Petre M (1996) Usability analysis of visual programming environments: A \u2018cognitive dimensions\u2019 framework. J Vis Lang Comput 7:131\u2013174","journal-title":"J Vis Lang Comput"},{"key":"10289_CR19","doi-asserted-by":"publisher","unstructured":"Head A, Hohman F, Barik T, Drucker SM, DeLine R (2019) Managing messes in computational notebooks. In: Proceedings of the 2019 CHI conference on human factors in computing systems, CHI \u201919. https:\/\/doi.org\/10.1145\/3290605.3300500. Association for Computing Machinery, New York, pp 1\u201312","DOI":"10.1145\/3290605.3300500"},{"key":"10289_CR20","doi-asserted-by":"crossref","unstructured":"Hill C, Bellamy R, Erickson T, Burnett M (2016) Trials and tribulations of developers of intelligent systems: A field study. In: 2016 IEEE symposium on visual languages and human-centric computing (VL\/HCC), pp 162\u2013170","DOI":"10.1109\/VLHCC.2016.7739680"},{"key":"10289_CR21","doi-asserted-by":"publisher","unstructured":"Hulkko H, Abrahamsson P (2005) A multiple case study on the impact of pair programming on product quality. In: Proceedings of the 27th international conference on software engineering, ICSE \u201905. https:\/\/doi.org\/10.1145\/1062455.1062545. Association for Computing Machinery, New York, pp 495\u2013504","DOI":"10.1145\/1062455.1062545"},{"key":"10289_CR22","unstructured":"Jupyter P (2015) Project Jupyter: Computational narratives as the engine of collaborative data science. https:\/\/blog.jupyter.org\/"},{"issue":"12","key":"10289_CR23","doi-asserted-by":"publisher","first-page":"2917","DOI":"10.1109\/TVCG.2012.219","volume":"18","author":"S Kandel","year":"2012","unstructured":"Kandel S, Paepcke A, Hellerstein JM, Heer J (2012) Enterprise data analysis and visualization: An interview study. IEEE Trans Vis Comput Graph 18 (12):2917\u20132926","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"10289_CR24","doi-asserted-by":"crossref","unstructured":"Kery MB, Myers BA (2018) Interactions for untangling messy history in a computational notebook. In: 2018 IEEE symposium on visual languages and human-centric computing (VL\/HCC), pp 147\u2013155","DOI":"10.1109\/VLHCC.2018.8506576"},{"key":"10289_CR25","doi-asserted-by":"publisher","unstructured":"Kery MB, Horvath A, Myers B (2017) Variolite: Supporting exploratory programming by data scientists. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI \u201917. https:\/\/doi.org\/10.1145\/3025453.3025626. ACM, New York, pp 1265\u20131276","DOI":"10.1145\/3025453.3025626"},{"key":"10289_CR26","doi-asserted-by":"publisher","unstructured":"Kery MB, Radensky M, Arya M, John BE, Myers BA (2018) The story in the notebook: Exploratory data science using a literate programming tool. In: Proceedings of the 2018 CHI conference on human factors in computing systems, CHI \u201918. https:\/\/doi.org\/10.1145\/3173574.3173748. Association for Computing Machinery, New York, pp 1\u201311","DOI":"10.1145\/3173574.3173748"},{"key":"10289_CR27","doi-asserted-by":"publisher","unstructured":"Kery MB, John BE, O\u2019Flaherty P, Horvath A, Myers BA (2019) Towards effective foraging by data scientists to find past analysis choices. In: Proceedings of the 2019 CHI conference on human factors in computing systems, CHI \u201919. https:\/\/doi.org\/10.1145\/3290605.3300322. Association for Computing Machinery, New York, pp 1\u201313","DOI":"10.1145\/3290605.3300322"},{"issue":"4","key":"10289_CR28","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1016\/j.scico.2009.10.007","volume":"75","author":"HM Kienle","year":"2010","unstructured":"Kienle HM, M\u00fcller HA (2010) Rigi-An environment for software reverse engineering, exploration, visualization, and redocumentation. Sci Comput Program 75(4):247\u2013263. https:\/\/doi.org\/10.1016\/j.scico.2009.10.007","journal-title":"Sci Comput Program"},{"issue":"11","key":"10289_CR29","doi-asserted-by":"publisher","first-page":"1024","DOI":"10.1109\/TSE.2017.2754374","volume":"44","author":"M Kim","year":"2017","unstructured":"Kim M, Zimmermann T, DeLine R, Begel A (2017) Data scientists in software teams: State of the art and challenges. IEEE Trans Softw Eng 44 (11):1024\u20131038","journal-title":"IEEE Trans Softw Eng"},{"issue":"12","key":"10289_CR30","doi-asserted-by":"publisher","first-page":"971","DOI":"10.1109\/TSE.2006.116","volume":"32","author":"AJ Ko","year":"2006","unstructured":"Ko AJ, Myers BA, Coblenz MJ, Aung H (2006) An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Trans Softw Eng 32(12):971\u2013987. https:\/\/doi.org\/10.1109\/TSE.2006.116","journal-title":"IEEE Trans Softw Eng"},{"key":"10289_CR31","unstructured":"Koop D, Patel J (2017) Dataflow notebooks: encoding and tracking dependencies of cells. In: 9th USENIX workshop on the theory and practice of provenance (TaPP)"},{"issue":"3","key":"10289_CR32","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1109\/MS.1986.233414","volume":"3","author":"S Letovsky","year":"1986","unstructured":"Letovsky S, Soloway E (1986) Delocalized plans and program comprehension. IEEE Softw 3(3):41","journal-title":"IEEE Softw"},{"issue":"4","key":"10289_CR33","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1016\/0164-1212(87)90033-1","volume":"7","author":"DC Littman","year":"1987","unstructured":"Littman DC, Pinto J, Letovsky S, Soloway E (1987) Mental models and software maintenance. J Syst Softw 7(4):341\u2013355. https:\/\/doi.org\/10.1016\/0164-1212(87)90033-1, http:\/\/www.sciencedirect.com\/science\/article\/pii\/0164121287900331","journal-title":"J Syst Softw"},{"issue":"1","key":"10289_CR34","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1109\/TVCG.2019.2934593","volume":"26","author":"J Liu","year":"2020","unstructured":"Liu J, Boukhelifa N, Eagan JR (2020a) Understanding the role of alternatives in data analysis practices. IEEE Trans Vis Comput Graph 26(1):66\u201376. https:\/\/doi.org\/10.1109\/TVCG.2019.2934593","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"10289_CR35","doi-asserted-by":"crossref","unstructured":"Liu Y, Althoff T, Heer J (2020b) Paths explored, paths omitted, paths obscured: Decision points & selective reporting in end-to-end data analysis. In: Proceedings of the 2020 CHI conference on human factors in computing systems, pp 1\u201314","DOI":"10.1145\/3313831.3376533"},{"key":"10289_CR36","doi-asserted-by":"crossref","unstructured":"Macke S, Gong H, Lee DJL, Head A, Xin D, Parameswaran A (2020) Fine-grained lineage for safer notebook interactions. arXiv:201206981","DOI":"10.14778\/3447689.3447712"},{"key":"10289_CR37","doi-asserted-by":"publisher","unstructured":"Merali Z (2010) Computational science: Error, why scientific programming does not compute. Nature https:\/\/doi.org\/10.1038\/467775a, https:\/\/www.nature.com\/articles\/467775a","DOI":"10.1038\/467775a"},{"key":"10289_CR38","doi-asserted-by":"crossref","unstructured":"Minelli R, Lanza M (2013) Visualizing the workflow of developers. In: 2013 First IEEE working conference on software visualization (VISSOFT), pp 1\u20134","DOI":"10.1109\/VISSOFT.2013.6650531"},{"key":"10289_CR39","doi-asserted-by":"crossref","unstructured":"Minelli R, Mocci A, Lanza M (2015) I know what you did last summer - an investigation of how developers spend their time. In: 2015 IEEE 23rd international conference on program comprehension, pp 25\u201335","DOI":"10.1109\/ICPC.2015.12"},{"key":"10289_CR40","doi-asserted-by":"crossref","unstructured":"Namaki MH, Floratou A, Psallidas F, Krishnan S, Agrawal A, Wu Y, Zhu Y, Weimer M (2020) Vamsa: Automated provenance tracking in data science scripts. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp 1542\u20131551","DOI":"10.1145\/3394486.3403205"},{"issue":"3","key":"10289_CR41","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1145\/272287.272333","volume":"41","author":"JT Nosek","year":"1998","unstructured":"Nosek JT (1998) The case for collaborative programming. Commun ACM 41(3):105\u2013108","journal-title":"Commun ACM"},{"key":"10289_CR42","doi-asserted-by":"crossref","unstructured":"Patel K, Fogarty J, Landay JA, Harrison B (2008) Investigating statistical machine learning as a tool for software development. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 667\u2013676","DOI":"10.1145\/1357054.1357160"},{"issue":"6","key":"10289_CR43","doi-asserted-by":"publisher","first-page":"9:1","DOI":"10.1147\/JRD.2017.2736278","volume":"61","author":"E Patterson","year":"2017","unstructured":"Patterson E, McBurney R, Schmidt H, Baldini I, Mojsilovi\u0107 A, Varshney KR (2017) Dataflow representation of data analyses: Toward a platform for collaborative data science. IBM J Res Dev 61(6):9:1\u20139:13. https:\/\/doi.org\/10.1147\/JRD.2017.2736278","journal-title":"IBM J Res Dev"},{"key":"10289_CR44","doi-asserted-by":"crossref","unstructured":"Pauw WD, Jensen E, Mitchell N, Sevitsky G, Vlissides JM, Yang J (2001) Visualizing the execution of java programs. In: Revised lectures on software visualization, International Seminar. Springer-Verlag, Berlin, pp 151\u2013162","DOI":"10.1007\/3-540-45875-1_12"},{"key":"10289_CR45","doi-asserted-by":"crossref","unstructured":"Perkel JM (2018) Why Jupyter is data scientists\u2019 computational notebook of choice. Nature. https:\/\/www.nature.com\/articles\/d41586-018-07196-1","DOI":"10.1038\/d41586-018-07196-1"},{"issue":"7772","key":"10289_CR46","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1038\/d41586-019-02619-z","volume":"573","author":"JM Perkel","year":"2019","unstructured":"Perkel JM (2019) Workflow systems turn raw data into scientific knowledge. Nature 573(7772):149\u2013151","journal-title":"Nature"},{"key":"10289_CR47","unstructured":"Pimentel JF, Braganholo V, Murta L, Freire J (2015) Collecting and analyzing provenance on interactive notebooks: When IPython meets noworkflow. In: TaPP"},{"issue":"3","key":"10289_CR48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3311955","volume":"52","author":"JF Pimentel","year":"2019","unstructured":"Pimentel JF, Freire J, Murta L, Braganholo V (2019) A survey on collecting, managing, and analyzing provenance from scripts. ACM Comput Surv (CSUR) 52(3):1\u201338","journal-title":"ACM Comput Surv (CSUR)"},{"key":"10289_CR49","doi-asserted-by":"crossref","unstructured":"Pimentel JF, Murta L, Braganholo V, Freire J (2019) A large-scale study about quality and reproducibility of Jupyter notebooks. In: 2019 IEEE\/ACM 16th international conference on mining software repositories (MSR), pp 507\u2013517","DOI":"10.1109\/MSR.2019.00077"},{"issue":"4","key":"10289_CR50","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10664-021-09961-9","volume":"26","author":"JF Pimentel","year":"2021","unstructured":"Pimentel JF, Murta L, Braganholo V, Freire J (2021) Understanding and improving the quality and reproducibility of Jupyter notebooks. Empir Softw Eng 26(4):1\u201355","journal-title":"Empir Softw Eng"},{"issue":"CSCW1","key":"10289_CR51","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3512934","volume":"6","author":"L Quaranta","year":"2022","unstructured":"Quaranta L, Calefato F, Lanubile F (2022) Eliciting best practices for collaboration with computational notebooks. Proc ACM Hum-Comput Interact 6(CSCW1):1\u201341","journal-title":"Proc ACM Hum-Comput Interact"},{"key":"10289_CR52","doi-asserted-by":"crossref","unstructured":"Rajlich V, Cowan GS (1997) Towards standard for experiments in program comprehension. In: Proceedings Fifth International Workshop on Program Comprehension. IWPC\u201997, pp 160\u2013161","DOI":"10.1109\/WPC.1997.601284"},{"key":"10289_CR53","doi-asserted-by":"crossref","unstructured":"Ramasamy D, Sarasua C, Bacchelli A, Bernstein A (2022) Workflow analysis of data science code in public Github repositories. To be published in EMSE","DOI":"10.1007\/s10664-022-10229-z"},{"key":"10289_CR54","doi-asserted-by":"crossref","unstructured":"Randles BM, Pasquetto IV, Golshan MS, Borgman CL (2017) Using the Jupyter notebook as a tool for open science: An empirical study. In: 2017 ACM\/IEEE joint conference on digital libraries (JCDL). IEEE, pp 1\u20132","DOI":"10.1109\/JCDL.2017.7991618"},{"key":"10289_CR55","unstructured":"Rule A, Birmingham A, Zuniga C, Altintas I, Huang S, Knight R, Moshiri N, Nguyen MH, Rosenthal SB, P\u00e9rez F, Rose PW (2018a) Ten simple rules for reproducible research in Jupyter notebooks. arXiv:1810.08055"},{"key":"10289_CR56","doi-asserted-by":"publisher","unstructured":"Rule A, Tabard A, Hollan JD (2018b) Exploration and explanation in computational notebooks. In: Proceedings of the 2018 CHI conference on human factors in computing systems, CHI \u201918. https:\/\/doi.org\/10.1145\/3173574.3173606. ACM, New York, pp 32:1\u201332:12","DOI":"10.1145\/3173574.3173606"},{"key":"10289_CR57","doi-asserted-by":"crossref","unstructured":"Rule A, Birmingham A, Zuniga C, Altintas I, Huang SC, Knight R, Moshiri N, Nguyen MH, Rosenthal SB, P\u00e9rez F et al (2019) Ten simple rules for writing and sharing computational analyses in Jupyter notebooks","DOI":"10.1371\/journal.pcbi.1007007"},{"key":"10289_CR58","volume-title":"The coding manual for qualitative researchers","author":"J Salda\u00f1a","year":"2015","unstructured":"Salda\u00f1a J (2015) The coding manual for qualitative researchers. Sage, Newbury Park"},{"key":"10289_CR59","unstructured":"Schweinsberg M, Feldman M, Staub N, van den Akker OR, van Aert RC, Van Assen MA, Liu Y, Althoff T, Heer J, Kale A et al (2021) Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organizational Behavior and Human Decision Processes"},{"key":"10289_CR60","doi-asserted-by":"publisher","unstructured":"Siegmund J (2016) Program comprehension: Past, present, and future. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER). https:\/\/doi.org\/10.1109\/SANER.2016.35, vol 5, pp 13\u201320","DOI":"10.1109\/SANER.2016.35"},{"issue":"3","key":"10289_CR61","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1177\/2515245917747646","volume":"1","author":"R Silberzahn","year":"2018","unstructured":"Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, Bahn\u00edk Bai F, Bannard C, Bonnier E, Carlsson R, Cheung F, Christensen G, Clay R, Craig MA, Rosa AD, Dam L, Evans MH, Cervantes IF, Fong N, Gamez-Djokic M, Glenz A, Gordon-McKeon S, Heaton TJ, Hederos K, Heene M, Mohr AJH, H\u00f6gden F, Hui K, Johannesson M, Kalodimos J, Kaszubowski E, Kennedy DM, Lei R, Lindsay TA, Liverani S, Madan CR, Molden D, Molleman E, Morey RD, Mulder LB, Nijstad BR, Pope NG, Pope B, Prenoveau JM, Rink F, Robusto E, Roderique H, Sandberg A, Schl\u00fcter E, Sch\u00f6nbrodt FD, Sherman MF, Sommer SA, Sotak K, Spain S, Sp\u00f6rlein C, Stafford T, Stefanutti L, Tauber S, Ullrich J, Vianello M, Wagenmakers EJ, Witkowiak M, Yoon S, Nosek BA (2018) Many analysts, one data set: Making transparent how variations in analytic choices affect results. Adv Methods Pract Psychol Sci 1(3):337\u2013356. https:\/\/doi.org\/10.1177\/2515245917747646","journal-title":"Adv Methods Pract Psychol Sci"},{"key":"10289_CR62","doi-asserted-by":"publisher","unstructured":"Srinivasa Ragavan S, Kuttal SK, Hill C, Sarma A, Piorkowski D, Burnett M (2016) Foraging among an overabundance of similar variants. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI \u201916. https:\/\/doi.org\/10.1145\/2858036.2858469. Association for Computing Machinery, New York, pp 3509\u20133521","DOI":"10.1145\/2858036.2858469"},{"key":"10289_CR63","doi-asserted-by":"publisher","first-page":"702","DOI":"10.1177\/1745691616658637","volume":"11","author":"S Steegen","year":"2016","unstructured":"Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W (2016) Increasing transparency through a multiverse analysis. Perspect Psychol Sci 11:702\u2013712. https:\/\/doi.org\/10.1177\/1745691616658637","journal-title":"Perspect Psychol Sci"},{"key":"10289_CR64","unstructured":"Storey MD, Fracchia FD, Muller HA (1997a) Cognitive design elements to support the construction of a mental model during software visualization. In: Proceedings 5th international workshop on program comprehension. IWPC\u201997, pp 17\u201328"},{"key":"10289_CR65","doi-asserted-by":"crossref","unstructured":"Storey MD, Wong K, Fracchia FD, Muller HA (1997b) On integrating visualization techniques for effective software exploration. In: Proceedings of VIZ \u201997: Visualization conference, information visualization symposium and parallel rendering symposium, pp 38\u201345","DOI":"10.1109\/INFVIS.1997.636784"},{"key":"10289_CR66","unstructured":"Storey MAD (1998) A cognitive framework for describing and evaluating software exploration tools. PhD thesis, Simon Fraser University, CAN, aAINQ37756"},{"key":"10289_CR67","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1002\/spe.386","volume":"31","author":"T Syst\u00e4","year":"2001","unstructured":"Syst\u00e4 T, Koskimies K, M\u00fcller H (2001) Shimba-an environment for reverse engineering Java software systems. Softw, Pract Exper 31:371\u2013394. https:\/\/doi.org\/10.1002\/spe.386","journal-title":"Softw, Pract Exper"},{"issue":"8","key":"10289_CR68","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1145\/208344.208348","volume":"38","author":"M Th\u00fcring","year":"1995","unstructured":"Th\u00fcring M, Hannemann J, Haake JM (1995) Hypermedia and cognition: Designing for comprehension. Commun ACM 38(8):57\u201366. https:\/\/doi.org\/10.1145\/208344.208348","journal-title":"Commun ACM"},{"key":"10289_CR69","doi-asserted-by":"publisher","first-page":"1332","DOI":"10.3389\/fpsyg.2017.01332","volume":"8","author":"J Wacker","year":"2017","unstructured":"Wacker J (2017) Increasing the reproducibility of science through close cooperation and forking path analysis. Front Psychol 8:1332. https:\/\/doi.org\/10.3389\/fpsyg.2017.01332","journal-title":"Front Psychol"},{"issue":"CSCW","key":"10289_CR70","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3359313","volume":"3","author":"D Wang","year":"2019","unstructured":"Wang D, Weisz JD, Muller M, Ram P, Geyer W, Dugan C, Tausczik Y, Samulowitz H, Gray A (2019) Human-ai collaboration in data science. Proc ACM Hum-Comput Interact 3 (CSCW):1\u201324. https:\/\/doi.org\/10.1145\/3359313","journal-title":"Proc ACM Hum-Comput Interact"},{"issue":"1","key":"10289_CR71","doi-asserted-by":"publisher","first-page":"e1001745","DOI":"10.1371\/journal.pbio.1001745","volume":"12","author":"G Wilson","year":"2014","unstructured":"Wilson G, Aruliah DA, Brown CT, Hong NPC, Davis M, Guy RT, Haddock SH, Huff KD, Mitchell IM, Plumbley MD et al (2014) Best practices for scientific computing. PLoS Biol 12(1):e1001745","journal-title":"PLoS Biol"},{"key":"10289_CR72","doi-asserted-by":"publisher","unstructured":"Ye D, Xing Z, Foo CY, Ang ZQ, Li J, Kapre N (2016) Software-specific named entity recognition in software engineering social content. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), vol 1, pp 90\u2013101, DOI https:\/\/doi.org\/10.1109\/SANER.2016.10","DOI":"10.1109\/SANER.2016.10"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10289-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-023-10289-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10289-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,16]],"date-time":"2024-10-16T22:24:25Z","timestamp":1729117465000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-023-10289-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,23]]},"references-count":72,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,5]]}},"alternative-id":["10289"],"URL":"https:\/\/doi.org\/10.1007\/s10664-023-10289-9","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,23]]},"assertion":[{"value":"5 January 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 March 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"58"}}