{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,16]],"date-time":"2025-09-16T18:51:36Z","timestamp":1758048696467,"version":"3.44.0"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2025,7,4]],"date-time":"2025-07-04T00:00:00Z","timestamp":1751587200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,4]],"date-time":"2025-07-04T00:00:00Z","timestamp":1751587200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001711","name":"Schweizerischer Nationalfonds zur F\u00f6rderung der Wissenschaftlichen Forschung","doi-asserted-by":"publisher","award":["184994","205975"],"award-info":[{"award-number":["184994","205975"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Despite the popularity of AI assistants for coding activities, there is limited empirical work on whether these coding assistants can help users complete data science tasks. Moreover, in data science programming, exploring alternative paths has been widely advocated, as such paths may lead to diverse understandings and conclusions (Gelman and Loken 2013; Kale et\u00a0al. 2019). Whether existing AI-based coding assistants can support data scientists in exploring the relevant alternative paths remains unexplored. To fill this gap, we conducted a mixed-methods study to understand how data scientists solved different data science tasks with the help of an AI-based coding assistant that provides explicit alternatives as recommendations throughout the data science workflow. Specifically, we quantitatively investigated whether the users accept the code recommendations, including alternative recommendations, by the AI assistant and whether the recommendations are helpful when completing descriptive and predictive data science tasks. Through the empirical study, we also investigated if including information about the data science step (e.g., data exploration) they seek recommendations for in a prompt leads to helpful recommendations. In our study, we found that including the data science step in a prompt had a statistically significant improvement in the acceptance of recommendations, whereas the presence of alternatives did not lead to any significant differences. Our study also shows a statistically significant difference in the acceptance and usefulness of recommendations between descriptive and predictive tasks. Participants generally had positive sentiments regarding AI assistance and our proposed interface. We share further insights on the interactions that emerged during the study and the challenges that our users encountered while solving their data science tasks.<\/jats:p>","DOI":"10.1007\/s10664-025-10622-4","type":"journal-article","created":{"date-parts":[[2025,7,4]],"date-time":"2025-07-04T04:21:10Z","timestamp":1751602870000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["AI support for data scientists: An empirical study on workflow and alternative code recommendations"],"prefix":"10.1007","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-9056-6907","authenticated-orcid":false,"given":"Dhivyabharathi","family":"Ramasamy","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2076-9584","authenticated-orcid":false,"given":"Cristina","family":"Sarasua","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0128-4602","authenticated-orcid":false,"given":"Abraham","family":"Bernstein","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,7,4]]},"reference":[{"key":"10622_CR1","unstructured":"Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, Almeida D, Altenschmidt J, Altman S, Anadkat S et\u00a0al (2023) Gpt-4 technical report. arXiv:2303.08774"},{"key":"10622_CR2","doi-asserted-by":"publisher","unstructured":"Amershi S, Weld D, Vorvoreanu M, Fourney A, Nushi B, Collisson P, Suh J, Iqbal S, Bennett PN, Inkpen K, Teevan J, Kikin-Gil R, Horvitz E (2019) Guidelines for human-ai interaction. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI \u201919, pp 1\u201313. https:\/\/doi.org\/10.1145\/3290605.3300233","DOI":"10.1145\/3290605.3300233"},{"key":"10622_CR3","doi-asserted-by":"publisher","unstructured":"Bar\u00a0El O, Milo T, Somech A (2019) Atena: an autonomous system for data exploration based on deep reinforcement learning. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM \u201919, pp 2873\u20132876. https:\/\/doi.org\/10.1145\/3357384.3357845","DOI":"10.1145\/3357384.3357845"},{"issue":"OOPSLA1","key":"10622_CR4","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1145\/3586030","volume":"7","author":"S Barke","year":"2023","unstructured":"Barke S, James MB, Polikarpova N (2023) Grounded copilot: how programmers interact with code-generating models. Proc ACM Program Lang 7(OOPSLA1):85\u2013111","journal-title":"Proc ACM Program Lang"},{"issue":"3","key":"10622_CR5","first-page":"189","volume":"189","author":"J Brooke","year":"1996","unstructured":"Brooke J (1996) Sus: a quick and dirty\u2019usability. Usability Eval Ind 189(3):189\u2013194","journal-title":"Usability Eval Ind"},{"issue":"240","key":"10622_CR6","first-page":"1","volume":"24","author":"A Chowdhery","year":"2023","unstructured":"Chowdhery A, Narang S, Devlin J, Bosma M, Mishra G, Roberts A, Barham P, Chung HW, Sutton C, Gehrmann S et al (2023) Palm: scaling language modeling with pathways. J Mach Learn Res 24(240):1\u2013113","journal-title":"J Mach Learn Res"},{"key":"10622_CR7","doi-asserted-by":"crossref","unstructured":"Cohen J (2013) Statistical power analysis for the behavioral sciences. Academic Press","DOI":"10.4324\/9780203771587"},{"key":"10622_CR8","doi-asserted-by":"crossref","unstructured":"Coignion T, Quinton C, Rouvoy R (2024) A performance study of llm-generated code on leetcode. In: Proceedings of the 28th international conference on evaluation and assessment in software engineering, pp 79\u201389","DOI":"10.1145\/3661167.3661221"},{"key":"10622_CR9","unstructured":"cop (2023) Github copilot. https:\/\/github.com\/features\/copilot"},{"key":"10622_CR10","doi-asserted-by":"publisher","unstructured":"Dragicevic P, Jansen Y, Sarma A, Kay M, Chevalier F (2019) Increasing the transparency of research papers with explorable multiverse analyses. In: Proceedings of the 2019 CHI conference on human factors in computing systems. https:\/\/doi.org\/10.1145\/3290605.3300295","DOI":"10.1145\/3290605.3300295"},{"key":"10622_CR11","doi-asserted-by":"crossref","unstructured":"Drosos I, Barik T, Guo PJ, DeLine R, Gulwani S (2020) Wrex: a unified programming-by-example interaction for synthesizing readable code for data scientists. In: Proceedings of the 2020 CHI conference on human factors in computing systems, pp 1\u201312","DOI":"10.1145\/3313831.3376442"},{"key":"10622_CR12","unstructured":"Gelman A, Loken E (2013) The garden of forking paths: why multiple comparisons can be a problem, even when there is no \u201cfishing expedition\u201d or \u201cp-hacking\u201d and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, vol 348, pp 1\u201317"},{"key":"10622_CR13","doi-asserted-by":"publisher","first-page":"64323","DOI":"10.1109\/access.2019.2917620","volume":"7","author":"Z Gong","year":"2019","unstructured":"Gong Z, Zhong P, Hu W (2019) Diversity in machine learning. IEEE Access 7:64323\u201364350. https:\/\/doi.org\/10.1109\/access.2019.2917620","journal-title":"IEEE Access"},{"key":"10622_CR14","unstructured":"hex (2023) Hex blog: introducing: \u201cno-code\u201d cells. https:\/\/hex.tech\/blog\/introducing-no-code-cells"},{"key":"10622_CR15","unstructured":"Jupyter (2015) Project jupyter: computational narratives as the engine of collaborative data science. https:\/\/blog.jupyter.org\/"},{"key":"10622_CR16","doi-asserted-by":"publisher","unstructured":"Kale A, Kay M, Hullman J (2019) Decision-making under uncertainty in research synthesis: designing for the garden of forking paths. In: Proceedings of the 2019 CHI conference on human factors in computing systems, CHI \u201919, pp 1\u201314. https:\/\/doi.org\/10.1145\/3290605.3300432","DOI":"10.1145\/3290605.3300432"},{"key":"10622_CR17","doi-asserted-by":"crossref","unstructured":"Kazemitabaar M, Hou X, Henley A, Ericson BJ, Weintrop D, Grossman T (2023) How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment. In: Proceedings of the 23rd Koli calling international conference on computing education research, pp 1\u201312","DOI":"10.1145\/3631802.3631806"},{"key":"10622_CR18","doi-asserted-by":"crossref","unstructured":"Kazemitabaar M, Ye R, Wang X, Henley AZ, Denny P, Craig M, Grossman T (2024) Codeaid: evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. In: Proceedings of the CHI conference on human factors in computing systems, pp 1\u201320","DOI":"10.1145\/3613904.3642773"},{"key":"10622_CR19","doi-asserted-by":"publisher","unstructured":"Kery MB, Horvath A, Myers B (2017) Variolite: supporting exploratory programming by data scientists. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI \u201917, pp 1265\u20131276. https:\/\/doi.org\/10.1145\/3025453.3025626","DOI":"10.1145\/3025453.3025626"},{"key":"10622_CR20","doi-asserted-by":"crossref","unstructured":"Kery MB, Ren D, Hohman F, Moritz D, Wongsuphasawat K, Patel K (2020) mage: fluid moves between code and graphical work in computational notebooks. In: Proceedings of the 33rd annual ACM symposium on user interface software and technology, pp 140\u2013151","DOI":"10.1145\/3379337.3415842"},{"key":"10622_CR21","unstructured":"Kietz JU, Serban F, Bernstein A, Fischer S, Vanschoren J, Brazdil P (2012) Designing kdd-workflows via htn-planning for intelligent discovery assistance"},{"issue":"11","key":"10622_CR22","doi-asserted-by":"publisher","first-page":"1024","DOI":"10.1109\/TSE.2017.2754374","volume":"44","author":"M Kim","year":"2017","unstructured":"Kim M, Zimmermann T, DeLine R, Begel A (2017) Data scientists in software teams: state of the art and challenges. IEEE Trans Softw Eng 44(11):1024\u20131038","journal-title":"IEEE Trans Softw Eng"},{"issue":"6624","key":"10622_CR23","doi-asserted-by":"publisher","first-page":"1092","DOI":"10.1126\/science.abq1158","volume":"378","author":"Y Li","year":"2022","unstructured":"Li Y, Choi D, Chung J, Kushman N, Schrittwieser J, Leblond R, Eccles T, Keeling J, Gimeno F, Dal Lago A et al (2022) Competition-level code generation with alphacode. Science 378(6624):1092\u20131097","journal-title":"Science"},{"key":"10622_CR24","doi-asserted-by":"crossref","unstructured":"Liang JT, Yang C, Myers BA (2023) A large-scale survey on the usability of ai programming assistants: successes and challenges. In: 2024 IEEE\/ACM 46th International Conference on Software Engineering (ICSE), IEEE Computer Society, pp 605\u2013617","DOI":"10.1145\/3597503.3608128"},{"issue":"2","key":"10622_CR25","doi-asserted-by":"publisher","first-page":"1753","DOI":"10.1109\/tvcg.2020.3028985","volume":"27","author":"Y Liu","year":"2021","unstructured":"Liu Y, Kale A, Althoff T, Heer J (2021) Boba: authoring and visualizing multiverse analyses. IEEE Trans Visual Comput Graph 27(2):1753\u20131763. https:\/\/doi.org\/10.1109\/tvcg.2020.3028985","journal-title":"IEEE Trans Visual Comput Graph"},{"key":"10622_CR26","unstructured":"Liu J, Xia CS, Wang Y, Zhang L (2024) Is your code generated by chatgpt really correct? Rigorous evaluation of large language models for code generation. Advances in neural information processing systems, vol 36"},{"key":"10622_CR27","doi-asserted-by":"crossref","unstructured":"McNutt AM, Wang C, Deline RA, Drucker SM (2023) On the design of ai-powered code assistants for notebooks. In: Proceedings of the 2023 CHI conference on human factors in computing systems, pp 1\u201316","DOI":"10.1145\/3544548.3580940"},{"key":"10622_CR28","doi-asserted-by":"publisher","unstructured":"Merrill MA, Zhang G, Althoff T (2021) Multiverse: mining collective data science knowledge from code on the web to suggest alternative analysis approaches. Proceedings of the 27th ACM SIGKDD conference on knowledge discovery and data mining, pp 1212\u20131222. https:\/\/doi.org\/10.1145\/3447548.3467455","DOI":"10.1145\/3447548.3467455"},{"key":"10622_CR29","doi-asserted-by":"crossref","unstructured":"Nahar N, Zhou S, Lewis G, K\u00e4stner C (2022) Collaboration challenges in building ml-enabled systems: communication, documentation, engineering, and process. In: Proceedings of the 44th international conference on software engineering, pp 413\u2013425","DOI":"10.1145\/3510003.3510209"},{"key":"10622_CR30","unstructured":"Nam D, Macvean A, Hellendoorn V, Vasilescu B, Myers B (2023) In-ide generation-based information support with a large language model. arXiv:2307.08177"},{"key":"10622_CR31","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1613\/jair.4377","volume":"51","author":"P Nguyen","year":"2014","unstructured":"Nguyen P, Hilario M, Kalousis A (2014) Using meta-mining to support data mining workflow planning and optimization. J Artif Intell Res 51:605\u2013644","journal-title":"J Artif Intell Res"},{"key":"10622_CR32","unstructured":"Olson RS, Moore JH (2016) Tpot: a tree-based pipeline optimization tool for automating machine learning. In: Workshop on automatic machine learning, PMLR, pp 66\u201374"},{"issue":"1","key":"10622_CR33","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1007\/s10664-022-10229-z","volume":"28","author":"D Ramasamy","year":"2022","unstructured":"Ramasamy D, Sarasua C, Bacchelli A, Bernstein A (2022) Workflow analysis of data science code in public Github repositories. Empir Softw Eng 28(1):7. https:\/\/doi.org\/10.1007\/s10664-022-10229-z","journal-title":"Empir Softw Eng"},{"key":"10622_CR34","doi-asserted-by":"publisher","unstructured":"Ramasamy D, Sarasua C, Bacchelli A, Bernstein A (2023) Visualising data science workflows to support third-party notebook comprehension: an empirical study. Empirical Softw Eng 28(3):1573\u20137616. https:\/\/doi.org\/10.1007\/s10664-023-10289-9","DOI":"10.1007\/s10664-023-10289-9"},{"key":"10622_CR35","doi-asserted-by":"publisher","unstructured":"Ross SI, Martinez F, Houde S, Muller M, Weisz JD (2023) The programmer\u2019s assistant: conversational interaction with a large language model for software development. In: Proceedings of the 28th international conference on intelligent user interfaces. Association for Computing Machinery, New York, NY, USA, IUI \u201923, pp 491\u2013514. https:\/\/doi.org\/10.1145\/3581641.3584037","DOI":"10.1145\/3581641.3584037"},{"key":"10622_CR36","unstructured":"Roziere B, Gehring J, Gloeckle F, Sootla S, Gat I, Tan XE, Adi Y, Liu J, Remez T, Rapin J et\u00a0al (2023) Code llama: open foundation models for code. arXiv:2308.12950"},{"key":"10622_CR37","doi-asserted-by":"publisher","unstructured":"Rule A, Tabard A, Hollan JD (2018) Exploration and explanation in computational notebooks. In: Proceedings of the 2018 CHI conference on human factors in computing systems, CHI \u201918, pp 1\u201312. https:\/\/doi.org\/10.1145\/3173574.3173606","DOI":"10.1145\/3173574.3173606"},{"key":"10622_CR38","unstructured":"Simonsohn U, Simmons JP, Nelson LD (2019) Specification curve: descriptive and inferential statistics on all reasonable specifications. Available at SSRN 2694998"},{"key":"10622_CR39","doi-asserted-by":"publisher","first-page":"702","DOI":"10.1177\/1745691616658637","volume":"11","author":"S Steegen","year":"2016","unstructured":"Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W (2016) Increasing transparency through a multiverse analysis. Perspect Psychol Sci 11:702\u2013712. https:\/\/doi.org\/10.1177\/1745691616658637","journal-title":"Perspect Psychol Sci"},{"key":"10622_CR40","unstructured":"tab (2023) Tabnine. https:\/\/www.tabnine.com\/"},{"key":"10622_CR41","unstructured":"Tukey JW et al (1977) Exploratory data analysis, vol 2. Reading, MA"},{"key":"10622_CR42","unstructured":"Tunstall L, Von\u00a0Werra L, Wolf T (2022) Natural language processing with transformers. O\u2019Reilly Media, Inc"},{"key":"10622_CR43","doi-asserted-by":"crossref","unstructured":"Vogelsang A, Borg M (2019) Requirements engineering for machine learning: perspectives from data scientists. In: 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW). IEEE, pp 245\u2013251","DOI":"10.1109\/REW.2019.00050"},{"key":"10622_CR44","doi-asserted-by":"crossref","unstructured":"Wang D, Weisz JD, Muller M, Ram P, Geyer W, Dugan C, Tausczik Y, Samulowitz H, Gray A (2019) Human-ai collaboration in data science: Exploring data scientists\u2019 perceptions of automated ai. Proceedings of the ACM on human-computer interaction 3(CSCW), pp 1\u201324","DOI":"10.1145\/3359313"},{"key":"10622_CR45","doi-asserted-by":"publisher","unstructured":"Xu FF, Alon U, Neubig G, Hellendoorn VJ (2022) A systematic evaluation of large language models of code. In: Proceedings of the 6th ACM SIGPLAN international symposium on machine programming. Association for Computing Machinery, New York, NY, USA, MAPS 2022, pp 1\u201310. https:\/\/doi.org\/10.1145\/3520312.3534862","DOI":"10.1145\/3520312.3534862"},{"issue":"1","key":"10622_CR46","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/0049124115610347","volume":"46","author":"C Young","year":"2017","unstructured":"Young C, Holsteen K (2017) Model uncertainty and robustness: a computational framework for multimodel analysis. Sociol Methods Res 46(1):3\u201340. https:\/\/doi.org\/10.1177\/0049124115610347","journal-title":"Sociol Methods Res"},{"key":"10622_CR47","doi-asserted-by":"crossref","unstructured":"Zamfirescu-Pereira J, Wong RY, Hartmann B, Yang Q (2023) Why johnny can\u2019t prompt: how non-ai experts try (and fail) to design llm prompts. In: Proceedings of the 2023 CHI conference on human factors in computing systems, pp 1\u201321","DOI":"10.1145\/3544548.3581388"},{"key":"10622_CR48","doi-asserted-by":"crossref","unstructured":"Zhang A, Muller M, Wang D (2020) How do data science workers collaborate. Roles, workflows, and tools ACM","DOI":"10.1145\/3392826"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-025-10622-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-025-10622-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-025-10622-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,13]],"date-time":"2025-09-13T08:54:11Z","timestamp":1757753651000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-025-10622-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,4]]},"references-count":48,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["10622"],"URL":"https:\/\/doi.org\/10.1007\/s10664-025-10622-4","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"type":"print","value":"1382-3256"},{"type":"electronic","value":"1573-7616"}],"subject":[],"published":{"date-parts":[[2025,7,4]]},"assertion":[{"value":"23 January 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 July 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors obtained approvals from the relevant institutional ethics committee to conduct the experiments. Additionally, explicit consent has been obtained from all participants for the use and publication of anonymised results.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Responsibilities"}},{"value":"The authors declare they have no financial or non-financial interests. The authors declare no conflict of interest with the suggested reviewers for this scientific article.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"133"}}