{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T11:31:49Z","timestamp":1750764709829,"version":"3.41.0"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,2,7]],"date-time":"2023-02-07T00:00:00Z","timestamp":1675728000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"ARC Discovery Project","award":["DP190102141"],"award-info":[{"award-number":["DP190102141"]}]},{"name":"ARC Training Centre for Information Resilience","award":["IC200100022"],"award-info":[{"award-number":["IC200100022"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>\n            Understanding how data workers interact with data, and various pieces of information related to data preparation, is key to designing systems that can better support them in exploring datasets. To date, however, there is a paucity of research studying the strategies adopted by data workers as they carry out data preparation activities. In this work, we investigate a specific data preparation activity, namely\n            <jats:italic>data quality discovery<\/jats:italic>\n            , and aim to (i)\u00a0understand the behaviors of data workers in discovering data quality issues, (ii)\u00a0explore what factors (e.g., prior experience) can affect their behaviors, as well as (iii)\u00a0understand how these behavioral observations relate to their performance. To this end, we collect a multi-modal dataset through a data-driven experiment that relies on the use of eye-tracking technology with a purpose-designed platform built on top of iPython Notebook. The experiment results reveal that: (i)\u00a0\u2018copy\u2013paste\u2013modify\u2019 is a typical strategy for writing code to complete tasks; (ii)\u00a0proficiency in writing code has a significant impact on the quality of task performance, while perceived difficulty and efficacy can influence task completion patterns; and (iii)\u00a0searching in external resources is a prevalent action that can be leveraged to achieve better performance. Furthermore, our experiment indicates that providing sample code within the system can help data workers get started with their task, and surfacing underlying data is an effective way to support exploration. By investigating data worker behaviors prior to each search action, we also find that the most common reasons that trigger external search actions are the need to seek assistance in writing or debugging code and to search for relevant code to reuse. Based on our experiment results, we showcase a systematic approach to select from the top best code snippets created by data workers and assemble them to achieve better performance than the best individual performer in the dataset. By doing so, our findings not only provide insights into patterns of interactions with various system components and information resources when performing data curation tasks, but also build effective and efficient data curation processes through data workers\u2019 collective intelligence.\n          <\/jats:p>","DOI":"10.1145\/3567419","type":"journal-article","created":{"date-parts":[[2022,10,7]],"date-time":"2022-10-07T13:16:03Z","timestamp":1665148563000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["A Data-Driven Analysis of Behaviors in Data Curation Processes"],"prefix":"10.1145","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7777-3592","authenticated-orcid":false,"given":"Lei","family":"Han","sequence":"first","affiliation":[{"name":"The University of Queensland, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5135-0313","authenticated-orcid":false,"given":"Tianwa","family":"Chen","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7311-3693","authenticated-orcid":false,"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2156-4097","authenticated-orcid":false,"given":"Marta","family":"Indulska","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6739-4145","authenticated-orcid":false,"given":"Shazia","family":"Sadiq","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,2,7]]},"reference":[{"key":"e_1_3_3_2_2","first-page":"99","volume-title":"Proceedings of the 12th Working Conference on MSR","author":"Ahmed Tarek M.","year":"2015","unstructured":"Tarek M. Ahmed, Weiyi Shang, and Ahmed E. Hassan. 2015. An empirical study of the copy and paste behavior during development. In Proceedings of the 12th Working Conference on MSR. IEEE Press, 99\u2013110."},{"key":"e_1_3_3_3_2","first-page":"51","volume-title":"Proceedings of WWW","author":"Aula Anne","year":"2010","unstructured":"Anne Aula, Rehan M. Khan, Zhiwei Guan, Paul Fontes, and Peter Hong. 2010. A comparison of visual and textual page previews in judging the helpfulness of web pages. In Proceedings of WWW. ACM, 51\u201360."},{"key":"e_1_3_3_4_2","first-page":"1","volume-title":"Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics","author":"Azuan Nurzety A.","year":"2017","unstructured":"Nurzety A. Azuan, Suzanne M. Embury, and Norman W. Paton. 2017. Observing the data scientist: Using manual corrections as implicit feedback. In Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. 1\u20136."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1736359"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3295750.3298926"},{"key":"e_1_3_3_7_2","first-page":"21","volume-title":"Proceedings of CHI","author":"Buscher Georg","year":"2009","unstructured":"Georg Buscher, Edward Cutrell, and Meredith Ringel Morris. 2009. What do you see when you\u2019re surfing? Using eye tracking to predict salient regions of web pages. In Proceedings of CHI. ACM, 21\u201330."},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.41.1.113"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376729"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.14778\/3115404.3115414"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.5555\/861869"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376442"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-21569-3_6"},{"key":"e_1_3_3_14_2","first-page":"6475","volume-title":"Proceedings of CHI","author":"Grusky Max","year":"2017","unstructured":"Max Grusky, Jeiran Jahani, Josh Schwartz, Dan Valente, Yoav Artzi, and Mor Naaman. 2017. Modeling sub-document attention using viewport time. In Proceedings of CHI. ACM, 6475\u20136480."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcss.2011.10.009"},{"key":"e_1_3_3_16_2","first-page":"65","volume-title":"Proceedings of UIST","author":"Guo Philip J.","year":"2011","unstructured":"Philip J. Guo, Sean Kandel, Joseph M. Hellerstein, and Jeffrey Heer. 2011. Proactive wrangling: Mixed-initiative end-user programming of data transformation scripts. In Proceedings of UIST. 65\u201374."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401059"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1177\/154193120605000909"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376798"},{"key":"e_1_3_3_20_2","first-page":"1225","volume-title":"Proceedings of CHI","author":"Huang Jeff","year":"2011","unstructured":"Jeff Huang, Ryen W. White, and Susan Dumais. 2011. No clicks, no problem: Using cursor movements to understand and improve search. In Proceedings of CHI. ACM, 1225\u20131234."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979444"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23187"},{"key":"e_1_3_3_23_2","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1109\/ISESE.2004.1334896","volume-title":"Proceedings of the 2004 International Symposium on Empirical Software Engineering, 2004. ISESE\u201904.","author":"Kim Miryung","year":"2004","unstructured":"Miryung Kim, Lawrence Bergman, Tessa Lau, and David Notkin. 2004. An ethnographic study of copy and paste programming practices in OOPL. In Proceedings of the 2004 International Symposium on Empirical Software Engineering, 2004. ISESE\u201904. IEEE, 83\u201392."},{"key":"e_1_3_3_24_2","doi-asserted-by":"crossref","unstructured":"Paul A. Kirschner. 2002. Cognitive Load Theory: Implications of cognitive load theory on the design of learning.","DOI":"10.1016\/S0959-4752(01)00014-7"},{"issue":"12","key":"e_1_3_3_25_2","first-page":"971","article-title":"An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks","author":"Ko Andrew J.","year":"2006","unstructured":"Andrew J. Ko, Brad A. Myers, Michael J. Coblenz, and Htet Htet Aung. 2006. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Transactions on Software Engineering12 (2006), 971\u2013987.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2020.102562"},{"key":"e_1_3_3_27_2","first-page":"1","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","author":"Koesten Laura","year":"2019","unstructured":"Laura Koesten, Emilia Kacprzak, Jeni Tennison, and Elena Simperl. 2019. Collaborative practices with structured data: Do tools support what users need? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1\u201314."},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1145\/2835776.2835833","volume-title":"Proceedings of WSDM","author":"Lagun Dmitry","year":"2016","unstructured":"Dmitry Lagun and Mounia Lalmas. 2016. Understanding user attention and engagement in online news reading. In Proceedings of WSDM. ACM, 113\u2013122."},{"key":"e_1_3_3_29_2","first-page":"1293","volume-title":"Proceedings of SIGIR","author":"Li Jing","year":"2018","unstructured":"Jing Li, Aixin Sun, Zhenchang Xing, and Lei Han. 2018. API caveat explorer\u2013surfacing negative usages from practice: An API-oriented interactive exploratory search system for programmers. In Proceedings of SIGIR. 1293\u20131296."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786871"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3295750.3298922"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730491"},{"key":"e_1_3_3_33_2","first-page":"737","volume-title":"Proceedings of CIKM","author":"Mehrotra R.","year":"2017","unstructured":"R. Mehrotra, A. H. Awadallah, M. Shokouhi, E. Yilmaz, I. Zitouni, A. El Kholy, and M. Khabsa. 2017. Deep sequential models for task satisfaction prediction. In Proceedings of CIKM. ACM, 737\u2013746."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2015.12"},{"key":"e_1_3_3_35_2","first-page":"126","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems","author":"Muller Michael","year":"2019","unstructured":"Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q. Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How data science workers work with data: Discovery, capture, curation, design, creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 126."},{"key":"e_1_3_3_36_2","doi-asserted-by":"crossref","first-page":"630","DOI":"10.1109\/ASE.2015.39","article-title":"Copy and paste redeemed (T)","author":"Narasimhan Krishna","year":"2015","unstructured":"Krishna Narasimhan and Christoph Reichenbach. 2015. Copy and paste redeemed (T). 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE) (2015), 630\u2013640.","journal-title":"2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE)"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/1083142.1083143"},{"key":"e_1_3_3_38_2","first-page":"3063","volume-title":"Proceedings of CHI","author":"Piorkowski David J.","year":"2013","unstructured":"David J. Piorkowski, Scott D. Fleming, Irwin Kwan, Margaret M. Burnett, Christopher Scaffidi, Rachel K. E. Bellamy, and Joshua Jordahl. 2013. The whats and hows of programmers\u2019 foraging diets. In Proceedings of CHI. ACM, 3063\u20133072."},{"key":"e_1_3_3_39_2","first-page":"618","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems","author":"Potluri Venkatesh","year":"2018","unstructured":"Venkatesh Potluri, Priyan Vaithilingam, Suresh Iyengar, Y. Vidya, Manohar Swaminathan, and Gopal Srinivasa. 2018. CodeTalk: Improving programming environment accessibility for visually impaired developers. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 618."},{"key":"e_1_3_3_40_2","first-page":"5","volume-title":"Proceedings of the Fourth International Workshop on Testing Database Systems","author":"Rabl Tilmann","year":"2011","unstructured":"Tilmann Rabl and Meikel Poess. 2011. Parallel data generation for performance analysis of large, complex RDBMS. In Proceedings of the Fourth International Workshop on Testing Database Systems. ACM, 5."},{"issue":"4","key":"e_1_3_3_41_2","first-page":"3","article-title":"Data cleaning: Problems and current approaches","volume":"23","author":"Rahm Erhard","year":"2000","unstructured":"Erhard Rahm and Hong Hai Do. 2000. Data cleaning: Problems and current approaches. IEEE Data Eng. Bull. 23, 4 (2000), 3\u201313.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_3_3_42_2","first-page":"133","volume-title":"Proceedings of the First Instructional Conference on Machine Learning","volume":"242","author":"Ramos Juan","year":"2003","unstructured":"Juan Ramos et\u00a0al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the First Instructional Conference on Machine Learning, Vol. 242. 133\u2013142."},{"issue":"7","key":"e_1_3_3_43_2","first-page":"46","article-title":"Pattern-matching: the Gestalt approach","volume":"13","author":"Ratcliff John W.","year":"1988","unstructured":"John W. Ratcliff and David E. Metzener. 1988. Pattern-matching: the Gestalt approach. Dr Dobbs Journal 13, 7 (1988), 46.","journal-title":"Dr Dobbs Journal"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3186549.3186559"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786855"},{"key":"e_1_3_3_46_2","first-page":"139","volume-title":"Proceedings of the 10th Working Conference on MSR","author":"Saha Ripon K.","year":"2013","unstructured":"Ripon K. Saha, Chanchal K. Roy, Kevin A. Schneider, and Dewayne E. Perry. 2013. Understanding the evolution of type-3 clones: An exploratory study. In Proceedings of the 10th Working Conference on MSR. IEEE Press, 139\u2013148."},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445518"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9286-4"},{"key":"e_1_3_3_49_2","first-page":"2279","volume-title":"Proceedings of KDD","author":"Sutton Charles","year":"2018","unstructured":"Charles Sutton, Timothy Hobson, James Geddes, and Rich Caruana. 2018. Data diff: Interpretable, executable summaries of changes in distributions for data wrangling. In Proceedings of KDD. 2279\u20132288."},{"key":"e_1_3_3_50_2","article-title":"Data generation for application-specific benchmarking","volume":"7","author":"Tay Y.","year":"2011","unstructured":"Y. Tay. 2011. Data generation for application-specific benchmarking. VLDB, Challenges and Visions 7 (2011).","journal-title":"VLDB, Challenges and Visions"},{"key":"e_1_3_3_51_2","volume-title":"Creating a Data-Driven Enterprise with DataOps","author":"Thusoo Ashish","year":"2017","unstructured":"Ashish Thusoo and Joydeep Sarma. 2017. Creating a Data-Driven Enterprise with DataOps. O\u2019Reilly Media, Incorporated."},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9868.00196"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2017.144"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313729"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9634-5"},{"key":"e_1_3_3_56_2","first-page":"425","volume-title":"Proceedings of SIGIR","author":"Xie Xiaohui","year":"2018","unstructured":"Xiaohui Xie, Jiaxin Mao, Maarten de Rijke, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2018. Constructing an interaction behavior model for web image search. In Proceedings of SIGIR. 425\u2013434."},{"key":"e_1_3_3_57_2","first-page":"2981","volume-title":"Proceedings of CHI","author":"Z\u00fcger Manuela","year":"2015","unstructured":"Manuela Z\u00fcger and Thomas Fritz. 2015. Interruptibility of software developers and its prediction using psycho-physiological sensors. In Proceedings of CHI. ACM, 2981\u20132990."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3567419","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3567419","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:29:38Z","timestamp":1750285778000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3567419"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,7]]},"references-count":56,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3567419"],"URL":"https:\/\/doi.org\/10.1145\/3567419","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2023,2,7]]},"assertion":[{"value":"2021-12-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-09-18","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-02-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}