{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,14]],"date-time":"2026-03-14T09:49:44Z","timestamp":1773481784733,"version":"3.50.1"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,6,13]],"date-time":"2023-06-13T00:00:00Z","timestamp":1686614400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2023,6,13]]},"abstract":"<jats:p>Data systems often present examples and solicit labels from users to learn a target model, i.e., active learning. However, due to the complexity of the underlying data, users may not initially have a perfect understanding of the effective model and do not know the accurate labeling. For example, a user who is training a model for detecting noisy or abnormal values may not perfectly know the properties of typical and clean values in the data. Users may improve their knowledge about the data and target model as they observe examples during training. As users gradually learn about the data and model, they may revise their labeling strategies. Current systems assume that users always provide correct labeling with potentially a fixed and small chance of annotation mistakes. Nonetheless, if the trainer revises its belief during training, such mistakes become significant and non-stationarity. Hence, current systems consume incorrect labels and may learn inaccurate models. In this paper, we build theoretical underpinnings and design algorithms to develop systems that collaborate with users to learn the target model accurately and efficiently. At the core of our proposal, a game-theoretic framework models the joint learning of user and system to reach a desirable eventual stable state, where both user and system share the same belief about the target model. We extensively evaluate our system using user studies over various real-world datasets and show that our algorithms lead to accurate results with a smaller number of interactions compared to existing methods.<\/jats:p>","DOI":"10.1145\/3589280","type":"journal-article","created":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T20:26:45Z","timestamp":1687292805000},"page":"1-25","source":"Crossref","is-referenced-by-count":5,"title":["Exploratory Training: When Annotators Learn About Data"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-4163-4071","authenticated-orcid":false,"given":"Rajesh","family":"Shrestha","sequence":"first","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-5093-0243","authenticated-orcid":false,"given":"Omeed","family":"Habibelahian","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2213-6303","authenticated-orcid":false,"given":"Arash","family":"Termehchy","sequence":"additional","affiliation":[{"name":"Oregon State University, Corvallis, OR, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0651-4128","authenticated-orcid":false,"given":"Paolo","family":"Papotti","sequence":"additional","affiliation":[{"name":"Eurecom, Biot, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,6,20]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.14778\/2856318.2856328"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994518"},{"key":"e_1_2_2_3_1","volume-title":"Foundations of Databases: The Logical Level","author":"Abiteboul Serge","unstructured":"Serge Abiteboul, Richard Hull, and Victor Vianu. 1994. Foundations of Databases: The Logical Level. Addison-Wesley."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3524077"},{"key":"e_1_2_2_5_1","volume-title":"Yu","author":"Aggarwal Charu C.","year":"2014","unstructured":"Charu C. Aggarwal, Xiangnan Kong, Quanquan Gu, Jiawei Han, and Philip S. Yu. 2014. Active Learning: A Survey. In Data Classification: Algorithms and Applications, Charu C. Aggarwal (Ed.). CRC Press, 571--606. http:\/\/www.crcnetbase.com\/doi\/abs\/10.1201\/b17320--23"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850578.2850579"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744818"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13406"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00371-018--1500--3"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.14778\/3204028.3204032"},{"key":"e_1_2_2_11_1","volume-title":"Advances in understanding strategic behaviour : game theory, experiments, and bounded rationality","author":"Camerer Colin F.","unstructured":"Colin F. Camerer, Teck-Hua Ho, and Juin Kuan Chong. 2004. Behavioural Game Theory: Thinking, Learning and Teaching. In Advances in understanding strategic behaviour : game theory, experiments, and bounded rationality. Palgrave Macmillan, 120--180."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2020.2967722"},{"key":"e_1_2_2_13_1","volume-title":"Holistic data cleaning: Putting violations into context","author":"Chu Xu","unstructured":"Xu Chu, Ihab F. Ilyas, and Paolo Papotti. 2013. Holistic data cleaning: Putting violations into context. In ICDE. IEEE Computer Society, 458--469."},{"key":"e_1_2_2_14_1","volume-title":"Proceedings of the 2021 International Conference on Machine Learning (ICML '21)","author":"Darrell Trevor","year":"2021","unstructured":"Trevor Darrell, Xin Wang, Li Erran Li, Fisher Yu, Zeynep Akata, Wenwu Zhu, Pradeep Ravikumar, Shiji Zhou, Shanghang Zhang, and Kalesha Bullard. 2021. HILL'21: ICML Workshop on Human in the Loop Learning. In Proceedings of the 2021 International Conference on Machine Learning (ICML '21)."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610523"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376916.1376940"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.2200\/S00439ED1V01Y201207DTM030"},{"key":"e_1_2_2_19_1","volume-title":"The Theory of Learning in Games","author":"Fudenberg Drew","unstructured":"Drew Fudenberg and David Levine. 1998. The Theory of Learning in Games. MIT Press."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aac6076"},{"key":"e_1_2_2_21_1","volume-title":"Proceedings of the 23rd International Conference on Neural Information Processing Systems -","volume":"1","author":"Golovin Daniel","year":"2010","unstructured":"Daniel Golovin, Andreas Krause, and Debajyoti Ray. 2010. Near-Optimal Bayesian Active Learning with Noisy Observations. In Proceedings of the 23rd International Conference on Neural Information Processing Systems - Volume 1 (Vancouver, British Columbia, Canada) (NIPS'10). Curran Associates Inc., Red Hook, NY, USA, 766--774."},{"key":"e_1_2_2_22_1","volume-title":"The Elements of Statistical Learning","author":"Hastie Trevor","unstructured":"Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning. Springer."},{"key":"e_1_2_2_23_1","unstructured":"Jian He Enzo Veltri Donatello Santoro Guoliang Li Giansalvatore Mecca Paolo Papotti and Nan Tang. 2016. Interactive and Deterministic Data Cleaning. In SIGMOD. ACM 893--907."},{"key":"e_1_2_2_24_1","doi-asserted-by":"crossref","unstructured":"Alireza Heidari Joshua McGrath Ihab F. Ilyas and Theodoros Rekatsinas. 2019. HoloDetect: Few-Shot Learning for Error Detection. In SIGMOD. ACM 829--846.","DOI":"10.1145\/3299869.3319888"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732240.2732248"},{"key":"e_1_2_2_26_1","first-page":"23","article-title":"Self-Service Data Preparation: Research to Practice","volume":"41","author":"Hellerstein Joseph M.","year":"2018","unstructured":"Joseph M. Hellerstein, Jeffrey Heer, and Sean Kandel. 2018. Self-Service Data Preparation: Research to Practice. IEEE Data Eng. Bull. 41, 2 (2018), 23--34.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1111\/1468-0262.00376"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2012.6400492"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/42.2.100"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007568.1007641"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-56039-4_34"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994514"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/13658816.2012.659184"},{"key":"e_1_2_2_34_1","volume-title":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence","author":"Lin Christopher H.","year":"1845","unstructured":"Christopher H. Lin, Mausam, and Daniel S. Weld. 2016. Re-Active Learning: Active Learning with Relabeling. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (Phoenix, Arizona) (AAAI'16). AAAI Press, 1845--1852."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.14778\/3401960.3401966"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360904"},{"key":"e_1_2_2_37_1","volume-title":"Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang.","author":"Mahdavi Mohammad","year":"2019","unstructured":"Mohammad Mahdavi, Ziawasch Abedjan, Raul Castro Fernandez, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang. 2019. Raha: A Configuration-Free Error Detection System. In SIGMOD. ACM, 865--882."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196899"},{"key":"e_1_2_2_39_1","unstructured":"Tom Mitchell. 1997. Machine Learning. McGraw-Hil."},{"key":"e_1_2_2_40_1","doi-asserted-by":"crossref","unstructured":"Yael Niv. 2009. The Neuroscience of Reinforcement Learning. In ICML.","DOI":"10.1145\/1553374.1553557"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmp.2008.12.005"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/3368289.3368293"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137628.3137631"},{"key":"e_1_2_2_44_1","volume-title":"Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence (Proceedings of Machine Learning Research","volume":"1717","author":"Rolf Esther","year":"2022","unstructured":"Esther Rolf, Nikolay Malkin, Alexandros Graikos, Ana Jojic, Caleb Robinson, and Nebojsa Jojic. 2022. Resolving label uncertainty with implicit posterior models. In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence (Proceedings of Machine Learning Research, Vol. 180), James Cussens and Kun Zhang (Eds.). PMLR, 1707--1717. https:\/\/proceedings.mlr.press\/v180\/rolf22a.html"},{"key":"e_1_2_2_45_1","volume-title":"Learning in extensive-form games: Experimental data and simple dynamic models in the intermediate term. Games and economic behavior 8, 1","author":"Roth Alvin E","year":"1995","unstructured":"Alvin E Roth and Ido Erev. 1995. Learning in extensive-form games: Experimental data and simple dynamic models in the intermediate term. Games and economic behavior 8, 1 (1995), 164--212."},{"key":"e_1_2_2_47_1","volume-title":"Active Learning","author":"Settles Burr","unstructured":"Burr Settles. 2012. Active Learning. Morgan & Claypool Publishers."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.5555\/2831071.2831072"},{"key":"e_1_2_2_49_1","volume-title":"Advances in Neural Information Processing Systems 11","author":"Tenenbaum Joshua B.","unstructured":"Joshua B. Tenenbaum. 1999. Bayesian Modeling of Human Concept Learning. In Advances in Neural Information Processing Systems 11, M. J. Kearns, S. A. Solla, and D. A. Cohn (Eds.). MIT Press, 59--68. http:\/\/papers.nips.cc\/paper\/1542-bayesian-modeling-of-human-concept-learning.pdf"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","unstructured":"Saravanan Thirumuruganathan Laure Berti-\u00c9quille Mourad Ouzzani Jorge-Arnulfo Quian\u00e9-Ruiz and Nan Tang. 2017. UGuide: User-Guided Discovery of FD-Detectable Errors. In SIGMOD. ACM 1385--1397. https:\/\/doi.org\/10.1145\/3035918.3064024","DOI":"10.1145\/3035918.3064024"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.14778\/3291264.3291268"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.14778\/1952376.1952378"},{"key":"e_1_2_2_53_1","unstructured":"Songbai Yan Kamalika Chaudhuri and Tara Javidi. 2016. Active Learning from Imperfect Labelers. In NIPS Daniel D. Lee Masashi Sugiyama Ulrike von Luxburg Isabelle Guyon and Roman Garnett (Eds.). 2128--2136."},{"key":"e_1_2_2_54_1","doi-asserted-by":"crossref","unstructured":"H Peyton Young. 2004. Strategic learning and its limits. OUP Oxford.","DOI":"10.1093\/acprof:oso\/9780199269181.001.0001"},{"key":"e_1_2_2_55_1","unstructured":"Chicheng Zhang and Kamalika Chaudhuri. 2015. Active Learning from Weak and Strong Labelers. In NIPS Corinna Cortes Neil D. Lawrence Daniel D. Lee Masashi Sugiyama and Roman Garnett (Eds.). 703--711."},{"key":"e_1_2_2_56_1","volume-title":"Zdonik","author":"\u00c7etintemel Ugur","year":"2013","unstructured":"Ugur \u00c7etintemel, Mitch Cherniack, Justin DeBrabant, Yanlei Diao, Kyriaki Dimitriadou, Alexander Kalinin, Olga Papaemmanouil, and Stanley B. Zdonik. 2013. Query Steering for Interactive Data Exploration. In CIDR."}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3589280","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3589280","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:54Z","timestamp":1750182534000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3589280"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,13]]},"references-count":54,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6,13]]}},"alternative-id":["10.1145\/3589280"],"URL":"https:\/\/doi.org\/10.1145\/3589280","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,13]]}}}