{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T13:19:50Z","timestamp":1768828790061,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":125,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,5,6]]},"DOI":"10.1145\/3411764.3445402","type":"proceedings-article","created":{"date-parts":[[2021,5,8]],"date-time":"2021-05-08T02:27:22Z","timestamp":1620440842000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":60,"title":["Designing Ground Truth and the Social Life of Labels"],"prefix":"10.1145","author":[{"given":"Michael","family":"Muller","sequence":"first","affiliation":[{"name":"AI Interactions IBM Research, United States"}]},{"given":"Christine T.","family":"Wolf","sequence":"additional","affiliation":[{"name":"Independent Consultant, United States"}]},{"given":"Josh","family":"Andres","sequence":"additional","affiliation":[{"name":"IBM Research Australia, Australia"}]},{"given":"Michael","family":"Desmond","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}]},{"given":"Narendra Nath","family":"Joshi","sequence":"additional","affiliation":[{"name":"IBM Research IBM, United States"}]},{"given":"Zahra","family":"Ashktorab","sequence":"additional","affiliation":[{"name":"Thomas J. Watson Center IBM Research, United States"}]},{"given":"Aabhas","family":"Sharma","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}]},{"given":"Kristina","family":"Brimijoin","sequence":"additional","affiliation":[{"name":"IBM Research Mrs., United States"}]},{"given":"Qian","family":"Pan","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}]},{"given":"Evelyn","family":"Duesterwald","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}]},{"given":"Casey","family":"Dugan","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}]}],"member":"320","published-online":{"date-parts":[[2021,5,7]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2008.4732651"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACHI.2009.17"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/pra2.103"},{"key":"e_1_3_2_1_5_1","volume-title":"Scenario-based XAI for Humanitarian Aid Forecasting. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1\u20138.","author":"Andres Josh","year":"2020","unstructured":"Josh Andres , Christine\u00a0 T Wolf , Sergio Cabrero\u00a0Barros , Erick Oduor , Rahul Nair , Alexander Kj\u00e6rum , Anders\u00a0Bech Tharsgaard , and Bo\u00a0Schwartz Madsen . 2020 . Scenario-based XAI for Humanitarian Aid Forecasting. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1\u20138. Josh Andres, Christine\u00a0T Wolf, Sergio Cabrero\u00a0Barros, Erick Oduor, Rahul Nair, Alexander Kj\u00e6rum, Anders\u00a0Bech Tharsgaard, and Bo\u00a0Schwartz Madsen. 2020. Scenario-based XAI for Humanitarian Aid Forecasting. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1\u20138."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2855518"},{"key":"e_1_3_2_1_7_1","unstructured":"Sinem Aslan Sinem\u00a0Emine Mete Eda Okur Ece Oktay Nese Alyuz Utku\u00a0Ergin Genc David Stanhill and Asli\u00a0Arslan Esme. 2017. Human expert labeling process (HELP): towards a reliable higher-order user state labeling process and tool to assess student engagement. Educational Technology(2017) 53\u201359.  Sinem Aslan Sinem\u00a0Emine Mete Eda Okur Ece Oktay Nese Alyuz Utku\u00a0Ergin Genc David Stanhill and Asli\u00a0Arslan Esme. 2017. Human expert labeling process (HELP): towards a reliable higher-order user state labeling process and tool to assess student engagement. Educational Technology(2017) 53\u201359."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300654"},{"key":"e_1_3_2_1_9_1","unstructured":"Sarah\u00a0Elsie Baker and Rosalind Edwards. 2012. How many qualitative interviews is enough? Expert voices and early career reflections on sampling and cases in qualitative research. (2012).  Sarah\u00a0Elsie Baker and Rosalind Edwards. 2012. How many qualitative interviews is enough? Expert voices and early career reflections on sampling and cases in qualitative research. (2012)."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2317956.2317959"},{"key":"e_1_3_2_1_11_1","volume-title":"Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution, and epistemology","author":"Bateson Gregory","unstructured":"Gregory Bateson . 2000. Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution, and epistemology . University of Chicago Press. Gregory Bateson. 2000. Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution, and epistemology. University of Chicago Press."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2065327.2065344"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_3_2_1_14_1","volume-title":"From the life-history approach to the transformation of sociological practice. Biography and society: The life history approach in the social sciences","author":"Bertaux Daniel","year":"1981","unstructured":"Daniel Bertaux . 1981. From the life-history approach to the transformation of sociological practice. Biography and society: The life history approach in the social sciences ( 1981 ), 29\u201345. Daniel Bertaux. 1981. From the life-history approach to the transformation of sociological practice. Biography and society: The life history approach in the social sciences (1981), 29\u201345."},{"key":"e_1_3_2_1_15_1","volume-title":"Science on the run: Information management and industrial geophysics at Schlumberger","author":"Bowker C","year":"1920","unstructured":"Geoffrey\u00a0 C Bowker , C Geoffrey , W\u00a0Bernard Carlson , 1994. Science on the run: Information management and industrial geophysics at Schlumberger , 1920 -1940. MIT press . Geoffrey\u00a0C Bowker, C Geoffrey, W\u00a0Bernard Carlson, 1994. Science on the run: Information management and industrial geophysics at Schlumberger, 1920-1940. MIT press."},{"key":"e_1_3_2_1_16_1","volume-title":"Demystifying theoretical sampling in grounded theory research.Grounded Theory Review 8, 2","author":"Breckenridge Jenna","year":"2009","unstructured":"Jenna Breckenridge and Derek Jones . 2009. Demystifying theoretical sampling in grounded theory research.Grounded Theory Review 8, 2 ( 2009 ). Jenna Breckenridge and Derek Jones. 2009. Demystifying theoretical sampling in grounded theory research.Grounded Theory Review 8, 2 (2009)."},{"key":"e_1_3_2_1_17_1","volume-title":"ACM Sigir forum, Vol.\u00a044. ACM New York","author":"Carvalho R","unstructured":"Vitor\u00a0 R Carvalho , Matthew Lease , and Emine Yilmaz . 2011. Crowdsourcing for search evaluation . In ACM Sigir forum, Vol.\u00a044. ACM New York , NY , USA , 17\u201322. Vitor\u00a0R Carvalho, Matthew Lease, and Emine Yilmaz. 2011. Crowdsourcing for search evaluation. In ACM Sigir forum, Vol.\u00a044. ACM New York, NY, USA, 17\u201322."},{"key":"e_1_3_2_1_18_1","unstructured":"Kathy Charmaz. 2014. Constructing grounded theory. sage.  Kathy Charmaz. 2014. Constructing grounded theory. sage."},{"key":"e_1_3_2_1_19_1","unstructured":"Kathy Charmaz and Antony Bryant. 2011. Grounded theory and credibility. Qualitative research 3(2011) 291\u2013309.  Kathy Charmaz and Antony Bryant. 2011. Grounded theory and credibility. Qualitative research 3(2011) 291\u2013309."},{"key":"e_1_3_2_1_20_1","volume-title":"Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis","author":"Cheplygina Veronika","unstructured":"Veronika Cheplygina and Josien\u00a0 PW Pluim . 2018. Crowd disagreement about medical images is informative . In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis . Springer , 105\u2013111. Veronika Cheplygina and Josien\u00a0PW Pluim. 2018. Crowd disagreement about medical images is informative. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. Springer, 105\u2013111."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSDIS.2015.113"},{"key":"e_1_3_2_1_22_1","volume-title":"Basics of qualitative research: Techniques and procedures for developing grounded theory","author":"Corbin Juliet","unstructured":"Juliet Corbin and Anselm Strauss . 2014. Basics of qualitative research: Techniques and procedures for developing grounded theory . Sage publications. Juliet Corbin and Anselm Strauss. 2014. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/DEST.2010.5610593"},{"key":"e_1_3_2_1_24_1","first-page":"697","article-title":"Automatic labeling of unlabeled text data","volume":"6","author":"Damerau J","year":"2004","unstructured":"Frederick\u00a0 J Damerau , David\u00a0 E Johnson , and Martin\u00a0 C Buskirk\u00a0Jr . 2004 . Automatic labeling of unlabeled text data . US Patent 6 , 697 ,998. Frederick\u00a0J Damerau, David\u00a0E Johnson, and Martin\u00a0C Buskirk\u00a0Jr. 2004. Automatic labeling of unlabeled text data. US Patent 6,697,998.","journal-title":"US Patent"},{"key":"e_1_3_2_1_25_1","volume-title":"Demo at NeurIPS","author":"Desmond Michael","year":"2020","unstructured":"Michael Desmond , Kristina Brimijoin , Evelyn Duesterwald , Narendra\u00a0Nath Joshi , Michael Muller , Zahra Ashktorab , Aabhas Sharma , Casey Dugan , and Qian Pan . 2020. AI= Assisted Data Labeling . Demo at NeurIPS 2020 . Michael Desmond, Kristina Brimijoin, Evelyn Duesterwald, Narendra\u00a0Nath Joshi, Michael Muller, Zahra Ashktorab, Aabhas Sharma, Casey Dugan, and Qian Pan. 2020. AI=Assisted Data Labeling. Demo at NeurIPS 2020."},{"key":"e_1_3_2_1_26_1","volume-title":"Focus on Bio-Image Informatics","author":"Dietz Christian","unstructured":"Christian Dietz and Michael\u00a0 R Berthold . 2016. KNIME for open-source bioimage analysis: a tutorial . In Focus on Bio-Image Informatics . Springer , 179\u2013197. Christian Dietz and Michael\u00a0R Berthold. 2016. KNIME for open-source bioimage analysis: a tutorial. In Focus on Bio-Image Informatics. Springer, 179\u2013197."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"crossref","unstructured":"Shari\u00a0L Dworkin. 2012. Sample size policy for qualitative studies using in-depth interviews.  Shari\u00a0L Dworkin. 2012. Sample size policy for qualitative studies using in-depth interviews.","DOI":"10.1007\/s10508-012-0016-6"},{"key":"e_1_3_2_1_28_1","volume-title":"Designing information spaces: The social navigation approach","author":"Erickson Thomas","unstructured":"Thomas Erickson and Wendy\u00a0 A Kellogg . 2003. Social translucence: using minimalist visualisations of social activity to support collective interaction . In Designing information spaces: The social navigation approach . Springer , 17\u201341. Thomas Erickson and Wendy\u00a0A Kellogg. 2003. Social translucence: using minimalist visualisations of social activity to support collective interaction. In Designing information spaces: The social navigation approach. Springer, 17\u201341."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Hao Fang Hao Cheng Maarten Sap Elizabeth Clark Ari Holtzman Yejin Choi Noah\u00a0A Smith and Mari Ostendorf. 2018. Sounding board: A user-centric and content-driven social chatbot. arXiv preprint arXiv:1804.10202(2018).  Hao Fang Hao Cheng Maarten Sap Elizabeth Clark Ari Holtzman Yejin Choi Noah\u00a0A Smith and Mari Ostendorf. 2018. Sounding board: A user-centric and content-driven social chatbot. arXiv preprint arXiv:1804.10202(2018).","DOI":"10.18653\/v1\/N18-5020"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025837"},{"key":"e_1_3_2_1_31_1","volume-title":"Material Vision. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 604\u2013617","author":"Feinberg Melanie","year":"2017","unstructured":"Melanie Feinberg . 2017 . Material Vision. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 604\u2013617 . Melanie Feinberg. 2017. Material Vision. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 604\u2013617."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2304496.2304501"},{"key":"e_1_3_2_1_33_1","volume-title":"Collaborative Annotation for Reliable Natural Language Processing: Technical and Sociological Aspects","author":"Fort Kar\u00ebn","unstructured":"Kar\u00ebn Fort . 2016. Collaborative Annotation for Reliable Natural Language Processing: Technical and Sociological Aspects . John Wiley & Sons . Kar\u00ebn Fort. 2016. Collaborative Annotation for Reliable Natural Language Processing: Technical and Sociological Aspects. John Wiley & Sons."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1057\/ejis.2011.24"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/214427.214431"},{"key":"e_1_3_2_1_36_1","unstructured":"Patty Gerstenblith. 2020. Provenience and Provenance Intersecting with International Law in the Market for Antiquities. NCJ Int\u2019l L. 45(2020) 457.  Patty Gerstenblith. 2020. Provenience and Provenance Intersecting with International Law in the Market for Antiquities. NCJ Int\u2019l L. 45(2020) 457."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208670"},{"key":"e_1_3_2_1_38_1","volume-title":"Raw data is an oxymoron","author":"Gitelman Lisa","unstructured":"Lisa Gitelman . 2013. Raw data is an oxymoron . MIT press . Lisa Gitelman. 2013. Raw data is an oxymoron. MIT press."},{"key":"e_1_3_2_1_39_1","volume-title":"Discovery of grounded theory: Strategies for qualitative research","author":"Glaser G","unstructured":"Barney\u00a0 G Glaser and Anselm\u00a0 L Strauss . 2017. Discovery of grounded theory: Strategies for qualitative research . Routledge . Barney\u00a0G Glaser and Anselm\u00a0L Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research. Routledge."},{"key":"e_1_3_2_1_40_1","unstructured":"Michele Goetz. 2017. 3 Ways Data Preparation Tools Help You Get Ahead Of Big Data. \u201dhttps:\/\/go.forrester.com\/blogs\/15-02-17-3_ways_data_preparation_tools_help_you_get_ahead_of_big_data\/\u201d.  Michele Goetz. 2017. 3 Ways Data Preparation Tools Help You Get Ahead Of Big Data. \u201dhttps:\/\/go.forrester.com\/blogs\/15-02-17-3_ways_data_preparation_tools_help_you_get_ahead_of_big_data\/\u201d."},{"key":"e_1_3_2_1_41_1","volume-title":"Practices of color classification. Mind, culture, and activity 7, 1-2","author":"Goodwin Charles","year":"2000","unstructured":"Charles Goodwin . 2000. Practices of color classification. Mind, culture, and activity 7, 1-2 ( 2000 ), 19\u201336. Charles Goodwin. 2000. Practices of color classification. Mind, culture, and activity 7, 1-2 (2000), 19\u201336."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866723"},{"key":"e_1_3_2_1_43_1","volume-title":"Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass","author":"Gray L","year":"2019","unstructured":"Mary\u00a0 L Gray and Siddharth Suri . 2019 . Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass . Eamon Dolan Books . Mary\u00a0L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Eamon Dolan Books."},{"key":"e_1_3_2_1_44_1","unstructured":"Ben Green. 2018. Data science as political action: grounding data science in a politics of justice. arXiv preprint arXiv:1811.03435(2018).  Ben Green. 2018. Data science as political action: grounding data science in a politics of justice. arXiv preprint arXiv:1811.03435(2018)."},{"key":"e_1_3_2_1_45_1","volume-title":"How many interviews are enough? An experiment with data saturation and variability. Field methods 18, 1","author":"Guest Greg","year":"2006","unstructured":"Greg Guest , Arwen Bunce , and Laura Johnson . 2006. How many interviews are enough? An experiment with data saturation and variability. Field methods 18, 1 ( 2006 ), 59\u201382. Greg Guest, Arwen Bunce, and Laura Johnson. 2006. How many interviews are enough? An experiment with data saturation and variability. Field methods 18, 1 (2006), 59\u201382."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047205"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-005X.2005.00151.x"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470744"},{"key":"e_1_3_2_1_49_1","unstructured":"Tony Hey Stewart Tansley Kristin Tolle 2009. The fourth paradigm: data-intensive scientific discovery. Vol.\u00a01. Microsoft research Redmond WA.  Tony Hey Stewart Tansley Kristin Tolle 2009. The fourth paradigm: data-intensive scientific discovery. Vol.\u00a01. Microsoft research Redmond WA."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"crossref","unstructured":"Humayun Irshad Eun-Yeong Oh Daniel Schmolze Liza\u00a0M Quintana Laura Collins Rulla\u00a0M Tamimi and Andrew\u00a0H Beck. 2017. Crowdsourcing scoring of immunohistochemistry images: Evaluating performance of the crowd and an automated computational method. Scientific reports 7(2017) 43286.  Humayun Irshad Eun-Yeong Oh Daniel Schmolze Liza\u00a0M Quintana Laura Collins Rulla\u00a0M Tamimi and Andrew\u00a0H Beck. 2017. Crowdsourcing scoring of immunohistochemistry images: Evaluating performance of the crowd and an automated computational method. Scientific reports 7(2017) 43286.","DOI":"10.1038\/srep43286"},{"key":"e_1_3_2_1_51_1","volume-title":"Fast and Automatic Visual Label Conflict Resolution. Demo at NeurIPS","author":"Joshi Narendra\u00a0Nath","year":"2020","unstructured":"Narendra\u00a0Nath Joshi , Aabhas Sharma , , Michael Muller , Qian Pan , Michael Desmond , Kristina Brimijoin , Zahra Ashktorab , Evelyn Duesterwald , and Casey Dugan . 2020. Fast and Automatic Visual Label Conflict Resolution. Demo at NeurIPS 2020 . Narendra\u00a0Nath Joshi, Aabhas Sharma, , Michael Muller, Qian Pan, Michael Desmond, Kristina Brimijoin, Zahra Ashktorab, Evelyn Duesterwald, and Casey Dugan. 2020. Fast and Automatic Visual Label Conflict Resolution. Demo at NeurIPS 2020."},{"key":"e_1_3_2_1_52_1","volume-title":"Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.","author":"Kajino Hiroshi","year":"2012","unstructured":"Hiroshi Kajino , Yuta Tsuboi , Issei Sato , and Hisashi Kashima . 2012 . Learning from crowds and experts . In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence. Hiroshi Kajino, Yuta Tsuboi, Issei Sato, and Hisashi Kashima. 2012. Learning from crowds and experts. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979444"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300322"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173748"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3214273"},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3323994.3369898"},{"key":"e_1_3_2_1_58_1","volume-title":"Computer Vision Metrics","author":"Krig Scott","unstructured":"Scott Krig . 2016. Ground truth data, content, metrics, and analysis . In Computer Vision Metrics . Springer , 247\u2013271. Scott Krig. 2016. Ground truth data, content, metrics, and analysis. In Computer Vision Metrics. Springer, 247\u2013271."},{"key":"e_1_3_2_1_59_1","volume-title":"Progress and its problems: Towards a theory of scientific growth. Vol.\u00a0282","author":"Laudan Larry","unstructured":"Larry Laudan . 1978. Progress and its problems: Towards a theory of scientific growth. Vol.\u00a0282 . Univ of California Press . Larry Laudan. 1978. Progress and its problems: Towards a theory of scientific growth. Vol.\u00a0282. Univ of California Press."},{"key":"e_1_3_2_1_60_1","volume-title":"Workshop on challenges in representation learning, ICML, Vol.\u00a03. 2.","author":"Lee Dong-Hyun","year":"2013","unstructured":"Dong-Hyun Lee . 2013 . Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks . In Workshop on challenges in representation learning, ICML, Vol.\u00a03. 2. Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, Vol.\u00a03. 2."},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2012-001110"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.24191\/smrj.v15i2.4972"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145301"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359174"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819978"},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"crossref","unstructured":"Janice\u00a0M Morse. 1995. The significance of saturation.  Janice\u00a0M Morse. 1995. The significance of saturation.","DOI":"10.1177\/104973239500500201"},{"key":"e_1_3_2_1_67_1","volume-title":"Ways of Knowing in HCI","author":"Muller Michael","unstructured":"Michael Muller . 2014. Curiosity , creativity, and surprise as analytic tools: Grounded theory method . In Ways of Knowing in HCI . Springer , 25\u201348. Michael Muller. 2014. Curiosity, creativity, and surprise as analytic tools: Grounded theory method. In Ways of Knowing in HCI. Springer, 25\u201348."},{"key":"e_1_3_2_1_68_1","volume-title":"Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, W15","author":"Muller Michael","year":"2019","unstructured":"Michael Muller , Melanie Feinberg , Timothy George , Steven\u00a0 J Jackson , Bonnie\u00a0 E John , Mary\u00a0Beth Kery , and Samir Passi . 2019 . Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, W15 . Michael Muller, Melanie Feinberg, Timothy George, Steven\u00a0J Jackson, Bonnie\u00a0E John, Mary\u00a0Beth Kery, and Samir Passi. 2019. Human-Centered Study of Data Science Work Practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, W15."},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300356"},{"key":"e_1_3_2_1_70_1","volume-title":"Critique and contribute: A practice-based framework for improving critical data studies and data science. Big data 5, 2","author":"Neff Gina","year":"2017","unstructured":"Gina Neff , Anissa Tanweer , Brittany Fiore-Gartland , and Laura Osburn . 2017. Critique and contribute: A practice-based framework for improving critical data studies and data science. Big data 5, 2 ( 2017 ), 85\u201397. Gina Neff, Anissa Tanweer, Brittany Fiore-Gartland, and Laura Osburn. 2017. Critique and contribute: A practice-based framework for improving critical data studies and data science. Big data 5, 2 (2017), 85\u201397."},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-013-1771-7"},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998331"},{"key":"e_1_3_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274405"},{"key":"e_1_3_2_1_74_1","article-title":"Comparison of various classification algorithms on iris datasets using WEKA","volume":"1","author":"Patel Kanu","year":"2014","unstructured":"Kanu Patel , Jay Vala , and Jaymit Pandya . 2014 . Comparison of various classification algorithms on iris datasets using WEKA . Int. J. Adv. Eng. Res. Dev.(IJAERD) 1 , 1 (2014). Kanu Patel, Jay Vala, and Jaymit Pandya. 2014. Comparison of various classification algorithms on iris datasets using WEKA. Int. J. Adv. Eng. Res. Dev.(IJAERD) 1, 1 (2014).","journal-title":"Int. J. Adv. Eng. Res. Dev.(IJAERD)"},{"key":"e_1_3_2_1_75_1","volume-title":"Workshop on Crowdsourcing and Human Computation at the Conference on Human Factors in Computing Systems (CHI).","author":"Paul A","year":"2011","unstructured":"Sharoda\u00a0 A Paul , Lichan Hong , and Ed\u00a0 H Chi . 2011 . What is a question? Crowdsourcing tweet categorization . In Workshop on Crowdsourcing and Human Computation at the Conference on Human Factors in Computing Systems (CHI). Sharoda\u00a0A Paul, Lichan Hong, and Ed\u00a0H Chi. 2011. What is a question? Crowdsourcing tweet categorization. In Workshop on Crowdsourcing and Human Computation at the Conference on Human Factors in Computing Systems (CHI)."},{"key":"e_1_3_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-40593-3_13"},{"key":"e_1_3_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702298"},{"key":"e_1_3_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2017.12.020"},{"key":"e_1_3_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376299"},{"key":"e_1_3_2_1_80_1","unstructured":"Krishna Rajan. 2013. Informatics for materials science and engineering: data-driven discovery for accelerated experimentation and application. Butterworth-Heinemann.  Krishna Rajan. 2013. Informatics for materials science and engineering: data-driven discovery for accelerated experimentation and application. Butterworth-Heinemann."},{"key":"e_1_3_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-019-00552-1"},{"key":"e_1_3_2_1_82_1","volume-title":"O\u2019Reilly Media","author":"Rattenbury Tye","unstructured":"Tye Rattenbury , Joseph\u00a0 M Hellerstein , Jeffrey Heer , Sean Kandel , and Connor Carreras . 2017. Principles of data wrangling: Practical techniques for data preparation . \u201d O\u2019Reilly Media , Inc .\u201d. Tye Rattenbury, Joseph\u00a0M Hellerstein, Jeffrey Heer, Sean Kandel, and Connor Carreras. 2017. Principles of data wrangling: Practical techniques for data preparation. \u201d O\u2019Reilly Media, Inc.\u201d."},{"key":"e_1_3_2_1_83_1","volume-title":"RE: Definitions of use. Design studies 29, 4","author":"Redstr\u00f6m Johan","year":"2008","unstructured":"Johan Redstr\u00f6m . 2008 . RE: Definitions of use. Design studies 29, 4 (2008), 410\u2013423. Johan Redstr\u00f6m. 2008. RE: Definitions of use. Design studies 29, 4 (2008), 410\u2013423."},{"key":"e_1_3_2_1_84_1","volume-title":"On lies, secrets, and silence: Selected prose 1966-1978","author":"Rich Adrienne","unstructured":"Adrienne Rich . 1995. On lies, secrets, and silence: Selected prose 1966-1978 . WW Norton & Company . Adrienne Rich. 1995. On lies, secrets, and silence: Selected prose 1966-1978. WW Norton & Company."},{"key":"e_1_3_2_1_85_1","volume-title":"A survey on data collection for machine learning: a big data-ai integration perspective","author":"Roh Yuji","year":"2019","unstructured":"Yuji Roh , Geon Heo , and Steven\u00a0Euijong Whang . 2019. A survey on data collection for machine learning: a big data-ai integration perspective . IEEE Transactions on Knowledge and Data Engineering ( 2019 ). Yuji Roh, Geon Heo, and Steven\u00a0Euijong Whang. 2019. A survey on data collection for machine learning: a big data-ai integration perspective. IEEE Transactions on Knowledge and Data Engineering (2019)."},{"key":"e_1_3_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173606"},{"key":"e_1_3_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300292"},{"key":"e_1_3_2_1_88_1","volume-title":"International Semantic Web Conference (P&D\/Industry\/BlueSky).","author":"Samuel Sheeba","year":"2018","unstructured":"Sheeba Samuel and Birgitta K\u00f6nig-Ries . 2018 . ProvBook: Provenance-based Semantic Enrichment of Interactive Notebooks for Reproducibility .. In International Semantic Web Conference (P&D\/Industry\/BlueSky). Sheeba Samuel and Birgitta K\u00f6nig-Ries. 2018. ProvBook: Provenance-based Semantic Enrichment of Interactive Notebooks for Reproducibility.. In International Semantic Web Conference (P&D\/Industry\/BlueSky)."},{"key":"e_1_3_2_1_89_1","unstructured":"Sheeba Samuel and Birgitta K\u00f6nig-Ries. 2020. ReproduceMeGit: A Visualization Tool for Analyzing Reproducibility of Jupyter Notebooks. arXiv preprint arXiv:2006.12110(2020).  Sheeba Samuel and Birgitta K\u00f6nig-Ries. 2020. ReproduceMeGit: A Visualization Tool for Analyzing Reproducibility of Jupyter Notebooks. arXiv preprint arXiv:2006.12110(2020)."},{"key":"e_1_3_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359178"},{"key":"e_1_3_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376290"},{"key":"e_1_3_2_1_92_1","volume-title":"Remarks on the complexity of cooperative work.Revue d\u2019intelligence artificielle 16, 4-5","author":"Schmidt Kjeld","year":"2002","unstructured":"Kjeld Schmidt . 2002. Remarks on the complexity of cooperative work.Revue d\u2019intelligence artificielle 16, 4-5 ( 2002 ), 443\u2013483. Kjeld Schmidt. 2002. Remarks on the complexity of cooperative work.Revue d\u2019intelligence artificielle 16, 4-5 (2002), 443\u2013483."},{"key":"e_1_3_2_1_93_1","volume-title":"Designing as reflective conversation with the materials of a design situation. Knowledge-based systems 5, 1","author":"Sch\u00f6n A","year":"1992","unstructured":"Donald\u00a0 A Sch\u00f6n . 1992. Designing as reflective conversation with the materials of a design situation. Knowledge-based systems 5, 1 ( 1992 ), 3\u201314. Donald\u00a0A Sch\u00f6n. 1992. Designing as reflective conversation with the materials of a design situation. Knowledge-based systems 5, 1 (1992), 3\u201314."},{"key":"e_1_3_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.29311\/mas.v18i1.3459"},{"key":"e_1_3_2_1_95_1","volume-title":"Machines as teammates: A research agenda on AI in team collaboration. Information & management 57, 2","author":"Seeber Isabella","year":"2020","unstructured":"Isabella Seeber , Eva Bittner , Robert\u00a0 O Briggs , Triparna de Vreede , Gert-Jan De\u00a0Vreede , Aaron Elkins , Ronald Maier , Alexander\u00a0 B Merz , Sarah Oeste-Rei\u00df , Nils Randrup , 2020. Machines as teammates: A research agenda on AI in team collaboration. Information & management 57, 2 ( 2020 ), 103174. Isabella Seeber, Eva Bittner, Robert\u00a0O Briggs, Triparna de Vreede, Gert-Jan De\u00a0Vreede, Aaron Elkins, Ronald Maier, Alexander\u00a0B Merz, Sarah Oeste-Rei\u00df, Nils Randrup, 2020. Machines as teammates: A research agenda on AI in team collaboration. Information & management 57, 2 (2020), 103174."},{"key":"e_1_3_2_1_96_1","unstructured":"Cathrine Seidelin Yvonne Dittrich and Eric Gr\u00f6nvall. [n.d.]. Co-designing data experiments. ([n.\u00a0d.]). (in preparation).  Cathrine Seidelin Yvonne Dittrich and Eric Gr\u00f6nvall. [n.d.]. Co-designing data experiments. ([n.\u00a0d.]). (in preparation)."},{"key":"e_1_3_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.14236\/ewic\/HCI2018.14"},{"key":"e_1_3_2_1_99_1","unstructured":"Ayush Singhal Pradeep Sinha and Rakesh Pant. 2017. Use of deep learning in modern recommendation system: A summary of recent works. arXiv preprint arXiv:1712.07525(2017).  Ayush Singhal Pradeep Sinha and Rakesh Pant. 2017. Use of deep learning in modern recommendation system: A summary of recent works. arXiv preprint arXiv:1712.07525(2017)."},{"key":"e_1_3_2_1_100_1","volume-title":"The ethnography of infrastructure. American behavioral scientist 43, 3","author":"Star Susan\u00a0Leigh","year":"1999","unstructured":"Susan\u00a0Leigh Star . 1999. The ethnography of infrastructure. American behavioral scientist 43, 3 ( 1999 ), 377\u2013391. Susan\u00a0Leigh Star. 1999. The ethnography of infrastructure. American behavioral scientist 43, 3 (1999), 377\u2013391."},{"key":"e_1_3_2_1_101_1","volume-title":"Steps toward an ecology of infrastructure: Design and access for large information spaces. Information systems research 7, 1","author":"Star Susan\u00a0Leigh","year":"1996","unstructured":"Susan\u00a0Leigh Star and Karen Ruhleder . 1996. Steps toward an ecology of infrastructure: Design and access for large information spaces. Information systems research 7, 1 ( 1996 ), 111\u2013134. Susan\u00a0Leigh Star and Karen Ruhleder. 1996. Steps toward an ecology of infrastructure: Design and access for large information spaces. Information systems research 7, 1 (1996), 111\u2013134."},{"key":"e_1_3_2_1_102_1","volume-title":"Layers of silence, arenas of voice: The ecology of visible and invisible work. Computer supported cooperative work (CSCW) 8, 1-2","author":"Star Susan\u00a0Leigh","year":"1999","unstructured":"Susan\u00a0Leigh Star and Anselm Strauss . 1999. Layers of silence, arenas of voice: The ecology of visible and invisible work. Computer supported cooperative work (CSCW) 8, 1-2 ( 1999 ), 9\u201330. Susan\u00a0Leigh Star and Anselm Strauss. 1999. Layers of silence, arenas of voice: The ecology of visible and invisible work. Computer supported cooperative work (CSCW) 8, 1-2 (1999), 9\u201330."},{"key":"e_1_3_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675298"},{"key":"e_1_3_2_1_104_1","volume-title":"Properties for growing grounded theory","author":"Stern P.N.","unstructured":"P.N. Stern . 2007. Properties for growing grounded theory . In The Sage handbook of grounded theory, A.\u00a0Bryant and K.\u00a0Charmaz (Eds.). Sage, Thousand Oaks, CA, USA. P.N. Stern. 2007. Properties for growing grounded theory. In The Sage handbook of grounded theory, A.\u00a0Bryant and K.\u00a0Charmaz (Eds.). Sage, Thousand Oaks, CA, USA."},{"key":"e_1_3_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1145\/2817721.2817734"},{"key":"e_1_3_2_1_106_1","volume-title":"Located accountabilities in technology production. Scandinavian journal of information systems 14, 2","author":"Suchman Lucy","year":"2002","unstructured":"Lucy Suchman . 2002. Located accountabilities in technology production. Scandinavian journal of information systems 14, 2 ( 2002 ), 7. Lucy Suchman. 2002. Located accountabilities in technology production. Scandinavian journal of information systems 14, 2 (2002), 7."},{"key":"e_1_3_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220057"},{"key":"e_1_3_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.5121\/ijsc.2012.3107"},{"key":"e_1_3_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.5771\/0943-7444-2019-7-558"},{"key":"e_1_3_2_1_111_1","volume-title":"Enterprise interoperability","author":"Van\u00a0der Aalst MP","unstructured":"Wil\u00a0 MP Van\u00a0der Aalst . 2014. Data scientist: The engineer of the future . In Enterprise interoperability VI. Springer , 13\u201326. Wil\u00a0MP Van\u00a0der Aalst. 2014. Data scientist: The engineer of the future. In Enterprise interoperability VI. Springer, 13\u201326."},{"key":"e_1_3_2_1_112_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-019-05855-6"},{"key":"e_1_3_2_1_113_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2008.4497403"},{"key":"e_1_3_2_1_114_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_3_2_1_115_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_3_2_1_116_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377325.3377538"},{"key":"e_1_3_2_1_117_1","unstructured":"Jacob Whitehill Paul Ruvolo Tingfan Wu Jacob Bergsma and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22 - Proceedings of the 2009 Conference. 2035\u20132043.  Jacob Whitehill Paul Ruvolo Tingfan Wu Jacob Bergsma and Javier Movellan. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22 - Proceedings of the 2009 Conference. 2035\u20132043."},{"key":"e_1_3_2_1_118_1","doi-asserted-by":"publisher","DOI":"10.1109\/eScienceW.2011.27"},{"key":"e_1_3_2_1_119_1","doi-asserted-by":"publisher","DOI":"10.1145\/1859799.1859814"},{"key":"e_1_3_2_1_120_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301019.3323879"},{"key":"e_1_3_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-43687-2_55"},{"key":"e_1_3_2_1_122_1","volume-title":"Human-Centered Data Science: A New Paradigm for Industrial IoT. In Ethnographic Praxis in Industry Conference Proceedings, Vol.\u00a02018","author":"Yapchain Matthew","year":"2018","unstructured":"Matthew Yapchain . 2018 . Human-Centered Data Science: A New Paradigm for Industrial IoT. In Ethnographic Praxis in Industry Conference Proceedings, Vol.\u00a02018 . Wiley Online Library, 53\u201361. Matthew Yapchain. 2018. Human-Centered Data Science: A New Paradigm for Industrial IoT. In Ethnographic Praxis in Industry Conference Proceedings, Vol.\u00a02018. Wiley Online Library, 53\u201361."},{"key":"e_1_3_2_1_123_1","unstructured":"Amy\u00a0X Zhang Michael Muller and Dakuo Wang. 2020. How do Data Science Workers Collaborate? Roles Workflows and Tools. arXiv preprint arXiv:2001.06684(2020).  Amy\u00a0X Zhang Michael Muller and Dakuo Wang. 2020. How do Data Science Workers Collaborate? Roles Workflows and Tools. arXiv preprint arXiv:2001.06684(2020)."},{"key":"e_1_3_2_1_124_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392826"},{"key":"e_1_3_2_1_125_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88688-4_52"},{"key":"e_1_3_2_1_126_1","unstructured":"Xiaojin Zhu and Zoubin Ghahramani. 2002. Learning from labeled and unlabeled data with label propagation. (2002).  Xiaojin Zhu and Zoubin Ghahramani. 2002. Learning from labeled and unlabeled data with label propagation. (2002)."},{"key":"e_1_3_2_1_127_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1053-5357(99)80114-0"}],"event":{"name":"CHI '21: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445402","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445402","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:34Z","timestamp":1750195714000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445402"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":125,"alternative-id":["10.1145\/3411764.3445402","10.1145\/3411764"],"URL":"https:\/\/doi.org\/10.1145\/3411764.3445402","relation":{},"subject":[],"published":{"date-parts":[[2021,5,6]]},"assertion":[{"value":"2021-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}