{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:26:22Z","timestamp":1750220782742,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":43,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,4,14]],"date-time":"2021-04-14T00:00:00Z","timestamp":1618358400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,14]]},"DOI":"10.1145\/3397481.3450647","type":"proceedings-article","created":{"date-parts":[[2021,4,14]],"date-time":"2021-04-14T06:07:18Z","timestamp":1618380438000},"page":"423-433","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["The Design and Development of a Game to Study Backdoor Poisoning Attacks: The Backdoor Game"],"prefix":"10.1145","author":[{"given":"Zahra","family":"Ashktorab","sequence":"first","affiliation":[{"name":"Thomas J. Watson Center IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Casey","family":"Dugan","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Johnson","sequence":"additional","affiliation":[{"name":"AI X-Lab IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aabhas","family":"Sharma","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dustin Ramsey","family":"Torres","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ingrid","family":"Lange","sequence":"additional","affiliation":[{"name":"Fjord Design, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin","family":"Hoover","sequence":"additional","affiliation":[{"name":"IBM Research AI, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Heiko","family":"Ludwig","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bryant","family":"Chen","sequence":"additional","affiliation":[{"name":"IBM Research, Almaden, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nathalie","family":"Baracaldo","sequence":"additional","affiliation":[{"name":"IBM Research, Almaden, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Werner","family":"Geyer","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qian","family":"Pan","sequence":"additional","affiliation":[{"name":"IBM Research, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,4,14]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Mart\u00edn Abadi Paul Barham Jianmin Chen Zhifeng Chen Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Geoffrey Irving Michael Isard 2016. Tensorflow: a system for large-scale machine learning.. In OSDI Vol.\u00a016. 265\u2013283.  Mart\u00edn Abadi Paul Barham Jianmin Chen Zhifeng Chen Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Geoffrey Irving Michael Isard 2016. Tensorflow: a system for large-scale machine learning.. In OSDI Vol.\u00a016. 265\u2013283."},{"key":"e_1_3_2_1_2_1","volume-title":"Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.","author":"Attenberg M","year":"2011","unstructured":"Josh\u00a0 M Attenberg , Pagagiotis\u00a0 G Ipeirotis , and Foster Provost . 2011 . Beat the machine: Challenging workers to find the unknown unknowns . In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. Josh\u00a0M Attenberg, Pagagiotis\u00a0G Ipeirotis, and Foster Provost. 2011. Beat the machine: Challenging workers to find the unknown unknowns. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3128572.3140450"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866218.1866220"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/2936924.2937066"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0925-2312(03)00433-8"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/365024.365078"},{"key":"e_1_3_2_1_8_1","unstructured":"Bryant Chen Wilka Carvalho Nathalie Baracaldo Heiko Ludwig Benjamin Edwards Taesung Lee Ian Molloy and Biplav Srivastava. 2018. Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering. arXiv preprint arXiv:1811.03728(2018).  Bryant Chen Wilka Carvalho Nathalie Baracaldo Heiko Ludwig Benjamin Edwards Taesung Lee Ian Molloy and Biplav Srivastava. 2018. Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering. arXiv preprint arXiv:1811.03728(2018)."},{"key":"e_1_3_2_1_9_1","unstructured":"Xinyun Chen Chang Liu Bo Li Kimberly Lu and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526(2017).  Xinyun Chen Chang Liu Bo Li Kimberly Lu and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526(2017)."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_3_2_1_11_1","volume-title":"Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.","author":"Chrons Otto","year":"2011","unstructured":"Otto Chrons and Sami Sundell . 2011 . Digitalkoot: Making old archives accessible using crowdsourcing . In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. Otto Chrons and Sami Sundell. 2011. Digitalkoot: Making old archives accessible using crowdsourcing. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature09304"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MMUL.2007.67"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1348246.1348248"},{"key":"e_1_3_2_1_15_1","volume-title":"Gamification: designing for motivation. interactions 19, 4","author":"Deterding Sebastian","year":"2012","unstructured":"Sebastian Deterding . 2012. Gamification: designing for motivation. interactions 19, 4 ( 2012 ), 14\u201317. Sebastian Deterding. 2012. Gamification: designing for motivation. interactions 19, 4 (2012), 14\u201317."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"crossref","unstructured":"Steven Dow Anand Kulkarni Brie Bunge Truc Nguyen Scott Klemmer and Bj\u00f6rn Hartmann. 2011. Shepherding the crowd: managing and providing feedback to crowd workers. In CHI\u201911 Extended Abstracts on Human Factors in Computing Systems. 1669\u20131674.  Steven Dow Anand Kulkarni Brie Bunge Truc Nguyen Scott Klemmer and Bj\u00f6rn Hartmann. 2011. Shepherding the crowd: managing and providing feedback to crowd workers. In CHI\u201911 Extended Abstracts on Human Factors in Computing Systems. 1669\u20131674.","DOI":"10.1145\/1979742.1979826"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00175"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2013.11.045"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Leo\u00a0A Goodman. 1961. Snowball sampling. The annals of mathematical statistics(1961) 148\u2013170.  Leo\u00a0A Goodman. 1961. Snowball sampling. The annals of mathematical statistics(1961) 148\u2013170.","DOI":"10.1214\/aoms\/1177705148"},{"key":"e_1_3_2_1_20_1","volume-title":"Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass","author":"Gray L","year":"2019","unstructured":"Mary\u00a0 L Gray and Siddharth Suri . 2019 . Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass . Eamon Dolan Books . Mary\u00a0L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Eamon Dolan Books."},{"key":"e_1_3_2_1_21_1","volume-title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861(2017).","author":"Howard G","year":"2017","unstructured":"Andrew\u00a0 G Howard , Menglong Zhu , Bo Chen , Dmitry Kalenichenko , Weijun Wang , Tobias Weyand , Marco Andreetto , and Hartwig Adam . 2017 . Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861(2017). Andrew\u00a0G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861(2017)."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_3_2_1_23_1","volume-title":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 405\u2013412","author":"Kloft Marius","year":"2010","unstructured":"Marius Kloft and Pavel Laskov . 2010 . Online anomaly detection under adversarial impact . In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 405\u2013412 . Marius Kloft and Pavel Laskov. 2010. Online anomaly detection under adversarial impact. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 405\u2013412."},{"key":"e_1_3_2_1_24_1","unstructured":"Ivan Krasin Tom Duerig Neil Alldrin Vittorio Ferrari Sami Abu-El-Haija Alina Kuznetsova Hassan Rom Jasper Uijlings Stefan Popov Shahab Kamali Matteo Malloci Jordi Pont-Tuset Andreas Veit Serge Belongie Victor Gomes Abhinav Gupta Chen Sun Gal Chechik David Cai Zheyun Feng Dhyanesh Narayanan and Kevin Murphy. 2017. OpenImages: A public dataset for large-scale multi-label and multi-class image classification.Dataset available from https:\/\/storage.googleapis.com\/openimages\/web\/index.html(2017).  Ivan Krasin Tom Duerig Neil Alldrin Vittorio Ferrari Sami Abu-El-Haija Alina Kuznetsova Hassan Rom Jasper Uijlings Stefan Popov Shahab Kamali Matteo Malloci Jordi Pont-Tuset Andreas Veit Serge Belongie Victor Gomes Abhinav Gupta Chen Sun Gal Chechik David Cai Zheyun Feng Dhyanesh Narayanan and Kevin Murphy. 2017. OpenImages: A public dataset for large-scale multi-label and multi-class image classification.Dataset available from https:\/\/storage.googleapis.com\/openimages\/web\/index.html(2017)."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531733"},{"key":"e_1_3_2_1_26_1","volume-title":"Toward a theory of intrinsically motivating instruction. Cognitive science 5, 4","author":"Malone W","year":"1981","unstructured":"Thomas\u00a0 W Malone . 1981. Toward a theory of intrinsically motivating instruction. Cognitive science 5, 4 ( 1981 ), 333\u2013369. Thomas\u00a0W Malone. 1981. Toward a theory of intrinsically motivating instruction. Cognitive science 5, 4 (1981), 333\u2013369."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_3_2_1_28_1","volume-title":"Proc. 6th International Conference on Virtual Learning ICVL, Vol.\u00a01. 323\u2013329","author":"Muntean Cristina\u00a0Ioana","year":"2011","unstructured":"Cristina\u00a0Ioana Muntean . 2011 . Raising engagement in e-learning through gamification . In Proc. 6th International Conference on Virtual Learning ICVL, Vol.\u00a01. 323\u2013329 . Cristina\u00a0Ioana Muntean. 2011. Raising engagement in e-learning through gamification. In Proc. 6th International Conference on Virtual Learning ICVL, Vol.\u00a01. 323\u2013329."},{"volume-title":"Machine learning in cyber trust","author":"Nelson Blaine","key":"e_1_3_2_1_29_1","unstructured":"Blaine Nelson , Marco Barreno , Fuching\u00a0Jack Chi , Anthony\u00a0 D Joseph , Benjamin\u00a0 IP Rubinstein , Udam Saini , Charles Sutton , JD Tygar , and Kai Xia . 2009. Misleading learners: Co-opting your spam filter . In Machine learning in cyber trust . Springer , 17\u201351. Blaine Nelson, Marco Barreno, Fuching\u00a0Jack Chi, Anthony\u00a0D Joseph, Benjamin\u00a0IP Rubinstein, Udam Saini, Charles Sutton, JD Tygar, and Kai Xia. 2009. Misleading learners: Co-opting your spam filter. In Machine learning in cyber trust. Springer, 17\u201351."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2019.100199"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/1067130.1067133"},{"key":"e_1_3_2_1_32_1","unstructured":"Ali Shafahi W\u00a0Ronny Huang Mahyar Najibi Octavian Suciu Christoph Studer Tudor Dumitras and Tom Goldstein. 2018. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems. 6103\u20136113.  Ali Shafahi W\u00a0Ronny Huang Mahyar Najibi Octavian Suciu Christoph Studer Tudor Dumitras and Tom Goldstein. 2018. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems. 6103\u20136113."},{"key":"e_1_3_2_1_33_1","volume-title":"Pang Wei\u00a0W Koh, and Percy\u00a0S Liang","author":"Steinhardt Jacob","year":"2017","unstructured":"Jacob Steinhardt , Pang Wei\u00a0W Koh, and Percy\u00a0S Liang . 2017 . Certified defenses for data poisoning attacks. In Advances in neural information processing systems. 3517\u20133529. Jacob Steinhardt, Pang Wei\u00a0W Koh, and Percy\u00a0S Liang. 2017. Certified defenses for data poisoning attacks. In Advances in neural information processing systems. 3517\u20133529."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/3298023.3298188"},{"key":"e_1_3_2_1_35_1","unstructured":"Brandon Tran Jerry Li and Aleksander Madry. 2018. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems. 8000\u20138010.  Brandon Tran Jerry Li and Aleksander Madry. 2018. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems. 8000\u20138010."},{"key":"e_1_3_2_1_36_1","unstructured":"MW Van\u00a0Someren YF Barnard and JAC Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive. (1994).  MW Van\u00a0Someren YF Barnard and JAC Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive. (1994)."},{"key":"e_1_3_2_1_37_1","unstructured":"Fabio Viola. 2011. Gamification-I Videogiochi nella Vita Quotidiana. Fabio Viola.  Fabio Viola. 2011. Gamification-I Videogiochi nella Vita Quotidiana. Fabio Viola."},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378704.1378719"},{"volume-title":"23rd {USENIX} Security Symposium ({USENIX} Security 14). 239\u2013254.","author":"Wang Gang","key":"e_1_3_2_1_41_1","unstructured":"Gang Wang , Tianyi Wang , Haitao Zheng , and Ben\u00a0 Y Zhao . 2014. Man vs. machine: Practical adversarial detection of malicious crowdsourcing workers . In 23rd {USENIX} Security Symposium ({USENIX} Security 14). 239\u2013254. Gang Wang, Tianyi Wang, Haitao Zheng, and Ben\u00a0Y Zhao. 2014. Man vs. machine: Practical adversarial detection of malicious crowdsourcing workers. In 23rd {USENIX} Security Symposium ({USENIX} Security 14). 239\u2013254."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1017\/S104909651500116X"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207708"}],"event":{"name":"IUI '21: 26th International Conference on Intelligent User Interfaces","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","SIGCHI ACM Special Interest Group on Computer-Human Interaction"],"location":"College Station TX USA","acronym":"IUI '21"},"container-title":["26th International Conference on Intelligent User Interfaces"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397481.3450647","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3397481.3450647","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:33Z","timestamp":1750200093000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397481.3450647"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,14]]},"references-count":43,"alternative-id":["10.1145\/3397481.3450647","10.1145\/3397481"],"URL":"https:\/\/doi.org\/10.1145\/3397481.3450647","relation":{},"subject":[],"published":{"date-parts":[[2021,4,14]]},"assertion":[{"value":"2021-04-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}