{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T04:05:02Z","timestamp":1768277102979,"version":"3.49.0"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2020,10,14]],"date-time":"2020-10-14T00:00:00Z","timestamp":1602633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000145","name":"Division of Information and Intelligent Systems","doi-asserted-by":"publisher","award":["1942229"],"award-info":[{"award-number":["1942229"]}],"id":[{"id":"10.13039\/100000145","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007522","name":"Office of Integrative Activities","doi-asserted-by":"publisher","award":["1557349"],"award-info":[{"award-number":["1557349"]}],"id":[{"id":"10.13039\/100007522","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,10,14]]},"abstract":"<jats:p>To prevent harmful AI behavior, people need to specify constraints that forbid undesirable actions. Unfortunately, this is a complex task, since writing rules that distinguish harmful from non-harmful actions tends to be quite difficult in real-world situations. Therefore, such decisions have historically been made by a small group of powerful AI companies and developers, with limited community input. In this paper, we study how to enable a crowd of non-AI experts to work together to communicate high-quality, reliable constraints to AI systems. We first focus on understanding how humans reason about temporal dynamics in the context of AI behavior, finding through experiments on a novel game-based testbed that participants tend to adopt a long-term notion of harm, even in uncertain situations that do not affect them directly. Building off of this insight, we explore task design for long-term constraint specification, developing new filtering approaches and new methods of promoting user reflection. Next, we develop a novel rule-based interface which allows people to craft rules in an accessible fashion without programming knowledge. We test our approaches on a real-world AI problem in the domain of education, and find that our new filtering mechanisms and interfaces significantly improve constraint quality and human efficiency. We also demonstrate how these systems can be applied to other real-world AI problems (e.g. in social networks).<\/jats:p>","DOI":"10.1145\/3415168","type":"journal-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T22:28:12Z","timestamp":1602800892000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Using the Crowd to Prevent Harmful AI Behavior"],"prefix":"10.1145","volume":"4","author":[{"given":"Travis","family":"Mandel","sequence":"first","affiliation":[{"name":"University of Hawai'i at Hilo, Hilo, HI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jahnu","family":"Best","sequence":"additional","affiliation":[{"name":"University of Hawai'i at Hilo, Hilo, HI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Randall H.","family":"Tanaka","sequence":"additional","affiliation":[{"name":"University of Hawai'i at Hilo, Hilo, HI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hiram","family":"Temple","sequence":"additional","affiliation":[{"name":"University of Hawai'i at Hilo, Hilo, HI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chansen","family":"Haili","sequence":"additional","affiliation":[{"name":"University of Hawai'i at Hilo, Hilo, HI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sebastian J.","family":"Carter","sequence":"additional","affiliation":[{"name":"University of Hawai'i at Hilo, Hilo, HI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kayla","family":"Schlechtinger","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Roy","family":"Szeto","sequence":"additional","affiliation":[{"name":"Center for Game Science, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,10,15]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359301"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300760"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3339904"},{"key":"e_1_2_2_4_1","volume-title":"Jean-Francc ois Bonnefon, and Iyad Rahwan","author":"Awad Edmond","year":"2018"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/843"},{"key":"e_1_2_2_6_1","volume-title":"Race after technology: Abolitionist tools for the new jim code. Social Forces","author":"Benjamin Ruha","year":"2019"},{"key":"e_1_2_2_7_1","volume-title":"Enough with the Trolley Problem. The Atlantic","author":"Bogost Ian","year":"2018"},{"key":"e_1_2_2_8_1","volume-title":"Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 966--974","author":"Bragg Jonathan","year":"2016"},{"key":"e_1_2_2_9_1","volume-title":"Unexpected Consequences of Self-Driving Cars. rodneybrooks.com","author":"Brooks Rodney","year":"2017"},{"key":"e_1_2_2_10_1","volume-title":"Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality data?","author":"Buhrmester Michael","year":"2016"},{"key":"e_1_2_2_11_1","volume-title":"Crowdsourcing Accurate and Creative Word Problems and Hints. AAAI HCOMP","author":"Chen Yvonne","year":"2016"},{"key":"e_1_2_2_12_1","volume-title":"COMPASS'91","author":"Kim Cheng Albert Mo","year":"1991"},{"key":"e_1_2_2_13_1","volume-title":"A Lyapunov-based Approach to Safe Reinforcement Learning. arXiv preprint arXiv:1805.07708","author":"Chow Yinlam","year":"2018"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.21606\/drs.2018.679"},{"key":"e_1_2_2_15_1","volume-title":"Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence. AAAI Press, 1153--1159","author":"Dai Peng","year":"2011"},{"key":"e_1_2_2_16_1","volume-title":"Safe Exploration in Continuous Action Spaces. arXiv preprint arXiv:1801.08757","author":"Dalal Gal","year":"2018"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1018033108"},{"key":"e_1_2_2_18_1","doi-asserted-by":"crossref","volume-title":"Introduction to Lattices and Order","author":"Davey B. A.","DOI":"10.1017\/CBO9780511809088"},{"key":"e_1_2_2_19_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"3","author":"Eon Greg","year":"2019"},{"key":"e_1_2_2_20_1","first-page":"1","article-title":"Trust but verify: A guide to algorithms and the law","volume":"31","author":"Desai Deven R","year":"2017","journal-title":"Harv. JL & Tech."},{"key":"e_1_2_2_21_1","series-title":"Harvard Business School Working Paper Series","volume-title":"Learning by Thinking: Overcoming Bias for Action through Reflection","author":"Stefano Giada Di","year":"2014"},{"key":"e_1_2_2_22_1","volume-title":"The moral machine is bad news for AI ethics. Mind Matters News","author":"Dixon Brendan","year":"2020"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_2_24_1","volume-title":"CARLA: An open urban driving simulator. arXiv preprint arXiv:1711.03938","author":"Dosovitskiy Alexey","year":"2017"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13270"},{"key":"e_1_2_2_26_1","volume-title":"Third AAAI Conference on Human Computation and Crowdsourcing.","author":"Gao Jie","year":"2015"},{"key":"e_1_2_2_27_1","volume-title":"Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.","author":"Gingold Yotam","year":"2012"},{"key":"e_1_2_2_28_1","unstructured":"Meghan Holohan. 2018. Her baby was stillborn but the ads just kept coming: One mother shares her pain. Today https:\/\/www.today.com\/parents\/gillian-brockell-s-open-letter-tech-companies-goes-viral-t145124.  Meghan Holohan. 2018. Her baby was stillborn but the ads just kept coming: One mother shares her pain. Today https:\/\/www.today.com\/parents\/gillian-brockell-s-open-letter-tech-companies-goes-viral-t145124."},{"key":"e_1_2_2_29_1","first-page":"1811","article-title":"LAW'S HALO AND THE MORAL MACHINE","volume":"119","author":"Huang Bert I","year":"2019","journal-title":"Columbia Law Review"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_2_2_31_1","volume-title":"Nature","volume":"583","author":"Kalluri Pratyusha","year":"2020"},{"key":"e_1_2_2_33_1","volume-title":"Dietary fats and health: dietary recommendations in the context of scientific evidence. Advances in nutrition","author":"Lawrence Glen D","year":"2013"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274373"},{"key":"e_1_2_2_35_1","volume-title":"Ai safety gridworlds. arXiv preprint arXiv:1711.09883","author":"Leike Jan","year":"2017"},{"key":"e_1_2_2_36_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"3","author":"Li Tianyi","year":"2019"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359297"},{"key":"e_1_2_2_38_1","volume-title":"Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods","author":"Mason Winter","year":"2012"},{"key":"e_1_2_2_39_1","volume-title":"Proceedings AAAI. 133--138","author":"McAllester David","year":"1993"},{"key":"e_1_2_2_40_1","volume-title":"The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review","author":"Miller George A","year":"1956"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702553"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359221"},{"key":"e_1_2_2_43_1","volume-title":"Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation","author":"Oleson David","year":"2011"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0890060402164043"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/3468.844354"},{"key":"e_1_2_2_46_1","volume-title":"The expanding circle: ethics and sociobiology","author":"Peter Singer","year":"1981"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1080\/07420520601085925"},{"key":"e_1_2_2_48_1","volume-title":"5th International symposium on imprecise probability: Theories and applications. 347--356","author":"Pfeifer Niki","year":"2007"},{"key":"e_1_2_2_49_1","volume-title":"Prisoner's dilemma: A study in conflict and cooperation","author":"Rapoport Anatol"},{"key":"e_1_2_2_50_1","volume-title":"Trial without Error: Towards Safe Reinforcement Learning via Human Intervention. arXiv preprint arXiv:1707.05173","author":"Saunders William","year":"2017"},{"key":"e_1_2_2_51_1","volume-title":"Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems","author":"Saunders William","year":"2018"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10539-006-9030-1"},{"key":"e_1_2_2_53_1","volume-title":"et almbox","author":"Sutton Richard S","year":"1998"},{"key":"e_1_2_2_54_1","volume-title":"Reward Constrained Policy Optimization. arXiv preprint arXiv:1805.11074","author":"Tessler Chen","year":"2018"},{"key":"e_1_2_2_55_1","volume-title":"Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill.","author":"Thomas Philip S","year":"2019"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359130"},{"key":"e_1_2_2_57_1","volume-title":"Adaptation and natural selection: A critique of some current evolutionary thought","author":"Williams George Christopher"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359245"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359158"},{"key":"e_1_2_2_60_1","unstructured":"Hankz Hankui Zhuo. 2015. Crowdsourced Action-Model Acquisition for Planning.. In AAAI. 3439--3446.  Hankz Hankui Zhuo. 2015. Crowdsourced Action-Model Acquisition for Planning.. In AAAI. 3439--3446."},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13752-013-0145-8"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415168","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415168","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415168","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:10Z","timestamp":1750197790000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415168"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,14]]},"references-count":60,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2020,10,14]]}},"alternative-id":["10.1145\/3415168"],"URL":"https:\/\/doi.org\/10.1145\/3415168","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,14]]},"assertion":[{"value":"2020-10-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}