{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T21:32:41Z","timestamp":1770327161456,"version":"3.49.0"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Online experimentation with volunteers relies on participants' non-financial motivations to complete a study, such as to altruistically support science or to compare oneself to others. Researchers rely on these motivations to attract study participants and often use incentives, like performance comparisons, to encourage participation. Often, these study incentives are advertised using a slogan (e.g., \"What is your thinking style?''). Research on framing effects suggests that advertisement slogans attract people with varying demographics and motivations. Could the slogan advertisements for studies risk attracting only specific users? To investigate the existence of potential sample biases, we measured how different slogan frames affected which participants self-selected into studies. We found that slogan frames impact recruitment significantly; changing the slogan frame from a 'supporting science' frame to a 'comparing oneself to others' frame lead to a 9% increase in recruitment for some studies. Additionally, slogans framed as learning more about oneself attract participants significantly more motivated by boredom compared to other slogan frames. We discuss design implications for using frames to improve recruitment and mitigate sources of sample bias in online research with volunteers.<\/jats:p>","DOI":"10.1145\/3274291","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Framing Effects"],"prefix":"10.1145","volume":"2","author":[{"given":"Tal","family":"August","sequence":"first","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"Nigini","family":"Oliveira","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"Chenhao","family":"Tan","sequence":"additional","affiliation":[{"name":"University of Colorado Boulder, Boulder, CO, USA"}]},{"given":"Noah","family":"Smith","sequence":"additional","affiliation":[{"name":"University of Washington &amp; Allen Institute for Artificial Intelligence, Seattle, WA, USA"}]},{"given":"Katharina","family":"Reinecke","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1177\/1088868309341564"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208699"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tmrv.2011.04.005"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1509\/jmr.10.0353"},{"key":"e_1_2_1_6_1","volume-title":"Lenz","author":"Berinsky Adam J.","year":"2012","unstructured":"Adam J. Berinsky, Gregory A. Huber, and Gabriel S. Lenz. 2012. Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk.PoliticalAnalysis20,3(2012),351--368."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature11421"},{"key":"e_1_2_1_8_1","unstructured":"Amber E Boydstun Dallas Card Justin Gross Philip Resnik and Noah A Smith. 2014. Tracking the development of media frames within and across policy issues. (2014)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1086\/374397"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1103343108"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-2072"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1559-1816.2009.00555.x"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.polisci.10.072805.103054"},{"key":"e_1_2_1_14_1","volume-title":"Motivation to Participate in an Online Citizen Science Game: A Study of Foldit. Science Communi- cation 37, 6 (Dec","author":"Curtis Vickie","year":"2015","unstructured":"Vickie Curtis. 2015. Motivation to Participate in an Online Citizen Science Game: A Study of Foldit. Science Communi- cation 37, 6 (Dec 2015), 723--746."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.ps.41.020190.002221"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1460-2466.1993.tb01304.x"},{"key":"e_1_2_1_17_1","first-page":"3","article-title":"The distribution of the partial correlation coe cient","volume":"3","author":"Fisher Ronald Aylmer","year":"1924","unstructured":"Ronald Aylmer Fisher et al. 1924. The distribution of the partial correlation coe cient. Metron 3, 3--4 (1924), 329--332.","journal-title":"Metron"},{"key":"e_1_2_1_18_1","volume-title":"Measures of e ect size for categorical data. The handbook of research synthesis","author":"Fleiss Joseph L","year":"1994","unstructured":"Joseph L Fleiss. 1994. Measures of e ect size for categorical data. The handbook of research synthesis (1994), 245--260."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-012-0296-9"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1093\/iwc\/iwt035"},{"key":"e_1_2_1_21_1","first-page":"58","article-title":"Incentives in web studies: Methodological issues and a review","volume":"1","author":"Goritz Anja S","year":"2006","unstructured":"Anja S Goritz. 2006. Incentives in web studies: Methodological issues and a review. International Journal of Internet Science 1, 1 (2006), 58--70. http:\/\/www.ijis.net\/ijis1_1\/ijis1_1_goeritz.pdf","journal-title":"International Journal of Internet Science"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1111\/0022-3816.00077"},{"key":"e_1_2_1_23_1","unstructured":"Joshua Hartshorne. 2018. Games With Words. https:\/\/www.gameswithwords.org\/"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1287\/mksc.1080.0459"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1080\/23808985.2009.11679084"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819936"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2929.2004.02012.x"},{"key":"e_1_2_1_28_1","unstructured":"Eric Jones Travis Oliphant Pearu Peterson et al. 2001--. SciPy: Open source scienti c tools for Python. http:\/\/www.scipy.org\/ Online."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134691"},{"key":"e_1_2_1_30_1","first-page":"2009","volume-title":"Proceedings of the Seventeenth Americas Conference on Information Systems 4","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowd-sourcing - A Study on Mechanical Turk. Proceedings of the Seventeenth Americas Conference on Information Systems 4, 2009 (2011), 1--11."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1101\/lm.1.2.106"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.22323\/2.17010202"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1006\/obhd.2001.2983"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1086\/209174"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3234695.3236360"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2124295.2124356"},{"key":"e_1_2_1_37_1","first-page":"132","article-title":"Donating context data to science: The e ects of social signals and perceptions on action-taking","volume":"29","author":"Liu Yong","year":"2017","unstructured":"Yong Liu, Denzil Ferreira, Jorge Goncalves, Simo Hosio, Pratyush Pandab, and Vassilis Kostakos. 2017. Donating context data to science: The e ects of social signals and perceptions on action-taking. Interacting with Computers 29, 2 (2017), 132--146.","journal-title":"Interacting with Computers"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0124-6"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10459-010-9222-y"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1940761.1940771"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0090375"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1093\/swr\/31.2.109"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2929.2005.02237.x"},{"key":"e_1_2_1_45_1","unstructured":"ProjectImplicit. 2011. ProjectImplicit. https:\/\/www.projectimplicit.net\/index.html"},{"key":"e_1_2_1_46_1","volume-title":"Galaxy Zoo: Motivations of citizen scientists. arXiv preprint arXiv:1303.6886","author":"Raddick M Jordan","year":"2013","unstructured":"M Jordan Raddick, Georgia Bracey, Pamela L Gay, Chris J Lintott, Carie Cardamone, Phil Murray, Kevin Schawinski, Alexander S Szalay, and Jan Vandenberg. 2013. Galaxy Zoo: Motivations of citizen scientists. arXiv preprint arXiv:1303.6886 (2013)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675246"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1080\/03637750701397098"},{"key":"e_1_2_1_49_1","volume-title":"Current Internet Science Trends, Techniques, Results","author":"Reips UD","unstructured":"UD Reips, B Batinic, W Bandilla, M Bosnjak, and L Graf. 1999. Financial incentives personal information and drop-out rate in online studies. In Current Internet Science Trends, Techniques, Results. Deutsche Gesellschaft fur Online-Forschung."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_2_1_51_1","volume-title":"Christopher R Johnson, and Jan Sche czyk.","author":"Ruppenhofer Josef","year":"2016","unstructured":"Josef Ruppenhofer, Michael Ellsworth, Miriam RL Petruck, Christopher R Johnson, and Jan Sche czyk. 2016. FrameNet II: Extended theory and practice. Institut fur Deutsche Sprache, Bibliothek."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 9th Python in Science Conference.","author":"Seabold J.S.","unstructured":"J.S. Seabold and J. Perktold. 2010. Statsmodels: Econometric and Statistical Modeling with Python. In Proceedings of the 9th Python in Science Conference."},{"key":"e_1_2_1_53_1","volume-title":"Robust and Optimal Ranking from Pairwise Comparisons. arXiv","author":"Shah Nihar B","year":"2015","unstructured":"Nihar B Shah and Martin J Wainwright. 2015. Simple, Robust and Optimal Ranking from Pairwise Comparisons. arXiv (2015), 1--17. arXiv:1512.08949 https:\/\/arxiv.org\/pdf\/1512.08949.pdfhttp:\/\/arxiv.org\/abs\/1512.08949"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1177\/0146167290163001"},{"key":"e_1_2_1_55_1","volume-title":"Public Opinion: Attitudes, onattitudes, Measurement Error, and Change (jan","author":"Sniderman P.M.","year":"2004","unstructured":"P.M. Sniderman and S.M. Theriault. 2004. The Structure of Political Argument and the Logic of Issue Framing. Studies in Public Opinion: Attitudes, onattitudes, Measurement Error, and Change (jan 2004), 133--164."},{"key":"e_1_2_1_56_1","unstructured":"Keiran Snyder. 2017. 1000 di erent people the same words - Textio Word Nerd. https:\/\/textio.ai\/1000-di erent-people-the-same-words-6149b5a1f351"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.4300\/JGME-D-12-00156.1"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL'14)","author":"Tan Chenhao","year":"2014","unstructured":"Chenhao Tan, Lillian Lee, and Bo Pang. 2014. The e ect of wording on message propagation: Topic-and author- controlled natural experiments on Twitter. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL'14) (2014)."},{"key":"e_1_2_1_59_1","unstructured":"TestMyBrain. 2017. TestMyBrain. http:\/\/dx.plos.org\/10.1371\/journal.pone.0165100"},{"key":"e_1_2_1_60_1","volume-title":"The framing of decisions and the psychology of choice. Science 211, 4481","author":"Tversky Amos","year":"1981","unstructured":"Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. Science 211, 4481 (1981), 453--458."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1287\/mksc.2013.0803"},{"key":"e_1_2_1_62_1","unstructured":"VolunteerScience. 2018. Volunteer Science. https:\/\/volunteerscience.com\/"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2010.04.011"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02506983"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274291","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274291","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:06Z","timestamp":1750207446000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274291"}},"subtitle":["Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases"],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":63,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274291"],"URL":"https:\/\/doi.org\/10.1145\/3274291","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}