{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:03:21Z","timestamp":1776107001652,"version":"3.50.1"},"reference-count":102,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T00:00:00Z","timestamp":1634083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"University of Minnesota Social Media & Business Analytics Collaborative"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,13]]},"abstract":"<jats:p>Online technologies offer great promise to expand models of delivery for therapeutic interventions to help users cope with increasingly common mental illnesses like anxiety and depression. For example, \"cognitive reappraisal\" is a skill that involves changing one's perspective on negative thoughts in order to improve one's emotional state. In this work, we present Flip*Doubt, a novel crowd-powered web application that provides users with cognitive reappraisals (\"reframes\") of negative thoughts. A one-month field deployment of Flip*Doubt with 13 graduate students yielded a data set of negative thoughts paired with positive reframes, as well as rich interview data about how participants interacted with the system. Through this deployment, our work contributes: (1) an in-depth qualitative understanding of how participants used a crowd-powered cognitive reappraisal system in the wild; and (2) detailed codebooks that capture informative context about negative input thoughts and reframes. Our results surface data-derived hypotheses that may help to explain what types of reframes are helpful for users, while also providing guidance to future researchers and developers interested in building collaborative systems for mental health. In our discussion, we outline implications for systems research to leverage peer training and support, as well as opportunities to integrate AI\/ML-based algorithms to support the cognitive reappraisal task. (Note: This paper includes potentially triggering mentions of mental health issues and suicide.)<\/jats:p>","DOI":"10.1145\/3479561","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T02:39:17Z","timestamp":1634611157000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Effective Strategies for Crowd-Powered Cognitive Reappraisal Systems: A Field Deployment of the Flip*Doubt Web Application for Mental Health"],"prefix":"10.1145","volume":"5","author":[{"given":"C. Estelle","family":"Smith","sequence":"first","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"given":"William","family":"Lane","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"given":"Hannah","family":"Miller Hillberg","sequence":"additional","affiliation":[{"name":"University of Wisconsin-Oshkosh, Oshkosh, WI, USA"}]},{"given":"Daniel","family":"Kluver","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"given":"Loren","family":"Terveen","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"given":"Svetlana","family":"Yarosh","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12888-016--1104--2"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1177\/1460458219896492"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02247-0_24"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.2196\/21819"},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Reout Arbel Marlyn Khouri Jasmine Sagi and Noga Cohen. 2020. Reappraising Others' Negative Emotions as a way to Enhance Coping during the COVID-19 Outbreak. Technical Report. PsyArXiv. https:\/\/doi.org\/10.31234\/osf.io\/y25gx type: article.","DOI":"10.31234\/osf.io\/y25gx"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.2196\/14392"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450122"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.5129"},{"key":"e_1_2_1_9_1","unstructured":"Jennifer Beckett. 2018. We need to talk about the mental health of content moderators. http:\/\/theconversation.com\/we-need-to-talk-about-the-mental-health-of-content-moderators-103830"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0005--7967(03)00174--8"},{"key":"e_1_2_1_12_1","volume-title":"Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Neural Information Processing Systems (NIPS)","author":"Bolukbasi Tolga","year":"2016","unstructured":"Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Neural Information Processing Systems (NIPS) (2016), 9. https:\/\/papers.nips.cc\/paper\/2016\/file\/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf"},{"key":"e_1_2_1_13_1","unstructured":"Jonaki Bose. 2017. Key Substance Use and Mental Health Indicators in the United States: Results from the 2017 National Survey on Drug Use and Health. (2017) 124."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-019-05364--1"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2014.07.006"},{"key":"e_1_2_1_16_1","volume-title":"Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Conference on Fairness, Accountability and Transparency. PMLR, 77--91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Conference on Fairness, Accountability and Transparency. PMLR, 77--91. http:\/\/proceedings.mlr.press\/v81\/buolamwini18a.html ISSN: 2640--3498."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287587"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858246"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359276"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376341"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1080\/j.1440--1614.2006.01743.x"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642713"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797618779971"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1177\/0146167217695558"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-019-03489--1"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000009"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.4089"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3370271"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media","volume":"14","author":"Fiesler Casey","year":"2020","unstructured":"Casey Fiesler, Nathan Beard, and Brian C. Keegan. 2020. No Robots, Spiders, or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social Media Terms of Service. Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14 (May 2020), 187--196. https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/7290"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173627"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.2196\/mental.7785"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.1053"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.paid.2005.12.009"},{"key":"e_1_2_1_34_1","volume-title":"Open algorithmic systems: lessons on opening the black box from Wikipedia. AoIR Selected Papers of Internet Research (Oct","author":"Stuart Geiger R.","year":"2016","unstructured":"R. Stuart Geiger and Aaron Halfaker. 2016. Open algorithmic systems: lessons on opening the black box from Wikipedia. AoIR Selected Papers of Internet Research (Oct. 2016). https:\/\/spir.aoir.org\/ojs\/index.php\/spir\/article\/view\/8772"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamapsychiatry.2020.1011"},{"key":"e_1_2_1_36_1","volume-title":"Emotion Regulation: Conceptual Foundations. Handbook of Emotion Regulation","author":"Gross J.J.","year":"2007","unstructured":"J.J. Gross and Ross Thompson. 2007. Emotion Regulation: Conceptual Foundations. Handbook of Emotion Regulation (2007), 3--27."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022--3514.74.1.224"},{"key":"e_1_2_1_38_1","volume-title":"ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia. arXiv:1909.05189 [cs] (Aug.","author":"Halfaker Aaron","year":"2020","unstructured":"Aaron Halfaker and R. Stuart Geiger. 2020. ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia. arXiv:1909.05189 [cs] (Aug. 2020). http:\/\/arxiv.org\/abs\/1909.05189 arXiv: 1909.05189."},{"key":"e_1_2_1_39_1","unstructured":"Department of Health Education & Welfare. 1979. The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research. Technical Report. Office of Human Research Protections. https:\/\/www.hhs.gov\/ohrp\/regulations-and-policy\/belmont-report\/read-the-belmont-report\/index.html Last Modified: 2018-01--15 00:00:00."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1006\/obhd.1996.0077"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1177\/0963721416686212"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338243"},{"key":"e_1_2_1_43_1","unstructured":"Khari Johnson. 2020. AI ethics pioneer's exit from Google involved research into risks and inequality in large language models. https:\/\/venturebeat.com\/2020\/12\/03\/ai-ethics-pioneers-exit-from-google-involved-research-into-risks-and-inequality-in-large-language-models\/"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393527"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1460563.1460639"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3--319--90092--6_10"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2365952.2365974"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1046\/j.1525--1497.2001.016009606.x"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1177\/0963721411408883"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1520340.1520646"},{"key":"e_1_2_1_51_1","volume-title":"Saumik Narayanan, Loren Terveen, and Svetlana Yarosh.","author":"Levonian Zachary","year":"2020","unstructured":"Zachary Levonian, Marco Dow, Drew Erikson, Sourojit Ghosh, Hannah Miller Hillberg, Saumik Narayanan, Loren Terveen, and Svetlana Yarosh. 2020. Patterns of Patient and Caregiver Mutual Support Connections in an Online Health Community. arXiv:2007.16172 [cs] (Sept. 2020). http:\/\/arxiv.org\/abs\/2007.16172 arXiv: 2007.16172."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025796"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0026351"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1177\/1368430207088035"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.genhosppsych.2013.03.008"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.6645"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1037\/tps0000168"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.2196\/10148"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1080\/17439760.2014.913671"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.4167"},{"key":"e_1_2_1_61_1","volume-title":"Crowdsourcing mental health and emotional well-being. Thesis","author":"Robert Robert","unstructured":"Robert (Robert Randall) Morris. 2015. Crowdsourcing mental health and emotional well-being. Thesis. Massachusetts Institute of Technology. http:\/\/dspace.mit.edu\/handle\/1721.1\/97972"},{"key":"e_1_2_1_62_1","unstructured":"Casey Newton. 2020. Half of all Facebook moderators may develop mental health issues. https:\/\/www.theverge.com\/interface\/2020\/5\/13\/21255994\/facebook-content-moderator-lawsuit-settlement-mental-health-issues"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2507157.2507188"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173987"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1176\/appi.prcp.20200029"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1037\/emo0000636"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998349"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173905"},{"key":"e_1_2_1_69_1","volume-title":"Olson and Wendy Kellogg (Eds.)","author":"Judith","year":"2014","unstructured":"Judith S. Olson and Wendy Kellogg (Eds.). 2014. Ways of knowing in HCI. Springer, New York. OCLC: ocn879418401."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1177\/107319110100800409"},{"key":"e_1_2_1_71_1","doi-asserted-by":"crossref","unstructured":"Pablo Paredes Ran Giald-Bachrach Mary Czerwinski Asta Roseway Kael Rowan and Javier Hernandez. 2014. PopTherapy: Coping with Stress through Pop-Culture. https:\/\/eudl.eu\/doi\/10.4108\/icst.pervasivehealth.2014.255070","DOI":"10.4108\/icst.pervasivehealth.2014.255070"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.amepre.2016.05.001"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376695"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1108\/MHRJ-12--2019-0048"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1305706110"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1080\/03069880902956975"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1080\/09515070.2012.703129"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpr.2006.05.002"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366815"},{"key":"e_1_2_1_80_1","volume-title":"Foy","author":"Rothbaum Barbara Olasov","year":"2000","unstructured":"Barbara Olasov Rothbaum, Elizabeth A. Meadows, Patricia Resick, and David W. Foy. 2000. Cognitive-behavioral therapy. In Effective treatments for PTSD: Practice guidelines from the International Society for Traumatic Stress Studies. Guilford Press, New York, NY, US, 320--325."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998246"},{"key":"e_1_2_1_82_1","doi-asserted-by":"crossref","unstructured":"Sabirat Rubya and Svetlana Yarosh. 2021. Comparing Generic and Community-Situated Crowdsourcing for Data Validation in the Context of Recovery from Substance Use Disorders. In CHI.","DOI":"10.1145\/3411764.3445399"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437259"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1086\/651235"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173972"},{"key":"e_1_2_1_86_1","volume-title":"Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach. arXiv:2101.07714 [cs] (Jan","author":"Sharma Ashish","year":"2021","unstructured":"Ashish Sharma, Inna W. Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2021. Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach. arXiv:2101.07714 [cs] (Jan. 2021). http:\/\/arxiv.org\/abs\/2101.07714 arXiv: 2101.07714."},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1177\/0146167207305537"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449117"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/3402855"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376744"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376783"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1001\/archinte.166.10.1092"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1145\/3398069"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1136\/eb-2018--102891"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797613496434"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0020262"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","unstructured":"Jim Warren Sarah Hopkins Andy Leung Sarah Hetrick and Sally Merry. 2020. Building a Digital Platform for Behavioral Intervention Technology Research and Deployment. https:\/\/doi.org\/10.24251\/HICSS.2020.414 Accepted: 2020-01-04T07:51:24Z.","DOI":"10.24251\/HICSS.2020.414"},{"key":"e_1_2_1_98_1","volume-title":"Understanding Abuse: A Typology of Abusive Language Detection Subtasks. (May","author":"Waseem Zeerak","year":"2017","unstructured":"Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. (May 2017). https:\/\/arxiv.org\/abs\/1705.09899v2"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3052764"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0033839"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274463"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-010--9227--5"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479561","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3479561","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:59:04Z","timestamp":1752469144000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479561"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":102,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2021,10,13]]}},"alternative-id":["10.1145\/3479561"],"URL":"https:\/\/doi.org\/10.1145\/3479561","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,13]]},"assertion":[{"value":"2021-10-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}