{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T21:57:49Z","timestamp":1775080669765,"version":"3.50.1"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"1-2","license":[{"start":{"date-parts":[[2023,6,26]],"date-time":"2023-06-26T00:00:00Z","timestamp":1687737600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Trans. Soc. Comput."],"published-print":{"date-parts":[[2023,6,30]]},"abstract":"<jats:p>Perspective is a publicly available, machine learning API that can score text for toxicity. It is available for use in online platforms and communities to limit toxicity and promote civil dialogue. In this work, we adopt a human-centered approach to evaluating Perspective by investigating if human ratings of toxicity align with Perspective\u2019s toxicity scores. We also test its transferability by making this comparison for comments from three platforms that have different commenting styles and moderation strategies: news websites, YouTube, and Twitter. Apart from toxicity, the main attribute, we collect participant ratings for three additional attributes: respectfulness, formality, and presence of stereotypes. While disrespect is part of how Perspective defines toxicity, formality and presence of stereotypes were included in the study to explore if they could be hidden\/latent attributes that affect toxicity scores from Perspective. We analyzed how participant ratings for these additional attributes vary with respect to Perspective\u2019s toxicity score for comments from each platform. We find that for high toxicity scores, Perspective strongly aligns with participant ratings of toxicity and disrespectfulness across all three platforms, providing weak evidence of its transferability. However, our evaluation also surfaced formality and presence of stereotypes as latent attributes that are unrecognized parts of Perspective\u2019s scores. We discuss how and why this evaluation is \u201chuman-centered,\u201d the importance of conducting such evaluations, and implications of these results for content moderation in social platforms.<\/jats:p>","DOI":"10.1145\/3582568","type":"journal-article","created":{"date-parts":[[2023,2,17]],"date-time":"2023-02-17T11:59:17Z","timestamp":1676635157000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["A Human-centered Evaluation of a Toxicity Detection API: Testing Transferability and Unpacking Latent Attributes"],"prefix":"10.1145","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4593-8038","authenticated-orcid":false,"given":"Meena Devii","family":"Muralikumar","sequence":"first","affiliation":[{"name":"Human-centered Design &amp; Engineering, University of Washington"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2403-2659","authenticated-orcid":false,"given":"Yun Shan","family":"Yang","sequence":"additional","affiliation":[{"name":"Human-centered Design &amp; Engineering, University of Washington"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5882-828X","authenticated-orcid":false,"given":"David W.","family":"McDonald","sequence":"additional","affiliation":[{"name":"Human-centered Design &amp; Engineering, University of Washington"}]}],"member":"320","published-online":{"date-parts":[[2023,6,26]]},"reference":[{"key":"e_1_3_4_2_2","unstructured":"Perspective API. [n.d.]. About the API. Retrieved November 3 2022 from https:\/\/developers.perspectiveapi.com\/s\/about-the-api-training-data?language=en_US."},{"key":"e_1_3_4_3_2","unstructured":"Perspective API. [n.d.]. About the API\u2014Attributes and Languages. Retrieved April 8 2021 https:\/\/developers.perspectiveapi.com\/s\/about-the-api-attributes-and-languages."},{"key":"e_1_3_4_4_2","unstructured":"Perspective API. [n.d.]. About the API\u2014Best Practices & Risks. Retrieved April 2 2021 from https:\/\/developers.perspectiveapi.com\/s\/about-the-api-best-practices-risks."},{"key":"e_1_3_4_5_2","article-title":"About the API\u2013FAQs","author":"Perspective API.","unstructured":"Perspective API. [n.d.]. About the API\u2013FAQs. Retrieved April 2, 2021 from https:\/\/developers.perspectiveapi.com\/s\/about-th e-api-faqs.","journal-title":"https:\/\/developers.perspectiveapi.com\/s\/about-th e-api-faqs"},{"key":"e_1_3_4_6_2","article-title":"About the API\u2013Model Cards","author":"Perspective API.","unstructured":"Perspective API. [n.d.]. About the API\u2013Model Cards. Retrieved November 3, 2022 from https:\/\/developers.perspectiveapi.com\/s\/about-th e-api-model-cards?language=en_US&tabset-20254=2.","journal-title":"https:\/\/developers.perspectiveapi.com\/s\/about-th e-api-model-cards?language=en_US&tabset-20254=2"},{"key":"e_1_3_4_7_2","article-title":"Conversation AI","author":"GitHub.","unstructured":"GitHub. [n.d.]. Conversation AI. Retrieved from April 2, 2021 from https:\/\/github.com\/conversationai.","journal-title":"https:\/\/github.com\/conversationai"},{"key":"e_1_3_4_8_2","article-title":"Conversationai.github.io\/toxicity_with_subattributes.md at Main","author":"GitHub.","unstructured":"GitHub. [n.d.]. Conversationai.github.io\/toxicity_with_subattributes.md at Main. Retrieved April 2, 2021 from https:\/\/github.com\/conversationai\/conversationai .github.io\/blob\/main\/crowdsourcing_annotation_schemes\/toxicity_with_subattribu tes.md.","journal-title":"https:\/\/github.com\/conversationai\/conversationai .github.io\/blob\/main\/crowdsourcing_annotation_schemes\/toxicity_with_subattribu tes.md"},{"key":"e_1_3_4_9_2","unstructured":"[n.d.]. Developers. https:\/\/developers.perspectiveapi.com\/s\/?language=en_US. (Accessed: March 14 2023)."},{"key":"e_1_3_4_10_2","unstructured":"The Verge. [n.d.]. Facebook Is Now Using AI to Sort Content for Quicker Moderation. Retrieved September 7 2021 from https:\/\/www.theverge.com\/2020\/11\/13\/21562596\/facebook-ai-moderation."},{"key":"e_1_3_4_11_2","unstructured":"Wikipedia. [n.d.]. Gamergate Controversy. Retrieved April 14 2021 from https:\/\/en.wikipedia.org\/wiki\/Gamergate_controversy."},{"key":"e_1_3_4_12_2","unstructured":"Wikipedia. [n.d.]. Goldilocks Principle. Retrieved April 8 2021 from https:\/\/en.wikipedia.org\/w\/index.php?title=Goldilocks_principle&oldid=985929960."},{"key":"e_1_3_4_13_2","unstructured":"Jigsaw. [n.d.]. Hello Neighbor: How a Local Newspaper Builds Community via Online Comments. Retrieved May 5 2022 from https:\/\/medium.com\/jigsaw\/hello-neighbor-how-a-local-newspaper-builds-community-via-online-comments-c6a0bbfccb3b."},{"key":"e_1_3_4_14_2","unstructured":"[n.d.]. Interpreting Reliability Results. Retrieved April 8 2021 from https:\/\/homepages.inf.ed.ac.uk\/jeanc\/maptask-coding-html\/node23.html."},{"key":"e_1_3_4_15_2","unstructured":"Kaggle. [n.d.]. Jigsaw Unintended Bias in Toxicity Classification. Retrieved May 27 2022 from https:\/\/www.kaggle.com\/competitions\/jigsaw-unintended-bias-in-toxicity-classification\/data."},{"key":"e_1_3_4_16_2","unstructured":"OpenWeb. [n.d.]. Nudge Theory Examples In Online Discussions. Retrieved May 5 2022 from https:\/\/www.openweb.com\/blog\/openweb-improves-community-health-with-real-time-feedback-powered-by-jigsaws-perspective-api."},{"key":"e_1_3_4_17_2","unstructured":"Science. [n.d.]. Overzealous Profanity Filter Bans Paleontologists from Talking about Bones. Retrieved September 7 2021 from https:\/\/www.theguardian.com\/science\/2020\/oct\/16\/profanity-filter-bones-paleontologists-conference#::text=Participants%20in%20a%20virtual%20paleontology beaver%20%E2%80%93%20during%20an%20online%20conference."},{"key":"e_1_3_4_18_2","unstructured":"Perspective API. [n.d.]. Retrieved April 2 2021 from https:\/\/perspectiveapi.com\/#\/home."},{"key":"e_1_3_4_19_2","unstructured":"Perspective API. [n.d.]. Case Studies. Retrieved November 28 2022 from https:\/\/www.perspectiveapi.com\/case-studies\/."},{"key":"e_1_3_4_20_2","article-title":"Research into Machine Learning","author":"Perspective API.","unstructured":"Perspective API. [n.d.]. Research into Machine Learning. Retrieved November 3, 2022 from https:\/\/www.perspectiveapi.com\/research\/.","journal-title":"https:\/\/www.perspectiveapi.com\/research\/"},{"key":"e_1_3_4_21_2","article-title":"Reddit Ban Endangered Thousands of Lives (re: r\/ProED): TrueOffMyChest","author":"Reddit.","unstructured":"Reddit. [n.d.]. Reddit Ban Endangered Thousands of Lives (re: r\/ProED): TrueOffMyChest. Retrieved April 2, 2021 from https:\/\/www.reddit.com\/r\/TrueOffMyChest\/comments \/9xa1dt\/reddit_ban_endangered_thousands_of_lives_re_rproed\/.","journal-title":"https:\/\/www.reddit.com\/r\/TrueOffMyChest\/comments \/9xa1dt\/reddit_ban_endangered_thousands_of_lives_re_rproed\/"},{"key":"e_1_3_4_22_2","article-title":"Research: Detox\/Data Release","author":"Meta.","unstructured":"Meta. [n.d.]. Research: Detox\/Data Release. Retrieved November 18, 2022 from https:\/\/meta.wikimedia.org\/wiki\/Research:Detox\/D ata_Release.","journal-title":"https:\/\/meta.wikimedia.org\/wiki\/Research:Detox\/D ata_Release"},{"key":"e_1_3_4_23_2","article-title":"The State of Online Harassment","author":"Pew Research Center.","unstructured":"Pew Research Center. [n.d.]. The State of Online Harassment. Retrieved April 2, 2021 from https:\/\/www.pewresearch.org\/internet\/2021\/01\/13\/ the-state-of-online-harassment\/.","journal-title":"https:\/\/www.pewresearch.org\/internet\/2021\/01\/13\/ the-state-of-online-harassment\/"},{"key":"e_1_3_4_24_2","article-title":"The Times Sharply Increases Articles Open for Comments, Using Google\u2019s Technology","author":"The New York Times.","unstructured":"The New York Times. [n.d.]. The Times Sharply Increases Articles Open for Comments, Using Google\u2019s Technology. Retrieved April 12, 2021 from https:\/\/www.nytimes.com\/2017\/06\/13\/insider\/have- a-comment-leave-a-comment.html#commentsContainer.","journal-title":"https:\/\/www.nytimes.com\/2017\/06\/13\/insider\/have- a-comment-leave-a-comment.html#commentsContainer"},{"key":"e_1_3_4_25_2","article-title":"Tumblr\u2019s New Anti-Porn Algorithm Is Flagging Non-Pornographic Content","author":"Buzzfeed.","unstructured":"Buzzfeed. [n.d.]. Tumblr\u2019s New Anti-Porn Algorithm Is Flagging Non-Pornographic Content. Retrieved April 4, 2021 from https:\/\/www.buzzfeednews.com\/article\/krishrach\/t umblr-porn-algorithm-ban.","journal-title":"https:\/\/www.buzzfeednews.com\/article\/krishrach\/t umblr-porn-algorithm-ban"},{"key":"e_1_3_4_26_2","article-title":"Tune (Experimental)\u2014Chrome Web Store","author":"Google.","unstructured":"Google. [n.d.]. Tune (Experimental)\u2014Chrome Web Store. Retrieved May 5, 2022 from https:\/\/chrome.google.com\/webstore\/detail\/tune-e xperimental\/gdfknffdmmjakmlikbpdngpcpbbfhbnp?hl=en.","journal-title":"https:\/\/chrome.google.com\/webstore\/detail\/tune-e xperimental\/gdfknffdmmjakmlikbpdngpcpbbfhbnp?hl=en"},{"key":"e_1_3_4_27_2","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1145\/3331184.3331262","volume-title":"Proceedings of the 42nd International acm Sigir Conference on Research and Development in Information Retrieval","author":"Arango Aym\u00e9","year":"2019","unstructured":"Aym\u00e9 Arango, Jorge P\u00e9rez, and Barbara Poblete. 2019. Hate speech detection is not as easy as you may think: A closer look at model validation. In Proceedings of the 42nd International acm Sigir Conference on Research and Development in Information Retrieval. 45\u201354."},{"key":"e_1_3_4_28_2","first-page":"1100","volume-title":"Companion Proceedings of the World Wide Web Conference","author":"Aroyo Lora","year":"2019","unstructured":"Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing subjective tasks: the case study of understanding toxicity in online discussions. In Companion Proceedings of the World Wide Web Conference. 1100\u20131105."},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3134659"},{"key":"e_1_3_4_30_2","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1145\/3308560.3317593","volume-title":"Companion Proceedings of the World Wide Web Conference (WWW\u201919)","author":"Borkan Daniel","year":"2019","unstructured":"Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of the World Wide Web Conference (WWW\u201919). Association for Computing Machinery, New York, NY, 491\u2013500. 10.1145\/3308560.3317593"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447535.3462484"},{"key":"e_1_3_4_32_2","first-page":"1201","volume-title":"Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing","author":"Chancellor Stevie","year":"2016","unstructured":"Stevie Chancellor, Jessica Annette Pater, Trustin Clear, Eric Gilbert, and Munmun De Choudhury. 2016. # thyghgapp: Instagram content moderation and lexical variation in pro-eating disorder communities. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 1201\u20131213."},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359276"},{"key":"e_1_3_4_34_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3134666","article-title":"You can\u2019t stay here: The efficacy of reddit\u2019s 2015 ban examined through hate speech","volume":"1","author":"Chandrasekharan Eshwar","year":"2017","unstructured":"Eshwar Chandrasekharan, Umashanthi Pavalanathan, Anirudh Srinivasan, Adam Glynn, Jacob Eisenstein, and Eric Gilbert. 2017. You can\u2019t stay here: The efficacy of reddit\u2019s 2015 ban examined through hate speech. Proc. ACM Hum.-Comput. Interact. 1, (2017), 1\u201322.","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3274301"},{"key":"e_1_3_4_36_2","first-page":"3175","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"Chandrasekharan Eshwar","year":"2017","unstructured":"Eshwar Chandrasekharan, Mattia Samory, Anirudh Srinivasan, and Eric Gilbert. 2017. The bag of communities: Identifying abusive behavior online with preexisting internet data. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 3175\u20133187."},{"key":"e_1_3_4_37_2","first-page":"1","volume-title":"Proceedings of the Chi Conference on Human Factors in Computing Systems","author":"D\u00edaz Mark","year":"2018","unstructured":"Mark D\u00edaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018. Addressing age-related bias in sentiment analysis. In Proceedings of the Chi Conference on Human Factors in Computing Systems. 1\u201314."},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278729"},{"issue":"1","key":"e_1_3_4_39_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3392845","article-title":"Conformity of eating disorders through content moderation","volume":"4","author":"Feuston Jessica L.","year":"2020","unstructured":"Jessica L. Feuston, Alex S. Taylor, and Anne Marie Piper. 2020. Conformity of eating disorders through content moderation. Proc. ACM Hum.-Comput. Interact. 4, 1 (2020), 1\u201328.","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/230538.230561"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3580494"},{"key":"e_1_3_4_42_2","volume-title":"Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media","author":"Gillespie Tarleton","year":"2018","unstructured":"Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media. Yale University Press."},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1177\/2053951719897945"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3270101.3270103"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3134687"},{"key":"e_1_3_4_46_2","article-title":"Deceiving google\u2019s perspective api built for detecting toxic comments","author":"Hosseini Hossein","year":"2017","unstructured":"Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google\u2019s perspective api built for detecting toxic comments. arXiv:1702.08138. Retrieved https:\/\/arxiv.org\/abs\/1702.08138.","journal-title":"arXiv:1702.08138"},{"key":"e_1_3_4_47_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"Huffaker Jordan S.","year":"2020","unstructured":"Jordan S. Huffaker, Jonathan K. Kummerfeld, Walter S. Lasecki, and Mark S. Ackerman. 2020. Crowdsourced detection of emotionally manipulative language. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1\u201314."},{"key":"e_1_3_4_48_2","first-page":"287","volume-title":"Conference Companion Publication of the Computer Supported Cooperative Work and Social Computing (CSCW\u201920 Companion)","author":"Jiang Jialun \u201cAaron\u201d","year":"2020","unstructured":"Jialun \u201cAaron\u201d Jiang, Skyler Middler, Jed R. Brubaker, and Casey Fiesler. 2020. Characterizing community guidelines on social media platforms. In Conference Companion Publication of the Computer Supported Cooperative Work and Social Computing (CSCW\u201920 Companion). Association for Computing Machinery, New York, NY. 287\u2013291."},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3375197"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.58.9.697"},{"key":"e_1_3_4_51_2","doi-asserted-by":"crossref","first-page":"347","DOI":"10.2307\/2669316","article-title":"Making the most of statistical analyses: Improving interpretation and presentation","author":"King Gary","year":"2000","unstructured":"Gary King, Michael Tomz, and Jason Wittenberg. 2000. Making the most of statistical analyses: Improving interpretation and presentation. Am. J. Pol. Sci. (2000), 347\u2013361.","journal-title":"Am. J. Pol. Sci."},{"key":"e_1_3_4_52_2","first-page":"299","volume-title":"Proceedings of the 17th Symposium on Usable Privacy and Security (SOUPS\u201921)","author":"Kumar Deepak","year":"2021","unstructured":"Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Proceedings of the 17th Symposium on Usable Privacy and Security (SOUPS\u201921). 299\u2013318."},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"issue":"3","key":"e_1_3_4_54_2","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1145\/3236386.3241340","article-title":"The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.","volume":"16","author":"Lipton Zachary C.","year":"2018","unstructured":"Zachary C. Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.Queue 16, 3 (2018), 31\u201357.","journal-title":"Queue"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702553"},{"key":"e_1_3_4_57_2","article-title":"Reducing gender bias in abusive language detection","author":"Park Ji Ho","year":"2018","unstructured":"Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. arXiv:1808.07231 (2018).","journal-title":"arXiv:1808.07231"},{"key":"e_1_3_4_58_2","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1145\/2957276.2957297","volume-title":"Proceedings of the 19th International Conference on Supporting Group Work","author":"Pater Jessica A.","year":"2016","unstructured":"Jessica A. Pater, Moon K. Kim, Elizabeth D. Mynatt, and Casey Fiesler. 2016. Characterizations of online harassment: Comparing policies across social media platforms. In Proceedings of the 19th International Conference on Supporting Group Work. 369\u2013374."},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.396"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1177\/20539517211046181"},{"key":"e_1_3_4_61_2","first-page":"41","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"R\u00f6ttger Paul","year":"2021","unstructured":"Paul R\u00f6ttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. Hatecheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 41\u201358. 10.18653\/v1\/2021.acl-long.4"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287598"},{"key":"e_1_3_4_63_2","volume-title":"The Design of Sites: Patterns for Creating Winning Web Sites","author":"Duyne Douglas K. Van","year":"2007","unstructured":"Douglas K. Van Duyne, James A. Landay, and Jason I. Hong. 2007. The Design of Sites: Patterns for Creating Winning Web Sites. Prentice Hall Professional."},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.woah-1.15"}],"container-title":["ACM Transactions on Social Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3582568","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3582568","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:47:16Z","timestamp":1750178836000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3582568"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,26]]},"references-count":63,"journal-issue":{"issue":"1-2","published-print":{"date-parts":[[2023,6,30]]}},"alternative-id":["10.1145\/3582568"],"URL":"https:\/\/doi.org\/10.1145\/3582568","relation":{},"ISSN":["2469-7818","2469-7826"],"issn-type":[{"value":"2469-7818","type":"print"},{"value":"2469-7826","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,26]]},"assertion":[{"value":"2021-12-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-12","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}