{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,18]],"date-time":"2026-10-18T02:24:36Z","timestamp":1792290276275,"version":"4.4.0"},"reference-count":139,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW1","license":[{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"NSF","award":["IIS-1651969"],"award-info":[{"award-number":["IIS-1651969"]}]},{"DOI":"10.13039\/100030807","name":"Virginia Commonwealth Cyber Initiative","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100030807","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,4,17]]},"abstract":"<jats:p>Open Source Intelligence (OSINT) investigations, which rely entirely on publicly available data such as social media, play an increasingly important role in solving crimes and holding governments accountable. The growing volume of data and complex nature of tasks, however, means there is a pressing need to scale and speed up OSINT investigations. Expert-led crowdsourcing approaches show promise, but tend to either focus on narrow tasks or domains, or require resource-intense, long-term relationships between expert investigators and crowds. We address this gap by providing a flexible framework that enables investigators across domains to enlist crowdsourced support for discovery and verification of OSINT. We use a design-based research (DBR) approach to develop OSINT Research Studios (ORS), a sociotechnical system in which novice crowds are trained to support professional investigators with complex OSINT investigations. Through our qualitative evaluation, we found that ORS facilitates ethical and effective OSINT investigations across multiple domains. We also discuss broader implications of expert-crowd collaboration and opportunities for future work.<\/jats:p>","DOI":"10.1145\/3637382","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T10:05:31Z","timestamp":1714385131000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence Investigations"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-0925-1084","authenticated-orcid":false,"given":"Anirban","family":"Mukhopadhyay","sequence":"first","affiliation":[{"name":"Department of Computer Science, Virginia Tech, Blacksburg, VA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3888-7693","authenticated-orcid":false,"given":"Sukrit","family":"Venkatagiri","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Swarthmore College, Swarthmore, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1809-6269","authenticated-orcid":false,"given":"Kurt","family":"Luther","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Virginia Tech, Arlington, VA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,4,26]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2015. A Call to Arms: Open Source Intelligence and Evidence Based Policymaking. https:\/\/www.bellingcat.com\/resources\/articles\/2015\/01\/20\/a-call-to-arms-open-source-intelligence-and-evidence-based-policymaking\/"},{"key":"e_1_2_1_2_1","unstructured":"2021. J298 OSINT Seminar - Open Source Investigations. https:\/\/journalism.berkeley.edu\/course-section\/j298-humanrights-center-seminar-f21\/"},{"key":"e_1_2_1_3_1","unstructured":"2021. WhatsApp can be a black box of misinformation but Maldita may have opened a window. https:\/\/www.poynter.org\/fact-checking\/2021\/whatsapp-can-be-a-black-box-of-misinformation-but-malditamay-have-opened-a-window\/"},{"key":"e_1_2_1_4_1","unstructured":"2022. Hunchly - OSINT Software for Cybersecurity Law Enforcement Journalists Private Investigators and more. https:\/\/www.hunch.ly\/"},{"key":"e_1_2_1_5_1","unstructured":"2022. Russia is losing so much equipment in Ukraine that weapons monitors can't keep up. https:\/\/www.independent. co.uk\/news\/world\/europe\/russia-ukraine-military-equipment-losses-b2049613.html Section: News."},{"key":"e_1_2_1_6_1","unstructured":"2022. TraceLabs Twitter Account. https:\/\/twitter.com\/tracelabs\/status\/1558831625986777088"},{"key":"e_1_2_1_7_1","unstructured":"2023. About Bellingcat. https:\/\/www.bellingcat.com\/about\/"},{"key":"e_1_2_1_8_1","unstructured":"2023. archive.is. https:\/\/archive.is\/"},{"key":"e_1_2_1_9_1","unstructured":"2023. Bot Sentinel - Dashboard. https:\/\/botsentinel.com\/"},{"key":"e_1_2_1_10_1","unstructured":"2023. Certified in Open Source Intelligence (C|OSINT) from McAfee Institute | NICCS. https:\/\/niccs.cisa.gov\/educationtraining\/ catalog\/mcafee-institute\/certified-open-source-intelligence-cosint"},{"key":"e_1_2_1_11_1","unstructured":"2023. Check (@checkdesk) \/ Twitter. https:\/\/twitter.com\/checkdesk"},{"key":"e_1_2_1_12_1","unstructured":"2023. EXIF Data Viewer. https:\/\/exifdata.com\/"},{"key":"e_1_2_1_13_1","unstructured":"2023. Google Images. https:\/\/images.google.com\/"},{"key":"e_1_2_1_14_1","unstructured":"2023. Join us in pushing back on misinformation. https:\/\/our.news\/"},{"key":"e_1_2_1_15_1","unstructured":"2023. Newsgathering and Monitoring on the Social Web. https:\/\/firstdraftnews.org:443\/long-form-article\/newsgathering-and-monitoring-on-the-social-web\/"},{"key":"e_1_2_1_16_1","unstructured":"2023. OSINT: Open-Source Intelligence. https:\/\/www.udemy.com\/course\/osint-open-source-intelligence\/"},{"key":"e_1_2_1_17_1","unstructured":"2023. Practical Open-Source Intelligence (OSINT) Training - SANS SEC497. https:\/\/www.sans.org\/cyber-securitycourses\/practical-open-source-intelligence\/"},{"key":"e_1_2_1_18_1","unstructured":"2023. Search Party Rules. https:\/\/www.tracelabs.org\/about\/search-party-rules"},{"key":"e_1_2_1_19_1","unstructured":"2023. Syrian Archive | Syrian Archive. https:\/\/syrianarchive.org\/"},{"key":"e_1_2_1_20_1","unstructured":"2023. TinEye Reverse Image Search. https:\/\/tineye.com\/"},{"key":"e_1_2_1_21_1","unstructured":"2023. Verifying Online Information. https:\/\/firstdraftnews.org:443\/long-form-article\/verifying-online-information\/"},{"key":"e_1_2_1_22_1","unstructured":"2023. Yandex. https:\/\/yandex.com\/"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--981--16--8059--5_24"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13235"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998271"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445272"},{"key":"e_1_2_1_27_1","volume-title":"Montse Cuadros, Sebastian Maier, Felipe Melero, and Matteo Raffaelli.","author":"Aliprandi Carlo","year":"2014","unstructured":"Carlo Aliprandi, Juan Arraiza Irujo, Montse Cuadros, Sebastian Maier, Felipe Melero, and Matteo Raffaelli. 2014. CAPER: Collaborative Information, Acquisition, Processing, Exploitation and Reporting for the Prevention of Organised Crime. Communications in Computer and Information Science (2014), 6."},{"key":"e_1_2_1_28_1","volume-title":"Scaling up fact-checking using the wisdom of crowds. Science advances 7, 36","author":"Allen Jennifer","year":"2021","unstructured":"Jennifer Allen, Antonio A Arechar, Gordon Pennycook, and David G Rand. 2021. Scaling up fact-checking using the wisdom of crowds. Science advances 7, 36 (2021), eabf4393."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.31234\/osf.io\/57e3q"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3434177"},{"key":"e_1_2_1_31_1","volume-title":"Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 17 (nov","author":"Garcia Adriana Alvarado","year":"2018","unstructured":"Adriana Alvarado Garcia and Christopher A. Le Dantec. 2018. Quotidian Report: Grassroots Data Practices to Address Public Safety. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 17 (nov 2018), 18 pages. https:\/\/doi.org\/10.1145\/ 3274286"},{"key":"e_1_2_1_32_1","unstructured":"Amnesty International. 2020. Syria: 'Nowhere is Safe for Us': Unlawful Attacks and Mass Displacement in North-West Syria. Technical Report. https:\/\/www.amnesty.org\/en\/documents\/document\/?indexNumber=MDE24%2f2089% 2f2020&language=en"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998294"},{"key":"e_1_2_1_34_1","volume-title":"Information Seeking Support Systems Workshop (ISSS","author":"Aula Anne","year":"2008","unstructured":"Anne Aula and Daniel M Russell. 2008. Complex and exploratory web search. In Information Seeking Support Systems Workshop (ISSS 2008), Chapel Hill, NC, USA. Citeseer."},{"key":"e_1_2_1_35_1","volume-title":"Wikitribune: can crowd-sourced journalism solve the crisis of trust in news? POLIS: journalism and society at the LSE","author":"Beckett Charlie","year":"2017","unstructured":"Charlie Beckett. 2017. Wikitribune: can crowd-sourced journalism solve the crisis of trust in news? POLIS: journalism and society at the LSE (2017)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517526"},{"key":"e_1_2_1_37_1","unstructured":"Bellingcat Investigation Team. 2015. Diversifying OSINT: Women Experts. https:\/\/www.bellingcat.com\/resources\/ articles\/2015\/12\/08\/women-in-osint-diversifying-the-field\/"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_39_1","first-page":"207","article-title":"Turning the spotlight on the crowd: Examining the participatory ethics and practices of crisis mapping","volume":"15","author":"Bittner Christian","year":"2016","unstructured":"Christian Bittner, Boris Michel, and Cate Turk. 2016. Turning the spotlight on the crowd: Examining the participatory ethics and practices of crisis mapping. ACME: An International Journal for Critical Geographies 15, 1 (2016), 207--229.","journal-title":"ACME: An International Journal for Critical Geographies"},{"key":"e_1_2_1_40_1","volume-title":"Social media sourcing practices: How Dutch newspapers use tweets in political news coverage. Managing democracy in the digital age: Internet regulation, social media use, and online civic engagement","author":"Brands Bert Jan","year":"2018","unstructured":"Bert Jan Brands, Todd Graham, and Marcel Broersma. 2018. Social media sourcing practices: How Dutch newspapers use tweets in political news coverage. Managing democracy in the digital age: Internet regulation, social media use, and online civic engagement (2018), 159--178."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_2_1_42_1","volume-title":"The use of social media in the recruitment process. (2014). 81","author":"Broughton Andrea","year":"2014","unstructured":"Andrea Broughton, Beth Foley, Stefanie Ledermaier, and Annette Cox. 2014. The use of social media in the recruitment process. (2014). 81 (2014)."},{"key":"e_1_2_1_43_1","volume-title":"Leveraging Large Language Model and Story-Based Gamification in Intelligent Tutoring System to Scaffold Introductory Programming Courses: A Design-Based Research Study. arXiv preprint arXiv:2302.12834","author":"Cao Chen","year":"2023","unstructured":"Chen Cao. 2023. Leveraging Large Language Model and Story-Based Gamification in Intelligent Tutoring System to Scaffold Introductory Programming Courses: A Design-Based Research Study. arXiv preprint arXiv:2302.12834 (2023)."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3317697.3323358"},{"key":"e_1_2_1_45_1","unstructured":"Southern Poverty Law Center. 2021. The Road to Jan. 6: A Year of Extremist Mobilization. https:\/\/www.splcenter. org\/news\/2021\/12\/30\/road-jan-6-year-extremist-mobilization"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274300"},{"key":"e_1_2_1_47_1","volume-title":"Design experiments in educational research. Educational researcher 32, 1","author":"Cobb Paul","year":"2003","unstructured":"Paul Cobb, Jere Confrey, Andrea DiSessa, Richard Lehrer, and Leona Schauble. 2003. Design experiments in educational research. Educational researcher 32, 1 (2003), 9--13."},{"key":"e_1_2_1_48_1","unstructured":"Josie Cochrane. 2022. Citizen OSINT Analysts: Motivations of Open-Source Intelligence Volunteers."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.3390\/app13042198"},{"key":"e_1_2_1_50_1","volume-title":"The Hackers Hunting Down Missing People: Nonprofit TraceLabs ran DEF CON's first crowdsourced event for tracking missing people through public information. Vice","author":"Cox Joseph","year":"2018","unstructured":"Joseph Cox. 2018. The Hackers Hunting Down Missing People: Nonprofit TraceLabs ran DEF CON's first crowdsourced event for tracking missing people through public information. Vice (2018). https:\/\/www.vice.com\/en_us\/article\/ qvmm3x\/hackers-hunting-missing-people-osint-defcon-tracelabs"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10606-"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2685553.2698995"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479550"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_56_1","volume-title":"Digital Witness: Using Open Source Information for Human Rights Investigation, Documentation, and Accountability","author":"Dubberley Sam","year":"2020","unstructured":"Sam Dubberley, Alexa Koenig, and Daragh Murray (Eds.). 2020. Digital Witness: Using Open Source Information for Human Rights Investigation, Documentation, and Accountability. Oxford University Press, Oxford, New York."},{"key":"e_1_2_1_57_1","volume-title":"Daniel Rees Lewis, and Elizabeth M Gerber","author":"Easterday Matthew W","year":"2014","unstructured":"Matthew W Easterday, Daniel Rees Lewis, and Elizabeth M Gerber. 2014. Design-based research process: Problems, phases, and applications. Boulder, CO: International Society of the Learning Sciences."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675182"},{"key":"e_1_2_1_59_1","unstructured":"Esteban Borges. 2019. SecurityTrails | OSINT Framework: The Perfect Cybersecurity Intel Gathering Tool. https: \/\/securitytrails.com\/blog\/osint-framework"},{"key":"e_1_2_1_60_1","unstructured":"Giancarlo Fiorella. 2021. First Steps to Getting Started in Open Source Research. https:\/\/www.bellingcat.com\/resources\/ 2021\/11\/09\/first-steps-to-getting-started-in-open-source-research\/"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207711"},{"key":"e_1_2_1_62_1","volume-title":"Measuring the reach of\" fake news\" and online disinformation in Europe. Australasian Policing 10, 2","author":"Fletcher Richard","year":"2018","unstructured":"Richard Fletcher, Alessio Cornia, Lucas Graves, and Rasmus Kleis Nielsen. 2018. Measuring the reach of\" fake news\" and online disinformation in Europe. Australasian Policing 10, 2 (2018)."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555616"},{"key":"e_1_2_1_64_1","volume-title":"Open Source Intelligence and AI: a Systematic Review of the GELSI Literature","author":"Ghioni Riccardo","year":"2022","unstructured":"Riccardo Ghioni, Mariarosaria Taddeo, and Luciano Floridi. 2022. Open Source Intelligence and AI: a Systematic Review of the GELSI Literature. AI & Society (2022)."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1016\/j"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.54501\/jots.v1i1.15"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSCWD.2019.8791923"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702395"},{"key":"e_1_2_1_69_1","volume-title":"How Digital Detectives Say They Proved Ukraine Attacks Came from Russia. The Guardian","author":"Gutierrez Pablo","year":"2015","unstructured":"Pablo Gutierrez and Paul Torpey. 2015. How Digital Detectives Say They Proved Ukraine Attacks Came from Russia. The Guardian (2015)."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858364"},{"key":"e_1_2_1_71_1","volume-title":"Ethics in the Age of OSINT Innocence","author":"Hanham Melissa","year":"2020","unstructured":"Melissa Hanham and Jaewoo Shin. 2020. Ethics in the Age of OSINT Innocence. Stanley Center for Peace and Security (May 2020), 6. https:\/\/stanleycenter.org\/publications\/ethics-osint-innocence\/"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359250"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137815"},{"key":"e_1_2_1_74_1","unstructured":"Annique Mossou Higgins Ross. 2021. A Beginner's Guide to Social Media Verification. https:\/\/www.bellingcat.com\/ resources\/2021\/11\/01\/a-beginners-guide-to-social-media-verification\/"},{"key":"e_1_2_1_75_1","volume-title":"We Are Bellingcat : An Intelligence Agency for the People","author":"Higgins Eliot","unstructured":"Eliot Higgins. 2021. We Are Bellingcat : An Intelligence Agency for the People. Bloomsbury Publishing, London."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1353\/lib.0.0046"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675202"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1080\/08850600290101767"},{"key":"e_1_2_1_79_1","doi-asserted-by":"crossref","unstructured":"Denis Iorga Octavian Grigorescu Mihai Predoiu Cristian Sandescu Mihai Dascalu and Razvan Rughinis. 2021. Early Usability Evaluation to Enhance User Interfaces-A Use Case on the Yggdrasil Cybersecurity Mockup.. In RoCHI. 103--110.","DOI":"10.37789\/rochi.2021.1.1.16"},{"key":"e_1_2_1_80_1","volume-title":"Objects of violence: synthetic data for practical ML in human rights investigations. arXiv:2004.01030 [cs] (April","author":"Kermode Lachlan","year":"2020","unstructured":"Lachlan Kermode, Jan Freyberg, Alican Akturk, Robert Trafford, Denis Kochetkov, Rafael Pardinas, Eyal Weizman, and Julien Cornebise. 2020. Objects of violence: synthetic data for practical ML in human rights investigations. arXiv:2004.01030 [cs] (April 2020), 12. http:\/\/arxiv.org\/abs\/2004.01030 arXiv: 2004.01030."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-65299-9_14"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173670"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531644"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_85_1","first-page":"1","article-title":"Experiential Learning Theory as a Guide for Experiential Educators in Higher Education","volume":"1","author":"Kolb Alice","year":"2017","unstructured":"Alice Kolb and David Kolb. 2017. Experiential Learning Theory as a Guide for Experiential Educators in Higher Education. Experiential Learning & Teaching in Higher Education 1, 1 (June 2017), 7--44. https:\/\/nsuworks.nova.edu\/ elthe\/vol1\/iss1\/7","journal-title":"Experiential Learning & Teaching in Higher Education"},{"key":"e_1_2_1_86_1","unstructured":"Meryl Kornfield. 2021. The wrong ID: Retired firefighter comedian and Chuck Norris falsely accused of being Capitol rioters. Washington Post. https:\/\/www.washingtonpost.com\/technology\/2021\/01\/16\/sleuths-falsely-identify-rioters\/"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511815355"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","unstructured":"Tianyi Li Kurt Luther and Chris North. 2018. CrowdIA: Solving Mysteries with Crowdsourced Sensemaking. 2 (2018) 105:1--105:29. Issue CSCW. https:\/\/doi.org\/10.1145\/3274374","DOI":"10.1145\/3274374"},{"key":"e_1_2_1_89_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction 3, CSCW","author":"Li Tianyi","year":"2019","unstructured":"Tianyi Li, Chandler J Manns, Chris North, and Kurt Luther. 2019. Dropping the baton? Understanding errors and bottlenecks in a crowdsourced sensemaking pipeline. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--26."},{"key":"e_1_2_1_90_1","unstructured":"Hana Matatov Adina Bechhofer Lora Aroyo Ofra Amir and Mor Naaman. 2018. DejaVu: A System for Journalists to Collaboratively Address Visual Misinformation."},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415204"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/2637002.2637023"},{"key":"e_1_2_1_93_1","doi-asserted-by":"crossref","unstructured":"Stephen C. Mercado. 2004. Sailing the sea of OSINT in the information age. Technical Report. dataset American Psychological Association. type. https:\/\/doi.org\/10.1037\/e741272011-005","DOI":"10.1037\/e741272011-005"},{"key":"e_1_2_1_94_1","unstructured":"Panagiotis Metaxas and Samantha T Finn. 2017. The infamous# Pizzagate conspiracy theory: Insight from a Twitter- Trails investigation. (2017)."},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/3512974"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702553"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1080\/17512786.2021.1919543"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1093\/bjc\/azv118"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047198"},{"key":"e_1_2_1_100_1","unstructured":"Department of Homeland Security. 2010. (U\/\/FOUO\/\/LES) DHS Terrorist Use of Social Networking Facebook Case Study | Public Intelligence. https:\/\/publicintelligence.net\/ufouoles-dhs-terrorist-use-of-social-networking-facebookcase- study\/"},{"key":"e_1_2_1_101_1","unstructured":"Massachusetts Institute of Technology. 2023. Urban Cyber Defense: Cybersecurity Clinic. http:\/\/urbancyberdefense. mit.edu\/cybersecurityclinic Accessed: 2023--10-02."},{"key":"e_1_2_1_102_1","first-page":"554","article-title":"Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks","volume":"8","author":"Oleson David","year":"2013","unstructured":"David Oleson, Alexander Sorokin, Greg Laughlin, Vaughn Hester, John Le, Christopher R Van Pelt, and Lukas A Biewald. 2013. Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks. US Patent 8,554,605.","journal-title":"US Patent"},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13230"},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1806781116"},{"key":"e_1_2_1_105_1","volume-title":"A conceptual framework for assessing motivation and self-regulated learning in college students. Educational psychology review 16, 4","author":"Pintrich Paul R","year":"2004","unstructured":"Paul R Pintrich. 2004. A conceptual framework for assessing motivation and self-regulated learning in college students. Educational psychology review 16, 4 (2004), 385--407."},{"key":"e_1_2_1_106_1","unstructured":"Tjeerd Plomp et al. 2013. Educational design research: An introduction. Educational design research (2013) 11--50."},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145394"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v8i1.7470"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134724"},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0263--2373(01)00069-X"},{"key":"e_1_2_1_111_1","volume-title":"According to New CareerBuilder Survey -","author":"Hunt Ryan","year":"2012","unstructured":"Ryan Hunt. 2012. Thirty-Seven Percent of Companies Use Social Networks to Research Potential Job Candidates, According to New CareerBuilder Survey - Apr 18, 2012. http:\/\/press.careerbuilder.com\/2012-04--18-Thirty- Seven-Percent-of-Companies-Use-Social-Networks-to-Research-Potential-Job-Candidates-According-to-New- CareerBuilder-Survey"},{"key":"e_1_2_1_112_1","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557279"},{"key":"e_1_2_1_113_1","volume-title":"Election 1: Fact Checking The NPR Fact Checkers. NPR (Oct","author":"Schumacher-Matos Edward","year":"2012","unstructured":"Edward Schumacher-Matos. 2012. Election 1: Fact Checking The NPR Fact Checkers. NPR (Oct. 2012). https: \/\/www.npr.org\/sections\/publiceditor\/2012\/10\/28\/161839145\/election-1-fact-checking-the-npr-fact-checkers"},{"key":"e_1_2_1_114_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2018.00062"},{"key":"e_1_2_1_115_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872518.2890098"},{"key":"e_1_2_1_116_1","volume-title":"Verification Handbook: A Definitive Guide to Verifying Digital Content for Emergency Coverage","author":"Silverman Craig","year":"2013","unstructured":"Craig Silverman. 2013. Verification Handbook: A Definitive Guide to Verifying Digital Content for Emergency Coverage. http:\/\/verificationhandbook.com\/"},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445092"},{"key":"e_1_2_1_118_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858121"},{"key":"e_1_2_1_119_1","doi-asserted-by":"crossref","unstructured":"Yla Tausczik and Mark Boons. 2018. Distributed Knowledge in Crowds: Crowd Performance on Hidden Profile Tasks. In Twelfth International AAAI Conference onWeb and Social Media. https:\/\/aaai.org\/ocs\/index.php\/ICWSM\/ICWSM18\/ paper\/view\/17817","DOI":"10.1609\/icwsm.v12i1.15005"},{"key":"e_1_2_1_120_1","volume-title":"Qualitative and quantitative feedback in the context of competency-based education. Medical teacher 39, 12","author":"Tekian Ara","year":"2017","unstructured":"Ara Tekian, Christopher J Watling, Trudie E Roberts, Yvonne Steinert, and John Norcini. 2017. Qualitative and quantitative feedback in the context of competency-based education. Medical teacher 39, 12 (2017), 1245--1249."},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13347-016-0216--4"},{"key":"e_1_2_1_122_1","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2010.527697"},{"key":"e_1_2_1_123_1","doi-asserted-by":"crossref","unstructured":"Sukrit Venkatagiri. 2022. Supporting and Transforming High-Stakes Investigations with Expert-Led Crowdsourcing. Ph.D. Dissertation. Virginia Tech.","DOI":"10.1145\/3565967.3571764"},{"key":"e_1_2_1_124_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449192"},{"key":"e_1_2_1_125_1","doi-asserted-by":"publisher","DOI":"10.1145\/3563657.3595997"},{"key":"e_1_2_1_126_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359209"},{"key":"e_1_2_1_127_1","doi-asserted-by":"publisher","DOI":"10.5555\/2042118.2042175"},{"key":"e_1_2_1_128_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274447"},{"key":"e_1_2_1_129_1","volume-title":"Verifying user-generated content. Verification handbook: A definitive guide to verifying digital content for emergency coverage","author":"Wardle Claire","year":"2014","unstructured":"Claire Wardle. 2014. Verifying user-generated content. Verification handbook: A definitive guide to verifying digital content for emergency coverage (2014), 24--33."},{"key":"e_1_2_1_130_1","unstructured":"Claire Wardle and Hossein Derakhshan. 2017. Information disorder: Toward an interdisciplinary framework for research and policymaking."},{"key":"e_1_2_1_131_1","unstructured":"Etienne Wenger et al. 1998. Communities of practice: Learning as a social system. Systems thinker 9 5 (1998) 2--3."},{"key":"e_1_2_1_132_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531633"},{"key":"e_1_2_1_133_1","volume-title":"Williams and Ilana Blum","author":"Heather","year":"2018","unstructured":"Heather J. Williams and Ilana Blum. 2018. Defining Second Generation Open Source Intelligence (OSINT) for the Defense Enterprise. Technical Report. RAND Corporation. https:\/\/www.rand.org\/pubs\/research_reports\/RR1964.html"},{"key":"e_1_2_1_134_1","volume-title":"Williams and Ilana Blum","author":"Heather","year":"2018","unstructured":"Heather J. Williams and Ilana Blum. 2018. Defining second generation open source intelligence (OSINT) for the defense enterprise. Technical Report. RAND Corporation Santa Monica United States."},{"key":"e_1_2_1_135_1","doi-asserted-by":"publisher","DOI":"10.1145\/2876034.2876042"},{"key":"e_1_2_1_136_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675140"},{"key":"e_1_2_1_137_1","volume-title":"Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 220--232","author":"Zhang Haoqi","year":"2017","unstructured":"Haoqi Zhang, Matthew W Easterday, Elizabeth M Gerber, Daniel Rees Lewis, and Leesha Maliakal. 2017. Agile research studios: Orchestrating communities of practice to advance research training. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 220--232."},{"key":"e_1_2_1_138_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531718"},{"key":"e_1_2_1_139_1","unstructured":"H. Akin \u00dcnver. 2018. Digital Open Source Intelligence and International Security: A Primer. Technical Report. Centre for Economics and Foreign Policy Studies. https:\/\/www.jstor.org\/stable\/resrep21048"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637382","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637382","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T17:18:10Z","timestamp":1755883090000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637382"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,17]]},"references-count":139,"journal-issue":{"issue":"CSCW1","published-print":{"date-parts":[[2024,4,17]]}},"alternative-id":["10.1145\/3637382"],"URL":"https:\/\/doi.org\/10.1145\/3637382","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,17]]},"assertion":[{"value":"2024-04-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}