{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,30]],"date-time":"2025-08-30T16:56:18Z","timestamp":1756572978957,"version":"3.41.0"},"reference-count":19,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2019,11,7]],"date-time":"2019-11-07T00:00:00Z","timestamp":1573084800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1651969, IIS-1527453"],"award-info":[{"award-number":["IIS-1651969, IIS-1527453"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2019,11,7]]},"abstract":"<jats:p>Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. In this paper, we introduce the concept of shared representations for crowd--augmented expert work, focusing on the complex sensemaking task of image geolocation performed by professional journalists and human rights investigators. We built GroundTruth, an online system that uses three shared representations-a diagram, grid, and heatmap-to allow experts to work with crowds in real time to geolocate images. Our mixed-methods evaluation with 11 experts and 567 crowd workers found that GroundTruth helped experts geolocate images, and revealed challenges and success strategies for expert-crowd interaction. We also discuss designing shared representations for visual search, sensemaking, and beyond.<\/jats:p>","DOI":"10.1145\/3359209","type":"journal-article","created":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T13:40:21Z","timestamp":1573220421000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["GroundTruth: Augmenting Expert Image Geolocation with Crowdsourcing and Shared Representations"],"prefix":"10.1145","volume":"3","author":[{"given":"Sukrit","family":"Venkatagiri","sequence":"first","affiliation":[{"name":"Virginia Tech, Arlington, VA, USA"}]},{"given":"Jacob","family":"Thebault-Spieker","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]},{"given":"Rachel","family":"Kohler","sequence":"additional","affiliation":[{"name":"Virginia Tech and BNSF Railway Company, Blacksburg, VA, USA"}]},{"given":"John","family":"Purviance","sequence":"additional","affiliation":[{"name":"Virginia Tech and Bloomberg LP, Blacksburg, VA, USA"}]},{"given":"Rifat Sabbir","family":"Mansur","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]},{"given":"Kurt","family":"Luther","sequence":"additional","affiliation":[{"name":"Virginia Tech, Arlington, VA, USA"}]}],"member":"320","published-online":{"date-parts":[[2019,11,7]]},"reference":[{"key":"e_1_2_2_1_1","article-title":"Motivation Factors in Crowdsourced Journalism: Social Impact, Social Change, and Peer Learning","volume":"9","author":"Aitamurto Tanja","year":"2015","unstructured":"Tanja Aitamurto . 2015 . Motivation Factors in Crowdsourced Journalism: Social Impact, Social Change, and Peer Learning . International Journal of Communication , Vol. 9 , 0 (2015). https:\/\/ijoc.org\/index.php\/ijoc\/article\/view\/3481 Tanja Aitamurto. 2015. Motivation Factors in Crowdsourced Journalism: Social Impact, Social Change, and Peer Learning. International Journal of Communication, Vol. 9, 0 (2015). https:\/\/ijoc.org\/index.php\/ijoc\/article\/view\/3481","journal-title":"International Journal of Communication"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3059454.3059477"},{"volume-title":"Proceedings of CSCW 2014 .","author":"Andr\u00e9 Paul","key":"e_1_2_2_3_1","unstructured":"Paul Andr\u00e9 , Aniket Kittur , and Steven P. Dow . 2014. Crowd Synthesis: Extracting Categories and Clusters from Complex Data . In Proceedings of CSCW 2014 . Paul Andr\u00e9, Aniket Kittur, and Steven P. Dow. 2014. Crowd Synthesis: Extracting Categories and Clusters from Complex Data. In Proceedings of CSCW 2014 ."},{"key":"e_1_2_2_4_1","unstructured":"Trushar Barot. 2014. Verifying Images. In Verification Handbook: A Definitive Guide to Verifying Digital Content for Emergency Coverage. http:\/\/verificationhandbook.com\/book\/chapter4.php  Trushar Barot. 2014. Verifying Images. In Verification Handbook: A Definitive Guide to Verifying Digital Content for Emergency Coverage. http:\/\/verificationhandbook.com\/book\/chapter4.php"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047201"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"volume-title":"Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18)","author":"Bragg Jonathan","key":"e_1_2_2_7_1","unstructured":"Jonathan Bragg , Mausam, and Daniel S. Weld . 2018. Sprout: Crowd-Powered Task Design for Crowdsourcing . In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18) . ACM, New York, NY, USA, 165--176. https:\/\/doi.org\/10.1145\/3242587.3242598 Jonathan Bragg, Mausam, and Daniel S. Weld. 2018. Sprout: Crowd-Powered Task Design for Crowdsourcing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 165--176. https:\/\/doi.org\/10.1145\/3242587.3242598"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1080\/17512786.2015.1020331"},{"key":"e_1_2_2_9_1","doi-asserted-by":"crossref","unstructured":"Sukrit Venkatagiri and Amy X Zhang. 2018. Response to \u201cHeuristics for the Online Curator\u201d Brian G Southwell and Vanessa Boudewyns (Eds.). Curbing the Spread of Misinformation: Insights Innovations and Interpretations From the Misinformation Solutions Forum 9--10. https:\/\/doi.org\/10.3768\/rtipress.2018.cp.0008.1812  Sukrit Venkatagiri and Amy X Zhang. 2018. Response to \u201cHeuristics for the Online Curator\u201d Brian G Southwell and Vanessa Boudewyns (Eds.). Curbing the Spread of Misinformation: Insights Innovations and Interpretations From the Misinformation Solutions Forum 9--10. https:\/\/doi.org\/10.3768\/rtipress.2018.cp.0008.1812","DOI":"10.3768\/rtipress.2018.cp.0008.1812"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.286"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274447"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46484-8_3"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531633"},{"key":"e_1_2_2_14_1","volume-title":"Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17)","author":"Whiting Mark E.","year":"1902","unstructured":"Mark E. Whiting , Dilrukshi Gamage , Snehalkumar (Neil) S. Gaikwad , Aaron Gilbee , Shirish Goyal , Alipta Ballav , Dinesh Majeti , Nalin Chhibber , Angela Richmond-Fuller , Freddie Vargus , Tejas Seshadri Sarma , Varshine Chandrakanthan , Teogenes Moura , Mohamed Hashim Salih , Gabriel Bayomi Tinoco Kalejaiye , Adam Ginzberg , Catherine A. Mullings , Yoni Dayan , Kristy Milland , Henrique Orefice , Jeff Regino , Sayna Parsi , Kunz Mainali , Vibhor Sehgal , Sekandar Matin , Akshansh Sinha , Rajan Vaish , and Michael S. Bernstein . 2017. Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms . In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17) . ACM, New York, NY, USA , 1902 --1913. https:\/\/doi.org\/10.1145\/2998181.2998234 event-place: Portland, Oregon, USA. Mark E. Whiting, Dilrukshi Gamage, Snehalkumar (Neil) S. Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, Tejas Seshadri Sarma, Varshine Chandrakanthan, Teogenes Moura, Mohamed Hashim Salih, Gabriel Bayomi Tinoco Kalejaiye, Adam Ginzberg, Catherine A. Mullings, Yoni Dayan, Kristy Milland, Henrique Orefice, Jeff Regino, Sayna Parsi, Kunz Mainali, Vibhor Sehgal, Sekandar Matin, Akshansh Sinha, Rajan Vaish, and Michael S. Bernstein. 2017. Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 1902--1913. https:\/\/doi.org\/10.1145\/2998181.2998234 event-place: Portland, Oregon, USA."},{"key":"e_1_2_2_15_1","doi-asserted-by":"crossref","unstructured":"Wesley Willett Shiry Ginosar Avital Steinitz Bj\u00f6rn Hartmann and Maneesh Agrawala. 2013. Identifying Redundancy and Exposing Provenance in Crowdsourced Data Analysis. (2013).  Wesley Willett Shiry Ginosar Avital Steinitz Bj\u00f6rn Hartmann and Maneesh Agrawala. 2013. Identifying Redundancy and Exposing Provenance in Crowdsourced Data Analysis. (2013).","DOI":"10.1109\/TVCG.2013.164"},{"volume-title":"Incorporating heuristically generated search patterns in search and rescue","author":"Wollan Helen","key":"e_1_2_2_16_1","unstructured":"Helen Wollan . 2004. Incorporating heuristically generated search patterns in search and rescue . University of Edinburgh (2004) . Helen Wollan. 2004. Incorporating heuristically generated search patterns in search and rescue. University of Edinburgh (2004)."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2012.07.007"},{"key":"e_1_2_2_18_1","first-page":"81","article-title":"What's the deal with `websleuthing'? News media representations of amateur detectives in networked spaces. Crime, Media","volume":"14","author":"Yardley Elizabeth","year":"2018","unstructured":"Elizabeth Yardley , Adam George Thomas Lynes , David Wilson , and Emma Kelly . 2018 . What's the deal with `websleuthing'? News media representations of amateur detectives in networked spaces. Crime, Media , Culture , Vol. 14 , 1 (2018), 81 -- 109 . https:\/\/doi.org\/10.1177\/1741659016674045 showeprinthttps:\/\/doi.org\/10.1177\/1741659016674045 Elizabeth Yardley, Adam George Thomas Lynes, David Wilson, and Emma Kelly. 2018. What's the deal with `websleuthing'? News media representations of amateur detectives in networked spaces. Crime, Media, Culture, Vol. 14, 1 (2018), 81--109. https:\/\/doi.org\/10.1177\/1741659016674045 showeprinthttps:\/\/doi.org\/10.1177\/1741659016674045","journal-title":"Culture"},{"key":"e_1_2_2_19_1","volume-title":"Wynne","author":"Yu Ling","year":"2015","unstructured":"Ling Yu , Sheryl Ball , Christine Blinn , Klaus Moeltner , Seth Peery , Valerie Thomas , Randolph Wynne , Ling Yu , Sheryl B. Ball , Christine E. Blinn , Klaus Moeltner , Seth Peery , Valerie A. Thomas , and Randolph H . Wynne . 2015 . Cloud-Sourcing: Using an Online Labor Force to Detect Clouds and Cloud Shadows in Landsat Images , Vol. 7 , 3 (2015), 2334--2351. https:\/\/doi.org\/10.3390\/rs70302334 Ling Yu, Sheryl Ball, Christine Blinn, Klaus Moeltner, Seth Peery, Valerie Thomas, Randolph Wynne, Ling Yu, Sheryl B. Ball, Christine E. Blinn, Klaus Moeltner, Seth Peery, Valerie A. Thomas, and Randolph H. Wynne. 2015. Cloud-Sourcing: Using an Online Labor Force to Detect Clouds and Cloud Shadows in Landsat Images, Vol. 7, 3 (2015), 2334--2351. https:\/\/doi.org\/10.3390\/rs70302334"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359209","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359209","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359209","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:12:54Z","timestamp":1750201974000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359209"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,7]]},"references-count":19,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2019,11,7]]}},"alternative-id":["10.1145\/3359209"],"URL":"https:\/\/doi.org\/10.1145\/3359209","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2019,11,7]]},"assertion":[{"value":"2019-11-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}