{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T11:29:20Z","timestamp":1771500560379,"version":"3.50.1"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"US National Historical Publications and Records Commission","award":["DH50013-15"],"award-info":[{"award-number":["DH50013-15"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Crowdsourcing more complex and creative tasks is seen as a desirable goal for both employers and workers, but these tasks traditionally require domain expertise. Employers can recruit only expert workers, but this approach does not scale well. Alternatively, employers can decompose complex tasks into simpler micro-tasks, but some domains, such as historical analysis, cannot be easily modularized in this way. A third approach is to train workers to learn the domain expertise. This approach offers clear benefits to workers, but is perceived as costly or infeasible for employers. In this paper, we explore the trade-offs between learning and productivity in training crowd workers to analyze historical documents. We compare CrowdSCIM, a novel approach that teaches historical thinking skills to crowd workers, with two crowd learning techniques from prior work and a baseline. Our evaluation (n=360) shows that CrowdSCIM allows workers to learn domain expertise while producing work of equal or higher quality versus other conditions, but efficiency is slightly lower.<\/jats:p>","DOI":"10.1145\/3274447","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History"],"prefix":"10.1145","volume":"2","author":[{"given":"Nai-Ching","family":"Wang","sequence":"first","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]},{"given":"David","family":"Hicks","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]},{"given":"Kurt","family":"Luther","sequence":"additional","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"{n. d.}. DIYHistory | Transcribe. https:\/\/diyhistory.lib.uiowa.edu\/"},{"key":"e_1_2_1_2_1","unstructured":"{n. d.}. Feature Article - Historical Thinking Winter 2010- Teaching with Primary Sources | Teacher Resources - Library of Congress. http:\/\/www.loc.gov\/teachers\/tps\/quarterly\/historical_thinking\/article.html"},{"key":"e_1_2_1_3_1","unstructured":"{n. d.}. Incite | Crowdsourced Analysis of Omeka Documents | Virginia Tech. http:\/\/incite.cs.vt.edu\/"},{"key":"e_1_2_1_4_1","unstructured":"{n. d.}. Smithsonian Digital Volunteers. https:\/\/transcription.si.edu\/"},{"key":"e_1_2_1_5_1","volume-title":"Levstik","author":"Barton Keith C.","year":"2004","unstructured":"Keith C. Barton and Linda S. Levstik. 2004. Teaching history for the common good. Routledge."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675251"},{"key":"e_1_2_1_9_1","volume-title":"van der Dussen","author":"Collingwood Robin George","year":"1993","unstructured":"Robin George Collingwood and Willem J. van der Dussen. 1993. The idea of history. Oxford University Press on Demand."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5840\/thinking19888129"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557217"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMS.2008.42"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702304"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820011"},{"key":"e_1_2_1_17_1","volume-title":"Fussell","author":"Goyal Nitesh","year":"2015","unstructured":"Nitesh Goyal and Susan R. Fussell. 2015. Designing for Collaborative Sensemaking: Leveraging Human Cognition For Complex Tasks. arXiv preprint arXiv:1511.05737 (2015)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820071"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2757226.2757249"},{"key":"e_1_2_1_20_1","volume-title":"Teaching and Learning in History","author":"Greene Stuart","unstructured":"Stuart Greene. 1994. Students as authors in the study of history. In Teaching and Learning in History. Routledge, 137--170."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1080\/00933104.2008.10473373"},{"key":"e_1_2_1_22_1","first-page":"3","article-title":"The SCIM-C strategy: expert historians, historical inquiry, and multimedia","volume":"68","author":"Hicks David","year":"2004","unstructured":"David Hicks, Peter E. Doolittle, and E. Thomas Ewing. 2004. The SCIM-C strategy: expert historians, historical inquiry, and multimedia. Social Education 68, 3 (April 2004), 221--226.","journal-title":"Social Education"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15548430jlr3602_2"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/227170.227173"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.2190\/4T1B-HBP0-3F7E-J4PN"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.22323\/2.15030205"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2468356.2468477"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531677"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998197"},{"key":"e_1_2_1_34_1","unstructured":"Doris Jung-Lin Lee Joanne Lo Moonhyok Kim and Eric Paulos. 2016. Crowdclass: Designing Classification-Based Citizen Science Learning Modules. In Fourth AAAI Conference on Human Computation and Crowdsourcing. https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP16\/paper\/view\/14027"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837907"},{"key":"e_1_2_1_36_1","volume-title":"Third AAAI Conference on Human Computation and Crowdsourcing. http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP15\/paper\/view\/11603","author":"Luther Kurt","year":"2015","unstructured":"Kurt Luther, Nathan Hahn, Steven P. Dow, and Aniket Kittur. 2015. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. In Third AAAI Conference on Human Computation and Crowdsourcing. http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP15\/paper\/view\/11603"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1093\/maghis\/22.2.55"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1080\/14682753.2017.1375252"},{"key":"e_1_2_1_39_1","first-page":"255","article-title":"The effects of training, modality, and redundancy on the development of a historical inquiry strategy in a multimedia learning environment","volume":"8","author":"McNeill Andrea L.","year":"2009","unstructured":"Andrea L. McNeill, Peter E. Doolittle, and David Hicks. 2009. The effects of training, modality, and redundancy on the development of a historical inquiry strategy in a multimedia learning environment. Journal of Interactive Online Learning 8, 3 (2009), 255--269.","journal-title":"Journal of Interactive Online Learning"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2724660.2728683"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025769"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02505026"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1080\/17512786.2013.765638"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40495-5_7"},{"key":"e_1_2_1_46_1","volume-title":"teaching, and learning history: National and international perspectives","author":"Stearns Peter N.","unstructured":"Peter N. Stearns, Peter C. Seixas, and Sam Wineburg. 2000. Knowing, teaching, and learning history: National and international perspectives. NYU Press."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1080\/15391523.2005.10782447"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025892"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675219"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2876034.2876042"},{"key":"e_1_2_1_51_1","first-page":"488","article-title":"Historical thinking and other unnatural acts","volume":"80","author":"Wineburg Sam","year":"1999","unstructured":"Sam Wineburg. 1999. Historical thinking and other unnatural acts. The Phi Delta Kappan 80, 7 (1999), 488--499. http:\/\/www.jstor.org\/stable\/20439490","journal-title":"The Phi Delta Kappan"},{"key":"e_1_2_1_52_1","volume-title":"Thinking like a historian. Teaching with primary sources quarterly 3, 1","author":"Wineburg Sam","year":"2010","unstructured":"Sam Wineburg. 2010. Thinking like a historian. Teaching with primary sources quarterly 3, 1 (2010), 2--4. https:\/\/www.weteachnyc.org\/media\/filer_public\/24\/4a\/244ab1eb-4540--4ca1-a60e-5ef96326b365\/research_history.pdf"},{"key":"e_1_2_1_53_1","volume-title":"Library of Congress","author":"Wineburg Sam","year":"2010","unstructured":"Sam Wineburg. 2010. Thinking Like a Historian. Library of Congress (2010). http:\/\/www.loc.gov\/teachers\/tps\/quarterly\/historical_thinking\/article.html"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.3102\/00028312028003495"},{"key":"e_1_2_1_55_1","volume-title":"Wilson","author":"Wineburg Samuel S.","year":"1991","unstructured":"Samuel S. Wineburg and Suzanne M. Wilson. 1991. Subject matter knowledge in the teaching of history. Advances in research on teaching 2 (1991), 305--347."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675140"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531718"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274447","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274447","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:36Z","timestamp":1750207476000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274447"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":55,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274447"],"URL":"https:\/\/doi.org\/10.1145\/3274447","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}