{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T07:09:16Z","timestamp":1750748956763,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":33,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,8,1]],"date-time":"2020-08-01T00:00:00Z","timestamp":1596240000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["SES-162863"],"award-info":[{"award-number":["SES-162863"]}]},{"DOI":"10.13039\/100000867","name":"Robert Wood Johnson Foundation","doi-asserted-by":"publisher","award":["74422"],"award-info":[{"award-number":["74422"]}],"id":[{"id":"10.13039\/100000867","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,8]]},"DOI":"10.1145\/3383583.3398537","type":"proceedings-article","created":{"date-parts":[[2020,8,1]],"date-time":"2020-08-01T04:18:32Z","timestamp":1596255512000},"page":"227-234","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Seeking Justification: How Expert Reviewers Validate Empirical Claims with Data Annotations"],"prefix":"10.1145","author":[{"given":"Nicholas","family":"Weber","sequence":"first","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sebastian","family":"Karcher","sequence":"additional","affiliation":[{"name":"Syracuse University, Syracuse, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,8]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Ethical reproducibility: towards transparent reporting in biomedical research. nature methods 10, 9","author":"Anderson James A","year":"2013","unstructured":"James A Anderson , Marleen Eijkholt , and Judy Illes . 2013. Ethical reproducibility: towards transparent reporting in biomedical research. nature methods 10, 9 ( 2013 ), 843--845. James A Anderson, Marleen Eijkholt, and Judy Illes. 2013. Ethical reproducibility: towards transparent reporting in biomedical research. nature methods 10, 9 (2013), 843--845."},{"key":"e_1_3_2_1_2_1","volume-title":"Corpus linguistics in the UK: Resources for sociolinguistic research. Language and Linguistics Compass2, 2","author":"Anderson Wendy","year":"2008","unstructured":"Wendy Anderson . 2008. Corpus linguistics in the UK: Resources for sociolinguistic research. Language and Linguistics Compass2, 2 ( 2008 ), 352--371. Wendy Anderson. 2008. Corpus linguistics in the UK: Resources for sociolinguistic research. Language and Linguistics Compass2, 2 (2008), 352--371."},{"key":"e_1_3_2_1_3_1","volume-title":"Bailar and Kay Patterson","author":"John","year":"1985","unstructured":"John C. Bailar and Kay Patterson . 1985 . Journal peer review: the need for a research agenda. The New England Journal of Medicine; Boston 312, 10 (March 1985), 654--657. https:\/\/doi.org\/10.1056\/nejm198503073121023 10.1056\/nejm198503073121023 John C. Bailar and Kay Patterson. 1985. Journal peer review: the need for a research agenda. The New England Journal of Medicine; Boston 312, 10 (March 1985), 654--657. https:\/\/doi.org\/10.1056\/nejm198503073121023"},{"key":"e_1_3_2_1_4_1","volume-title":"Using thematic analysis in psychology. Qualitative research in psychology 3, 2","author":"Braun Virginia","year":"2006","unstructured":"Virginia Braun and Victoria Clarke . 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 ( 2006 ), 77--101. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101."},{"key":"e_1_3_2_1_5_1","volume-title":"Rose","author":"Christensen Garret","year":"2019","unstructured":"Garret Christensen , Allan Dafoe , Edward Miguel , Don A. Moore , and Andrew K . Rose . 2019 . A study of the impact of data sharing on article citations using journal policies as a natural experiment. PLOS ONE 14, 12 (Dec. 2019), e0225883. https:\/\/doi.org\/10.1371\/journal.pone.0225883 10.1371\/journal.pone.0225883 Garret Christensen, Allan Dafoe, Edward Miguel, Don A. Moore, and Andrew K. Rose. 2019. A study of the impact of data sharing on article citations using journal policies as a natural experiment. PLOS ONE14, 12 (Dec. 2019), e0225883. https:\/\/doi.org\/10.1371\/journal.pone.0225883"},{"key":"e_1_3_2_1_6_1","volume-title":"An open, large-scale, collaborative effort to estimate the reproducibility of psychological science.Perspectives on Psychological Science 7, 6","author":"Collaboration Open Science","year":"2012","unstructured":"Open Science Collaboration . 2012. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science.Perspectives on Psychological Science 7, 6 ( 2012 ), 657--660. Open Science Collaboration. 2012. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science.Perspectives on Psychological Science 7, 6 (2012), 657--660."},{"key":"e_1_3_2_1_7_1","volume-title":"Data policies of highly-ranked social science journals. SocArXiv(March","author":"Crosas Merc\u00e8","year":"2018","unstructured":"Merc\u00e8 Crosas , Julian Gautier , Sebastian Karcher , Dessi Kirilova , Gerard Otalora , and Abigail Schwartz . 2018. Data policies of highly-ranked social science journals. SocArXiv(March 2018 ). https:\/\/doi.org\/10.17605\/OSF.IO\/9H7AY 10.17605\/OSF.IO Merc\u00e8 Crosas, Julian Gautier, Sebastian Karcher, Dessi Kirilova, Gerard Otalora, and Abigail Schwartz. 2018. Data policies of highly-ranked social science journals. SocArXiv(March 2018). https:\/\/doi.org\/10.17605\/OSF.IO\/9H7AY"},{"key":"e_1_3_2_1_8_1","volume-title":"Data access and research transparency in the qualitative tradition. PS: Political Science & Politics47, 1","author":"Elman Colin","year":"2014","unstructured":"Colin Elman and Diana Kapiszewski . 2014. Data access and research transparency in the qualitative tradition. PS: Political Science & Politics47, 1 ( 2014 ), 43--47. Colin Elman and Diana Kapiszewski. 2014. Data access and research transparency in the qualitative tradition. PS: Political Science & Politics47, 1 (2014), 43--47."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_9_1","DOI":"10.1017\/S1049096517001755"},{"volume-title":"Politics, values, and public policy: The problem of methodology","author":"Fischer Frank","unstructured":"Frank Fischer . 2019. Politics, values, and public policy: The problem of methodology . Routledge . Frank Fischer. 2019. Politics, values, and public policy: The problem of methodology. Routledge.","key":"e_1_3_2_1_10_1"},{"doi-asserted-by":"crossref","unstructured":"Tom Elis Hardwicke Maya B Mathur Kyle Earl MacDonald Gustav Nilsonne George Christopher Banks Mallory Kidwell Alicia Hofelich Mohr Elizabeth Clayton Erica J. Yoon Michael Henry Tessler Richie L Lenne Sara Kai Altman Bria Long and Michael C. Frank. 2018. Data availability reusability and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition. preprint. BITSS. https:\/\/doi.org\/10.31222\/osf.io\/39cfb 10.31222\/osf.io","key":"#cr-split#-e_1_3_2_1_11_1.1","DOI":"10.1098\/rsos.180448"},{"doi-asserted-by":"crossref","unstructured":"Tom Elis Hardwicke Maya B Mathur Kyle Earl MacDonald Gustav Nilsonne George Christopher Banks Mallory Kidwell Alicia Hofelich Mohr Elizabeth Clayton Erica J. Yoon Michael Henry Tessler Richie L Lenne Sara Kai Altman Bria Long and Michael C. Frank. 2018. Data availability reusability and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition. preprint. BITSS. https:\/\/doi.org\/10.31222\/osf.io\/39cfb","key":"#cr-split#-e_1_3_2_1_11_1.2","DOI":"10.31222\/osf.io\/39cfb"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.29173\/iq959","article-title":"Annotation for transparent inquiry: Transparent data and analysis for qualitative research","volume":"43","author":"Karcher Sebastian","year":"2019","unstructured":"Sebastian Karcher and Nicholas Weber . 2019 . Annotation for transparent inquiry: Transparent data and analysis for qualitative research . IASSIST Quarterly 43 , 2(2019), 1 -- 9 . Sebastian Karcher and Nicholas Weber. 2019. Annotation for transparent inquiry: Transparent data and analysis for qualitative research. IASSIST Quarterly 43, 2(2019), 1--9.","journal-title":"IASSIST Quarterly"},{"doi-asserted-by":"crossref","unstructured":"John Ernest Kratz and Carly Strasser. [n.d.]. Researcher Perspectives on Publication and Peer Review of Data. 10 2 ([n. d.]) e0117619. https:\/\/doi.org\/10.1371\/journal.pone.0117619    10.1371\/journal.pone.0117619\nJohn Ernest Kratz and Carly Strasser. [n.d.]. Researcher Perspectives on Publication and Peer Review of Data. 10 2 ([n. d.]) e0117619. https:\/\/doi.org\/10.1371\/journal.pone.0117619","key":"e_1_3_2_1_13_1","DOI":"10.1371\/journal.pone.0117619"},{"doi-asserted-by":"crossref","unstructured":"Carole J. Lee and David Moher. [n.d.]. Promote scientific integrity via journal peer review data. 357 6348 ([n. d.]) 256--257. https:\/\/doi.org\/10.1126\/science.aan4141    10.1126\/science.aan4141\nCarole J. Lee and David Moher. [n.d.]. Promote scientific integrity via journal peer review data. 357 6348 ([n. d.]) 256--257. https:\/\/doi.org\/10.1126\/science.aan4141","key":"e_1_3_2_1_14_1","DOI":"10.1126\/science.aan4141"},{"key":"e_1_3_2_1_15_1","first-page":"115","article-title":"Science and technology in a democratic order","volume":"1","author":"Merton Robert K","year":"1942","unstructured":"Robert K Merton . 1942 . Science and technology in a democratic order . Journal of legal and political sociology 1 , 1 (1942), 115 -- 126 . Robert K Merton. 1942. Science and technology in a democratic order. Journal of legal and political sociology 1, 1 (1942), 115--126.","journal-title":"Journal of legal and political sociology"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_16_1","DOI":"10.1017\/S1049096513001789"},{"doi-asserted-by":"crossref","unstructured":"Richard D. Morey Christopher D. Chambers Peter J. Etchells Christine R.Harris Rink Hoekstra Dani\u00ebl Lakens Stephan Lewandowsky Candice CokerMorey Daniel P. Newman Felix D. Sch\u00f6nbrodt Wolf Vanpaemel Eric-Jan Wagenmakers and Rolf A. Zwaan. [n.d.]. The Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review. ([n. d.]). https:\/\/royalsocietypublishing.org\/doi\/abs\/10.1098\/rsos.150547  Richard D. Morey Christopher D. Chambers Peter J. Etchells Christine R.Harris Rink Hoekstra Dani\u00ebl Lakens Stephan Lewandowsky Candice CokerMorey Daniel P. Newman Felix D. Sch\u00f6nbrodt Wolf Vanpaemel Eric-Jan Wagenmakers and Rolf A. Zwaan. [n.d.]. The Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review. ([n. d.]). https:\/\/royalsocietypublishing.org\/doi\/abs\/10.1098\/rsos.150547","key":"e_1_3_2_1_17_1","DOI":"10.1098\/rsos.150547"},{"key":"e_1_3_2_1_18_1","volume-title":"Reporting transparency: making the ethical mandate explicit. BMC medicine14,1","author":"Nicholls Stuart G","year":"2016","unstructured":"Stuart G Nicholls , Sin\u00e9ad M Langan , Eric I Benchimol , and David Moher . 2016. Reporting transparency: making the ethical mandate explicit. BMC medicine14,1 ( 2016 ), 44. Stuart G Nicholls, Sin\u00e9ad M Langan, Eric I Benchimol, and David Moher. 2016. Reporting transparency: making the ethical mandate explicit. BMC medicine14,1 (2016), 44."},{"key":"e_1_3_2_1_19_1","volume-title":"Data for: Making the real: Rhetorical adduction and the Bangladesh Liberation War. (Feb","author":"O'Mahoney Joseph","year":"2019","unstructured":"Joseph O'Mahoney . 2019. Data for: Making the real: Rhetorical adduction and the Bangladesh Liberation War. (Feb . 2019 ). https:\/\/doi.org\/10.5064\/F6M2H9VQ 10.5064\/F6M2H9VQ Joseph O'Mahoney. 2019. Data for: Making the real: Rhetorical adduction and the Bangladesh Liberation War. (Feb. 2019). https:\/\/doi.org\/10.5064\/F6M2H9VQ"},{"doi-asserted-by":"crossref","unstructured":"Mark A. Parsons Ruth Duerr and Jean-Bernard Minster. [n.d.]. Data Citation and Peer Review. 91 34 ([n. d.]) 297--298. https:\/\/doi.org\/10.1029\/2010EO340001    10.1029\/2010EO340001\nMark A. Parsons Ruth Duerr and Jean-Bernard Minster. [n.d.]. Data Citation and Peer Review. 91 34 ([n. d.]) 297--298. https:\/\/doi.org\/10.1029\/2010EO340001","key":"e_1_3_2_1_20_1","DOI":"10.1029\/2010EO340001"},{"doi-asserted-by":"crossref","unstructured":"George Peat Richard D Riley Peter Croft Katherine I Morley Panayiotis A Kyzas Karel GM Moons Pablo Perel Ewout W Steyerberg Sara Schroter Douglas G Altman et al.2014. Improving the transparency of prognosis research: the role of reporting data sharing registration and protocols. PLoS medicine11 7 (2014).  George Peat Richard D Riley Peter Croft Katherine I Morley Panayiotis A Kyzas Karel GM Moons Pablo Perel Ewout W Steyerberg Sara Schroter Douglas G Altman et al.2014. Improving the transparency of prognosis research: the role of reporting data sharing registration and protocols. PLoS medicine11 7 (2014).","key":"e_1_3_2_1_21_1","DOI":"10.1371\/journal.pmed.1001671"},{"doi-asserted-by":"crossref","unstructured":"Ted Pedersen. [n.d.]. Empiricism Is Not a Matter of Faith. 34 3 ([n. d.]) 465--470. https:\/\/doi.org\/10.1162\/coli.2008.34.3.465    10.1162\/coli.2008.34.3.465\nTed Pedersen. [n.d.]. Empiricism Is Not a Matter of Faith. 34 3 ([n. d.]) 465--470. https:\/\/doi.org\/10.1162\/coli.2008.34.3.465","key":"e_1_3_2_1_22_1","DOI":"10.1162\/coli.2008.34.3.465"},{"key":"e_1_3_2_1_23_1","volume-title":"Fridsma","author":"Piwowar Heather A.","year":"2007","unstructured":"Heather A. Piwowar , Roger S. Day , and Douglas B . Fridsma . 2007 . Sharing detailed research data is associated with increased citation rate. PLoS ONE2, 3 (March2007), e308. https:\/\/doi.org\/10.1371\/journal.pone.0000308 10.1371\/journal.pone.0000308 Heather A. Piwowar, Roger S. Day, and Douglas B. Fridsma. 2007. Sharing detailed research data is associated with increased citation rate. PLoS ONE2, 3 (March2007), e308. https:\/\/doi.org\/10.1371\/journal.pone.0000308"},{"unstructured":"Royal Society (Great Britain) Science Policy Centre and Royal Society (Great Britain). [n.d.]. Science as an open enterprise. OCLC: 840187578.  Royal Society (Great Britain) Science Policy Centre and Royal Society (Great Britain). [n.d.]. Science as an open enterprise. OCLC: 840187578.","key":"e_1_3_2_1_24_1"},{"key":"e_1_3_2_1_25_1","volume-title":"Martin Hrabe de Angelis, and Nadia Rosenthal","author":"Schofield Paul N","year":"2009","unstructured":"Paul N Schofield , Tania Bubela , Thomas Weaver , Lili Portilla , Stephen D Brown , John M Hancock , David Einhorn , Glauco Tocchini-Valentini , Martin Hrabe de Angelis, and Nadia Rosenthal . 2009 . Post-publication sharing of data and tools. Nature 461, 7261 (2009), 171--173. Paul N Schofield, Tania Bubela, Thomas Weaver, Lili Portilla, Stephen D Brown, John M Hancock, David Einhorn, Glauco Tocchini-Valentini, Martin Hrabe de Angelis, and Nadia Rosenthal. 2009. Post-publication sharing of data and tools. Nature 461, 7261 (2009), 171--173."},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_26_1","DOI":"10.1016\/S0167-7799(02)01985-6"},{"doi-asserted-by":"publisher","key":"e_1_3_2_1_27_1","DOI":"10.1002\/asi.23336"},{"doi-asserted-by":"crossref","unstructured":"Mike Thelwall and Kayvan Kousha. 2017. Do journal data sharing mandates work? Life sciences evidence from Dryad. Aslib Journal of Information Management(2017).  Mike Thelwall and Kayvan Kousha. 2017. Do journal data sharing mandates work? Life sciences evidence from Dryad. Aslib Journal of Information Management(2017).","key":"e_1_3_2_1_28_1","DOI":"10.1108\/AJIM-09-2016-0159"},{"key":"e_1_3_2_1_29_1","volume-title":"Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC medical research methodology 12, 1","author":"Tong Allison","year":"2012","unstructured":"Allison Tong , Kate Flemming , Elizabeth McInnes , Sandy Oliver , and Jonathan Craig . 2012. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC medical research methodology 12, 1 ( 2012 ), 181. Allison Tong, Kate Flemming, Elizabeth McInnes, Sandy Oliver, and Jonathan Craig. 2012. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC medical research methodology 12, 1 (2012), 181."},{"key":"e_1_3_2_1_30_1","first-page":"1","article-title":"Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra","volume":"1","author":"Vanpaemel Wolf","year":"2015","unstructured":"Wolf Vanpaemel , Maarten Vermorgen , Leen Deriemaecker , and Gert Storms . 2015 . Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra : Psychology 1 , 1 (Oct. 2015), Art. 3. https:\/\/doi.org\/10.1525\/collabra.13 10.1525\/collabra.13 Wolf Vanpaemel, Maarten Vermorgen, Leen Deriemaecker, and Gert Storms. 2015. Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra: Psychology 1, 1 (Oct. 2015), Art. 3. https:\/\/doi.org\/10.1525\/collabra.13","journal-title":"Psychology"},{"key":"e_1_3_2_1_31_1","volume-title":"Reproducible and reusable research: are journal data sharing policies meeting the mark? Peer J5","author":"Vasilevsky Nicole A","year":"2017","unstructured":"Nicole A Vasilevsky , Jessica Minnier , Melissa A Haendel , and Robin E Champieux . 2017. Reproducible and reusable research: are journal data sharing policies meeting the mark? Peer J5 ( 2017 ), e3208. Nicole A Vasilevsky, Jessica Minnier, Melissa A Haendel, and Robin E Champieux. 2017. Reproducible and reusable research: are journal data sharing policies meeting the mark? Peer J5 (2017), e3208."},{"key":"e_1_3_2_1_32_1","volume-title":"Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Boninoda Silva Santos, Philip E Bourne, et al.","author":"Wilkinson Mark D","year":"2016","unstructured":"Mark D Wilkinson , Michel Dumontier , IJsbrand Jan Aalbersberg , Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Boninoda Silva Santos, Philip E Bourne, et al. 2016 . The FAIR Guiding Principles for scientific data management and stewardship.Scientific data3 (2016). Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Boninoda Silva Santos, Philip E Bourne, et al. 2016. The FAIR Guiding Principles for scientific data management and stewardship.Scientific data3 (2016)."}],"event":{"sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval","IEEE Institute of Electrical and Electronics Engineers"],"acronym":"JCDL '20","name":"JCDL '20: The ACM\/IEEE Joint Conference on Digital Libraries in 2020","location":"Virtual Event China"},"container-title":["Proceedings of the ACM\/IEEE Joint Conference on Digital Libraries in 2020"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383583.3398537","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3383583.3398537","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:11Z","timestamp":1750197671000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3383583.3398537"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8]]},"references-count":33,"alternative-id":["10.1145\/3383583.3398537","10.1145\/3383583"],"URL":"https:\/\/doi.org\/10.1145\/3383583.3398537","relation":{},"subject":[],"published":{"date-parts":[[2020,8]]},"assertion":[{"value":"2020-08-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}