{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T17:58:50Z","timestamp":1770227930464,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":19,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,4,25]],"date-time":"2022-04-25T00:00:00Z","timestamp":1650844800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["W911NF-19-2- 0272"],"award-info":[{"award-number":["W911NF-19-2- 0272"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000208","name":"Institute of Museum and Library Services","doi-asserted-by":"publisher","award":["LG-37-19-0078-19"],"award-info":[{"award-number":["LG-37-19-0078-19"]}],"id":[{"id":"10.13039\/100000208","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,4,25]]},"DOI":"10.1145\/3487553.3524658","type":"proceedings-article","created":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T22:41:30Z","timestamp":1660689690000},"page":"784-788","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["A Study of Computational Reproducibility using URLs Linking to Open Access Datasets and Software"],"prefix":"10.1145","author":[{"given":"Lamia","family":"Salsabil","sequence":"first","affiliation":[{"name":"Old Dominion University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jian","family":"Wu","sequence":"additional","affiliation":[{"name":"Old Dominion University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Muntabir Hasan","family":"Choudhury","sequence":"additional","affiliation":[{"name":"Old Dominion University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William A.","family":"Ingram","sequence":"additional","affiliation":[{"name":"Virginia Polytechnic Institute and State University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Edward A.","family":"Fox","sequence":"additional","affiliation":[{"name":"Virginia Polytechnic Institute and State University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sarah M.","family":"Rajtmajer","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"C. Lee","family":"Giles","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,8,16]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"John\u00a0T Cacioppo Robert\u00a0M Kaplan Jon\u00a0A Krosnick James\u00a0L Olds and Heather Dean. 2015. Social behavioral and economic sciences perspectives on robust and reliable science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social Behavioral and Economic Sciences 1(2015).  John\u00a0T Cacioppo Robert\u00a0M Kaplan Jon\u00a0A Krosnick James\u00a0L Olds and Heather Dean. 2015. Social behavioral and economic sciences perspectives on robust and reliable science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National Science Foundation Directorate for Social Behavioral and Economic Sciences 1(2015)."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-018-0399-z"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.207"},{"key":"e_1_3_2_1_4_1","volume-title":"Estimating the reproducibility of psychological science. Science 349, 6251","author":"Collaboration Open\u00a0Science","year":"2015","unstructured":"Open\u00a0Science Collaboration . 2015. Estimating the reproducibility of psychological science. Science 349, 6251 ( 2015 ). https:\/\/doi.org\/10.1126\/science.aac4716 arXiv:https:\/\/science.sciencemag.org\/content\/349\/6251\/aac4716.full.pdf 10.1126\/science.aac4716 Open\u00a0Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015). https:\/\/doi.org\/10.1126\/science.aac4716 arXiv:https:\/\/science.sciencemag.org\/content\/349\/6251\/aac4716.full.pdf"},{"key":"e_1_3_2_1_5_1","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","volume":"1","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019 , Minneapolis, MN, USA , June 2-7, 2019, Volume 1 (Long and Short Papers). 4171\u20134186. https:\/\/aclweb.org\/anthology\/papers\/N\/N19\/N19-1423\/ Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). 4171\u20134186. https:\/\/aclweb.org\/anthology\/papers\/N\/N19\/N19-1423\/"},{"key":"e_1_3_2_1_6_1","volume-title":"Analyzing the GitHub Repositories of Research Papers","author":"F\u00e4rber Michael","unstructured":"Michael F\u00e4rber . 2020. Analyzing the GitHub Repositories of Research Papers . Association for Computing Machinery , New York, NY, USA , 491\u2013492. https:\/\/doi.org\/10.1145\/3383583.3398578 10.1145\/3383583.3398578 Michael F\u00e4rber. 2020. Analyzing the GitHub Repositories of Research Papers. Association for Computing Machinery, New York, NY, USA, 491\u2013492. https:\/\/doi.org\/10.1145\/3383583.3398578"},{"key":"e_1_3_2_1_7_1","unstructured":"Behnam Ghavimi Philipp Mayr Sahar Vahdati and Christoph Lange. 2016. Identifying and improving dataset references in social sciences full texts. arXiv preprint arXiv:1603.01774(2016).  Behnam Ghavimi Philipp Mayr Sahar Vahdati and Christoph Lange. 2016. Identifying and improving dataset references in social sciences full texts. arXiv preprint arXiv:1603.01774(2016)."},{"key":"e_1_3_2_1_8_1","volume-title":"\u00a0A. Ioannidis","author":"Goodman N.","year":"2016","unstructured":"Steven\u00a0 N. Goodman , Daniele Fanelli , and John P . \u00a0A. Ioannidis . 2016 . What does research reproducibility mean?Science Translational Medicine 8, 341 (2016), 341ps12\u2013341ps12. https:\/\/doi.org\/10.1126\/scitranslmed.aaf5027 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/scitranslmed.aaf5027 10.1126\/scitranslmed.aaf5027 Steven\u00a0N. Goodman, Daniele Fanelli, and John P.\u00a0A. Ioannidis. 2016. What does research reproducibility mean?Science Translational Medicine 8, 341 (2016), 341ps12\u2013341ps12. https:\/\/doi.org\/10.1126\/scitranslmed.aaf5027 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/scitranslmed.aaf5027"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-010-0202-z"},{"key":"e_1_3_2_1_10_1","volume-title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints (July","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu , Myle Ott , Naman Goyal , Jingfei Du , Mandar Joshi , Danqi Chen , Omer Levy , Mike Lewis , Luke Zettlemoyer , and Veselin Stoyanov . 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints (July 2019 ). arxiv:1907.11692\u00a0[cs.CL] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints (July 2019). arxiv:1907.11692\u00a0[cs.CL]"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-4007"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13222-019-00317-8"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00077"},{"key":"e_1_3_2_1_14_1","volume-title":"a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR","author":"Sanh Victor","year":"2019","unstructured":"Victor Sanh , Lysandre Debut , Julien Chaumond , and Thomas Wolf . 2019. DistilBERT , a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR ( 2019 ). arXiv:1910.01108 Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR (2019). arXiv:1910.01108"},{"key":"e_1_3_2_1_15_1","volume-title":"Extraction and Evaluation of Statistical Information from Social and Behavioral Science Papers","author":"Teja\u00a0Lanka Sree\u00a0Sai","unstructured":"Sree\u00a0Sai Teja\u00a0Lanka , Sarah Rajtmajer , Jian Wu , and C.\u00a0 Lee Giles . 2021. Extraction and Evaluation of Statistical Information from Social and Behavioral Science Papers . Association for Computing Machinery , New York, NY, USA , 426\u2013430. https:\/\/doi.org\/10.1145\/3442442.3451363 10.1145\/3442442.3451363 Sree\u00a0Sai Teja\u00a0Lanka, Sarah Rajtmajer, Jian Wu, and C.\u00a0Lee Giles. 2021. Extraction and Evaluation of Statistical Information from Social and Behavioral Science Papers. Association for Computing Machinery, New York, NY, USA, 426\u2013430. https:\/\/doi.org\/10.1145\/3442442.3451363"},{"key":"e_1_3_2_1_16_1","volume-title":"Building A Large Collection of Multi-domain Electronic Theses and Dissertations. In 2021 IEEE International Conference on Big Data (Big Data). IEEE Computer Society, 6043\u20136045","author":"Uddin Sami","year":"2021","unstructured":"Sami Uddin , Bipasha Banerjee , Jian Wu , William\u00a0 A Ingram , and Edward\u00a0 A Fox . 2021 . Building A Large Collection of Multi-domain Electronic Theses and Dissertations. In 2021 IEEE International Conference on Big Data (Big Data). IEEE Computer Society, 6043\u20136045 . DOI: 10.1109\/BigData52589.2021.9672058 10.1109\/BigData52589.2021.9672058 Sami Uddin, Bipasha Banerjee, Jian Wu, William\u00a0A Ingram, and Edward\u00a0A Fox. 2021. Building A Large Collection of Multi-domain Electronic Theses and Dissertations. In 2021 IEEE International Conference on Big Data (Big Data). IEEE Computer Society, 6043\u20136045. DOI: 10.1109\/BigData52589.2021.9672058"},{"key":"e_1_3_2_1_17_1","volume-title":"CORD-19: The Covid-19 Open Research Dataset. CoRR","author":"Wang Lucy\u00a0Lu","year":"2020","unstructured":"Lucy\u00a0Lu Wang , Kyle Lo , Yoganand Chandrasekhar , Russell Reas , Jiangjiang Yang , Darrin Eide , Kathryn Funk , Rodney Kinney , Ziyang Liu , William Merrill , Paul Mooney , Dewey\u00a0 A. Murdick , Devvret Rishi , Jerry Sheehan , Zhihong Shen , Brandon Stilson , Alex\u00a0 D. Wade , Kuansan Wang , Chris Wilhelm , Boya Xie , Douglas Raymond , Daniel\u00a0 S. Weld , Oren Etzioni , and Sebastian Kohlmeier . 2020. CORD-19: The Covid-19 Open Research Dataset. CoRR ( 2020 ). arXiv:2004.10706 Lucy\u00a0Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, Paul Mooney, Dewey\u00a0A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex\u00a0D. Wade, Kuansan Wang, Chris Wilhelm, Boya Xie, Douglas Raymond, Daniel\u00a0S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020. CORD-19: The Covid-19 Open Research Dataset. CoRR (2020). arXiv:2004.10706"},{"key":"e_1_3_2_1_18_1","volume-title":"Arjun\u00a0Manoj Menon, Sai\u00a0Ajay Modukuri, Nishanth Nakshatri, Xin Wei, Zhuoer Wang, James Caverlee, Sarah\u00a0Michele Rajtmajer, and C.\u00a0Lee Giles.","author":"Wu Jian","year":"2021","unstructured":"Jian Wu , Rajal Nivargi , Sree Sai\u00a0Teja Lanka , Arjun\u00a0Manoj Menon, Sai\u00a0Ajay Modukuri, Nishanth Nakshatri, Xin Wei, Zhuoer Wang, James Caverlee, Sarah\u00a0Michele Rajtmajer, and C.\u00a0Lee Giles. 2021 . Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models. CoRR ( 2021). arXiv:2104.04580 Jian Wu, Rajal Nivargi, Sree Sai\u00a0Teja Lanka, Arjun\u00a0Manoj Menon, Sai\u00a0Ajay Modukuri, Nishanth Nakshatri, Xin Wei, Zhuoer Wang, James Caverlee, Sarah\u00a0Michele Rajtmajer, and C.\u00a0Lee Giles. 2021. Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models. CoRR (2021). arXiv:2104.04580"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1909046117"}],"event":{"name":"WWW '22: The ACM Web Conference 2022","location":"Virtual Event, Lyon France","acronym":"WWW '22","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"]},"container-title":["Companion Proceedings of the Web Conference 2022"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487553.3524658","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3487553.3524658","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3487553.3524658","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:22Z","timestamp":1750188622000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487553.3524658"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,25]]},"references-count":19,"alternative-id":["10.1145\/3487553.3524658","10.1145\/3487553"],"URL":"https:\/\/doi.org\/10.1145\/3487553.3524658","relation":{},"subject":[],"published":{"date-parts":[[2022,4,25]]},"assertion":[{"value":"2022-08-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}