{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T23:08:40Z","timestamp":1768000120230,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":18,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,6,21]],"date-time":"2021-06-21T00:00:00Z","timestamp":1624233600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Sciences and Engineering Research Council of Canada","award":["RGPIN-2020-05001"],"award-info":[{"award-number":["RGPIN-2020-05001"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,6,21]]},"DOI":"10.1145\/3463274.3463359","type":"proceedings-article","created":{"date-parts":[[2021,6,18]],"date-time":"2021-06-18T23:43:11Z","timestamp":1624059791000},"page":"353-358","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Towards a More Structured Peer Review Process with Empirical Standards"],"prefix":"10.1145","author":[{"given":"Arham","family":"Arshad","sequence":"first","affiliation":[{"name":"Dalhousie University, Canada"}]},{"given":"Taher","family":"Ghaleb","sequence":"additional","affiliation":[{"name":"Queens University, Canada"}]},{"given":"Paul","family":"Ralph","sequence":"additional","affiliation":[{"name":"Dalhousie University, Canada"}]}],"member":"320","published-online":{"date-parts":[[2021,6,21]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2013.09.001"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"crossref","unstructured":"John Bohannon. 2013. Who\u2019s afraid of peer review?  John Bohannon. 2013. Who\u2019s afraid of peer review?","DOI":"10.1126\/science.2013.342.6154.342_60"},{"key":"e_1_3_2_1_4_1","volume-title":"The case meta-analysis method for OD. Research in organizational change and development 1","author":"Bullock RJ","year":"1987","unstructured":"RJ Bullock and Mark\u00a0 E Tubbs . 1987. The case meta-analysis method for OD. Research in organizational change and development 1 ( 1987 ), 171\u2013228. RJ Bullock and Mark\u00a0E Tubbs. 1987. The case meta-analysis method for OD. Research in organizational change and development 1 (1987), 171\u2013228."},{"key":"e_1_3_2_1_5_1","volume-title":"Chance and consensus in peer review. Science 214, 4523","author":"Cole Stephen","year":"1981","unstructured":"Stephen Cole , Gary\u00a0 A Simon , 1981. Chance and consensus in peer review. Science 214, 4523 ( 1981 ), 881\u2013886. Stephen Cole, Gary\u00a0A Simon, 1981. Chance and consensus in peer review. Science 214, 4523 (1981), 881\u2013886."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sapharm.2012.04.004"},{"key":"e_1_3_2_1_7_1","volume-title":"Ethical dilemmas in scientific publication: pitfalls and solutions for editors","author":"Gollogly Laragh","year":"2006","unstructured":"Laragh Gollogly and Hooman Momen . 2006. Ethical dilemmas in scientific publication: pitfalls and solutions for editors . Revista de Sa\u00fade P\u00fablica 40, SPE ( 2006 ), 24\u201329. Laragh Gollogly and Hooman Momen. 2006. Ethical dilemmas in scientific publication: pitfalls and solutions for editors. Revista de Sa\u00fade P\u00fablica 40, SPE (2006), 24\u201329."},{"key":"e_1_3_2_1_8_1","volume-title":"Answering the call for a standard reliability measure for coding data. Communication methods and measures 1, 1","author":"Hayes F","year":"2007","unstructured":"Andrew\u00a0 F Hayes and Klaus Krippendorff . 2007. Answering the call for a standard reliability measure for coding data. Communication methods and measures 1, 1 ( 2007 ), 77\u201389. Andrew\u00a0F Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication methods and measures 1, 1 (2007), 77\u201389."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.22784"},{"key":"e_1_3_2_1_10_1","volume-title":"Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research 1, 2","author":"Mahoney J","year":"1977","unstructured":"Michael\u00a0 J Mahoney . 1977. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research 1, 2 ( 1977 ), 161\u2013175. Michael\u00a0J Mahoney. 1977. Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive therapy and research 1, 2 (1977), 161\u2013175."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X00011183"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Paul Ralph. 2016. Practical suggestions for improving scholarly peer review quality and reducing cycle times. Communications of the association for information systems 38 1(2016) 13.  Paul Ralph. 2016. Practical suggestions for improving scholarly peer review quality and reducing cycle times. Communications of the association for information systems 38 1(2016) 13.","DOI":"10.17705\/1CAIS.03813"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3437479.3437483"},{"key":"e_1_3_2_1_14_1","unstructured":"Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard\u00a0Nicolau de Fran\u00e7a Carlo\u00a0Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Mart\u00ednez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarim\u00e4ki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2021. Empirical Standards for Software Engineering Research. arxiv:2010.03525\u00a0[cs.SE]  Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard\u00a0Nicolau de Fran\u00e7a Carlo\u00a0Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Mart\u00ednez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarim\u00e4ki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2021. Empirical Standards for Software Engineering Research. arxiv:2010.03525\u00a0[cs.SE]"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3385678.3385681"},{"key":"e_1_3_2_1_16_1","volume-title":"Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?Brain 123, 9","author":"Rothwell M","year":"2000","unstructured":"Peter\u00a0 M Rothwell and Christopher\u00a0 N Martyn . 2000. Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?Brain 123, 9 ( 2000 ), 1964\u20131969. Peter\u00a0M Rothwell and Christopher\u00a0N Martyn. 2000. Reproducibility of peer review in clinical neuroscience: Is agreement between reviewers any greater than would be expected by chance alone?Brain 123, 9 (2000), 1964\u20131969."},{"key":"e_1_3_2_1_17_1","volume-title":"Rating the ratings: Assessing the psychometric quality of rating data.Psychological bulletin 88, 2","author":"Saal E","year":"1980","unstructured":"Frank\u00a0 E Saal , Ronald\u00a0 G Downey , and Mary\u00a0 A Lahey . 1980. Rating the ratings: Assessing the psychometric quality of rating data.Psychological bulletin 88, 2 ( 1980 ), 413. Frank\u00a0E Saal, Ronald\u00a0G Downey, and Mary\u00a0A Lahey. 1980. Rating the ratings: Assessing the psychometric quality of rating data.Psychological bulletin 88, 2 (1980), 413."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.3233\/ISU-2008-0568"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"crossref","unstructured":"John\u00a0M Ziman. 1968. Public knowledge: An essay concerning the social dimension of science. CUP Archive.  John\u00a0M Ziman. 1968. Public knowledge: An essay concerning the social dimension of science. CUP Archive.","DOI":"10.1063\/1.3035233"}],"event":{"name":"EASE 2021: Evaluation and Assessment in Software Engineering","location":"Trondheim Norway","acronym":"EASE 2021"},"container-title":["Evaluation and Assessment in Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3463274.3463359","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3463274.3463359","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:27Z","timestamp":1750195887000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3463274.3463359"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,21]]},"references-count":18,"alternative-id":["10.1145\/3463274.3463359","10.1145\/3463274"],"URL":"https:\/\/doi.org\/10.1145\/3463274.3463359","relation":{},"subject":[],"published":{"date-parts":[[2021,6,21]]},"assertion":[{"value":"2021-06-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}