{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:35:31Z","timestamp":1760232931791,"version":"build-2065373602"},"reference-count":55,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,12,2]],"date-time":"2022-12-02T00:00:00Z","timestamp":1669939200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Publications"],"abstract":"<jats:p>Performance evaluation is a broad discipline within computer science, combining deep technical work in experimentation, simulation, and modeling. The field\u2019s subjects encompass all aspects of computer systems, including computer architecture, networking, energy efficiency, and machine learning. This wide methodological and topical focus can make it difficult to discern what attracts the community\u2019s attention and how this attention evolves over time. As a first attempt to quantify and qualify this attention, using the proxy metric of paper citations, this study looks at the premier conference in the field, SIGMETRICS. We analyze citation frequencies at monthly intervals over a five-year period and examine possible associations with myriad other factors, such as time since publication, comparable conferences, peer review, self-citations, author demographics, and textual properties of the papers. We found that in several ways, SIGMETRICS is distinctive not only in its scope, but also in its citation phenomena: papers generally exhibit a strongly linear rate of citation growth over time, few if any uncited papers, a large gamut of topics of interest, and a possible disconnect between peer-review outcomes and eventual citations. The two most-cited papers in the dataset also exhibit larger author teams, higher than typical self-citations, and distinctive citation growth curves. These two papers, sharing some coauthors and a research focus, could either signal the area where SIGMETRICS had the most research impact, or they could represent outliers; their omission from the analysis reduces some of the otherwise distinctive observed metrics to nonsignificant levels.<\/jats:p>","DOI":"10.3390\/publications10040047","type":"journal-article","created":{"date-parts":[[2022,12,5]],"date-time":"2022-12-05T04:40:09Z","timestamp":1670215209000},"page":"47","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Multifactor Citation Analysis over Five Years: A Case Study of SIGMETRICS Papers"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3709-1829","authenticated-orcid":false,"given":"Eitan","family":"Frachtenberg","sequence":"first","affiliation":[{"name":"Department of Computer Science, Reed College, 3203 Woodstock Blvd, Portland, OR 97202, USA"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,2]]},"reference":[{"key":"ref_1","unstructured":"(2022, October 26). ACM Special Interest Group on Performance Evaluation. Available online: http:\/\/www.sigmetrics.org."},{"key":"ref_2","unstructured":"(2022, July 26). ACM Conference Statistics for SIGMETRICS. Available online: https:\/\/dl.acm.org\/conference\/metrics."},{"key":"ref_3","unstructured":"Moed, H.F. (2006). Citation Analysis in Research Evaluation, Springer Science & Business Media."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Broch, E. (2001, January 9\u201313). Cite me, cite my references? (Scholarly use of the ACM SIGIR proceedings based on two citation indexes). Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA, USA.","DOI":"10.1145\/383952.384090"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1145\/3371934.3371948","article-title":"Five decades of the ACM special interest group on data communications (SIGCOMM) a bibliometric perspective","volume":"49","author":"Iqbal","year":"2019","journal-title":"ACM SIGCOMM Comput. Commun. Rev."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1007\/s11192-019-03086-z","article-title":"A bibliometric analysis of publications in computer networking research","volume":"119","author":"Iqbal","year":"2019","journal-title":"Scientometrics"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1145\/1107499.1107505","article-title":"Citation analysis of database publications","volume":"34","author":"Rahm","year":"2005","journal-title":"ACM Sigmod Rec."},{"key":"ref_8","first-page":"43","article-title":"Getting noticed in economics: The determinants of academic citations","volume":"41","author":"Johnson","year":"1997","journal-title":"Am. Econ."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"858","DOI":"10.1002\/asi.21011","article-title":"The decline in the concentration of citations, 1900\u20132007","volume":"60","author":"Gingras","year":"2009","journal-title":"J. Am. Soc. Inf. Sci. Technol."},{"key":"ref_10","unstructured":"Bernius, S., and Hanauske, M. (2009, January 5\u20138). Open access to scientific literature-increasing citations as an incentive for authors to make their publications freely accessible. Proceedings of the 42nd Hawaii International Conference on System Sciences, Waikoloa, HI, USA."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1284","DOI":"10.1111\/ecin.12064","article-title":"Identifying the effect of open access on citations using a panel of science journals","volume":"52","author":"McCabe","year":"2014","journal-title":"Econ. Inq."},{"key":"ref_12","first-page":"823","article-title":"Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation\u2014Review of the literature","volume":"11","author":"Halevi","year":"2017","journal-title":"J. Inf."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"787","DOI":"10.1007\/s11192-015-1798-9","article-title":"Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison","volume":"106","author":"Harzing","year":"2016","journal-title":"Scientometrics"},{"key":"ref_14","first-page":"1160","article-title":"Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories","volume":"12","author":"Thelwall","year":"2018","journal-title":"J. Inf."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1145\/3516431.3516438","article-title":"How Inclusive are We? An Analysis of Gender Diversity in Database Venues","volume":"50","author":"Bonifati","year":"2022","journal-title":"ACM SIGMOD Rec."},{"key":"ref_16","first-page":"332","article-title":"What is gender, anyway: A review of the options for operationalising gender","volume":"12","author":"Lindqvist","year":"2020","journal-title":"Psychol. Sex."},{"key":"ref_17","first-page":"77","article-title":"Data and Techniques Used for Analysis of Women Authorship in STEMM: A Review","volume":"2","author":"Bhagat","year":"2018","journal-title":"Fem. Res."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1145\/1978542.1978561","article-title":"Gender and Computing Conference Papers","volume":"54","author":"Cohoon","year":"2011","journal-title":"Commun. ACM"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Holman, L., Stuart-Fox, D., and Hauser, C.E. (2018). The gender gap in science: How long until women are equally represented?. PLoS Biol., 16.","DOI":"10.1371\/journal.pbio.2004956"},{"key":"ref_20","unstructured":"National Science Board (US) (2020). The State of U.S. Science and Engineering, National Science Board (US)."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/3430803","article-title":"Gender Trends in Computer Science Authorship","volume":"64","author":"Wang","year":"2021","journal-title":"Commun. ACM"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Way, S.F., Larremore, D.B., and Clauset, A. (2016, January 11\u201315). Gender, productivity, and prestige in computer science faculty hiring networks. Proceedings of the 25th International Conference on World Wide Web, Montreal, QC, Canada.","DOI":"10.1145\/2872427.2883073"},{"key":"ref_23","first-page":"1","article-title":"2017 CRA Taulbee Survey","volume":"30","author":"Zweben","year":"2018","journal-title":"Comput. Res. News"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1145\/3376901","article-title":"A Bibliometric Approach for Detecting the Gender Gap in Computer Science","volume":"63","author":"Mattauch","year":"2020","journal-title":"Commun. ACM"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e156","DOI":"10.7717\/peerj-cs.156","article-title":"Comparison and benchmark of name-to-gender inference services","volume":"4","author":"Santamaria","year":"2018","journal-title":"PeerJ Comput. Sci."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1007\/s100510050359","article-title":"How popular is your paper? An empirical study of the citation distribution","volume":"4","author":"Redner","year":"1998","journal-title":"Eur. Phys. J. B-Condens. Matter Complex Syst."},{"key":"ref_27","unstructured":"Wang, D., and Barab\u00e1si, A.L. (2021). The Science of Science, Cambridge University Press."},{"key":"ref_28","unstructured":"Wu, L.L., Luesukprasert, L., and Lee, L. (2009, January 5\u20138). Research and the long tail: A large-scale citation analysis. Proceedings of the 42nd Hawaii International Conference on System Sciences, Waikoloa, HI, USA."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1126\/science.1986409","article-title":"Who\u2019s uncited now?","volume":"251","author":"Hamilton","year":"1991","journal-title":"Science"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1258\/shorts.2009.100020","article-title":"The impact of article titles on citation hits: An analysis of general and specialist medical journals","volume":"1","author":"Jacques","year":"2010","journal-title":"JRSM Short Rep."},{"key":"ref_31","unstructured":"Pichappan, P., and Ponnudurai, R. (1999, January 5\u20138). Skewness in citation peak. Proceedings of the 7th Conference of the International Society for Scientometrics and Informetrics, Colima, Mexico."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"851","DOI":"10.1007\/s11192-012-0775-9","article-title":"Citation time window choice for research impact evaluation","volume":"94","author":"Wang","year":"2013","journal-title":"Scientometrics"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1023\/A:1021919228368","article-title":"A macro study of self-citation","volume":"56","author":"Aksnes","year":"2003","journal-title":"Scientometrics"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1177\/016555159802400606","article-title":"Patterns of self-citation across disciplines (1980\u20131989)","volume":"24","author":"Snyder","year":"1998","journal-title":"J. Inf. Sci."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1023\/B:SCIE.0000013299.38210.74","article-title":"A bibliometric approach to the role of author self-citations in scientific communication","volume":"59","author":"Wolfgang","year":"2004","journal-title":"Scientometrics"},{"key":"ref_36","first-page":"365","article-title":"A review of the literature on citation impact indicators","volume":"10","author":"Waltman","year":"2016","journal-title":"J. Inf."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"774","DOI":"10.21105\/joss.00774","article-title":"quanteda: An R package for the quantitative analysis of textual data","volume":"3","author":"Benoit","year":"2018","journal-title":"J. Open Source Softw."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Sch\u00fctze, H., Manning, C.D., and Raghavan, P. (2008). Introduction to Information Retrieval, Cambridge University Press.","DOI":"10.1017\/CBO9780511809071"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"147470490900700301","DOI":"10.1177\/147470490900700301","article-title":"Hot topics and popular papers in evolutionary psychology: Analyses of title words and citation counts in Evolution and Human Behavior, 1979\u20132008","volume":"7","author":"Webster","year":"2009","journal-title":"Evol. Psychol."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"763","DOI":"10.1007\/s11192-019-03015-0","article-title":"The correlation between paper length and citations: A meta-analysis","volume":"118","author":"Xie","year":"2019","journal-title":"Scientometrics"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"7717","DOI":"10.1002\/ece3.2505","article-title":"Citations increase with manuscript length, author number, and references cited in ecology journals","volume":"6","author":"Fox","year":"2016","journal-title":"Ecol. Evol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.econlet.2019.02.017","article-title":"Readability and research impact","volume":"180","author":"McCannon","year":"2019","journal-title":"Econ. Lett."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Kincaid, J.P., Fishburne, R.P., Rogers, R.L., and Chissom, B.S. (1975). Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel, Naval Technical Training Command Millington TN Research Branch. Technical report.","DOI":"10.21236\/ADA006655"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1007\/s11192-011-0412-z","article-title":"Article title type and its relation with the number of downloads and citations","volume":"88","author":"Jamali","year":"2011","journal-title":"Scientometrics"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1016\/j.respol.2012.05.004","article-title":"Peer review versus citations\u2013An analysis of best paper prizes","volume":"42","year":"2013","journal-title":"Res. Policy"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1007\/s11192-018-2943-z","article-title":"Predictive power of conference-related factors on citation rates of conference papers","volume":"118","author":"Lee","year":"2019","journal-title":"Scientometrics"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Wainer, J., Eckmann, M., and Rocha, A. (2015). Peer-selected \u201cbest papers\u201d\u2014Are they really that \u201cgood\u201d?. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0118446"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"e299","DOI":"10.7717\/peerj-cs.299","article-title":"A Survey of Accepted Authors in Computer Systems Conferences","volume":"6","author":"Frachtenberg","year":"2020","journal-title":"PeerJ Comput. Sci."},{"key":"ref_49","unstructured":"Clarivate Analytics (2018). Global State of Peer Review, Clarivate."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1195","DOI":"10.1007\/s11192-016-1889-2","article-title":"Factors affecting number of citations: A comprehensive review of the literature","volume":"107","author":"Tahamtan","year":"2016","journal-title":"Scientometrics"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Frachtenberg, E., and Kaner, R.D. (2022). Underrepresentation of women in computer systems research. PLoS ONE, 17.","DOI":"10.1371\/journal.pone.0266439"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/2378023117738903","article-title":"Men set their own cites high: Gender and self-citation across fields and over time","volume":"3","author":"King","year":"2017","journal-title":"Socius"},{"key":"ref_53","unstructured":"Garc\u00eda-Portugu\u00e9s, E. (2022, October 26). Notes for Predictive Modeling, Version 5.8.6; 2021; ISBN 978-84-09-29679-8. Available online: https:\/\/bookdown.org\/egarpor\/PM-UC3M\/."},{"key":"ref_54","unstructured":"Fran\u00e7ois, O. (2015). Arbitrariness of peer review: A Bayesian analysis of the NIPS experiment. arXiv."},{"key":"ref_55","unstructured":"Frachtenberg, E. (2022, October 26). Systems Conferences Analysis Dataset. Available online: https:\/\/doi.org\/10.5281\/zenodo.5590574."}],"container-title":["Publications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2304-6775\/10\/4\/47\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:32:48Z","timestamp":1760146368000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2304-6775\/10\/4\/47"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,2]]},"references-count":55,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["publications10040047"],"URL":"https:\/\/doi.org\/10.3390\/publications10040047","relation":{},"ISSN":["2304-6775"],"issn-type":[{"type":"electronic","value":"2304-6775"}],"subject":[],"published":{"date-parts":[[2022,12,2]]}}}