{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T07:27:28Z","timestamp":1772868448039,"version":"3.50.1"},"reference-count":20,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2024,4,25]],"date-time":"2024-04-25T00:00:00Z","timestamp":1714003200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,4,25]],"date-time":"2024-04-25T00:00:00Z","timestamp":1714003200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000083","name":"Directorate for Computer and Information Science and Engineering","doi-asserted-by":"publisher","award":["2138259"],"award-info":[{"award-number":["2138259"]}],"id":[{"id":"10.13039\/100000083","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000085","name":"Directorate for Geosciences","doi-asserted-by":"publisher","award":["1852977"],"award-info":[{"award-number":["1852977"]}],"id":[{"id":"10.13039\/100000085","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The Allocations Service for the Advanced Cyberinfrastructure Coordination Ecosystem: Services &amp; Support (ACCESS) program is charged with accepting, reviewing, and processing researchers\u2019 requests to use resources that are integrated into the ACCESS ecosystem. We present as a case study the metrics framework used to evaluate the Allocations Service project, a metrics framework that aligns with the project\u2019s goals and identifies key performance indicators (KPIs). Several of our top-level KPIs reflect complex concepts and are composite measures built from suites of metrics compiled from two primary sources: a well-instrumented allocations and accounting system and an annual survey of the ACCESS researcher community. This approach allows us to describe and measure complex concepts such as \u201cdemocratization\u201d and \u201cecosystem access time\u201d in a quantitative manner and to target improvements to project activities. The metrics framework is augmented by metrics to measure the performance of the project team, to describe general ecosystem and allocations activity, and to capture publications from the researcher community. We used this framework to gather and present data as part of the ACCESS Allocations Service first annual NSF panel review. The metrics were largely successful at communicating our progress, but we also encountered a few unexpected technical issues with the data and calculations themselves, which we are continuing to refine. Presented here as a case study, this approach to a metrics framework for the Allocations Service has proved valuable in complementing more subjective descriptions of the project, its accomplishments, and progress toward our goals.<\/jats:p>","DOI":"10.1007\/s42979-024-02787-4","type":"journal-article","created":{"date-parts":[[2024,4,25]],"date-time":"2024-04-25T11:01:37Z","timestamp":1714042897000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["From Vision to Evaluation: A Metrics Framework for the ACCESS Allocations Service"],"prefix":"10.1007","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6393-0967","authenticated-orcid":false,"given":"David L.","family":"Hart","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3763-9436","authenticated-orcid":false,"given":"Stephen L.","family":"Deems","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1948-2199","authenticated-orcid":false,"given":"Laura T.","family":"Herriott","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,4,25]]},"reference":[{"key":"2787_CR1","unstructured":"ACCESS Allocations (n.d.) ACCESS allocations policies. https:\/\/allocations.access-ci.org\/access-allocations-policies. Accessed 5 Sep 2023."},{"key":"2787_CR2","unstructured":"American Council on Education. Carnegie classification of institutions of higher education. https:\/\/carnegieclassifications.acenet.edu\/ (2023). Accessed 16 Sep 2023"},{"key":"2787_CR3","doi-asserted-by":"publisher","unstructured":"Boerner TJ, Deems S, Furlani TR, Knuth SL, Towns J. ACCESS: advancing innovation: NSF\u2019s advanced cyberinfrastructure coordination ecosystem: services & support. In Practice and Experience in Advanced Research Computing (PEARC23). Association for Computing Machinery, New York, NY, USA, 2023. 173\u2013176. https:\/\/doi.org\/10.1145\/3569951.3597559.","DOI":"10.1145\/3569951.3597559."},{"key":"2787_CR4","unstructured":"Committee for Scientific and Technological Policy. Revised field of science and technology (FOS) classification in the frascati manual. Organisation for Economic Co-operation and Development. https:\/\/www.oecd.org\/science\/inno\/38235147.pdf (2007). Accessed 17 Sep 2023."},{"key":"2787_CR5","doi-asserted-by":"publisher","unstructured":"Hart DL, Schuele A, Soriano E, Dahan M, Hanlon M. XRAS: allocations software as a service in XSEDE. XSEDE14: Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment. 2014. https:\/\/doi.org\/10.1145\/2616498.2616562.","DOI":"10.1145\/2616498.2616562."},{"key":"2787_CR6","unstructured":"Jensen E, Jones N, Medina L, Perry M, Bolender B, Battle K. Measuring racial and ethnic diversity for the 2020 census. U.S. Census Bureau https:\/\/www.census.gov\/newsroom\/blogs\/random-samplings\/2021\/08\/measuring-racial-ethnic-diversity-2020-census.html (2021). Accessed 17 Sep 2023."},{"key":"2787_CR7","unstructured":"Jones N, Marks R, Ramirez R, R\u00edos-Vargas M. 2020 Census illuminates racial and ethnic composition of the country. U.S. Census Bureau, https:\/\/www.census.gov\/library\/stories\/2021\/08\/improved-race-ethnicity-measures-reveal-united-states-population-much-more-multiracial.html (2021). Accessed 25 Sep 2023."},{"issue":"1","key":"2787_CR8","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1515\/ntrev-2012-0043","volume":"2","author":"K Madhavan","year":"2013","unstructured":"Madhavan K, Zentner L, Farnsworth V, Shivarajapura S, Zentner M, Denny N, Klimeck G. nanoHUB.org: Cloud-based services for nanoscale modeling, simulation, and education. Nanotechnol Rev. 2013;2(1):107\u201317. https:\/\/doi.org\/10.1515\/ntrev-2012-0043.","journal-title":"Nanotechnol Rev"},{"key":"2787_CR9","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1108\/978-1-78973-269-620191002","volume-title":"The new metrics: practical assessment of research impact","author":"K Maull","year":"2019","unstructured":"Maull K, Mayernik M. Scholarly metrics at NCAR. In: Lasda EM, editor. The new metrics: practical assessment of research impact. Emerald Publishing: Bingley; 2019. p. 15\u201338. https:\/\/doi.org\/10.1108\/978-1-78973-269-620191002."},{"key":"2787_CR10","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23721","author":"M Mayernik","year":"2016","unstructured":"Mayernik M, Hart DL, Maull K, Weber NM. Assessing and tracing the outcomes and impact of research infrastructures. J Assoc Inf Sci Technol. 2016. https:\/\/doi.org\/10.1002\/asi.23721.","journal-title":"J Assoc Inf Sci Technol"},{"key":"2787_CR11","unstructured":"National Science Foundation. Advanced cyberinfrastructure coordination ecosystem: services & support (ACCESS). https:\/\/www.nsf.gov\/pubs\/2021\/nsf21555\/nsf21555.htm (2021). Accessed Sep 5 2023."},{"key":"2787_CR12","unstructured":"Okrent A, Burke A. The STEM labor force of today: scientists, engineers, and skilled technical workers. National Science Board, https:\/\/ncses.nsf.gov\/pubs\/nsb20212\/participation-of-demographic-groups-in-stem (2021). Accessed 25 Sep 2023."},{"issue":"9","key":"2787_CR13","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1109\/MC.2022.3174928","volume":"55","author":"M Parashar","year":"2022","unstructured":"Parashar M. Democratizing science through advanced cyberinfrastructure. Computer. 2022;55(9):79\u201384. https:\/\/doi.org\/10.1109\/MC.2022.3174928.","journal-title":"Computer"},{"key":"2787_CR14","doi-asserted-by":"publisher","DOI":"10.1038\/163688a0","author":"EH Simpson","year":"1949","unstructured":"Simpson EH. Measurement of diversity. Nature. 1949. https:\/\/doi.org\/10.1038\/163688a0.","journal-title":"Nature"},{"key":"2787_CR15","unstructured":"Singh P. Simpson\u2019s diversity index calculator. Omni calculator. https:\/\/www.omnicalculator.com\/statistics\/simpsons-diversity-index (2023). Accessed 5 Sep 2023."},{"issue":"5","key":"2787_CR16","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1109\/MCSE.2014.80","volume":"16","author":"J Towns","year":"2014","unstructured":"Towns J, Cockerill T, Dahan M, et al. XSEDE: accelerating scientific discovery. Comput Sci Eng. 2014;16(5):62\u201374. https:\/\/doi.org\/10.1109\/MCSE.2014.80.","journal-title":"Comput Sci Eng"},{"key":"2787_CR17","doi-asserted-by":"publisher","DOI":"10.1101\/2023.06.16.545223","author":"L Tyson","year":"2023","unstructured":"Tyson L, Swetnam PB, Antin RB, et al. CyVerse: cyberinfrastructure for open science. bioRrxiv. 2023. https:\/\/doi.org\/10.1101\/2023.06.16.545223.","journal-title":"bioRrxiv"},{"key":"2787_CR18","unstructured":"U.S. Census Bureau (2022) Population. https:\/\/www.census.gov\/topics\/population.html. Accessed 5 Sep 2023."},{"key":"2787_CR19","unstructured":"XSEDE. XSEDE allocations practices and procedures version 1.1. http:\/\/hdl.handle.net\/2142\/110005 (2021). Accessed 17 Sep 2023."},{"key":"2787_CR20","unstructured":"XSEDE. XSEDE: the extreme science and engineering discovery environment (OAC 15-48562) Annual report: report year 6; May 1, 2021\u2013April 30, 2022. https:\/\/hdl.handle.net\/2142\/114407 (2022). Accessed 5 Sep 2023."}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-024-02787-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-024-02787-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-024-02787-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,25]],"date-time":"2024-04-25T11:28:36Z","timestamp":1714044516000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-024-02787-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,25]]},"references-count":20,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["2787"],"URL":"https:\/\/doi.org\/10.1007\/s42979-024-02787-4","relation":{},"ISSN":["2661-8907"],"issn-type":[{"value":"2661-8907","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,25]]},"assertion":[{"value":"26 September 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 March 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 April 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Financial Interests: The authors declare they have no financial interests. Non-Financial Interests: The authors, through the ACCESS Allocations Service award, are responsible for the eXtensible Resource Allocations Service (XRAS), which is available to third parties via a for-fee service agreement with the University of Illinois, Urbana-Champaign.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"No human subjects research was conducted as part of the preparation of this manuscript. Allocations activity data in this report are gathered and used in accordance with the ACCESS Privacy Policy (). Surveys and survey methodology were approved by the NCAR Institutional Review Board.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}}],"article-number":"490"}}