{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:16:37Z","timestamp":1750306597398,"version":"3.41.0"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,1,20]],"date-time":"2015-01-20T00:00:00Z","timestamp":1421712000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2015,1,20]]},"abstract":"<jats:p>A framework of open interoperable simulators for computer architecture is long overdue. Today there are many separate, uncoordinated efforts to develop simulation and modeling artifacts (tools) for computer architecture research. The artifacts are used to empirically evaluate new computer architecture innovations and compare them with the state of the art. The artifacts are usually developed by individual groups, often for a specific purpose, and may not be publicly released. Consequently, it is difficult to leverage investment in artifact development and to repeat or reproduce experiments. In this position paper, we present recommendations and a roadmap for sharing and building open-source, interoperable simulation and modeling artifacts. The recommendations are the outcome of a community workshop involving industry, government and academia to determine how to coordinate effort, share tools and improve methodology.<\/jats:p>","DOI":"10.1145\/2723872.2723886","type":"journal-article","created":{"date-parts":[[2015,1,28]],"date-time":"2015-01-28T14:05:51Z","timestamp":1422453951000},"page":"108-117","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["A Roadmap and Plan of Action for Community-Supported Empirical Evaluation in Computer Architecture"],"prefix":"10.1145","volume":"49","author":[{"given":"Bruce R.","family":"Childers","sequence":"first","affiliation":[{"name":"University of Pittsburgh, Pittsburgh, PA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alex K.","family":"Jones","sequence":"additional","affiliation":[{"name":"University of Pittsburgh, Pittsburgh, PA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel","family":"Moss\u00e9","sequence":"additional","affiliation":[{"name":"University of Pittsburgh, Pittsburgh, PA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2015,1,20]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"International Symposium on Computer Architecture (ISCA), http:\/\/www.gpgpu-sim.org\/isca2012-tutorial\/, 2012","author":"Aaamodt T. M.","year":"2012","unstructured":"T. M. Aaamodt and A. Boktor . GPGPU-Sim 3.x: A performance simulator for many-core accelerator research . International Symposium on Computer Architecture (ISCA), http:\/\/www.gpgpu-sim.org\/isca2012-tutorial\/, 2012 . {Online; accessed 30- Sep- 2012 }. T. M. Aaamodt and A. Boktor. GPGPU-Sim 3.x: A performance simulator for many-core accelerator research. International Symposium on Computer Architecture (ISCA), http:\/\/www.gpgpu-sim.org\/isca2012-tutorial\/, 2012. {Online; accessed 30-Sep-2012}."},{"key":"e_1_2_1_2_1","volume-title":"International Conference on Parallel Architectures and Compilation Techniques (PACT), http:\/\/www.gpgpu-sim.org\/pact2012-tutorial\/, 2012","author":"Aaamodt T. M.","year":"2012","unstructured":"T. M. Aaamodt and W.W. Fung . GPGPU- Sim 3.x : A performance simulator for manycore accelerator research . International Conference on Parallel Architectures and Compilation Techniques (PACT), http:\/\/www.gpgpu-sim.org\/pact2012-tutorial\/, 2012 . {Online; accessed 30- Sep- 2012 }. T. M. Aaamodt andW.W. Fung. GPGPU-Sim 3.x: A performance simulator for manycore accelerator research. International Conference on Parallel Architectures and Compilation Techniques (PACT), http:\/\/www.gpgpu-sim.org\/pact2012-tutorial\/, 2012. {Online; accessed 30-Sep-2012}."},{"key":"e_1_2_1_3_1","volume-title":"http:\/\/www.artist-embedded.org\/","author":"European ArtistDesign","year":"2012","unstructured":"ArtistDesign NoE -- European Network of Excellence on High Performance and Embedded Architecture and Compilation. http:\/\/www.artist-embedded.org\/ , 2012 . {Online; accessed 21-Sep-2012}. ArtistDesign NoE -- European Network of Excellence on High Performance and Embedded Architecture and Compilation. http:\/\/www.artist-embedded.org\/, 2012. {Online; accessed 21-Sep-2012}."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPASS.2009.4919648"},{"key":"e_1_2_1_5_1","volume-title":"gem5: A multiple-ISA full system simulator with detailed memory modeling. Tutorial at the International Symposium on Computer Architecture (ISCA),http:\/\/www.m5sim.org\/. {Online","author":"Beckmann B.","year":"2012","unstructured":"B. Beckmann , N. Binkert , A. Saidi , J. Hestness , G. Black , K. Sewell , and D. Hower . gem5: A multiple-ISA full system simulator with detailed memory modeling. Tutorial at the International Symposium on Computer Architecture (ISCA),http:\/\/www.m5sim.org\/. {Online ; accessed 12- Jul- 2012 }. B. Beckmann, N. Binkert, A. Saidi, J. Hestness, G. Black, K. Sewell, and D. Hower. gem5: A multiple-ISA full system simulator with detailed memory modeling. Tutorial at the International Symposium on Computer Architecture (ISCA),http:\/\/www.m5sim.org\/. {Online; accessed 12-Jul-2012}."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/1247360.1247401"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2024716.2024718"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 2009 conference on Hot topics in cloud computing, HotCloud'09","author":"Campbell R.","year":"2009","unstructured":"R. Campbell , I. Gupta , M. Heath , S. Y. Ko , M. Kozuch , M. Kunze , T. Kwan , K. Lai , H. Y. Lee , M. Lyons , D. Milojicic , D. O'Hallaron , and Y. C. Soh . Open Cirrus(TM) cloud computing testbed: federated data centers for open source systems and services research . In Proceedings of the 2009 conference on Hot topics in cloud computing, HotCloud'09 , Berkeley, CA, USA , 2009 . USENIX Association. R. Campbell, I. Gupta, M. Heath, S. Y. Ko, M. Kozuch, M. Kunze, T. Kwan, K. Lai, H. Y. Lee, M. Lyons, D. Milojicic, D. O'Hallaron, and Y. C. Soh. Open Cirrus(TM) cloud computing testbed: federated data centers for open source systems and services research. In Proceedings of the 2009 conference on Hot topics in cloud computing, HotCloud'09, Berkeley, CA, USA, 2009. USENIX Association."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1508865.1508905"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063384.2063454"},{"volume-title":"http:\/\/www.chameleoncloud.org\/","year":"2014","key":"e_1_2_1_11_1","unstructured":"Chameleon. http:\/\/www.chameleoncloud.org\/ , 2014 . {Online; accessed 13-Nov-2014}. Chameleon. http:\/\/www.chameleoncloud.org\/, 2014. {Online; accessed 13-Nov-2014}."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/1331699.1331723"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/956993.956995"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1344671.1344684"},{"volume-title":"http:\/\/www.cloudlab.us\/","year":"2014","key":"e_1_2_1_15_1","unstructured":"CloudLab. http:\/\/www.cloudlab.us\/ , 2014 . {Online; accessed 13-Nov-2014}. CloudLab. http:\/\/www.cloudlab.us\/, 2014. {Online; accessed 13-Nov-2014}."},{"key":"e_1_2_1_16_1","volume-title":"http:\/\/www.cs.cmu.edu\/afs\/cs\/project\/ai-repository\/ai\/0.html","author":"Artificial Intelligence Repository CMU","year":"2012","unstructured":"CMU Artificial Intelligence Repository . http:\/\/www.cs.cmu.edu\/afs\/cs\/project\/ai-repository\/ai\/0.html , 2012 . {Online; accessed 02-Jan-2012}. CMU Artificial Intelligence Repository. http:\/\/www.cs.cmu.edu\/afs\/cs\/project\/ai-repository\/ai\/0.html, 2012. {Online; accessed 02-Jan-2012}."},{"key":"e_1_2_1_17_1","volume-title":"Sniper: Fast, accurate and scalable multicore simulation. Tutorial at the International Symposium on Computer Architecture (ISCA),\/\/http:\/\/snipersim.org\/w\/Tutorial:ISCA_2012. {Online","author":"Eeckhout L.","year":"2012","unstructured":"L. Eeckhout , W. Heirman , T. E. Carlson , and I. Hur . Sniper: Fast, accurate and scalable multicore simulation. Tutorial at the International Symposium on Computer Architecture (ISCA),\/\/http:\/\/snipersim.org\/w\/Tutorial:ISCA_2012. {Online ; accessed 30- Sep- 2012 }. L. Eeckhout, W. Heirman, T. E. Carlson, and I. Hur. Sniper: Fast, accurate and scalable multicore simulation. Tutorial at the International Symposium on Computer Architecture (ISCA),\/\/http:\/\/snipersim.org\/w\/Tutorial:ISCA_2012. {Online; accessed 30-Sep-2012}."},{"key":"e_1_2_1_18_1","volume-title":"http:\/\/www.emulab.net","author":"Emulab","year":"2012","unstructured":"Emulab --- total network testbed. http:\/\/www.emulab.net , 2012 . {Online; accessed 18-Sep-2012}. Emulab --- total network testbed. http:\/\/www.emulab.net, 2012. {Online; accessed 18-Sep-2012}."},{"key":"e_1_2_1_19_1","volume-title":"http:\/\/users.ece.utexas.edu\/~derek\/FABRIC.html. {Online","author":"Infrastructure Accelerator Research","year":"2012","unstructured":"FPGA- Accelerator Research Infrastructure Cloud (FAbRIC). http:\/\/users.ece.utexas.edu\/~derek\/FABRIC.html. {Online ; accessed 29- Sep- 2012 }. FPGA-Accelerator Research Infrastructure Cloud (FAbRIC). http:\/\/users.ece.utexas.edu\/~derek\/FABRIC.html. {Online; accessed 29-Sep-2012}."},{"key":"e_1_2_1_20_1","volume-title":"http:\/\/www.gem5.org\/. {Online","author":"The","year":"2012","unstructured":"The gem5 simulator system. http:\/\/www.gem5.org\/. {Online ; accessed 12- Jul- 2012 }. The gem5 simulator system. http:\/\/www.gem5.org\/. {Online; accessed 12-Jul-2012}."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2010.5416636"},{"key":"e_1_2_1_22_1","unstructured":"K. Ghose A. Patel F. Afram H. Zheng J. Tringali and J. Greensky. Marssx86 tutorial. Tutorial at Conference on Parallel Architecture and Compilation Techniques (PACT) http:\/\/marss86.org\/~marss86\/index.php\/PACT_Tutorial_2012. {Online; accessed24-Sep-2012}.  K. Ghose A. Patel F. Afram H. Zheng J. Tringali and J. Greensky. Marssx86 tutorial. Tutorial at Conference on Parallel Architecture and Compilation Techniques (PACT) http:\/\/marss86.org\/~marss86\/index.php\/PACT_Tutorial_2012. {Online; accessed24-Sep-2012}."},{"key":"e_1_2_1_23_1","first-page":"113","volume-title":"USENIX 2008 Annual Technical Conference on Annual Technical Conference, ATC'08","author":"Hibler M.","year":"2008","unstructured":"M. Hibler , R. Ricci , L. Stoller , J. Duerig , S. Guruprasad , T. Stack , K. Webb , and J. Lepreau . Large-scale virtualization in the emulab network testbed . In USENIX 2008 Annual Technical Conference on Annual Technical Conference, ATC'08 , pages 113 -- 128 , Berkeley, CA, USA , 2008 . USENIX Association. M. Hibler, R. Ricci, L. Stoller, J. Duerig, S. Guruprasad, T. Stack, K. Webb, and J. Lepreau. Large-scale virtualization in the emulab network testbed. In USENIX 2008 Annual Technical Conference on Annual Technical Conference, ATC'08, pages 113--128, Berkeley, CA, USA, 2008. USENIX Association."},{"key":"e_1_2_1_24_1","volume-title":"http:\/\/www.hipeac.net\/","author":"Compilation PEAC","year":"2012","unstructured":"Hi PEAC Compilation Architecture --- European Network of Excellence on High Performance and Embedded Architecture and Compilation. http:\/\/www.hipeac.net\/ , 2012 . {Online; accessed 27-Sep-2012}. HiPEAC Compilation Architecture --- European Network of Excellence on High Performance and Embedded Architecture and Compilation. http:\/\/www.hipeac.net\/, 2012. {Online; accessed 27-Sep-2012}."},{"key":"e_1_2_1_25_1","volume-title":"http:\/\/communities.intel.com\/community\/marc","author":"Applications Research Intel","year":"2012","unstructured":"Intel Many-core Applications Research Community (MARC). http:\/\/communities.intel.com\/community\/marc , 2012 . {Online; accessed 17-Mar-2012}. Intel Many-core Applications Research Community (MARC). http:\/\/communities.intel.com\/community\/marc, 2012. {Online; accessed 17-Mar-2012}."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1964218.1964220"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/IISWC.2009.5306801"},{"key":"e_1_2_1_28_1","volume-title":"GPU application development, debugging, and performance tuning with GPU Ocelot","author":"Kerr A.","year":"2011","unstructured":"A. Kerr , G. Diamos , and S. Yalamanchili . GPU application development, debugging, and performance tuning with GPU Ocelot . In Wen-mei Hwu, editor, GPU Computing GEMS, vol. 2 . Morgan Kaufmann , 2011 . A. Kerr, G. Diamos, and S. Yalamanchili. GPU application development, debugging, and performance tuning with GPU Ocelot. In Wen-mei Hwu, editor, GPU Computing GEMS, vol. 2. Morgan Kaufmann, 2011."},{"key":"e_1_2_1_29_1","volume-title":"Tutorial on Ocelot and SST-MacSim Simulator. International Symposium on Computer Architecture, http:\/\/comparch.gatech.edu\/hparch\/isca12_gt.html, 2012","author":"Kim H.","year":"2012","unstructured":"H. Kim , S. Yalamanchili , J. Lee , N. Lakshminarayana , A. Kerr , A. Rodrigues , and G. Hsieh . Tutorial on Ocelot and SST-MacSim Simulator. International Symposium on Computer Architecture, http:\/\/comparch.gatech.edu\/hparch\/isca12_gt.html, 2012 . {Online; accessed 15- Sep- 2012 }. H. Kim, S. Yalamanchili, J. Lee, N. Lakshminarayana, A. Kerr, A. Rodrigues, and G. Hsieh. Tutorial on Ocelot and SST-MacSim Simulator. International Symposium on Computer Architecture, http:\/\/comparch.gatech.edu\/hparch\/isca12_gt.html, 2012. {Online; accessed 15-Sep-2012}."},{"key":"e_1_2_1_30_1","volume-title":"Tutorial on Ocelot and SST-MacSim Simulator. Conference on High-Performance Computer Architecture, http:\/\/comparch.gatech.edu\/hparch\/OcelotMacSim_tutorial.html, 2012","author":"Kim H.","year":"2012","unstructured":"H. Kim , S. Yalamanchili , J. Lee , N. Lakshminarayana , A. Kerr , A. Rodrigues , and G. Hsieh . Tutorial on Ocelot and SST-MacSim Simulator. Conference on High-Performance Computer Architecture, http:\/\/comparch.gatech.edu\/hparch\/OcelotMacSim_tutorial.html, 2012 . {Online; accessed 15- Sep- 2012 }. H. Kim, S. Yalamanchili, J. Lee, N. Lakshminarayana, A. Kerr, A. Rodrigues, and G. Hsieh. Tutorial on Ocelot and SST-MacSim Simulator. Conference on High-Performance Computer Architecture, http:\/\/comparch.gatech.edu\/hparch\/OcelotMacSim_tutorial.html, 2012. {Online; accessed 15-Sep-2012}."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.982916"},{"key":"e_1_2_1_32_1","volume-title":"http:\/\/manifold.gatech.edu. {Online","author":"Manifold","year":"2012","unstructured":"Manifold parallel discrete event simulation. http:\/\/manifold.gatech.edu. {Online ; accessed 29- Sep- 2012 }. Manifold parallel discrete event simulation. http:\/\/manifold.gatech.edu. {Online; accessed 29-Sep-2012}."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/SBAC-PAD.2010.37"},{"key":"e_1_2_1_34_1","volume-title":"nanohub.org. {Online","author":"Simulation Online","year":"2012","unstructured":"nanoHUB.org -- Online Simulation and More for Nanotechnology . nanohub.org. {Online ; accessed 20- Sep- 2012 }. nanoHUB.org -- Online Simulation and More for Nanotechnology. nanohub.org. {Online; accessed 20-Sep-2012}."},{"key":"e_1_2_1_35_1","volume-title":"http:\/\/opencirrus.org","author":"Computing Research Testbed Open Cloud","year":"2012","unstructured":"OpenCirrus(TM) the HP\/Intel\/Yahoo! Open Cloud Computing Research Testbed . http:\/\/opencirrus.org , 2012 . {Online; accessed 15-Sep-2012}. OpenCirrus(TM) the HP\/Intel\/Yahoo! Open Cloud Computing Research Testbed. http:\/\/opencirrus.org, 2012. {Online; accessed 15-Sep-2012}."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1344671.1344685"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2006.1598110"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/1298455.1298489"},{"key":"e_1_2_1_39_1","unstructured":"PlanetLab -- An Open Platform for Developing Deploying and Accessing Planetary-Scale Services. http:\/\/www.planet-lab.org\/ 2012. {Online; accessed 27-Sep-2012}.  PlanetLab -- An Open Platform for Developing Deploying and Accessing Planetary-Scale Services. http:\/\/www.planet-lab.org\/ 2012. {Online; accessed 27-Sep-2012}."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2012.2184760"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1964218.1964225"},{"key":"e_1_2_1_42_1","volume-title":"http:\/\/seattle.cs.washington.edu","author":"Open Seattle","year":"2012","unstructured":"Seattle -- Open Peer-to- Peer Computing . http:\/\/seattle.cs.washington.edu , 2012 . {Online; accessed 18-Sep-2012}. Seattle -- Open Peer-to-Peer Computing. http:\/\/seattle.cs.washington.edu, 2012. {Online; accessed 18-Sep-2012}."},{"key":"e_1_2_1_43_1","unstructured":"SST\n  : The Structural Simulation Toolkit. http:\/\/sst.sandia.gov 2012. {Online; accessed 12-Aug-2012}.  SST: The Structural Simulation Toolkit. http:\/\/sst.sandia.gov 2012. {Online; accessed 12-Aug-2012}."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2370816.2370865"},{"key":"e_1_2_1_45_1","unstructured":"XSEDE\n  : Extreme Science and Engineering Discovery Environment. http:\/\/www.xsede.org 2012. {Online; accessed 15-Sep-2012}.  XSEDE: Extreme Science and Engineering Discovery Environment. http:\/\/www.xsede.org 2012. {Online; accessed 15-Sep-2012}."}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723872.2723886","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2723872.2723886","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:16:43Z","timestamp":1750227403000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723872.2723886"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,1,20]]},"references-count":45,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,1,20]]}},"alternative-id":["10.1145\/2723872.2723886"],"URL":"https:\/\/doi.org\/10.1145\/2723872.2723886","relation":{},"ISSN":["0163-5980"],"issn-type":[{"type":"print","value":"0163-5980"}],"subject":[],"published":{"date-parts":[[2015,1,20]]},"assertion":[{"value":"2015-01-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}