{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:11:27Z","timestamp":1750306287809,"version":"3.41.0"},"reference-count":23,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,2,25]],"date-time":"2016-02-25T00:00:00Z","timestamp":1456358400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGMETRICS Perform. Eval. Rev."],"published-print":{"date-parts":[[2016,2,25]]},"abstract":"<jats:p>Software engineering considers performance evaluation to be one of the key portions of software quality assurance. Unfortunately, there seems to be a lack of standard methodologies for performance evaluation even in the scope of experimental computer science. Inspired by the concept of \"instantiation\" in object-oriented programming, we distinguish the generic performance evaluation logic from the distributed and ad-hoc relevant studies, and develop an abstract evaluation methodology (by analogy of \"class\") we name Domain Knowledge-driven Methodology (DoKnowMe). By replacing five predefined domain-specific knowledge artefacts, DoKnowMe could be instantiated into specific methodologies (by analogy of \"object\") to guide evaluators in performance evaluation of different software and even computing systems. We also propose a generic validation framework with four indicators (i.e. usefulness, feasibility, effectiveness and repeatability), and use it to validate DoKnowMe in the Cloud services evaluation domain. Given the positive and promising validation result, we plan to integrate more common evaluation strategies to improve DoKnowMe and further focus on the performance evaluation of Cloud autoscaler systems.<\/jats:p>","DOI":"10.1145\/2897356.2897360","type":"journal-article","created":{"date-parts":[[2016,2,26]],"date-time":"2016-02-26T14:29:03Z","timestamp":1456496943000},"page":"23-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["DoKnowMe"],"prefix":"10.1145","volume":"43","author":[{"given":"Zheng","family":"Li","sequence":"first","affiliation":[{"name":"Lund University, Lund, Sweden"}]},{"given":"Liam","family":"O'Brien","sequence":"additional","affiliation":[{"name":"ICT Innovation and Services Geoscience Australia, Canberra, Australia"}]},{"given":"Maria","family":"Kihl","sequence":"additional","affiliation":[{"name":"Lund University, Lund, Sweden"}]}],"member":"320","published-online":{"date-parts":[[2016,2,25]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.amepre.2009.02.002"},{"key":"e_1_2_1_2_1","volume-title":"Soft Systems Methodology in Action","author":"Checkland P.","year":"1999","unstructured":"P. Checkland and J. Scholes . Soft Systems Methodology in Action . John Wiley & Sons Ltd ., New York, NY, September 1999 . P. Checkland and J. Scholes. Soft Systems Methodology in Action. John Wiley & Sons Ltd., New York, NY, September 1999."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1010629.805480"},{"key":"e_1_2_1_4_1","unstructured":"D. G. Feitelson. Experimental computer science: The need for a cultural change. http:\/\/www.cs.huji.ac.il\/~feit\/papers\/exp05.pdf 3 December 2006.  D. G. Feitelson. Experimental computer science: The need for a cultural change. http:\/\/www.cs.huji.ac.il\/~feit\/papers\/exp05.pdf 3 December 2006."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723872.2723875"},{"key":"e_1_2_1_6_1","volume-title":"Computer Systems Performance Evaluation and Prediction","author":"Fortier P. J.","year":"2003","unstructured":"P. J. Fortier and H. E. Michel . Computer Systems Performance Evaluation and Prediction . Digital Press , Burlington, MA , July 2003 . P. J. Fortier and H. E. Michel. Computer Systems Performance Evaluation and Prediction. Digital Press, Burlington, MA, July 2003."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1281700.1281711"},{"key":"e_1_2_1_8_1","unstructured":"G. Heiser. Systems benchmarking crimes. http:\/\/www.cse.unsw.edu.au\/~gernot\/benchmarking-crimes.html 13 March 2015.  G. Heiser. Systems benchmarking crimes. http:\/\/www.cse.unsw.edu.au\/~gernot\/benchmarking-crimes.html 13 March 2015."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2188286.2188301"},{"key":"e_1_2_1_10_1","volume-title":"Measurement, Simulation, and Modeling","author":"Jain R. K.","year":"1991","unstructured":"R. K. Jain . The Art of Computer Systems Performance Analysis: Techniques for Experimental Design , Measurement, Simulation, and Modeling . Wiley Computer Publishing , John Wiley & Sons, Inc., New York, NY, April 1991 . R. K. Jain. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley Computer Publishing, John Wiley & Sons, Inc., New York, NY, April 1991."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1201\/b16328"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.peva.2009.07.007"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/AINA.2013.99"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2012.6427525"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/Grid.2012.15"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2013.39"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2013.04.021"},{"key":"e_1_2_1_18_1","volume-title":"Design and Analysis of Experiments","author":"Montgomery D. C.","year":"2009","unstructured":"D. C. Montgomery . Design and Analysis of Experiments . John Wiley & Sons, Inc. , Hoboken, NJ , 7 th edition edition, January 2009 . D. C. Montgomery. Design and Analysis of Experiments. John Wiley & Sons, Inc., Hoboken, NJ, 7th edition edition, January 2009.","edition":"7"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1824372"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.peva.2012.05.006"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1006\/jvlc.1993.1015"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9102-8"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2577036.2577037"}],"container-title":["ACM SIGMETRICS Performance Evaluation Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897356.2897360","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2897356.2897360","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:39:02Z","timestamp":1750221542000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897356.2897360"}},"subtitle":["Towards a Domain Knowledge-driven Methodology for Performance Evaluation"],"short-title":[],"issued":{"date-parts":[[2016,2,25]]},"references-count":23,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,2,25]]}},"alternative-id":["10.1145\/2897356.2897360"],"URL":"https:\/\/doi.org\/10.1145\/2897356.2897360","relation":{},"ISSN":["0163-5999"],"issn-type":[{"type":"print","value":"0163-5999"}],"subject":[],"published":{"date-parts":[[2016,2,25]]},"assertion":[{"value":"2016-02-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}