{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:39:10Z","timestamp":1750307950162,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":24,"publisher":"ACM","license":[{"start":{"date-parts":[[2007,6,13]],"date-time":"2007-06-13T00:00:00Z","timestamp":1181692800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2007,6,13]]},"DOI":"10.1145\/1281700.1281713","type":"proceedings-article","created":{"date-parts":[[2012,10,11]],"date-time":"2012-10-11T15:35:23Z","timestamp":1349969723000},"page":"13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Performance testing of combinatorial solvers with isomorph class instances"],"prefix":"10.1145","author":[{"given":"Franc","family":"Brglez","sequence":"first","affiliation":[{"name":"NC State University, Raleigh, NC"}]},{"given":"Jason A.","family":"Osborne","sequence":"additional","affiliation":[{"name":"NC State University, Raleigh, NC"}]}],"member":"320","published-online":{"date-parts":[[2007,6,13]]},"reference":[{"key":"e_1_3_2_1_1_1","first-page":"201","volume-title":"Operations Research","author":"Hooker J. N.","year":"1994","unstructured":"J. N. Hooker . Needed : An empirical science of algorithms . Operations Research , pages 42(2): 201 -- 212 , 1994 . J. N. Hooker. Needed: An empirical science of algorithms. Operations Research, pages 42(2):201--212, 1994."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02430363"},{"key":"e_1_3_2_1_3_1","first-page":"33","volume-title":"Testing heuristics: We have it all wrong. J. of Heuristics","author":"Hooker J.","year":"1996","unstructured":"J. Hooker . Testing heuristics: We have it all wrong. J. of Heuristics , pages 1: 33 -- 42 , 1996 . J. Hooker. Testing heuristics: We have it all wrong. J. of Heuristics, pages 1:33--42, 1996."},{"key":"e_1_3_2_1_4_1","volume-title":"Design of Experiments to Evaluate CAD Algorithms: Which Improvements Are Due to Improved Heuristic and Which Are Merely Due to Chance?","author":"Brglez F.","year":"1998","unstructured":"F. Brglez . Design of Experiments to Evaluate CAD Algorithms: Which Improvements Are Due to Improved Heuristic and Which Are Merely Due to Chance? Technical Report 1998-TR@CBL-04-Brglez, Computer Science , NCSU, Raleigh, NC 27695, April 1998 . F. Brglez. Design of Experiments to Evaluate CAD Algorithms: Which Improvements Are Due to Improved Heuristic and Which Are Merely Due to Chance? Technical Report 1998-TR@CBL-04-Brglez, Computer Science, NCSU, Raleigh, NC 27695, April 1998."},{"key":"e_1_3_2_1_5_1","first-page":"238","volume-title":"UAI-98","author":"Hoos H. H.","year":"1998","unstructured":"H. H. Hoos and T. Stuetzle . Evaluating Las Vegas Algorithms -- Pitfalls and Remedies . In UAI-98 , pages 238 -- 245 . Morgan Kaufmann Publishers , 1998 . H. H. Hoos and T. Stuetzle. Evaluating Las Vegas Algorithms -- Pitfalls and Remedies. In UAI-98, pages 238--245. Morgan Kaufmann Publishers, 1998."},{"key":"e_1_3_2_1_6_1","volume-title":"Handbook of Global Optimization, Volume 2: Heuristic Approaches","author":"McGeoch C. C.","year":"2001","unstructured":"C. C. McGeoch . Experimental Analysis of Algorithms . In P. Pardalos and E. Romeijn, editor, Handbook of Global Optimization, Volume 2: Heuristic Approaches . Kluwer Academic Publishers , 2001 . C. C. McGeoch. Experimental Analysis of Algorithms. In P. Pardalos and E. Romeijn, editor, Handbook of Global Optimization, Volume 2: Heuristic Approaches. Kluwer Academic Publishers, 2001."},{"key":"e_1_3_2_1_7_1","first-page":"215","volume-title":"Fifth and Sixth DIMACS Implementation Challenges","author":"Johnson D. S.","year":"2002","unstructured":"D. S. Johnson . A Theoretician's Guide to the Experimental Analysis of Algorithms. In M. H. Goldwasser and D. S. Johnson and C. C. McGeoch, editor , Fifth and Sixth DIMACS Implementation Challenges , pages 215 -- 250 . Am. Math. Society , 2002 . D. S. Johnson. A Theoretician's Guide to the Experimental Analysis of Algorithms. In M. H. Goldwasser and D. S. Johnson and C. C. McGeoch, editor, Fifth and Sixth DIMACS Implementation Challenges, pages 215--250. Am. Math. Society, 2002."},{"key":"e_1_3_2_1_8_1","volume-title":"Experimental Computer Science: The Need for a Cultural Change","author":"Feitelson D. G.","year":"2005","unstructured":"D. G. Feitelson . Experimental Computer Science: The Need for a Cultural Change , 2005 . Manuscript , from http:\/\/www.cs.huji.ac.il\/feit\/pub.html. D. G. Feitelson. Experimental Computer Science: The Need for a Cultural Change, 2005. Manuscript, from http:\/\/www.cs.huji.ac.il\/feit\/pub.html."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1137\/S0097539799349948"},{"key":"e_1_3_2_1_10_1","unstructured":"Home page for cplex 2007. http:\/\/www.ilog.com\/products\/cplex\/.  Home page for cplex 2007. http:\/\/www.ilog.com\/products\/cplex\/."},{"key":"e_1_3_2_1_11_1","unstructured":"Home page for lp_solve 2007. http:\/\/lpsolve.sourceforge.net\/5.5\/.  Home page for lp_solve 2007. http:\/\/lpsolve.sourceforge.net\/5.5\/."},{"key":"e_1_3_2_1_12_1","unstructured":"About cplex and lp_solve file formats 2007. http:\/\/lpsolve.sourceforge.net\/5.5\/CPLEX-format.htm.  About cplex and lp_solve file formats 2007. http:\/\/lpsolve.sourceforge.net\/5.5\/CPLEX-format.htm."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1065579.1065682"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10472-004-9417-0"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/42805"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/288548.289103"},{"key":"e_1_3_2_1_17_1","volume-title":"International Journal on Software Tools for Technology Transfer (STTT), 3(2):193--206","author":"Brglez J. E.","year":"2001","unstructured":"J. E. Harlow III and F. Brglez . Design of experiments and evaluation of BDD ordering heuristics . International Journal on Software Tools for Technology Transfer (STTT), 3(2):193--206 , May 2001 . Springer-Verlag Heidelberg . http:\/\/springerlink.metapress.com\/, ISSN: 1433--2779 (Paper) 1433--2787 (Online). J. E. Harlow III and F. Brglez. Design of experiments and evaluation of BDD ordering heuristics. International Journal on Software Tools for Technology Transfer (STTT), 3(2):193--206, May 2001. Springer-Verlag Heidelberg. http:\/\/springerlink.metapress.com\/, ISSN: 1433--2779 (Paper) 1433--2787 (Online)."},{"key":"e_1_3_2_1_19_1","volume-title":"Eighth International Symposium on Artificial Intelligence and Mathematics, January 4--6, 2004","author":"Guo Y.","year":"2004","unstructured":"Y. Guo , A. Lim , B. Rodrigues , and Y. Zhu . Heuristics for a brokering set packing problem . In Eighth International Symposium on Artificial Intelligence and Mathematics, January 4--6, 2004 , Fort Lauderdale, Florida, USA. ACM , January 2004 . Y. Guo, A. Lim, B. Rodrigues, and Y. Zhu. Heuristics for a brokering set packing problem. In Eighth International Symposium on Artificial Intelligence and Mathematics, January 4--6, 2004, Fort Lauderdale, Florida, USA. ACM, January 2004."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2005.01.007"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.39.3.378"},{"key":"e_1_3_2_1_22_1","volume-title":"Logic Synthesis and Verification Algorithms","author":"Hachtel G. D.","year":"1996","unstructured":"G. D. Hachtel and F. Somenzi . Logic Synthesis and Verification Algorithms . Kluwer Academic Publishers , 1996 . G. D. Hachtel and F. Somenzi. Logic Synthesis and Verification Algorithms. Kluwer Academic Publishers, 1996."},{"key":"e_1_3_2_1_23_1","volume-title":"ACM-FCRC, 2007. Proceedings of Workshop on Experimental Computer Science, Part of ACM FCRC","author":"Stallmann M. F. M.","year":"2007","unstructured":"M. F. M. Stallmann and F. Brglez . High-contrast algorithm behavior: Observation, conjecture, and experimental design . In ACM-FCRC, 2007. Proceedings of Workshop on Experimental Computer Science, Part of ACM FCRC , San Diego, 13- -14 June 2007 . M. F. M. Stallmann and F. Brglez. High-contrast algorithm behavior: Observation, conjecture, and experimental design. In ACM-FCRC, 2007. Proceedings of Workshop on Experimental Computer Science, Part of ACM FCRC, San Diego, 13--14 June 2007."},{"key":"e_1_3_2_1_24_1","volume-title":"Statistical Theory and Methodology In Science and Engineering","author":"Brownlee K. A.","year":"1984","unstructured":"K. A. Brownlee . Statistical Theory and Methodology In Science and Engineering . Krieger Publishing , 1984 . Reprinted, with revisons, from second edition, 1965. K. A. Brownlee. Statistical Theory and Methodology In Science and Engineering. Krieger Publishing, 1984. Reprinted, with revisons, from second edition, 1965."},{"key":"e_1_3_2_1_25_1","volume-title":"Duxbury","author":"Bain L. J.","year":"1987","unstructured":"L. J. Bain and M. EngelHardt . Introduction to Probability and Mathematical Statistics . Duxbury , 1987 . L. J. Bain and M. EngelHardt. Introduction to Probability and Mathematical Statistics. Duxbury, 1987."}],"event":{"name":"ExpCS07: Workshop on Experimental Computer Science","sponsor":["SIGOPS ACM Special Interest Group on Operating Systems","SIGARCH ACM Special Interest Group on Computer Architecture","SIGMETRICS ACM Special Interest Group on Measurement and Evaluation","SIGCOMM ACM Special Interest Group on Data Communication"],"location":"San Diego California","acronym":"ExpCS07"},"container-title":["Proceedings of the 2007 workshop on Experimental computer science"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1281700.1281713","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1281700.1281713","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T14:57:47Z","timestamp":1750258667000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1281700.1281713"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2007,6,13]]},"references-count":24,"alternative-id":["10.1145\/1281700.1281713","10.1145\/1281700"],"URL":"https:\/\/doi.org\/10.1145\/1281700.1281713","relation":{},"subject":[],"published":{"date-parts":[[2007,6,13]]},"assertion":[{"value":"2007-06-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}