{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:24:51Z","timestamp":1750307091386,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":20,"publisher":"ACM","license":[{"start":{"date-parts":[[2010,9,28]],"date-time":"2010-09-28T00:00:00Z","timestamp":1285632000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2010,9,28]]},"DOI":"10.1145\/2377576.2377603","type":"proceedings-article","created":{"date-parts":[[2012,9,25]],"date-time":"2012-09-25T23:48:43Z","timestamp":1348616923000},"page":"136-145","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["The multi-relationship evaluation design framework"],"prefix":"10.1145","author":[{"given":"Brian A.","family":"Weiss","sequence":"first","affiliation":[{"name":"National Institute of Standards and Technology, Gaithersburg, Maryland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Linda C.","family":"Schmidt","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, Maryland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2010,9,28]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.604198"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1865909.1865944"},{"key":"e_1_3_2_1_3_1","unstructured":"Calisi D. Iocchi L. and Nardi D. 2008 \"A Unified Benchmark Framework for Autonomous Mobile Robots and Vehicles Motion Algorithms (MoVeMA benchmarks) \" In Workshop on Experimental Methodology and Benchmarking in Robotics Research (RSS 2008).  Calisi D. Iocchi L. and Nardi D. 2008 \"A Unified Benchmark Framework for Autonomous Mobile Robots and Vehicles Motion Algorithms (MoVeMA benchmarks) \" In Workshop on Experimental Methodology and Benchmarking in Robotics Research (RSS 2008)."},{"key":"e_1_3_2_1_4_1","unstructured":"Fontana R. Richley E. and Barney J. 2003 \"Commercialization of an Ultra Wideband Precision Asset Location System \" In 2003 IEEE Conference on Ultra Wideband Systems and Technologies.  Fontana R. Richley E. and Barney J. 2003 \"Commercialization of an Ultra Wideband Precision Asset Location System \" In 2003 IEEE Conference on Ultra Wideband Systems and Technologies ."},{"key":"e_1_3_2_1_5_1","unstructured":"Galvin C. 2010 \"Gulf Oil Spill: Responding with Robots \" Robotics Trends. DOI = http:\/\/www.roboticstrends.com\/service_robotics\/article\/gulf_oil_spill_responding_with_robots  Galvin C. 2010 \"Gulf Oil Spill: Responding with Robots \" Robotics Trends . DOI = http:\/\/www.roboticstrends.com\/service_robotics\/article\/gulf_oil_spill_responding_with_robots"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.719692"},{"issue":"3","key":"e_1_3_2_1_7_1","first-page":"42","article-title":"Robots to the Rescue","volume":"5","author":"Messina E.","year":"2009","unstructured":"Messina , E. , 2009 , \" Robots to the Rescue ,\" Crisis Response Journal , 5 ( 3 ), pp. 42 -- 43 . Messina, E., 2009, \"Robots to the Rescue,\" Crisis Response Journal, 5(3), pp. 42--43.","journal-title":"Crisis Response Journal"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/rob.v24:8\/9"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1865909.1865955"},{"key":"e_1_3_2_1_10_1","unstructured":"Schlenoff C. I. Weiss B. A. Steves M. P. Virts A. M. and Shneier M. O. 2006 \"Overview of the First Advanced Technology Evaluations for ASSIST \" Proc. of the Performance Metrics for Intelligent Systems (PerMIS) Workshop.  Schlenoff C. I. Weiss B. A. Steves M. P. Virts A. M. and Shneier M. O. 2006 \"Overview of the First Advanced Technology Evaluations for ASSIST \" Proc. of the Performance Metrics for Intelligent Systems (PerMIS) Workshop ."},{"key":"e_1_3_2_1_11_1","unstructured":"Scholtz J. C. Antonishek B. and Young J. D. 2004 \"Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas \" Proc. of the 2004 Performance Metrics for Intelligent Systems (PerMIS) Workshop.   Scholtz J. C. Antonishek B. and Young J. D. 2004 \"Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas \" Proc. of the 2004 Performance Metrics for Intelligent Systems (PerMIS) Workshop ."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1121241.1121246"},{"issue":"6","key":"e_1_3_2_1_13_1","first-page":"1","article-title":"Performance Analysis for Stable Mobile Robot Navigation Solutions","volume":"6962","author":"Scrapper C. J.","year":"2008","unstructured":"Scrapper , C. J. , Madhavan , R. , Balakirsky , S. B. , 2008 , \" Performance Analysis for Stable Mobile Robot Navigation Solutions ,\" Proc. of the Unmanned Systems Technology X -- SPIE Conference , 6962 ( 6 ), pp. 1 -- 12 . Scrapper, C. J., Madhavan, R., Balakirsky, S. B., 2008, \"Performance Analysis for Stable Mobile Robot Navigation Solutions,\" Proc. of the Unmanned Systems Technology X -- SPIE Conference, 6962(6), pp. 1--12.","journal-title":"Proc. of the Unmanned Systems Technology X -- SPIE Conference"},{"key":"e_1_3_2_1_14_1","unstructured":"Steves M. P. 2007 \"Utility Assessments of Soldier-Worn Sensor Systems for ASSIST \" Proc. of the 2006 Performance Metrics for Intelligent Systems (PerMIS) Workshop.  Steves M. P. 2007 \"Utility Assessments of Soldier-Worn Sensor Systems for ASSIST \" Proc. of the 2006 Performance Metrics for Intelligent Systems (PerMIS) Workshop ."},{"key":"e_1_3_2_1_15_1","first-page":"558","article-title":"An Evaluation Methodology for Autonomous Mobile Robots for Planetary Exploration","author":"Sukhatme G. S.","year":"1995","unstructured":"Sukhatme , G. S. and Bekey , G. A. , 1995 , \" An Evaluation Methodology for Autonomous Mobile Robots for Planetary Exploration ,\" Proc. of the First ECPD International Conference on Advanced Robotics and Intelligent Automation , pp. 558 -- 563 . Sukhatme, G. S. and Bekey, G. A., 1995, \"An Evaluation Methodology for Autonomous Mobile Robots for Planetary Exploration,\" Proc. of the First ECPD International Conference on Advanced Robotics and Intelligent Automation, pp. 558--563.","journal-title":"Proc. of the First ECPD International Conference on Advanced Robotics and Intelligent Automation"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1774674.1774676"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1865909.1865957"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Weiss B. A. Schmidt L. C. Scott H. A. and Schlenoff C. I. 2010 \"The Multi-Relationship Evaluation Design Framework: Designing Testing Plans to Comprehensively Assess Advanced and Intelligent Technologies \" Proc. of the ASME 2010 International Design Engineering Technical Conferences (IDETC) -- 22nd International Conference on Design Theory and Methodology (DTM).  Weiss B. A. Schmidt L. C. Scott H. A. and Schlenoff C. I. 2010 \"The Multi-Relationship Evaluation Design Framework: Designing Testing Plans to Comprehensively Assess Advanced and Intelligent Technologies \" Proc. of the ASME 2010 International Design Engineering Technical Conferences (IDETC) -- 22nd International Conference on Design Theory and Methodology (DTM) .","DOI":"10.1115\/DETC2010-28928"},{"key":"e_1_3_2_1_19_1","unstructured":"Weiss B. A. Schlenoff C. I. Sanders G. A. Steves M. P. Condon S. Phillips J. and Parvaz D. 2008 \"Performance Evaluation of Speech Translation Systems \" Proc. of the 6th edition of the Language Resources and Evaluation Conference.  Weiss B. A. Schlenoff C. I. Sanders G. A. Steves M. P. Condon S. Phillips J. and Parvaz D. 2008 \"Performance Evaluation of Speech Translation Systems \" Proc. of the 6th edition of the Language Resources and Evaluation Conference ."},{"key":"e_1_3_2_1_20_1","unstructured":"Weiss B. A. Schlenoff C. I. Shneier M. O. and Virts A. M. 2006 \"Technology Evaluations and Performance Metrics for Soldier-Worn Sensors for ASSIST \" Proc. of the 2006 Performance Metrics for Intelligent Systems (PerMIS) Workshop.  Weiss B. A. Schlenoff C. I. Shneier M. O. and Virts A. M. 2006 \"Technology Evaluations and Performance Metrics for Soldier-Worn Sensors for ASSIST \" Proc. of the 2006 Performance Metrics for Intelligent Systems (PerMIS) Workshop ."}],"event":{"name":"PerMIS '10: Performance Metrics for Intelligent Systems","sponsor":["NIST National Institute of Standards & Technology","SIGAI ACM Special Interest Group on Artificial Intelligence"],"location":"Baltimore Maryland","acronym":"PerMIS '10"},"container-title":["Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2377576.2377603","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2377576.2377603","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T09:34:33Z","timestamp":1750239273000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2377576.2377603"}},"subtitle":["creating evaluation blueprints to assess advanced and intelligent technologies"],"short-title":[],"issued":{"date-parts":[[2010,9,28]]},"references-count":20,"alternative-id":["10.1145\/2377576.2377603","10.1145\/2377576"],"URL":"https:\/\/doi.org\/10.1145\/2377576.2377603","relation":{},"subject":[],"published":{"date-parts":[[2010,9,28]]},"assertion":[{"value":"2010-09-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}