{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T21:00:13Z","timestamp":1767992413301,"version":"3.49.0"},"reference-count":67,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2017,10,1]],"date-time":"2017-10-01T00:00:00Z","timestamp":1506816000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Evaluation"],"published-print":{"date-parts":[[2017,10]]},"abstract":"<jats:p> This work proposes a method for assessment of evaluation processes, applicable to systems in which both measurable and policy-based performance criteria coexist. It intends to provide a mapping procedure, based solely on measurable criteria, aiming at visual inspection of the outcome, which allows the introduction of two types of indicators at different phases of the process. The definition of performance-based zones is suggested: two of them clearly correspond to high\/low performance, in which the overall evaluation result is essentially established on the basis of measurable indicators, i.e. on the basis of objective criteria, and an in-between area, more prone to a less deterministic outcome. The method of evaluation is applied in three successive steps: (i) build the representation and identify the performance zones, (ii) introduce additional policy criteria to complete a first draft of the outcome, and, finally (iii) assess relative positioning, correct if necessary, and finalize. Performance maps are presented and inspected for a set of Research and Development institutions, to illustrate examples of application of the methodology proposed. <\/jats:p>","DOI":"10.1177\/1356389017733336","type":"journal-article","created":{"date-parts":[[2017,10,16]],"date-time":"2017-10-16T12:26:47Z","timestamp":1508156807000},"page":"479-494","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":4,"title":["Seeing is believing: A graphical reference framework for multi-criteria evaluation"],"prefix":"10.1177","volume":"23","author":[{"given":"T\u00e2nia F.G.G.","family":"Cova","sequence":"first","affiliation":[{"name":"University of Coimbra, Portugal"}]},{"given":"Susana","family":"Jarmelo","sequence":"additional","affiliation":[{"name":"University of Coimbra, Portugal"}]},{"given":"Sandra C.C.","family":"Nunes","sequence":"additional","affiliation":[{"name":"University of Coimbra, Portugal"}]},{"given":"Sebasti\u00e3o J.","family":"Formosinho","sequence":"additional","affiliation":[{"name":"University of Coimbra, Portugal"}]},{"given":"J. S\u00e9rgio Seixas","family":"de Melo","sequence":"additional","affiliation":[{"name":"University of Coimbra, Portugal"}]},{"given":"Alberto","family":"Pais","sequence":"additional","affiliation":[{"name":"University of Coimbra, Portugal"}]}],"member":"179","published-online":{"date-parts":[[2017,10,16]]},"reference":[{"key":"bibr1-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1038\/465860a"},{"key":"bibr2-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2009.01.001"},{"key":"bibr3-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2009.04.001"},{"key":"bibr4-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1177\/01655515030294003"},{"key":"bibr5-1356389017733336","volume-title":"Bibliometrics: Use and Abuse in the Review of Research Performance","author":"Blockman W","year":"2014"},{"key":"bibr6-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1093\/reseval\/rvw004"},{"key":"bibr7-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1177\/014920639201800206"},{"key":"bibr8-1356389017733336","doi-asserted-by":"publisher","DOI":"10.2307\/248755"},{"key":"bibr9-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.aos.2013.06.002"},{"key":"bibr10-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2014.11.005"},{"key":"bibr11-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-012-0792-8"},{"key":"bibr12-1356389017733336","volume-title":"CHER annual conference, ISEG","author":"Deem R","year":"2015"},{"key":"bibr13-1356389017733336","doi-asserted-by":"publisher","DOI":"10.3152\/095820211X12941371876346"},{"key":"bibr14-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1093\/scipol\/sct007"},{"key":"bibr15-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-006-0144-7"},{"key":"bibr16-1356389017733336","unstructured":"European Physics Society (2012) On the use of bibliometric indices during assessment. Available at: http:\/\/c.ymcdn.com\/sites\/www.eps.org\/resource\/collection\/B77D91E8-2370-43C3-9814-250C65E13549\/EPS_statement_June2012.pdf (accessed May 2014)."},{"key":"bibr17-1356389017733336","unstructured":"European Science Foundation (2011) European Peer Review Guide, Integrating Policies and Practices for Coherent Procedures. Available at: http:\/\/www.esf.org\/activities\/mo-fora\/peer-review.html; http:\/\/www.esf.org\/fileadmin\/Public_documents\/Publications\/European_Peer_Review_Guide_01.pdf (accessed May 2014)."},{"key":"bibr18-1356389017733336","unstructured":"European Science Foundation (2015) ESF report on the Foundation for Science and Technology (FCT) evaluation of R&D Units 2013\u20132014. Strasbourg."},{"key":"bibr19-1356389017733336","unstructured":"Ferreira N, Firmino T (2015) FCT foi avaliada e Nuno Crato diz que resultados apoiam as suas politicas [FCT was evaluated and Nuno Crato says that results support his policy]. Available at: http:\/\/www.publico.pt\/ciencia\/noticia\/a-fct-tambem-foi-avaliada-e-o-resultado-apoia-as-suas-politicas-1703572.29 (accessed August 2015)."},{"key":"bibr20-1356389017733336","unstructured":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia (FCT) (2014a) Available at: http:\/\/www.fct.pt\/apoios\/unidades\/avaliacoes"},{"key":"bibr21-1356389017733336","unstructured":"Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia (FCT) (2014b) Available at: http:\/\/www.fct.pt\/apoios\/unidades\/avaliacoes\/2013\/analise_bibliometrica.phtml.en"},{"key":"bibr22-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/58.3.453"},{"issue":"1","key":"bibr23-1356389017733336","first-page":"12","volume":"10","author":"Galindo MP","year":"1986","journal-title":"Q\u00fcesti\u00f3n"},{"key":"bibr24-1356389017733336","doi-asserted-by":"publisher","DOI":"10.2308\/accr.2004.79.2.409"},{"key":"bibr25-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-015-1813-1"},{"key":"bibr26-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbiomech.2010.11.006"},{"key":"bibr27-1356389017733336","doi-asserted-by":"publisher","DOI":"10.2307\/249291"},{"key":"bibr28-1356389017733336","doi-asserted-by":"publisher","DOI":"10.2307\/249329"},{"key":"bibr29-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/S0263-2373(98)00029-2"},{"key":"bibr30-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0507655102"},{"key":"bibr31-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.aos.2013.11.003"},{"key":"bibr32-1356389017733336","unstructured":"Institut de France, Acad\u00e9mie des Sciences (2011) On the proper use of Bibliometrics to evaluate individual researchers. Available at: http:\/\/www.academie-sciences.fr\/activite\/rapport\/avis170111gb.pdf (accessed May 2014)."},{"key":"bibr33-1356389017733336","doi-asserted-by":"publisher","DOI":"10.2308\/accr.2003.78.3.725"},{"key":"bibr34-1356389017733336","volume-title":"Principal Component Analysis","author":"Joliffe IT","year":"2002","edition":"2"},{"key":"bibr35-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1038\/422259a"},{"key":"bibr36-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2007.06.014"},{"issue":"12","key":"bibr37-1356389017733336","first-page":"1228","volume":"15","author":"Leydesdorff L","year":"2014","journal-title":"Science and Society"},{"key":"bibr38-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2010.09.002"},{"key":"bibr39-1356389017733336","doi-asserted-by":"publisher","DOI":"10.2308\/jmar-50837"},{"key":"bibr40-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1007\/BF02129599"},{"key":"bibr41-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.aos.2003.11.001"},{"issue":"4","key":"bibr42-1356389017733336","first-page":"416","volume":"25","author":"Mutz R","year":"2016","journal-title":"Research Evaluation"},{"key":"bibr43-1356389017733336","unstructured":"National Health and Medical Research Council, Australia (2010) Removes journal impact factor from peer review of individual research grant and fellowship applications. Available at: http:\/\/www.nhmrc.gov.au\/_files_nhmrc\/file\/grants\/peer\/impact%20factors%20in%20peer%20review.pdf (accessed May 2014)."},{"key":"bibr44-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-007-2135-7"},{"key":"bibr45-1356389017733336","unstructured":"Open Researcher and Contributor ID (2014) Available at: https:\/\/orcid.org\/"},{"issue":"2","key":"bibr46-1356389017733336","first-page":"650","volume-title":"Proceedings of ISSI 2011: The 13th Conference of the International Society for Scientometrics and Informetrics","volume":"1","author":"Parra C","year":"2011"},{"key":"bibr47-1356389017733336","unstructured":"Peer review in scientific publications (2011) Science and Technology Committee, House of Commons, UK. Available at: http:\/\/www.publications.parliament.uk\/pa\/cm201012\/cmselect\/cmsctech\/856\/85602.htm (accessed May 2014)."},{"key":"bibr48-1356389017733336","volume-title":"Rapid Instructional Design: Learning ID Fast and Right","author":"Piskurich GM","year":"2011","edition":"2"},{"key":"bibr49-1356389017733336","doi-asserted-by":"publisher","DOI":"10.2307\/25148678"},{"key":"bibr50-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1093\/reseval\/rvv023"},{"key":"bibr51-1356389017733336","unstructured":"San Francisco Declaration on Research Assessment (DORA) (2013) Available at: http:\/\/am.ascb.org\/dora\/ (accessed May 2014)."},{"key":"bibr52-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2008.05.001"},{"key":"bibr53-1356389017733336","unstructured":"Scopus Database (2014) Available at: https:\/\/www.elsevier.com\/solutions\/scopus"},{"key":"bibr54-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.mar.2004.08.002"},{"key":"bibr55-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2011.06.002"},{"key":"bibr56-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1108\/17410400710731437"},{"key":"bibr57-1356389017733336","unstructured":"Swedish Research Council (2009) Quality assessment in peer review. Available at: http:\/\/www.cm.se\/webbshop_vr\/pdfer\/2011_01L.pdf (accessed May 2014)."},{"key":"bibr58-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1093\/reseval\/rvu021"},{"key":"bibr59-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1023\/A:1024141819302"},{"key":"bibr60-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.aos.2009.02.002"},{"key":"bibr61-1356389017733336","unstructured":"Venables WN, Smith DM (2013) An introduction to R. Notes on R: A programming environment for data analysis and graphics, (electronic manual of version 3.0.1.)."},{"key":"bibr62-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2010.06.006"},{"key":"bibr63-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1177\/0165551506072165"},{"key":"bibr64-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.aos.2012.06.001"},{"key":"bibr65-1356389017733336","volume-title":"The Financial Times Essential Guide to Leading Your Team: How to Set Goals, Measure Performance and Reward Talent","author":"Yemm G","year":"2013"},{"key":"bibr66-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0059912"},{"key":"bibr67-1356389017733336","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2011.05.004"}],"container-title":["Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1356389017733336","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1356389017733336","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1356389017733336","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T08:09:50Z","timestamp":1740902990000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1356389017733336"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,10]]},"references-count":67,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2017,10]]}},"alternative-id":["10.1177\/1356389017733336"],"URL":"https:\/\/doi.org\/10.1177\/1356389017733336","relation":{},"ISSN":["1356-3890","1461-7153"],"issn-type":[{"value":"1356-3890","type":"print"},{"value":"1461-7153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,10]]}}}