{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:30:39Z","timestamp":1740123039047,"version":"3.37.3"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,9,14]],"date-time":"2020-09-14T00:00:00Z","timestamp":1600041600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,9,14]],"date-time":"2020-09-14T00:00:00Z","timestamp":1600041600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003407","name":"Ministero dell\u2019Istruzione, dell\u2019Universit\u00e0 e della Ricerca","doi-asserted-by":"publisher","award":["PRIN n. 2015RJARX7"],"award-info":[{"award-number":["PRIN n. 2015RJARX7"]}],"id":[{"id":"10.13039\/501100003407","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Clarivate Analytics","award":["KOL PRoject"],"award-info":[{"award-number":["KOL PRoject"]}]},{"name":"Universit\u00e0 degli Studi di Roma La Sapienza"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2020,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The quantitative evaluation of research is currently carried out by means of indicators calculated on data extracted and integrated by analysts who elaborate them by creating illustrative tables and plots of results. In this approach, the robustness of the metrics used and the possibility for users of the metrics to intervene in the evaluation process are completely neglected. We propose a new approach which is able to move forward, from indicators\u2019 development to an interactive performance model\u2019s development. It combines the advantages of the ontology-based data access paradigm with the flexibility and robustness of a visual analytics environment putting the consumer\/stakeholder at the centre of the evaluation. A detailed description of such an approach is presented in the paper. The approach is illustrated and evaluated trough a comprehensive user\u2019s study that proves the added capabilities and the benefits that a user of performance models can have by using this approach.<\/jats:p>","DOI":"10.1007\/s11192-020-03689-x","type":"journal-article","created":{"date-parts":[[2020,9,14]],"date-time":"2020-09-14T20:02:42Z","timestamp":1600113762000},"page":"865-892","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Performance model\u2019s development: a novel approach encompassing ontology-based data access and visual analytics"],"prefix":"10.1007","volume":"125","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9051-6972","authenticated-orcid":false,"given":"Marco","family":"Angelini","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cinzia","family":"Daraio","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maurizio","family":"Lenzerini","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Francesco","family":"Leotta","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giuseppe","family":"Santucci","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,9,14]]},"reference":[{"issue":"1","key":"3689_CR1","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1109\/TVCG.2018.2865028","volume":"25","author":"M Angelini","year":"2019","unstructured":"Angelini, M., Blasilli, G., Catarci, T., Lenti, S., & Santucci, G. (2019a). Vulnus: Visual vulnerability analysis for network security. IEEE Transactions on Visualization and Computer Graphics, 25(1), 183\u2013192. https:\/\/doi.org\/10.1109\/TVCG.2018.2865028.","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"3689_CR2","unstructured":"Angelini, M., Daraio, C., Lenzerini, M., Leotta, F., & Santucci, G. (2019b). Performance model\u2019s development: A novel approach encompassing ontology-based data access and visual analytics. In 2019 17th International conference on scientometrics and infometrics, ISSI 2019\u2014Proceedings (Vol. 2, pp. 1912\u20131923)."},{"issue":"6","key":"3689_CR3","doi-asserted-by":"publisher","first-page":"1077","DOI":"10.1016\/j.ipm.2018.04.006","volume":"54","author":"M Angelini","year":"2018","unstructured":"Angelini, M., Fazzini, V., Ferro, N., Santucci, G., & Silvello, G. (2018). CLAIRE: A combinatorial visual analytics system for information retrieval evaluation. Information Processing & Management, 54(6), 1077\u20131100.","journal-title":"Information Processing & Management"},{"issue":"3","key":"3689_CR4","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1007\/s12650-016-0377-3","volume":"20","author":"M Angelini","year":"2017","unstructured":"Angelini, M., & Santucci, G. (2017). Cyber situational awareness: from geographical alerts to high-level management. Journal of Visualization, 20(3), 453\u2013459.","journal-title":"Journal of Visualization"},{"key":"3689_CR5","unstructured":"Antonioli, N., Castan\u00f2, F., Civili, C., Coletta, S., Grossi, S., Lembo, D., et al. (2013). Ontology-based data access: The Experience at the Italian Department of Treasury. In Conference on advanced information systems engineering, CAiSE 2013-Co-located with 25th international conference on advanced information systems engineering, CAiSE 2013 (Vol. 1017, pp. 9\u201316)."},{"key":"3689_CR6","unstructured":"Antonioli, N., Castan\u00f2, F., Coletta, S., Grossi, S., Lembo, D., Lenzerini, M., et al. (2014). Ontology-based data management for the Italian Public Debt. In Formal ontology in information systems: Proceedings of the eighth international conference (FOIS 2014). IOS Press."},{"key":"3689_CR7","doi-asserted-by":"crossref","unstructured":"Aracri, R. M., Radini, R., Scannapieco, M., & Tosco, L. (2017, June). Using ontologies for official statistics: The ISTAT experience. In International conference on web engineering (pp. 166\u2013172). Cham: Springer.","DOI":"10.1007\/978-3-319-74433-9_15"},{"issue":"9","key":"3689_CR8","doi-asserted-by":"publisher","first-page":"1078","DOI":"10.1287\/mnsc.30.9.1078","volume":"30","author":"RD Banker","year":"1984","unstructured":"Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Science, 30(9), 1078\u20131092.","journal-title":"Management Science"},{"key":"3689_CR200","doi-asserted-by":"publisher","unstructured":"Battle, L., Angelini, M., Binnig, C., Catarci, T., Eichmann, P., Fekete, J. D., Santucci, G., Sedlmair, M., & Willett, W. (2018). Evaluating visual data analysis systems: A discussion report. In Proceedings of the workshop on human-in-the-loop data analytics (HILDA\u201918) (Vol. 4, pp. 1\u20136). New York, NY, USA: Association for Computing Machinery. https:\/\/doi.org\/10.1145\/3209900.3209901,","DOI":"10.1145\/3209900.3209901"},{"issue":"9","key":"3689_CR9","first-page":"883","volume":"44","author":"V Belton","year":"1993","unstructured":"Belton, V., Stephen, P., & Vickers, S. P. (1993). Demystifying DEA\u2014A visual interactive approach based on multiple criteria analysis. Journal of the Operational Research Society, 44(9), 883\u2013896.","journal-title":"Journal of the Operational Research Society"},{"issue":"194","key":"3689_CR10","first-page":"4","volume":"189","author":"J Brooke","year":"1996","unstructured":"Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4\u20137.","journal-title":"Usability Evaluation in Industry"},{"issue":"3","key":"3689_CR11","doi-asserted-by":"publisher","first-page":"471","DOI":"10.3233\/SW-160217","volume":"8","author":"D Calvanese","year":"2017","unstructured":"Calvanese, D., Cogrel, B., Komla-Ebri, S., Kontchakov, R., Lanti, D., Rezk, M., et al. (2017). Ontop: Answering SPARQL queries over relational databases. Semantic Web, 8(3), 471\u2013487.","journal-title":"Semantic Web"},{"issue":"1","key":"3689_CR12","doi-asserted-by":"publisher","first-page":"43","DOI":"10.3233\/SW-2011-0029","volume":"2","author":"D Calvanese","year":"2011","unstructured":"Calvanese, D., De Giacomo, G., Lembo, D., Lenzerini, M., Poggi, A., Rodriguez-Muro, M., et al. (2011). The MASTRO system for ontology-based data access. Semantic Web, 2(1), 43\u201353.","journal-title":"Semantic Web"},{"key":"3689_CR14","unstructured":"Calvanese, D., De Giacomo, G., Lenzerini, M., Nardi, D., & Rosati, R. (1998). Description logic framework for information integration. In Proceedings of KR (pp. 2\u201313)."},{"key":"3689_CR15","doi-asserted-by":"crossref","unstructured":"Catarci, T., Di Mascio, T., Franconi, E., Santucci, G., & Tessaris, S. (2003). An ontology based visual tool for query formulation support. In European conference on Artificial Intelligence (Vol. 2889, pp. 32\u201333). Lecture notes in computer science.","DOI":"10.1007\/978-3-540-39962-9_15"},{"key":"3689_CR16","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1016\/0377-2217(78)90138-8","volume":"2","author":"A Charnes","year":"1978","unstructured":"Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the inefficiency of decision making units. European Journal of Operational Research, 2, 429\u2013444.","journal-title":"European Journal of Operational Research"},{"key":"3689_CR17","volume-title":"An introduction to efficiency analysis","author":"T Coelli","year":"2004","unstructured":"Coelli, T., Rao, D. S. P., & Battese, G. E. (2004). An introduction to efficiency analysis (2nd ed.). Dordrecht: Kluwer Academic Publishers.","edition":"2"},{"key":"3689_CR18","unstructured":"Cook, K. A., & Thomas, J. J. (2005). Illuminating the path: The research and development agenda for visual analytics. Tech. rep., Pacific Northwest National Lab. (PNNL), Richland, WA, USA."},{"key":"3689_CR19","volume-title":"Data envelopment analysis: A comprehensive text with models, applications, references and DEA-Solver software","author":"WW Cooper","year":"1999","unstructured":"Cooper, W. W., Seiford, L. M., & Tone, K. (1999). Data envelopment analysis: A comprehensive text with models, applications, references and DEA-Solver software. Boston: Kluwer Academic Publishers."},{"issue":"4","key":"3689_CR20","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1515\/jdis-2017-0018","volume":"2","author":"C Daraio","year":"2017","unstructured":"Daraio, C. (2017). A framework for the assessment of research and its impacts. Journal of Data and Information Science, 2(4), 7\u201342.","journal-title":"Journal of Data and Information Science"},{"key":"3689_CR21","volume-title":"Springer handbook of science and technology indicators","author":"C Daraio","year":"2019","unstructured":"Daraio, C. (2019). Econometric approaches to the measurement of research productivity. In W. Gl\u00e4nzel, H. F. Moed, H. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators. Cham: Springer."},{"key":"3689_CR22","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23679","author":"C Daraio","year":"2017","unstructured":"Daraio, C., & Bonaccorsi, A. (2017). Beyond university rankings? Generating new indicators on universities by linking data in open platforms. Journal of the Association for Information Science and Technology. https:\/\/doi.org\/10.1002\/asi.23679.","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"1","key":"3689_CR23","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1007\/s11192-016-1914-5","volume":"108","author":"C Daraio","year":"2016","unstructured":"Daraio, C., & Gl\u00e4nzel, W. (2016). Grand challenges in data integration. State of the art and future perspectives: An introduction. Scientometrics, 108(1), 391\u2013400.","journal-title":"Scientometrics"},{"key":"3689_CR24","unstructured":"Daraio, C., Lenzerini, M., Leporelli, C., Moed, H. F., Naggar, P., Bonaccorsi, A., et al. (2015). Sapientia: The ontology of multi-dimensional research assessment. In A. A. Salah, Y. Tonta, A. A. Akdag Salah, C. Sugimoto, & U. Al (Eds.), Proceedings of ISSI 2015 Istanbul: 15th international society of scientometrics and informetrics conference, Istanbul, Turkey, 29 June to 3 July, 2015, Bogazi\u00e7i University Printhouse (pp. 965\u2013977)."},{"issue":"2","key":"3689_CR25","doi-asserted-by":"publisher","first-page":"857","DOI":"10.1007\/s11192-015-1814-0","volume":"106","author":"C Daraio","year":"2016","unstructured":"Daraio, C., Lenzerini, M., Leporelli, C., Moed, H. F., Naggar, P., Bonaccorsi, A., et al. (2016a). Data integration for research and innovation policy: An ontology-based data management approach. Scientometrics, 106(2), 857\u2013871.","journal-title":"Scientometrics"},{"issue":"1","key":"3689_CR26","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1007\/s11192-016-1913-6","volume":"108","author":"C Daraio","year":"2016","unstructured":"Daraio, C., Lenzerini, M., Leporelli, C., Naggar, P., Bonaccorsi, A., & Bartolucci, A. (2016b). The advantages of an ontology-based data management approach: Openness, interoperability and data quality. Scientometrics, 108(1), 441\u2013455.","journal-title":"Scientometrics"},{"key":"3689_CR27","volume-title":"Advanced robust and nonparametric methods in efficiency analysis. Methodology and applications","author":"C Daraio","year":"2007","unstructured":"Daraio, C., & Simar, L. (2007). Advanced robust and nonparametric methods in efficiency analysis. Methodology and applications. New York: Springer."},{"key":"3689_CR28","first-page":"243","volume-title":"The performance of public enterprises: Concepts and measurements","author":"D Deprins","year":"1984","unstructured":"Deprins, D., Simar, L., & Tulkens, H. (1984). Measuring labor inefficiency in post offices. In M. Marchand, P. Pestieau, & H. Tulkens (Eds.), The performance of public enterprises: Concepts and measurements (pp. 243\u2013267). Amsterdam: North-Holland."},{"key":"3689_CR29","unstructured":"Erhan, H., Woodbury, R., & Salmasi, N. H. (2009). Visual sensitivity analysis of parametric design models: Improving agility in design. In T. Tidafi & T. Dorta (Eds.), Proceedings of joining languages, cultures and visions: CAAD futures 2009, PUM, 2009 (pp. 815\u2013829)."},{"key":"3689_CR30","volume-title":"Production frontiers","author":"R Fare","year":"1994","unstructured":"Fare, R., Grosskopf, S., & Lovell, C. A. K. (1994). Production frontiers. Cambridge: Cambridge University Press."},{"key":"3689_CR201","unstructured":"Floridi, L. (2014). The fourth revolution: How the infosphere is reshaping human reality. OUP Oxford."},{"issue":"3","key":"3689_CR31","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1109\/MC.2015.82","volume":"48","author":"M Giese","year":"2015","unstructured":"Giese, M., Soylu, A., Vega-Gorgojo, G., Waaler, A., Haase, P., Jim\u00e9nez-Ruiz, E., et al. (2015). Optique: Zooming in on big data. Computer, 48(3), 60\u201367.","journal-title":"Computer"},{"key":"3689_CR32","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-68628-8","volume-title":"Parallel coordinates: Visual multidimensional geometry and its applications","author":"A Inselberg","year":"2009","unstructured":"Inselberg, A. (2009). Parallel coordinates: Visual multidimensional geometry and its applications (Vol. 20). Berlin: Springer."},{"key":"3689_CR33","first-page":"154","volume-title":"Visual analytics: Definition, process, and challenges","author":"D Keim","year":"2008","unstructured":"Keim, D., Andrienko, G., Fekete, J. D., Gorg, C., Kohlhammer, J., & Melancon, G. (2008). Visual analytics: Definition, process, and challenges (pp. 154\u2013175). Berlin: Springer."},{"key":"3689_CR34","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.websem.2017.05.005","volume":"44","author":"E Kharlamov","year":"2017","unstructured":"Kharlamov, E., Hovland, D., Skj\u00e6veland, M. G., Bilidas, D., Jim\u00e9nez-Ruiz, E., Xiao, G., et al. (2017). Ontology based data access in Statoil. Journal of Web Semantics, 44, 3\u201336.","journal-title":"Journal of Web Semantics"},{"key":"3689_CR35","unstructured":"Lembo, D., Pantaleone, D., Santarelli, V., & Savo, D. F. (2016). Easy OWL drawing with the Graphol visual ontology language. In KR (pp. 573\u2013576)."},{"key":"3689_CR36","volume-title":"Springer handbook of science and technology indicators","author":"M Lenzerini","year":"2019","unstructured":"Lenzerini, M., & Daraio, C. (2019). Challenges, approaches and solutions in data integration for research and innovation. In W. Gl\u00e4nzel, H. F. Moed, H. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators. Cham: Springer."},{"key":"3689_CR37","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1007\/978-3-030-02511-3_7","volume-title":"Springer handbook of science and technology indicators","author":"JA Moral-Mu\u00f1oz","year":"2019","unstructured":"Moral-Mu\u00f1oz, J. A., L\u00f3pez-Herrera, A. G., Herrera-Viedma, E., & Cobo, M. J. (2019). Science mapping analysis software tools: A review. In W. Gl\u00e4nzel, H. F. Moed, H. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 159\u2013185). Cham: Springer."},{"key":"3689_CR38","unstructured":"Mosca, A., Roda, F., & Rull, G. (2018). The ontology for research and innovation policy making. In Formal ontology in information systems: Proceedings of the 10th international conference (FOIS 2018) (Vol. 306, p. 200). IOS Press."},{"key":"3689_CR39","unstructured":"Santarelli, V., Lembo, D., Ruzzi, M., Ronconi, G., Bouquet, P., Molinari, A., et al. (2019). Semantic technologies for the production and publication of open data in ACI\u2014Automobile Club d\u2019Italia. In Proceedings of the ISWC 2019 satellite tracks (posters & demonstrations, industry, and outrageous ideas) co-located with 18th international semantic web conference (ISWC) (Vol. 2456, pp. 307\u2013308)."},{"key":"3689_CR40","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1007\/978-3-030-02511-3_12","volume-title":"Springer handbook of science and technology indicators","author":"A Schubert","year":"2019","unstructured":"Schubert, A., & Schubert, G. (2019). All along the h-index-related literature: A guided tour. In W. Gl\u00e4nzel, H. F. Moed, H. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 301\u2013334). Cham: Springer."},{"key":"3689_CR41","doi-asserted-by":"publisher","first-page":"28","DOI":"10.1016\/j.cola.2019.01.004","volume":"51","author":"ICS Silva","year":"2019","unstructured":"Silva, I. C. S., Santucci, G., & Freitas, C. M. D. S. (2019). Visualization and analysis of schema and instances of ontologies for improving user tasks and knowledge discovery. Journal of Computer Languages, 51, 28\u201347.","journal-title":"Journal of Computer Languages"},{"key":"3689_CR42","doi-asserted-by":"publisher","first-page":"667","DOI":"10.1007\/978-3-030-02511-3_25","volume-title":"Springer handbook of science and technology indicators","author":"G Sivertsen","year":"2019","unstructured":"Sivertsen, G. (2019). Developing current research information systems (CRIS) as data sources for studies of research. In W. Gl\u00e4nzel, H. F. Moed, H. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 667\u2013683). Cham: Springer."},{"issue":"5","key":"3689_CR43","doi-asserted-by":"publisher","first-page":"627","DOI":"10.3233\/SW-180293","volume":"9","author":"A Soylu","year":"2018","unstructured":"Soylu, A., Kharlamov, E., Zheleznyakov, D., Jimenez-Ruiz, E., Giese, M., Skj\u00e6veland, M. G., et al. (2018). OptiqueVQS: A visual query system over ontologies for industry. Semantic Web, 9(5), 627\u2013660.","journal-title":"Semantic Web"},{"key":"3689_CR44","doi-asserted-by":"publisher","DOI":"10.1515\/9783486724653","volume-title":"Die Gesetzm\u00e4\u00dfigkeit im Gesellschaftsleben: Statistische Studien","author":"G von Mayr","year":"1877","unstructured":"von Mayr, G. (1877). Die Gesetzm\u00e4\u00dfigkeit im Gesellschaftsleben: Statistische Studien. M\u00fcnchen: Oldenbourg."},{"key":"3689_CR45","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1007\/978-3-030-02511-3_14","volume-title":"Springer handbook of science and technology indicators","author":"L Wildgaard","year":"2019","unstructured":"Wildgaard, L. (2019). An overview of author-level indicators of research performance. In W. Gl\u00e4nzel, H. F. Moed, H. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 361\u2013396). Cham: Springer."}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03689-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11192-020-03689-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03689-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,14]],"date-time":"2021-09-14T01:34:15Z","timestamp":1631583255000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11192-020-03689-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,14]]},"references-count":46,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,11]]}},"alternative-id":["3689"],"URL":"https:\/\/doi.org\/10.1007\/s11192-020-03689-x","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"type":"print","value":"0138-9130"},{"type":"electronic","value":"1588-2861"}],"subject":[],"published":{"date-parts":[[2020,9,14]]},"assertion":[{"value":"15 December 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 September 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}