{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T14:29:06Z","timestamp":1776090546986,"version":"3.50.1"},"reference-count":57,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2022,6,4]],"date-time":"2022-06-04T00:00:00Z","timestamp":1654300800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Content and metadata concerning a specialized field such as Art and Cultural Heritage are often scattered throughout the World Wide Web, making it hard for end-users to find, especially amid the vast and often commercialized general content of the Web. This paper presents the process of designing and developing a Federated Search Engine (FSE) that collects such content from multiple credible sources of the world of Art and Culture and presents it to the user in a unified user-oriented manner, enhancing it with added functionality. The study focuses on the challenges such an endeavor presents and the technological tools, design decisions and methodology that lead to a fully functional, Web-based platform. This implemented search engine was evaluated by a group of stakeholders from the wider fields of art, culture and media during a closed test and the insights and feedback gained by these tests are herein analyzed and presented. These insights contain both the quantitative metrics of user engagement during the testing period and the qualitative information presented by the stakeholders through interviews. The above findings are thoroughly discussed and lead to conclusions regarding the usefulness and viability of Web applications in the aggregation and diffusion of Art and Cultural Heritage related content.<\/jats:p>","DOI":"10.3390\/info13060285","type":"journal-article","created":{"date-parts":[[2022,6,4]],"date-time":"2022-06-04T09:42:32Z","timestamp":1654335752000},"page":"285","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["User Evaluation and Metrics Analysis of a Prototype Web-Based Federated Search Engine for Art and Cultural Heritage"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1316-5448","authenticated-orcid":false,"given":"Minas","family":"Pergantis","sequence":"first","affiliation":[{"name":"Department of Audio and Visual Arts, Ionian University, 7 Tsirigoti Square, 49100 Corfu, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0876-8167","authenticated-orcid":false,"given":"Iraklis","family":"Varlamis","sequence":"additional","affiliation":[{"name":"Department of Informatics and Telematics, Harokopio University of Athens, Omirou 9, 17778 Tavros Athens, Greece"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0456-2217","authenticated-orcid":false,"given":"Andreas","family":"Giannakoulopoulos","sequence":"additional","affiliation":[{"name":"Department of Audio and Visual Arts, Ionian University, 7 Tsirigoti Square, 49100 Corfu, Greece"}]}],"member":"1968","published-online":{"date-parts":[[2022,6,4]]},"reference":[{"key":"ref_1","unstructured":"Respini, E. (2018). Art in the Age of the Internet: 1989 to Today, Yale University Press."},{"key":"ref_2","unstructured":"Manjoo, F. (The New York Times, 2017). How the internet is saving culture, not killing it, The New York Times."},{"key":"ref_3","unstructured":"Jones, L.S. (2013). Art Information and the Internet: How to Find It, How to Use It, Routledge."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1016\/j.culher.2015.04.006","article-title":"Framing the past: How virtual experience affects bodily description of artefacts","volume":"17","author":"Matthews","year":"2016","journal-title":"J. Cult. Herit."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.culher.2016.10.016","article-title":"Evaluating websites of museums: State of the art","volume":"24","author":"Kabassi","year":"2017","journal-title":"J. Cult. Herit."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Spink, A., and Jansen, B.J. (2005). Web Search: Public Searching of the Web, Springer.","DOI":"10.1007\/1-4020-2269-7"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1145\/792550.792552","article-title":"A Taxonomy of Web Search","volume":"36","author":"Broder","year":"2002","journal-title":"ACM SIGIR Forum"},{"key":"ref_8","unstructured":"Purcell, K., Rainie, L., and Brenner, J. (2022, April 25). Search Engine Use 2012. Available online: https:\/\/www.pewresearch.org\/internet\/2012\/03\/09\/search-engine-use-2012\/."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"744","DOI":"10.1002\/asi.20555","article-title":"Web searcher interaction with the Dogpile. com metasearch engine","volume":"58","author":"Jansen","year":"2007","journal-title":"J. Am. Soc. Inf. Sci. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1500000010","article-title":"Federated Search","volume":"5","author":"Shokouhi","year":"2011","journal-title":"Found. Trends Inf. Retr."},{"key":"ref_11","unstructured":"(2022, April 25). The Editors of Encyclopaedia Britannica. Encyclopaedia Britannica. Art Collection 2022. Available online: https:\/\/www.britannica.com\/art\/art-collection."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1145\/792550.792553","article-title":"Challenges in web search engines","volume":"36","author":"Henzinger","year":"2002","journal-title":"ACM SIGIR Forum"},{"key":"ref_13","unstructured":"Silverstein, C., Henzinger, M., Marais, H., and Moricz, M. (1998). Analysis of a Very Large AltaVista Query Log, Digital SRC. Technical Report 1998-014."},{"key":"ref_14","unstructured":"Beus, J. (2022, April 25). Why (Almost) Everything You Knew about Google CTR is No Longer Valid\u2014SISTRIX. SISTRIX, Available online: https:\/\/www.sistrix.com\/blog\/why-almost-everything-you-knew-about-google-ctr-is-no-longer-valid."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1109\/MC.2015.297","article-title":"Understanding search-engine optimization","volume":"48","author":"Gudivada","year":"2015","journal-title":"Computer"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Song, Y., Ma, H., Wang, H., and Wang, K. (2013, January 13\u201317). Exploring and Exploiting User Search Behavior on Mobile and Tablet Devices to Improve Search Relevance. Proceedings of the 22nd International Conference on World Wide Web, Rio de Janeiro, Brazil.","DOI":"10.1145\/2488388.2488493"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1016\/j.dss.2007.06.002","article-title":"A machine learning approach to web page filtering using content and structure analysis","volume":"44","author":"Chau","year":"2008","journal-title":"Decis. Support Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1079\/PNS2004399","article-title":"Focus-group interview and data analysis","volume":"63","author":"Rabiee","year":"2004","journal-title":"Proc. Nutr. Soc."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1080\/03634529009378794","article-title":"Assessing educational effectiveness: The focus group interview as a technique for data collection","volume":"39","author":"Lederman","year":"1990","journal-title":"Commun. Educ."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"654","DOI":"10.1046\/j.1365-2648.1998.00708.x","article-title":"An analysis of the effectiveness of focus groups as a method of qualitative data collection with Chinese populations in nursing research","volume":"28","author":"Twinn","year":"1998","journal-title":"J. Adv. Nurs."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kontio, J., Lehtola, L., and Bragge, J. (2004, January 20\u201320). Using the focus group method in software engineering: Obtaining practitioner and user experiences. Proceedings of the 2004 International Symposium on Empirical Software Engineering, 2004. ISESE\u201904, Redondo Beach, CA, USA.","DOI":"10.1109\/ISESE.2004.1334914"},{"key":"ref_22","unstructured":"Fine, M.R. (2002). Beta Testing for Better Software, John Wiley & Sons."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wurangian, N.C. (1993). Testing\u2013Alpha, Beta\u2026. OCLC Systems & Services: International Digital Library Perspectives, Emerald Group Publishing.","DOI":"10.1108\/EUM0000000003718"},{"key":"ref_24","unstructured":"Pergantis, M., Lamprogeorgos, A., and Giannakoulopoulos, A. (2021, January 28\u201329). Results of a Qualitative Survey of Habits and Needs of Users Searching the Web for Art and Cultural Heritage. Proceedings of the 3rd International Conference Digital Culture & AudioVisual Challenges Interdisciplinary Creativity in Arts and Technology, Online."},{"key":"ref_25","unstructured":"Schoch, K. (2020). Case Study Research. Research Design and Methods: An Applied Guide for the Scholar-Practitioner, SAGE Publications."},{"key":"ref_26","unstructured":"McCaffery, F., O\u2019Connor, R.V., and Messnarz, R. (2013). Research Prototypes Versus Products: Lessons Learned from Software Development Processes in Research Projects. Systems, Software and Services Process Improvement. EuroSPI. Communications in Computer and Information Science, Springer."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Pissinati, P.D.S.C., \u00c9vora, Y.D.M., Rossaneis, M.A., Gvozd, R., Santos, M.S.D., and Haddad, M.D.C.F.L. (2019). Development of a web software prototype to support retirement planning. Rev. Lat.-Am. De Enferm., 27.","DOI":"10.1590\/1518-8345.3024.3169"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Carvajal-Ortiz, L., Florian-Gaviria, B., and D\u00edaz, J.F. (October, January 30). Models, Methods and Software Prototype to Support the Design, Evaluation, and Analysis in the Curriculum Management of Competency-Based for Higher Education. Proceedings of the 2019 XLV Latin American Computing Conference (CLEI), Panama, Panama.","DOI":"10.1109\/CLEI47609.2019.235114"},{"key":"ref_29","unstructured":"Begic, M., Cirimotic, M., Farkas, I., Skoric, I., Car, Z., Rasan, I., and Zilak, M. (October, January 28). Software Prototype Based on Augmented Reality for Mastering Vocabulary. Proceedings of the 43rd International Convention on Information, Communication and Electronic Technology (MIPRO), Opatija, Croatia."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Terho, H., Suonsyrj\u00e4, S., Syst\u00e4, K., and Mikkonen, T. (2017, January 4\u20137). Understanding the Relations between Iterative Cycles in Software Engineering. Proceedings of the 50th Hawaii International Conference on System Sciences, Hilton Waikoloa Village, HI, USA.","DOI":"10.24251\/HICSS.2017.710"},{"key":"ref_31","unstructured":"Suonsyrj\u00e4, S., Syst\u00e4, K., Mikkonen, T., and Terho, H. (2022, April 30). Collecting Usage Data for Software Development: Selection Framework for Technological Approaches. Available online: https:\/\/www.ksiresearch.org\/seke\/seke16paper\/seke16paper_186.pdf."},{"key":"ref_32","unstructured":"Litosseliti, L. (2003). Using Focus Groups in Research, A&C Black. Continuum, London."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Kunze, J., and Baker, T. (2022, April 25). The Dublin Core Metadata Element Set, Dublin Core Metadata Initiative, 2007. Available online: https:\/\/www.ietf.org\/rfc\/rfc5013.txt.","DOI":"10.17487\/rfc5013"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2311","DOI":"10.1109\/TVCG.2018.2830759","article-title":"Visualization of Cultural Heritage Collection Data: State of the Art and Future Challenges","volume":"25","author":"Windhager","year":"2019","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_35","unstructured":"Clayphan, R., Charles, V., and Isaac, A. (2022, April 25). Europeana Data Model\u2014Mapping Guidelines 2.4. Available online: https:\/\/pro.europeana.eu\/page\/edm-documentation."},{"key":"ref_36","unstructured":"Feigenbaum, G., and Reist, I. (2012). Provenance: An Alternate History of Art, Getty Research Institute."},{"key":"ref_37","unstructured":"Ng, A. (2022, April 25). The Author\u2019s Rights in Literary and Artistic Works. In John Marshall Review of Intellectual Property Law, Forthcoming, Mississippi College School of Law Research Paper 2009. Available online: https:\/\/ssrn.com\/abstract=1427989."},{"key":"ref_38","first-page":"142","article-title":"Text-to-speech audio description: Towards wider availability of AD","volume":"15","author":"Szarkowska","year":"2011","journal-title":"J. Spec. Transl."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Corbett, E., and Weber, A. (2016, January 6\u20139). What Can I Say? Addressing User Experience Challenges of a Mobile Voice User Interface for Accessibility. Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services, Florence, Italy.","DOI":"10.1145\/2935334.2935386"},{"key":"ref_40","unstructured":"Sharma, H. (2022, April 25). Social Media Engagement Metrics\u20146 Things You Can Learn. Available online: http:\/\/www.optimizesmart.com\/6-learn-postrank-social-engagement-metrics\/#ixzz3OZtPcqmC."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1108\/EL-03-2015-0046","article-title":"Trends and challenges of visual search interfaces in digital libraries and repositories","volume":"35","year":"2017","journal-title":"Electron. Libr."},{"key":"ref_42","unstructured":"Sunikka, A. (2004). Usability Evaluation of the Helsinki School of Economics Website. [Master\u2019s Thesis, Aalto University]."},{"key":"ref_43","unstructured":"Kapsalis, E. (2016, January 11\u201320). The Impact of Open Access on Galleries, Libraries, Museums, & Archives. Proceedings of the SXSW Conference 2016, Austin, TX, USA."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Premchand, A., and Choudhry, A. (2018, January 15\u201317). Open Banking & APIs for Transformation in Banking. Proceedings of the 2018 International Conference on Communication, Computing and Internet of Things (IC3IoT), Chennai, India.","DOI":"10.1109\/IC3IoT.2018.8668107"},{"key":"ref_45","unstructured":"Pergantis, M., Lamprogeorgos, A., and Giannakoulopoulos, A. (2022, January 13\u201314). Comparative Analysis of Popular APIs Providing Access to Content Related to Art and Cultural Heritage. Proceedings of the 4th International Conference Digital Culture & AudioVisual Challenges Interdisciplinary Creativity in Arts and Technology, Corfu, Greece."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Krishna, R., and Jayakrishnan, R. (2013). Impact of Cloud Services on Software Development Life Cycle. Software Engineering Frameworks for the Cloud Computing Paradigm, Springer.","DOI":"10.1007\/978-1-4471-5031-2_4"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Li, T., Pavlou, P., and Santos, G.D. (2013, January 15\u201318). What Drives Users\u2019 Website Registration? A Randomized Field Experiment. Proceedings of the 34th International Conference on Information Systems (ICIS), Milano, Italy.","DOI":"10.2139\/ssrn.2369444"},{"key":"ref_48","first-page":"57","article-title":"To Social Login or not Login? Exploring Factors Affecting the Decision","volume":"11","author":"Gafni","year":"2014","journal-title":"Issues Inf. Sci. Inf. Technol."},{"key":"ref_49","unstructured":"Church, K., and Oliver, N. (September, January 30). Understanding Mobile Web and Mobile Search Use in Today\u2019s Dynamic Mobile Landscape. Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Service, Stockholm, Sweden."},{"key":"ref_50","unstructured":"Ceccarelli, D., Gordea, S., Lucchese, C., Nardini, F.M., Perego, R., and Tolomei, G. (2011). Discovering Europeana Users\u2019 Search Behavior. ERCIM News, Available online: https:\/\/ercim-news.ercim.eu\/en86\/special\/discovering-europeana-users-search-behavior."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Dimoulas, C., and Veglis, A. (2019, January 28\u201330). Factors and Models Contributing to the Optimization of Search Engine Results Credibility and Application on News Content: The Cross-Credibility Engine Optimization (CCEO) Model. Proceedings of the 23rd Pan-Hellenic Conference of Informatics, Nicosia, Cyprus.","DOI":"10.1145\/3368640.3368670"},{"key":"ref_52","first-page":"27","article-title":"The accuracy and thoroughness of a federated search engine in the health sciences","volume":"12","author":"Appelt","year":"2007","journal-title":"Internet Ref. Serv. Q."},{"key":"ref_53","unstructured":"Rehm, G., Marheinecke, K., Hegele, S., Piperidis, S., Bontcheva, K., Haji\u010d, J., Choukri, K., Vasi\u013cjevs, A., Backfried, G., and Prinz, C. (2020). The European Language Technology Landscape in 2020: Language-Centric and Human-Centric AI for Cross-Cultural Communication in Multilingual Europe. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Zhang, J., and Kamps, J. (2010, January 18\u201321). Search Log Analysis of User Stereotypes, Information Seeking Behavior, and Contextual Evaluation. Proceedings of the Third Symposium on Information Interaction in Context, New Brunswick, NJ, USA.","DOI":"10.1145\/1840784.1840820"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"e280","DOI":"10.1002\/pra2.280","article-title":"Evaluating the User Experience in a Digital Library","volume":"57","author":"Barifah","year":"2020","journal-title":"Proc. Assoc. Inf. Sci. Technol."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"862","DOI":"10.1002\/asi.20564","article-title":"Defining a session on web search engines","volume":"58","author":"Jansen","year":"2007","journal-title":"J. Am. Soc. Inf. Sci. Technol."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1007\/s007999900022","article-title":"A transaction log analysis of a digital library","volume":"3","author":"Jones","year":"2000","journal-title":"Int. J. Digit. Libr."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/6\/285\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:24:26Z","timestamp":1760138666000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/6\/285"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,4]]},"references-count":57,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["info13060285"],"URL":"https:\/\/doi.org\/10.3390\/info13060285","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,4]]}}}