{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T01:56:04Z","timestamp":1785462964993,"version":"3.56.0"},"reference-count":74,"publisher":"SAGE Publications","issue":"3","license":[{"start":{"date-parts":[[2012,4,19]],"date-time":"2012-04-19T00:00:00Z","timestamp":1334793600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Journal of Information Science"],"published-print":{"date-parts":[[2012,6]]},"abstract":"<jats:p>In the past two decades, the use of digital libraries (DLs) has grown significantly. Accordingly, questions about the utility, usability and cost of DLs have started to arise, and greater attention is being paid to the quality evaluation of this type of information system. Since DLs are destined to serve user communities, one of the main aspects to be considered in DL evaluation is the user\u2019s opinion. The literature on this topic has produced a set of varied criteria to judge DLs from the user\u2019s perspective, measuring instruments to elicit users\u2019 opinions, and approaches to analyse the elicited data to conclude an evaluation. This paper provides a literature review of the quality evaluation of DLs based on users\u2019 perceptions. Its main contribution is to bring together previously disparate streams of work to help shed light on this thriving area. In addition, the various studies are discussed, and some challenges to be faced in the future are proposed.<\/jats:p>","DOI":"10.1177\/0165551512438359","type":"journal-article","created":{"date-parts":[[2012,4,20]],"date-time":"2012-04-20T20:14:07Z","timestamp":1334952847000},"page":"269-283","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":61,"title":["A review of quality evaluation of digital libraries based on users\u2019 perceptions"],"prefix":"10.1177","volume":"38","author":[{"given":"Rub\u00e9n","family":"Heradio","sequence":"first","affiliation":[{"name":"Distance Learning University of Spain (UNED), Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David","family":"Fern\u00e1ndez-Amor\u00f3s","sequence":"additional","affiliation":[{"name":"Distance Learning University of Spain (UNED), Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Francisco Javier","family":"Cabrerizo","sequence":"additional","affiliation":[{"name":"Distance Learning University of Spain (UNED), Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Enrique","family":"Herrera-Viedma","sequence":"additional","affiliation":[{"name":"University of Granada, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2012,4,19]]},"reference":[{"key":"bibr1-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1007\/s00799-004-0100-1"},{"key":"bibr2-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1177\/0165551510386173"},{"key":"bibr3-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1177\/0165551510383086"},{"key":"bibr4-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1177\/0165551510378811"},{"key":"bibr5-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1177\/0165551510382071"},{"key":"bibr6-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1177\/0165551510377709"},{"key":"bibr7-0165551512438359","doi-asserted-by":"crossref","first-page":"119","DOI":"10.7551\/mitpress\/2424.003.0009","volume-title":"Digital library use: Social practice in design and evaluation","author":"Marchionini G","year":"2003"},{"key":"bibr8-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/S1071-5819(02)91024-4"},{"issue":"3","key":"bibr9-0165551512438359","first-page":"350","volume":"49","author":"Saracevic T","year":"2000","journal-title":"Library Trends"},{"key":"bibr10-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1177\/0165551509353251"},{"key":"bibr11-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1108\/02640470910966899"},{"key":"bibr12-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2008.12.003"},{"key":"bibr13-0165551512438359","first-page":"45","volume-title":"Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries","author":"Tsakonas G","year":"2004"},{"key":"bibr14-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21559"},{"key":"bibr15-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1007\/s00799-007-0011-z"},{"key":"bibr16-0165551512438359","doi-asserted-by":"crossref","first-page":"85","DOI":"10.7551\/mitpress\/2424.003.0008","volume-title":"Digital library use: social practice in design and evaluation","author":"Borgman CL","year":"2003"},{"key":"bibr17-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1177\/0165551507087711"},{"key":"bibr18-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21220"},{"key":"bibr19-0165551512438359","first-page":"13","volume-title":"Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries","author":"Saracevic T","year":"2004"},{"key":"bibr20-0165551512438359","unstructured":"Kitchenham B. Procedures for performing systematic reviews. Technical Report TR\/SE0401 Software Engineering Group. Department of Computer Science, Keele University, 2004."},{"issue":"2","key":"bibr21-0165551512438359","first-page":"13","volume":"26","author":"Webster J","year":"2002","journal-title":"MIS Quarterly"},{"key":"bibr22-0165551512438359","author":"Candela L","year":"2007","journal-title":"The DELOS Digital Library Reference Model. Foundations for digital libraries"},{"key":"bibr23-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1007\/s007990050044"},{"issue":"2","key":"bibr24-0165551512438359","first-page":"304","volume":"49","author":"Marchionini G","year":"2001","journal-title":"Library Trends"},{"key":"bibr25-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-44796-2_17"},{"key":"bibr26-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1145\/503112.503114"},{"key":"bibr27-0165551512438359","first-page":"79","volume-title":"4th DELOS Workshop on Evaluation of Digital Libraries: Testbeds, Measurements, and Metrics","author":"Evans J","year":"2002"},{"key":"bibr28-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1108\/10650750410551451"},{"key":"bibr29-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1145\/996350.996360"},{"key":"bibr30-0165551512438359","first-page":"31","volume-title":"Notes of the DELOS WP7 Workshop on the Evaluation of Digital Libraries","author":"Blandford A","year":"2004"},{"issue":"2","key":"bibr31-0165551512438359","first-page":"96","volume":"55","author":"Jeng J","year":"2005","journal-title":"International Journal of Libraries and Information Services"},{"key":"bibr32-0165551512438359","doi-asserted-by":"publisher","DOI":"10.6017\/ital.v24i2.3365"},{"key":"bibr33-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1080\/10572317.2005.10762690"},{"key":"bibr34-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14504201161"},{"key":"bibr35-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/j.lisr.2006.06.002"},{"key":"bibr36-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1108\/14684520610686292"},{"key":"bibr37-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1177\/0165551506077661"},{"key":"bibr38-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1108\/02640470810864136"},{"key":"bibr39-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2007.01.021"},{"key":"bibr40-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1108\/02640470810921628"},{"key":"bibr41-0165551512438359","doi-asserted-by":"publisher","DOI":"10.6017\/ital.v27i4.3238"},{"key":"bibr42-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2007.07.008"},{"key":"bibr43-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2007.10.003"},{"issue":"1","key":"bibr44-0165551512438359","first-page":"333","volume":"13","author":"Zuccala A","year":"2008","journal-title":"Information Research"},{"key":"bibr45-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2008.01.012"},{"key":"bibr46-0165551512438359","first-page":"92","volume":"2009","author":"Kostkova P","journal-title":"Lecture Notes in Artificial Intelligence"},{"key":"bibr47-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1142\/9789814295062_0090"},{"key":"bibr48-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-84334-484-1.50004-1"},{"key":"bibr49-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1142\/S0219622010003907"},{"key":"bibr50-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/j.acalib.2010.01.002"},{"key":"bibr51-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1109\/ISDA.2010.5687193"},{"key":"bibr52-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1109\/ISKE.2010.5680768"},{"key":"bibr53-0165551512438359","first-page":"RW_202_1","volume-title":"2011 International Fuzzy Systems Association World Congress and the 2011 Asian Fuzzy Systems Society","author":"Cabrerizo FJ","year":"2011"},{"key":"bibr54-0165551512438359","first-page":"21","volume-title":"Human factors for informatics usability","author":"Shackel B","year":"1991"},{"key":"bibr55-0165551512438359","volume-title":"Human ease of use criteria and their tradeoffs","author":"Miller RB","year":"1971"},{"key":"bibr56-0165551512438359","volume-title":"Ergonomic requirements for office work with visual display terminals","author":"ISO","year":"1994"},{"issue":"2","key":"bibr57-0165551512438359","first-page":"39","volume":"14","author":"Kano N","year":"1984","journal-title":"Hinshitsu Quality Journal of the Japanese Society for Quality Control"},{"key":"bibr58-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1007\/s007999900022"},{"key":"bibr59-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/S0740-8188(02)00126-3"},{"key":"bibr60-0165551512438359","first-page":"231","volume":"2458","author":"Michalis S","year":"2002","journal-title":"Lecture Notes in Computer Science"},{"key":"bibr61-0165551512438359","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1145\/544220.544276","volume-title":"Proceedings of the 2nd ACM\/IEEECS Joint Conference on Digital Libraries","author":"Gon\u00e7alves MA","year":"2002"},{"key":"bibr62-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1145\/984321.984325"},{"key":"bibr63-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.2000.0416"},{"issue":"140","key":"bibr64-0165551512438359","first-page":"1","volume":"22","author":"Likert R","year":"1932","journal-title":"Archives of Psychology"},{"key":"bibr65-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/S0261-5177(01)00050-4"},{"key":"bibr66-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/0165-0114(95)00162-X"},{"key":"bibr67-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1109\/3468.618263"},{"key":"bibr68-0165551512438359","doi-asserted-by":"publisher","DOI":"10.4135\/9781849208604"},{"key":"bibr69-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2929.2004.02012.x"},{"key":"bibr70-0165551512438359","volume-title":"Simple statistics: A course book for the social sciences","author":"Clegg F","year":"1983"},{"key":"bibr71-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1002\/int.20332"},{"key":"bibr72-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1142\/S0219622009003296"},{"issue":"1","key":"bibr73-0165551512438359","first-page":"62","volume":"16","author":"Cabrerizo FJ","year":"2010","journal-title":"Journal of Universal Computer Science"},{"key":"bibr74-0165551512438359","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2009.11.019"}],"container-title":["Journal of Information Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0165551512438359","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0165551512438359","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0165551512438359","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T23:08:19Z","timestamp":1777504099000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0165551512438359"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,4,19]]},"references-count":74,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2012,6]]}},"alternative-id":["10.1177\/0165551512438359"],"URL":"https:\/\/doi.org\/10.1177\/0165551512438359","relation":{},"ISSN":["0165-5515","1741-6485"],"issn-type":[{"value":"0165-5515","type":"print"},{"value":"1741-6485","type":"electronic"}],"subject":[],"published":{"date-parts":[[2012,4,19]]}}}