{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T11:12:17Z","timestamp":1777893137117,"version":"3.51.4"},"reference-count":48,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,3,24]],"date-time":"2022-03-24T00:00:00Z","timestamp":1648080000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Evaluating public services has become an important task in order to direct actions that may positively affect with the quality of the service provided by governments. To undertake an effective evaluation, it is necessary to analyze data and information on the impact of the services. A wide range of studies have been proposed to measure how an organization delivers its services according to the expectations of the stakeholders. This paper investigates approaches for evaluating public services from the perspective of users. The goal is to identify and describe evaluation-based models, as well as the instruments and tools employed in the service evaluation process. A systematic literature review was conducted to search and analyze studies published in the last 15 years. From the analysis of 31 studies, we identified four main dimensions regarding service evaluation: quality, success and acceptance of information systems, user satisfaction, and user experience. This work contributes to the identification of models, dimensions, instruments, and tools to evaluate public services from the perspective of users. The results of this work can be used as a guide to the Public Administration in the construction of effective models to evaluate their public services and to guarantee quality standards that meet the expectations of the users.<\/jats:p>","DOI":"10.3390\/info13040162","type":"journal-article","created":{"date-parts":[[2022,3,27]],"date-time":"2022-03-27T21:27:51Z","timestamp":1648416471000},"page":"162","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Evaluation of Public Services Considering the Expectations of Users\u2014A Systematic Literature Review"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0229-7421","authenticated-orcid":false,"given":"V\u00edtor G. de","family":"Menezes","sequence":"first","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Brasilia 72444-240, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5573-6830","authenticated-orcid":false,"given":"Glauco V.","family":"Pedrosa","sequence":"additional","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Brasilia 72444-240, Brazil"},{"name":"Post-Graduate Program in Applied Computing, University of Brasilia (UnB), Brasilia 70910-900, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marcos P. P. da","family":"Silva","sequence":"additional","affiliation":[{"name":"Post-Graduate Program in Applied Computing, University of Brasilia (UnB), Brasilia 70910-900, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8243-7924","authenticated-orcid":false,"given":"Rejane M. da C.","family":"Figueiredo","sequence":"additional","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Brasilia 72444-240, Brazil"},{"name":"Post-Graduate Program in Applied Computing, University of Brasilia (UnB), Brasilia 70910-900, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Gardenghi, J.L., Pereira, L.G., Alcantara, S.M., Figueiredo, R.M.C., Ramos, C.S., and Ribeiro, L.C.M. (2020). Digitalization by Means of a Prototyping Process: The Case of a Brazilian Public Service. Information, 11.","DOI":"10.3390\/info11090413"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1016\/j.ijinfomgt.2009.03.013","article-title":"Measuring the Success of the Greek Taxation Information System","volume":"30","author":"Floropoulos","year":"2010","journal-title":"Int. J. Inf. Manag."},{"key":"ref_3","unstructured":"Soares, V.d.A., Iwama, G.Y., Menezes, V.G., Gomes, M.M.F., Pedrosa, G.V., Silva, W.C.M.P.d., and Figueiredo, R.M.D.C. (2019, January 18\u201320). Evaluating Government Services Based on User Perspective. Proceedings of the 20th Annual International Conference on Digital Government Research, Dubai, United Arab Emirates."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Pedrosa, G., Menezes, V., and Figueiredo, R. (2020, January 7\u201310). A Quality Management-Based Approach to Evaluate Public Services: A Case Study in Brazil. Proceedings of the 53rd Hawaii International Conference on System Sciences, HICSS 2020, Maui, HI, USA.","DOI":"10.24251\/HICSS.2020.277"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1177\/002224298504900403","article-title":"A Conceptual Model of Service Quality and Its Implications for Future Research","volume":"49","author":"Parasuraman","year":"1985","journal-title":"J. Mark."},{"key":"ref_6","first-page":"12","article-title":"SERVQUAL: A Multiple-Item Scale for Measuring Consumer Perceptions of Service Quality","volume":"64","author":"Parasuraman","year":"1988","journal-title":"J. Retail."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"982","DOI":"10.1287\/mnsc.35.8.982","article-title":"User Acceptance of Computer Technology: A Comparison of Two Theoretical Models","volume":"35","author":"Davis","year":"1989","journal-title":"Manag. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1080\/07421222.2003.11045748","article-title":"The DeLone and McLean Model of Information Systems Success: A Ten-Year Update","volume":"19","author":"DeLone","year":"2003","journal-title":"J. Manag. Inf. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1177\/1094670504271156","article-title":"E-S-QUAL: A Multiple-Item Scale for Assessing Electronic Service Quality","volume":"7","author":"Parasuraman","year":"2005","journal-title":"J. Serv. Res."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Pedrosa, G.V., Kosloski, R.A.D., Menezes, V.G.d., Iwama, G.Y., Silva, W.C.M.P.d., and Figueiredo, R.M.d.C. (2020). A Systematic Review of Indicators for Evaluating the Effectiveness of Digital Public Services. Information, 11.","DOI":"10.3390\/info11100472"},{"key":"ref_11","unstructured":"Kitchenham, B., and Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering, Keele University. Keele University and Durham University Joint Report."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1016\/j.jss.2006.07.009","article-title":"Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain","volume":"80","author":"Brereton","year":"2007","journal-title":"J. Syst. Softw."},{"key":"ref_13","unstructured":"Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele University."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2049","DOI":"10.1016\/j.infsof.2013.07.010","article-title":"A Systematic Review of Systematic Review Process Research in Software Engineering","volume":"55","author":"Kitchenham","year":"2013","journal-title":"Inf. Softw. Technol."},{"key":"ref_15","unstructured":"Petticrew, M., and Roberts, H. (2008). Systematic Reviews in the Social Sciences: A Practical Guide, John Wiley & Sons."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Dieste, O., and Padua, A.G. (2007, January 20\u201321). Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), Madrid, Spain.","DOI":"10.1109\/ESEM.2007.19"},{"key":"ref_17","first-page":"243","article-title":"Evaluation of E-health Services: User\u2019s Perspective Criteria","volume":"2","author":"Hamid","year":"2008","journal-title":"Transform. Gov. People Process Policy"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Huai, J. (2011, January 12\u201314). Quality Evaluation of E-Government Public Service. Proceedings of the International Conference on Management and Service Science, Wuhan, China.","DOI":"10.1109\/ICMSS.2011.5999011"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Fgee, E.B., and Alkallas, M.I. (2013, January 20\u201322). E-Government in Libya: Constraints, Potentials and Implementation. Proceedings of the International Conference on Computer Applications Technology (ICCAT), Sousse, Tunisia.","DOI":"10.1109\/ICCAT.2013.6521992"},{"key":"ref_20","unstructured":"Langham, J., Paulsen, N., and Hartel, C. (2017, January 21\u201322). Evaluating Design Effectiveness for Public Sector Services: An Introduction to XE. Proceedings of the 12th European Conference on Innovation and Entrepreneurship (ECIE), Paris, France."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"717","DOI":"10.1016\/j.giq.2007.06.002","article-title":"Assessing eGovernment Systems Success: A Validation of the DeLone and McLean Model of Information Systems Success","volume":"25","author":"Wang","year":"2008","journal-title":"Gov. Inf. Q."},{"key":"ref_22","unstructured":"Scott, M., Delone, W., and Golden, W. (2011, January 9\u201311). IT Quality and Egovernment Net Benefits: A Citizen Perspective. Proceedings of the 19th European Conference on Information Systems, ECIS 2011, Helsinki, Finland."},{"key":"ref_23","unstructured":"Okunola, O.M., and Rowley, J. (2013, January 13\u201314). Dimensions of the User Experience of E-Government Services: The Nigeria Immigration Service Website. Proceedings of the 13th European Conference on EGovernment (ECEG 2013), Como, Italy."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1504\/IJPQM.2017.085848","article-title":"E-Government Services Quality and Citizens\u2019 Satisfaction: A Multi-Criteria Satisfaction Analysis of TAXISnet Information System in Greece","volume":"22","author":"Skordoulis","year":"2017","journal-title":"Int. J. Product. Qual. Manag."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1007\/s12208-009-0030-0","article-title":"Quality Dimensions in the Public Sector: Municipal Services and Citizen\u2019s Perception","volume":"6","author":"Vaughan","year":"2009","journal-title":"Int. Rev. Public Nonprofit Mark."},{"key":"ref_26","unstructured":"Srivastava, S., Teo, T., and Nishant, R. What Is Electronic Service Quality? In Proceedings of the 19th European Conference on Information Systems (ECIS), Helsinki, Finland, 9\u201311 June 2011."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Jing, F., and Wenting, Y. (2014, January 25\u201327). A Study of G2C E-Government Citizen\u2019s Satisfaction: Perspectives of Online Service Quality and Offline Service Quality. Proceedings of the 11th International Conference on Service Systems and Service Management (ICSSSM), Beijing, China.","DOI":"10.1109\/ICSSSM.2014.6943345"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Uthaman, V.S., and Ramankutty, V. (2017, January 7\u20139). E-Governance Service Quality of Common Service Centers: A Review of Existing Models and Key Dimensions. Proceedings of the ICEGOV \u201917: 10th International Conference on Theory and Practice of Electronic Governance, New Delhi, India.","DOI":"10.1145\/3047273.3047280"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/j.1681-4835.2012.tb00382.x","article-title":"E-Government Service Quality: A Qualitative Evaluation in the Case of Saudi Arabia","volume":"54","author":"Alanezi","year":"2012","journal-title":"Electron. J. Inf. Syst. Dev. Ctries."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1504\/IJSTM.2015.073924","article-title":"Electronic Governance Service Quality: A Study in the State of Odisha","volume":"21","author":"Patra","year":"2015","journal-title":"Int. J. Serv. Technol. Manag."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Uthaman, V.S., and Vasanthagopal, R. (2017, January 7\u20139). A Comprehensive Multidimensional Conceptual Model to Assess the E-Governance Service Quality at Common Service Centers in India. Proceedings of the Special Collection on eGovernment Innovations in India-ICEGOV \u201917, New Delhi, India.","DOI":"10.1145\/3055219.3055239"},{"key":"ref_32","first-page":"224","article-title":"Measuring Electronic Government Procurement Success and Testing for the Moderating Effect of Computer Self-Efficacy","volume":"4","author":"Jang","year":"2010","journal-title":"Int. J. Digit. Content Technol. Its Appl."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1108\/17410391211224408","article-title":"E-service Quality Model for Indian Government Portals: Citizens\u2019 Perspective","volume":"25","author":"Bhattacharya","year":"2012","journal-title":"J. Enterp. Inf. Manag."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1007\/s10796-014-9504-7","article-title":"Investigating Success of an E-Government Initiative: Validation of an Integrated IS Success Model","volume":"17","author":"Rana","year":"2015","journal-title":"Inf. Syst. Front."},{"key":"ref_35","unstructured":"Alfadli, I., and Munro, M. (2018, January 25\u201326). Citizen Centered E-Government Services Assessment Framework. Proceedings of the European Conference on Digital Government, Santiago, Spain."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1007\/978-3-642-14799-9_18","article-title":"Success of Government E-Service Delivery: Does Satisfaction Matter?","volume":"Volume 6228","author":"Wimmer","year":"2010","journal-title":"Electronic Government"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Zheng, F., and Lu, Y. (2012, January 18\u201321). Influencing Factors of Public Satisfaction in Local Governments\u2019 Overall Performance Evaluation in China. Proceedings of the Fifth International Conference on Business Intelligence and Financial Engineering, Lanzhou, China.","DOI":"10.1109\/BIFE.2012.110"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Ssemugabi, S., and de Villiers, M.R. (2016, January 26\u201328). Make Your Choice: Dimensionality of an Open Integrated Conceptual Model for Evaluating E-Service Quality, Usability and User Experience (e-SQUUX) of Web-Based Applications. Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists on\u2014SAICSIT \u201916, Johannesburg, South Africa.","DOI":"10.1145\/2987491.2987527"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Arias, M.I., and Ma\u00e7ada, A.C.G. (2018, January 4\u20136). Digital Government for E-Government Service Quality: A Literature Review. Proceedings of the 11th International Conference on Theory and Practice of Electronic Governance\u2014ICEGOV \u201918, Galway, Ireland.","DOI":"10.1145\/3209415.3209422"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Alanezi, M.A., Mahmood, A.K., and Basri, S. (2011, January 25\u201328). Conceptual Model for Measuring E-Government Service Quality. Proceedings of the IEEE Conference on Open Systems, Langkawi, Malaysia.","DOI":"10.1109\/ICOS.2011.6079243"},{"key":"ref_41","unstructured":"Almalki, O., Duan, Y., and Frommholz, I. (2013, January 13\u201314). Developing a Conceptual Framework to Evaluate E-Government Portals\u2019 Success. Proceedings of the 13th European Conference on E-Government, Varese, Italy."},{"key":"ref_42","first-page":"139","article-title":"E-Government Services Evaluation from Citizen Satisfaction Perspective: A Case of Afghanistan","volume":"10","author":"Esichaikul","year":"2016","journal-title":"Transform. Gov. People Process Policy"},{"key":"ref_43","first-page":"20","article-title":"Citizen Centric Assessment Framework for E-Governance Services Quality","volume":"27","author":"Singh","year":"2018","journal-title":"Int. J. Bus. Inf. Syst."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Alias, E.S., Mohd Idris, S.H., Ashaari, N.S., and Kasimin, H. (2011, January 17\u201319). Evaluating E-Government Services in Malaysia Using the EGOVSAT Model. Proceedings of the 2011 International Conference on Electrical Engineering and Informatics, Bandung, Indonesia.","DOI":"10.1109\/ICEEI.2011.6021740"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"43","DOI":"10.4018\/jegr.2013070103","article-title":"Applying Gap Model for Bringing Effectiveness to E-Government Services: A Case of NeGP Deployment in India","volume":"9","author":"Amritesh","year":"2013","journal-title":"Int. J. Electron. Gov. Res."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/j.1681-4835.2016.tb00542.x","article-title":"A Citizen-Centric Framework For Assessing E-Government Effectiveness","volume":"74","author":"Sigwejo","year":"2016","journal-title":"Electron. J. Inf. Syst. Dev. Ctries"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1287\/isre.11.4.342.11872","article-title":"Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model","volume":"11","author":"Venkatesh","year":"2000","journal-title":"Inf. Syst. Res."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Modesto, A.S.C., Figueiredo, R.M.d.C., Ramos, C.S., Santos, L.d.S., Venson, E., and Pedrosa, G.V. (2021). Organizational Strategies for End-User Development\u2014A Systematic Literature Mapping. Informatics, 8.","DOI":"10.3390\/informatics8010015"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/4\/162\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:42:02Z","timestamp":1760136122000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/4\/162"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,24]]},"references-count":48,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["info13040162"],"URL":"https:\/\/doi.org\/10.3390\/info13040162","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,24]]}}}