{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T11:14:58Z","timestamp":1780053298545,"version":"3.54.0"},"reference-count":49,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2020,10,6]],"date-time":"2020-10-06T00:00:00Z","timestamp":1601942400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Effectiveness is a key feature of good governance, as the public sector must make the best use of resources to comply with the needs of the population. Several indicators can be analyzed to evaluate the effectiveness of a service. This study analyzes theoretical references and presents a systematic research of indicators to assess the effectiveness of digital public services in the perspective of the user. First, a literature review was carried out to identify the most common indicators employed to evaluate effectiveness in the public sector; then, the perception of academics and professionals regarding digital government was assessed to analyze the relevance of these indicators. As a result, two groups of indicators were found: technical factors based on service quality and usefulness of the service. This work contributes to enrich the discussion on how to create an effective model to evaluate the effectiveness of public services to guarantee quality standards and comply with the expectations of users.<\/jats:p>","DOI":"10.3390\/info11100472","type":"journal-article","created":{"date-parts":[[2020,10,7]],"date-time":"2020-10-07T09:12:58Z","timestamp":1602061978000},"page":"472","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["A Systematic Review of Indicators for Evaluating the Effectiveness of Digital Public Services"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5573-6830","authenticated-orcid":false,"given":"Glauco Vitor","family":"Pedrosa","sequence":"first","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Federal District, 72444-240 Brasilia, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ricardo A. D.","family":"Kosloski","sequence":"additional","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Federal District, 72444-240 Brasilia, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vitor G. de","family":"Menezes","sequence":"additional","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Federal District, 72444-240 Brasilia, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriela Y.","family":"Iwama","sequence":"additional","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Federal District, 72444-240 Brasilia, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wander C. M. P. da","family":"Silva","sequence":"additional","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Federal District, 72444-240 Brasilia, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8243-7924","authenticated-orcid":false,"given":"Rejane M. da C.","family":"Figueiredo","sequence":"additional","affiliation":[{"name":"Information Technology Research and Application Center (ITRAC), University of Brasilia (UnB), Federal District, 72444-240 Brasilia, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,10,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1108\/10662240910998869","article-title":"E-government adoption in ASEAN: The case of Cambodia","volume":"19","author":"Sang","year":"2009","journal-title":"Internet Res."},{"key":"ref_2","unstructured":"AlKalbani, A., Deng, H., and Kam, B. (2015, January 6\u20139). Organisational security culture and information security compliance for e-government development: The moderating effect of social pressure. Proceedings of the 19th Pacific Asia Conference on Information Systems, Singapore."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1108\/IntR-10-2016-0296","article-title":"Evaluating the performance of e-government in developing countries: A public value perspective","volume":"28","author":"Deng","year":"2018","journal-title":"Internet Res."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"101385","DOI":"10.1016\/j.giq.2019.06.002","article-title":"Defining digital transformation: Results from expert interviews","volume":"36","author":"Mergel","year":"2019","journal-title":"Gov. Inf. Q."},{"key":"ref_5","first-page":"633","article-title":"Understanding e-government research: A perspective from the information and library science field of knowledge","volume":"4","year":"2015","journal-title":"Internet Res."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"131","DOI":"10.3233\/IP-2004-0055","article-title":"Evaluating the progress of e-government development: A critical analysis","volume":"9","author":"Kunstelj","year":"2004","journal-title":"Inf. Polity"},{"key":"ref_7","unstructured":"Kearns, I. (2004). Public Value and E-Government, Institute for Public Policy Research."},{"key":"ref_8","unstructured":"(2020, September 07). European eGovernment Action Plan. Available online: https:\/\/ec.europa.eu\/digital-single-market\/en\/policies\/egovernment."},{"key":"ref_9","first-page":"394","article-title":"Evaluation of regional government portal on the basis of public value concept: Case study from Russian Federation","volume":"232","author":"Golubeva","year":"2007","journal-title":"Proc. ACM Int. Conf."},{"key":"ref_10","unstructured":"Carrara, W. (2007). Value Creation Analysis for Government Transformation Projects, Ministry of Budget, Public Accounts and Civil Service. Technical Report."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1007\/978-3-540-85204-9_8","article-title":"E-government project evaluation: An integrated framework","volume":"5184","author":"Liu","year":"2008","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_12","unstructured":"Harmon, M.M., and Mayer, R.T. (1986). Organization Theory for Public Administration, Little Brown and Company."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.omega.2016.12.007","article-title":"Measuring effectiveness of production in the public sector","volume":"73","author":"Forsund","year":"2017","journal-title":"Omega"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1177\/1094428114562629","article-title":"Bibliometric Methods in Management and Organization","volume":"18","author":"Zupic","year":"2015","journal-title":"Organ. Res. Methods"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2049","DOI":"10.1016\/j.infsof.2013.07.010","article-title":"A systematic review of systematic review process research in software engineering","volume":"55","author":"Kitchenham","year":"2013","journal-title":"Inf. Softw. Technol."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Methley, A.M., Campbell, S., Chew-Graham, C., McNally, R., and Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv. Res., 579.","DOI":"10.1186\/s12913-014-0579-0"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Dieste, O., and Padua, A.G. (2007, January 20\u201321). Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), Madrid, Spain.","DOI":"10.1109\/ESEM.2007.19"},{"key":"ref_18","unstructured":"Davis, F. (1986). A Technology Acceptance Model for Empirically Testing New End-User Information Systems: Theory and Results, Sloan School of Management, Massachusetts Institute of Technology."},{"key":"ref_19","first-page":"114","article-title":"A critical survey of selected government interoperability frameworks","volume":"5","author":"Ray","year":"2011","journal-title":"Transform. Gov. People Process D Policy"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1572","DOI":"10.1007\/s10796-015-9613-y","article-title":"Citizens\u2019 adoption of an electronic government system: Towards a unified view","volume":"19","author":"Rana","year":"2017","journal-title":"Inf. Syst. Front."},{"key":"ref_21","unstructured":"Alotaibi, S., and Roussinov, D. (2017, January 12). User acceptance of m-government services in Saudi Arabia: An SEM approach. Proceedings of the 17th European Conference on Digital Government (ECDG), Lisbon, Portugal."},{"key":"ref_22","unstructured":"Almalki, O., Duan, Y., and Frommholz, I. (2013, January 13\u201314). Developing a Conceptual Framework to Evaluate E-Government Portals\u2019 Success. Proceedings of the 13th European Conference on e-Government, Como, Italy."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/j.1681-4835.2016.tb00542.x","article-title":"A Citizen-Centric Framework For Assessing E-Government Effectiveness","volume":"74","author":"Sigwejo","year":"2016","journal-title":"Electron. J. Inf. Syst. Dev. Ctries."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1108\/09593841211232712","article-title":"Predictors for e-government adoption in Jordan: Deployment of an empirical evaluation based on a citizen-centric approach","volume":"25","author":"Alomari","year":"2012","journal-title":"Inf. Technol. People"},{"key":"ref_25","unstructured":"Delone, W., and McLean, E. (2002, January 10). Information Systems Success Revisited. Proceedings of the 35th Annual Hawaii International Conference on System Sciences, Big Island, HI, USA."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.dss.2005.05.009","article-title":"Value-based adoption of mobile internet: An empirical investigation","volume":"43","author":"Kim","year":"2007","journal-title":"Decis. Support Syst."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"107","DOI":"10.3401\/poms.1070.0006","article-title":"An exploratory study of the impact of e-service process on online customer satisfaction","volume":"17","author":"Ba","year":"2008","journal-title":"Prod. Oper. Manag."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Uthaman, V.S., and Ramankutty, V. (2017, January 7\u20139). E-Governance Service Quality of Common Service Centers: A Review of Existing Models and Key Dimensions. Proceedings of the 10th International Conference on Theory and Practice of Electronic Governance, New Delhi, India.","DOI":"10.1145\/3047273.3047280"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Wong, M.S., and Jackson, S. (2017, January 27\u201329). User satisfaction evaluation of Malaysian e-government education services. Proceedings of the 2017 International Conference on Engineering, Technology and Innovation (ICE\/ITMC), Funchal, Portugal.","DOI":"10.1109\/ICE.2017.8279931"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1108\/09604520310506513","article-title":"E-service quality and the public sector","volume":"13","author":"Buckley","year":"2003","journal-title":"Manag. Serv. Qual. Int. J."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1108\/ICS-09-2016-0074","article-title":"Understanding of e-service security dimensions and its effect on quality and intention to use","volume":"25","author":"Taherdoost","year":"2017","journal-title":"Inf. Comput. Secur."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Shajari, M., and Ismail, Z. (2011, January 25\u201327). Key Factors Influencing the Adoption of E-government in Iran. Proceedings of the 2011 Fourth International Conference on Information and Computing, Phuket Island, Thailand.","DOI":"10.1109\/ICIC.2011.72"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Roy, M.C., Chartier, A., Cr\u00eate, J., and Poulin, D. (2015). Factors influencing e-government use in non-urban areas. Electron. Commer. Res., 15.","DOI":"10.1007\/s10660-015-9193-4"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1111\/j.1467-8500.2009.00643.x","article-title":"The participation divide? Political participation, trust in government, and e-government in Australia and New Zealand","volume":"68","author":"Goldfinch","year":"2009","journal-title":"Aust. J. Public Adm."},{"key":"ref_35","first-page":"1","article-title":"Moderating Factors in Adopting Local e-Government in Spain","volume":"38","author":"Rufin","year":"2011","journal-title":"Local Gov. Stud."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1108\/09604520510575236","article-title":"Towards quality e-service in the public sector: The evolution of web sites in the local public service sector","volume":"15","author":"Ancarani","year":"2005","journal-title":"Manag. Serv. Qual. Int. J."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1016\/j.elerap.2011.12.004","article-title":"Transforming e-services evaluation data into business analytics using value models","volume":"11","author":"Loukis","year":"2012","journal-title":"Electron. Commer. Res. Appl."},{"key":"ref_38","unstructured":"European Commission (2001). Web-Based Survey on Electronic Public Services, European Commission. Cap Gemini and Ernst and Young Report."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"65","DOI":"10.7146\/hjlcb.v20i38.25905","article-title":"What Do People Expect from Public Services? Requests in Public Service Encounters","volume":"20","year":"2017","journal-title":"HERMES-Lang. Commun. Bus."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1016\/j.giq.2010.03.001","article-title":"Using ICTs to create a culture of transparency: E-government and social media as openness and anti-corruption tools for societies","volume":"27","author":"Bertot","year":"2010","journal-title":"Gov. Inf. Q."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1016\/j.csi.2018.01.003","article-title":"Support resource based on standards for accessible e-Government transactional services","volume":"58","author":"Moreno","year":"2018","journal-title":"Comput. Stand. Interfaces"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Ylinen, M., and Pekkola, S. (2019, January 8\u201311). A Process Model for Public Sector It Management to Answer the Needs of Digital Transformation. Proceedings of the 52nd Hawaii International Conference on System Sciences, Maui, HI, USA.","DOI":"10.24251\/HICSS.2019.748"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"628","DOI":"10.1108\/FS-05-2017-0013","article-title":"Factors influencing perceptions on corruption in public service delivery via e-government platform","volume":"19","author":"Saxena","year":"2017","journal-title":"Foresight"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"360","DOI":"10.1590\/01047760201824042570","article-title":"Public Interest versus Forests","volume":"24","author":"Birben","year":"2018","journal-title":"CERNE"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1108\/JOSM-01-2017-0020","article-title":"Toward a public service management: Past, present, and future directions","volume":"28","author":"Hodgkinson","year":"2017","journal-title":"J. Serv. Manag."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.giq.2019.01.001","article-title":"The public value of E-Government\u2014A literature review","volume":"36","author":"Twizeyimana","year":"2019","journal-title":"Gov. Inf. Q."},{"key":"ref_47","first-page":"1066","article-title":"An analysis of the e-service literature: Towards a research agenda","volume":"16","author":"Jennifer","year":"2006","journal-title":"Internet Res."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Nardi, B.A. (1997). The use of ethnographic methods in design and evaluation. Handbook of Human-Computer Interaction, North-Holland.","DOI":"10.1016\/B978-044481862-1\/50081-9"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1016\/j.giq.2011.04.005","article-title":"Critical factors for evaluating the public value of e-government in Sri Lanka","volume":"29","author":"Karunasena","year":"2012","journal-title":"Gov. Inf. Q."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/10\/472\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:16:58Z","timestamp":1760177818000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/10\/472"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,6]]},"references-count":49,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2020,10]]}},"alternative-id":["info11100472"],"URL":"https:\/\/doi.org\/10.3390\/info11100472","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,6]]}}}