{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T04:21:08Z","timestamp":1778646068868,"version":"3.51.4"},"reference-count":140,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T00:00:00Z","timestamp":1725494400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Digital"],"abstract":"<jats:p>This systematic literature review aimed to assess the methodological quality of user-centered usability evaluation of digital applications to promote citizens\u2019 engagement and participation in public governance by (i) systematizing their purposes; (ii) analyzing the evaluation procedures, methods, and instruments that were used; (iii) determining their conformance with recommended usability evaluation good practices; and (iv) identifying the implications of the reported results for future developments. An electronic search was conducted on Web of Science, Scopus, and IEEE Xplore databases, and after a screening procedure considering predefined eligibility criteria, 34 studies were reviewed. These studies performed user-centered usability evaluation of digital applications related to (i) participatory reporting of urban issues, (ii) environmental sustainability, (iii) civic participation, (iv) urban planning, (v) promotion of democratic values, (vi) electronic voting, and (vii) chatbots. In terms of the methodological quality of the included studies, the results suggest that there is a high heterogeneity of the user-centered usability evaluation. Therefore, there is a need for recommendations to support user-centered usability evaluations of digital applications to promote citizens\u2019 engagement and participation in public governance to improve the planning and conduction of future research.<\/jats:p>","DOI":"10.3390\/digital4030038","type":"journal-article","created":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T08:36:08Z","timestamp":1725525368000},"page":"740-761","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Methodological Quality of User-Centered Usability Evaluation of Digital Applications to Promote Citizens\u2019 Engagement and Participation in Public Governance: A Systematic Literature Review"],"prefix":"10.3390","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3207-3445","authenticated-orcid":false,"given":"Rute","family":"Bastardo","sequence":"first","affiliation":[{"name":"UNIDCOM, Science and Technology School, University of Tr\u00e1s-os-Montes and Alto Douro, Quinta de Prado, 5001-801 Vila Real, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9042-2730","authenticated-orcid":false,"given":"Jo\u00e3o","family":"Pav\u00e3o","sequence":"additional","affiliation":[{"name":"INESC-TEC, Science and Technology School, University of Tr\u00e1s-os-Montes and Alto Douro, Quinta de Prado, 5001-801 Vila Real, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3801-7249","authenticated-orcid":false,"given":"Nelson Pacheco","family":"Rocha","sequence":"additional","affiliation":[{"name":"IEETA-Institute of Electronics and Informatics Engineering of Aveiro, Department of Medical Sciences, University of Aveiro, 3810-193 Aveiro, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2024,9,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1111\/0033-3352.00196","article-title":"The evolution of e-government among municipalities: Rhetoric or reality?","volume":"62","author":"Moon","year":"2002","journal-title":"Public Adm. Rev."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"95","DOI":"10.3233\/IP-170065","article-title":"Smart governance: Opportunities for technologically-mediated citizen co-production","volume":"23","author":"Webster","year":"2018","journal-title":"Inf. Polity"},{"key":"ref_3","unstructured":"Barcevi\u010dius, E., Cibait\u0117, G., Codagnone, C., Gineikyt\u0117, V., Klimavi\u010di\u016bt\u0117, L., Liva, G., Matulevi\u010d, L., Misuraca, G., and Vanini, I. (2019). Exploring Digital Government Transformation in the EU, Publications Office of the European Union."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"104480","DOI":"10.1016\/j.cities.2023.104480","article-title":"Co-production as a strategy for enhanced equal representation in public service delivery: The case of Rotterdam","volume":"141","author":"Cornips","year":"2023","journal-title":"Cities"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1080\/19460171.2022.2153071","article-title":"Policy translations of citizen participation and new public governance: The case of school governing bodies","volume":"17","author":"Gerrard","year":"2023","journal-title":"Crit. Policy Stud."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"905","DOI":"10.1080\/07352166.2022.2094799","article-title":"A citizen participation model for co-creation of public value in a smart city","volume":"46","author":"Cao","year":"2024","journal-title":"J. Urban Aff."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1016\/j.giq.2005.02.001","article-title":"E-government success factors: Mapping practical tools to theoretical foundations","volume":"22","author":"Pardo","year":"2005","journal-title":"Gov. Inf. Q."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Rav\u0161elj, D., Umek, L., Todorovski, L., and Aristovnik, A. (2022). A review of digital era governance research in the first two decades: A bibliometric study. Future Internet, 14.","DOI":"10.3390\/fi14050126"},{"key":"ref_9","unstructured":"(2018). Resolution Adopted by the General Assembly on 26 November 2018 (A\/73\/L.20 and A\/73\/L.20\/Add.1), United Nations."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/j.giq.2015.07.001","article-title":"Digital government evolution: From transformation to contextualization","volume":"32","author":"Janowski","year":"2015","journal-title":"Gov. Inf. Q."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Charalabidis, Y., Sarantis, D., and Askounis, D. (2009). Knowledge-driven project management in government transformation. Handbook of Research on ICT-Enabled Transformational Government: A Global Perspective, IGI Global.","DOI":"10.4018\/978-1-60566-390-6.ch012"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/S0740-624X(01)00066-1","article-title":"Developing fully functional E-government: A four stage model","volume":"18","author":"Layne","year":"2001","journal-title":"Gov. Inf. Q."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.giq.2003.11.004","article-title":"A two-stage model of e-government growth: Theories and empirical evidence for US cities","volume":"21","author":"Reddick","year":"2004","journal-title":"Gov. Inf. Q."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1016\/j.infsof.2012.07.007","article-title":"The maturity of maturity model research: A systematic mapping study","volume":"54","author":"Wendler","year":"2012","journal-title":"Inf. Softw. Technol."},{"key":"ref_15","first-page":"71","article-title":"E-government maturity models: A comparative study","volume":"5","author":"Cheikhi","year":"2014","journal-title":"Int. J. Softw. Eng. Appl."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Joshi, P.R., and Islam, S. (2018). E-government maturity model for sustainable E-government services from the perspective of developing countries. Sustainability, 10.","DOI":"10.3390\/su10061882"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"3074","DOI":"10.1109\/ACCESS.2021.3137799","article-title":"Assessing transparency in eGovernment electronic processes","volume":"10","author":"Hochstetter","year":"2021","journal-title":"IEEE Access"},{"key":"ref_18","first-page":"133","article-title":"UX in E-government Services for Citizens: A Systematic Literature Review","volume":"18","author":"Aldrees","year":"2023","journal-title":"J. User Exp."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1016\/j.giq.2019.02.005","article-title":"From conventional governance to e-democracy: Tracing the evolution of e-governance research trends using network analysis tools","volume":"36","author":"Bindu","year":"2019","journal-title":"Gov. Inf. Q."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"741","DOI":"10.1177\/10659129221100260","article-title":"Participatory governance and responsiveness: Do motivational interventions increase engagement with citizen input?","volume":"76","author":"Wittels","year":"2023","journal-title":"Political Res. Q."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Vrabie, C. (2023). E-Government 3.0: An AI model to use for enhanced local democracies. Sustainability, 15.","DOI":"10.3390\/su15129572"},{"key":"ref_22","first-page":"1257","article-title":"A review in E-government service quality measurement","volume":"10","author":"Maan","year":"2018","journal-title":"Indones. J. Electr. Eng. Comput. Sci."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"100950","DOI":"10.1016\/j.infoecopol.2021.100950","article-title":"Are emerging technologies helping win the fight against corruption? A review of the state of evidence","volume":"57","author":"Adam","year":"2021","journal-title":"Inf. Econ. Policy"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"101485","DOI":"10.1016\/j.giq.2020.101485","article-title":"Citizens\u2019 continuous use of eGovernment services: The role of self-efficacy, outcome expectations and satisfaction","volume":"37","author":"Alruwaie","year":"2020","journal-title":"Gov. Inf. Q."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1017\/S0007123419000322","article-title":"Escaping the disengagement dilemma: Two field experiments on motivating citizens to report on public services","volume":"51","author":"Buntaine","year":"2021","journal-title":"Br. J. Political Sci."},{"key":"ref_26","first-page":"94","article-title":"Exploring the smart future of participation: Community, inclusivity, and people with disabilities","volume":"10","author":"Bricout","year":"2021","journal-title":"Int. J. E-Plan. Res. (IJEPR)"},{"key":"ref_27","first-page":"211","article-title":"Are satisfied citizens willing to participate more? An analysis of citizens\u2019 life satisfaction in the public service domain and public participation","volume":"27","author":"Kim","year":"2022","journal-title":"Int. Rev. Public Adm."},{"key":"ref_28","unstructured":"(2019). Human-Centred Design for Interactive Systems (Standard No. ISO 9241-210)."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1994). Usability Engineering, Morgan Kaufmann.","DOI":"10.1016\/B978-0-08-052029-2.50009-7"},{"key":"ref_30","unstructured":"(1999). Guidance on Usability (Standard No. ISO 9241-11)."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1145\/953460.953488","article-title":"Understanding usability in mobile commerce","volume":"46","author":"Venkatesh","year":"2003","journal-title":"Commun. ACM"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Srivastava, P., and Mostafavi, A. (2018). Challenges and opportunities of crowdsourcing and participatory planning in developing infrastructure systems of smart cities. Infrastructures, 3.","DOI":"10.3390\/infrastructures3040051"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Hyv\u00e4rinen, T., Kaikkonen, A., and Hiltunen, M. (2005, January 19\u201322). Placing links in mobile banking application. Proceedings of the 7th International Conference on Human Computer Interaction with Mobile Devices & Services, Salzburg, Austria.","DOI":"10.1145\/1085777.1085788"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1177\/0972150915610702","article-title":"Service quality parameters for mobile government services in India","volume":"17","author":"Chanana","year":"2016","journal-title":"Glob. Bus. Rev."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1108\/JEIM-01-2016-0004","article-title":"Exploring mobile government from the service quality perspective","volume":"30","author":"Ahmad","year":"2017","journal-title":"J. Enterp. Inf. Manag."},{"key":"ref_36","first-page":"476","article-title":"Prioritization of citizens\u2019 preferences for using mobile government services: The analytic hierarchy process (AHP) approach","volume":"11","author":"Saadi","year":"2017","journal-title":"Transform. Gov. People Process Policy"},{"key":"ref_37","first-page":"31","article-title":"Review of performance assessment frameworks of e-government projects","volume":"14","author":"Singh","year":"2020","journal-title":"Transform. Gov. People Process Policy"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"e1026","DOI":"10.7717\/peerj-cs.1026","article-title":"Exploration of the usability quality attributes of mobile government services: A literature review","volume":"8","author":"Desmal","year":"2022","journal-title":"PeerJ Comput. Sci."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"197","DOI":"10.29379\/jedem.v14i2.717","article-title":"A Review of E-Government Assessment Frameworks: E-Readiness, Adoption, Citizen Engagement and Quality: E-Readiness, Adoption, Citizen Engagement and Quality","volume":"14","author":"Sheoran","year":"2022","journal-title":"JeDEM-Ejournal Edemocracy Open Gov."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/j.giq.2017.01.003","article-title":"A usability study of the obamacare website: Evaluation and recommendations","volume":"34","author":"Venkatesh","year":"2017","journal-title":"Gov. Inf. Q."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"S106","DOI":"10.1016\/j.giq.2016.04.001","article-title":"Deploying governance networks for societal challenges","volume":"35","author":"Ojo","year":"2018","journal-title":"Gov. Inf. Q."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Ababneh, R., and Alrefaie, L. (2022). Evaluating the quality of public administration institutes\u2019 websites in the Arab world. Global Knowledge, Memory and Communication, Emerald.","DOI":"10.1108\/GKMC-12-2021-0208"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1007\/s10209-021-00800-8","article-title":"Assessing the usability, accessibility, and mobile readiness of e-government websites: A case study in India","volume":"21","author":"Agrawal","year":"2022","journal-title":"Univers. Access Inf. Soc."},{"key":"ref_44","first-page":"1","article-title":"The impact of usability on e-government usage in the Peruvian context","volume":"23","year":"2022","journal-title":"Issues Inf. Syst."},{"key":"ref_45","first-page":"59","article-title":"Research on the Usability Test of Interface Design in e-Government-Focused on Qingdao e-Government Website","volume":"36","author":"Liu","year":"2023","journal-title":"Arch. Des. Res."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Qonita, F., Budiman, M.F., Sari, V.M., and Limantara, N. (2023, January 11\u201312). Analysis of User Experience on The Government Application of Indonesian Higher Education Institutional Information Systems Using Usability Method. Proceedings of the 4th International Conference on Innovative Trends in Information Technology (ICITIIT), Kerala, India.","DOI":"10.1109\/ICITIIT57246.2023.10068674"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1","DOI":"10.4018\/IJEGR.322307","article-title":"Usability and Accessibility of Open Government Data Portals of Countries Worldwide: An Application of TOPSIS and Entropy Weight Method","volume":"19","author":"Sheoran","year":"2023","journal-title":"Int. J. Electron. Gov. Res. (IJEGR)"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"e1074","DOI":"10.7717\/peerj-cs.1074","article-title":"A user satisfaction model for mobile government services: A literature review","volume":"8","author":"Desmal","year":"2022","journal-title":"PeerJ Comput. Sci."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Menezes, V.G.D., Pedrosa, G.V., Silva, M.P.D., and Figueiredo, R.M.D. (2022). Evaluation of public services considering the expectations of users-A systematic literature review. Information, 13.","DOI":"10.3390\/info13040162"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Lyzara, R., Purwandari, B., Zulfikar, M.F., Santoso, H.B., and Solichah, I. (2019, January 10\u201312). E-government usability evaluation: Insights from a systematic literature review. Proceedings of the 2nd International Conference on Software Engineering and Information Management, Bali, Indonesia.","DOI":"10.1145\/3305160.3305178"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Monz\u00f3n, F.H., Tupia, M., and Bruzza, M. (2020, January 29\u201331). Security versus usability in e-government: Insights from the literature. Proceedings of the MICRADS 2020, Quito, Ecuador.","DOI":"10.1007\/978-981-15-4875-8_3"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Alshamsi, A., Williams, N., and Andras, P. (2016, January 11\u201315). The Trade-off between Usability and Security in the Context of eGovernment: A Mapping Study. Proceedings of the 30th International BCS Human Computer Interaction Conference, Fern Barrow, UK.","DOI":"10.14236\/ewic\/HCI2016.36"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Yerlikaya, Z., and Onay Durdu, P. (2017, January 9\u201314). Usability of university websites: A systematic review. Proceedings of the 11th International Conference of Universal Access in Human-Computer Interaction, Vancouver, BC, Canada.","DOI":"10.1007\/978-3-319-58706-6_22"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Cisneros, D., Huam\u00e1n Monz\u00f3n, F., and Paz, F. (2021, January 24\u201329). Accessibility evaluation of E-Government web applications: A systematic review. Proceedings of the International Conference on Human-Computer Interaction, Virtual Event.","DOI":"10.1007\/978-3-030-78224-5_15"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Zhang, J., Chang, D., and Zhang, Z. (2021, January 13\u201316). Review on the Application of Eye-tracking Technology in Usability Evaluation of E-government Apps. Proceedings of the IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), Singapore.","DOI":"10.1109\/IEEM50564.2021.9672792"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1016\/j.ijsu.2010.02.007","article-title":"Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement","volume":"8","author":"Moher","year":"2010","journal-title":"Int. J. Surg."},{"key":"ref_57","unstructured":"Booth, A., Sutton, A., Clowes, M., and Martyn-St James, M. (2021). Systematic Approaches to a Successful Literature Review, Sage."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"e14829","DOI":"10.2196\/14829","article-title":"A scale to assess the methodological quality of studies assessing usability of electronic health products and services: Delphi study followed by validity and reliability testing","volume":"21","author":"Silva","year":"2019","journal-title":"J. Med. Internet Res."},{"key":"ref_59","first-page":"139","article-title":"Participation to public e-service development: A systematic literature review","volume":"8","author":"Madyatmadja","year":"2016","journal-title":"J. Telecommun. Electron. Comput. Eng. (JTEC)"},{"key":"ref_60","first-page":"141","article-title":"Sistemas de l\u00f3gica difusa para la evaluaci\u00f3n de usabilidad de sitios web de gobierno electr\u00f3nico: Una revisi\u00f3n sistem\u00e1tica","volume":"E41","author":"Delgado","year":"2021","journal-title":"Rev. Ib\u00e9rica De Sist. E Tecnol. De Informa\u00e7\u00e3o"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Mahou-Lago, X.M., and Varela-\u00c1lvarez, E.J. (2016). Innovation and opportunities for citizen participation in Spanish smart cities. Smarter as the New Urban Agenda, Springer.","DOI":"10.1007\/978-3-319-17620-8_20"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Li, E., Chen, Q., Zhang, X., and Zhang, C. (2023). Digital government development, local governments\u2019 attention distribution and enterprise total factor productivity: Evidence from China. Sustainability, 15.","DOI":"10.3390\/su15032472"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"101382","DOI":"10.1016\/j.softx.2023.101382","article-title":"Digital government: Mobile applications and their impact on access to public information","volume":"22","author":"Castilla","year":"2023","journal-title":"SoftwareX"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1016\/j.giq.2013.05.007","article-title":"Bringing in the users: The role for usability evaluation in eGovernment","volume":"30","year":"2013","journal-title":"Gov. Inf. Q."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Sachs, M., and Schossb\u00f6ck, J. (2019, January 2\u20134). Acceptance of tools for electronic citizen participation. Proceedings of the International Conference on Electronic Participation, San Benedetto Del Tronto, Italy.","DOI":"10.1007\/978-3-030-27397-2_4"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1108\/IJPSM-09-2020-0227","article-title":"Perceived usability of local government (LG) financial statements by local councillors: Comparative study of Finland and Germany","volume":"34","author":"Haustein","year":"2021","journal-title":"Int. J. Public Sect. Manag."},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Rey, W.P. (2023, January 27\u201329). Assessing MABIS Mobile App Based on People at the Center of Mobile Application Development (PACMAD) Usability Model: Empirical Investigation. Proceedings of the 13th International Conference on Software Technology and Engineering (ICSTE), Osaka, Japan.","DOI":"10.1109\/ICSTE61649.2023.00014"},{"key":"ref_68","doi-asserted-by":"crossref","first-page":"204","DOI":"10.1504\/IJBCRM.2023.131873","article-title":"Fuzzy expert system and information systems auditing: An approach for risk assessment in audit pre-planning","volume":"13","author":"Ouaadi","year":"2023","journal-title":"Int. J. Bus. Contin. Risk Manag."},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"4164","DOI":"10.1080\/17538947.2023.2264827","article-title":"City digital twins for urban resilience","volume":"16","author":"Therias","year":"2023","journal-title":"Int. J. Digit. Earth"},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"691","DOI":"10.3897\/jucs.97112","article-title":"Evaluation of a Legally Binding Smart-Contract Language for Blockchain Applications","volume":"29","author":"Dwivedi","year":"2023","journal-title":"J. Univers. Comput. Sci."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"842","DOI":"10.1108\/14684520810923962","article-title":"User evaluations of tax filing web sites: A comparative study of South Korea and Turkey","volume":"32","author":"Kirlidog","year":"2008","journal-title":"Online Inf. Rev."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1300\/J179v06n03_06","article-title":"Dubai e-government: An evaluation of G2B websites","volume":"6","author":"Awan","year":"2008","journal-title":"J. Internet Commer."},{"key":"ref_73","first-page":"265","article-title":"Evaluating the usability of developing countries\u2019 e-government sites: A user perspective","volume":"6","author":"Choudrie","year":"2009","journal-title":"Electron. Gov. Int. J."},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Yuan, L., and Zhongling, L. (2010, January 26\u201329). Experimental evaluation on government portal website\u2019s usability to 11 government websites of Zhejiang province. Proceedings of the 2nd International Conference on Information Science and Engineering, Kerbala, Iraq.","DOI":"10.1109\/ICISE.2010.5688953"},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Israel, D., and Tiwari, R. (2011, January 26\u201328). Empirical study of factors influencing acceptance of e-government services in India. Proceedings of the 5th International Conference on Theory and Practice of Electronic Governance, Tallinn, Estonia.","DOI":"10.1145\/2072069.2072093"},{"key":"ref_76","doi-asserted-by":"crossref","unstructured":"Gonz\u00e1lez Mart\u00ednez, S., Luna-Reyes, L.F., Luna, D.E., Gil-Garcia, J.R., and Sandoval-Almaz\u00e1n, R. (2011, January 12\u201315). Comparing usability of government web portals during governor change of terms. Proceedings of the 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times, College Park, MD, USA.","DOI":"10.1145\/2037556.2037609"},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1108\/ARLA-08-2013-0118","article-title":"An evaluation of citizen service web portals in Latin America","volume":"28","year":"2015","journal-title":"Acad. Rev. Latinoam. De Adm."},{"key":"ref_78","doi-asserted-by":"crossref","unstructured":"Baguma, R. (2018, January 4\u20136). Usability evaluation of the etax portal for Uganda. Proceedings of the 11th International Conference on Theory and Practice of Electronic Governance, Galway, Ireland.","DOI":"10.1145\/3209415.3209470"},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1016\/j.giq.2018.12.005","article-title":"User experience of government documents: A framework for informing design decisions","volume":"36","author":"Madariaga","year":"2019","journal-title":"Gov. Inf. Q."},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"169","DOI":"10.3991\/ijim.v13i10.11286","article-title":"An Interview Report on Users\u2019 Perception about the Usability Performance of a Mobile E-Government Application","volume":"13","author":"Hussain","year":"2019","journal-title":"Int. J. Interact. Mob. Technol."},{"key":"ref_81","doi-asserted-by":"crossref","unstructured":"Patsoulis, G., Promikyridis, R., and Tambouris, E. (2021, January 26\u201328). Integration of chatbots with Knowledge Graphs in eGovernment: The case of Getting a Passport. Proceedings of the 25th Pan-Hellenic Conference on Informatics, Volos, Greece.","DOI":"10.1145\/3503823.3503901"},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1007\/s12518-015-0160-5","article-title":"Towards Co-PPGIS-a collaborative public participatory GIS-based measure for transparency in housing schemes: A case of Lahore, Pakistan","volume":"8","author":"Butt","year":"2016","journal-title":"Appl. Geomat."},{"key":"ref_83","doi-asserted-by":"crossref","unstructured":"Birghan, F., Hettenhausen, R., Meschede, C., and Siebenlist, T. (2019, January 18\u201320). Informing Citizens via Council Information Systems. Proceedings of the 20th Annual International Conference on Digital Government Research, Dubai, United Arab Emirates.","DOI":"10.1145\/3325112.3325220"},{"key":"ref_84","unstructured":"Sofyan, Z. (2019, January 18\u201320). User requirements elicitation in web-based Participatory Geographic Information System interface design. Proceedings of the 9th Annual International Conference on Sciences & Engineering, Banda Aceh, Indonesia."},{"key":"ref_85","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1016\/j.ijhcs.2010.06.003","article-title":"An adaptive e-questionnaire for measuring user perceived portal quality","volume":"68","author":"Magoutas","year":"2010","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_86","first-page":"343","article-title":"An AHP method for evaluating usability of electronic government portals","volume":"8","author":"Byun","year":"2011","journal-title":"Electron. Gov. Int. J."},{"key":"ref_87","doi-asserted-by":"crossref","unstructured":"Faisal, M., Al-Qouz, H., and Husain, F. (2016, January 8\u201310). A direct method for measuring user experience in E-government portals. Proceedings of the 15th International Conference on Information Technology Based Higher Education and Training (ITHET), Istanbul, Turkey.","DOI":"10.1109\/ITHET.2016.7760706"},{"key":"ref_88","doi-asserted-by":"crossref","unstructured":"Schneider, H., Frison, K., Wagner, J., and Butz, A. (2016, January 4\u20138). CrowdUX: A case for using widespread and lightweight tools in the quest for UX. Proceedings of the 2016 ACM Conference on Designing Interactive Systems, Brisbane, Australia.","DOI":"10.1145\/2901790.2901814"},{"key":"ref_89","doi-asserted-by":"crossref","unstructured":"Burian, M.F., Fauvety, P., Aquino, N., Gonz\u00e1lez, M., Romero, D., Cernuzzi, L., Paniagua, J., and Chen\u00fa-Abente, R. (2020, January 19\u201323). Design of SmartMoving, an application for pedestrians with reduced mobility. Proceedings of the XLVI Latin American Computing Conference (CLEI), Loja, Ecuador.","DOI":"10.1109\/CLEI52000.2020.00049"},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"2989","DOI":"10.1007\/s11606-020-06506-6","article-title":"Development and pilot testing of a patient-centered web-based reproductive decision support tool for primary care","volume":"36","author":"Callegari","year":"2021","journal-title":"J. Gen. Intern. Med."},{"key":"ref_91","doi-asserted-by":"crossref","first-page":"e2","DOI":"10.4108\/ew.v9i39.1191","article-title":"Sustainable smart parking solution in a campus environment","volume":"9","author":"Remelhe","year":"2022","journal-title":"EAI Endorsed Trans. Energy Web"},{"key":"ref_92","doi-asserted-by":"crossref","first-page":"925","DOI":"10.1109\/TMC.2022.3213125","article-title":"Towards accessible shared autonomous electric mobility with dynamic deadlines","volume":"23","author":"Wang","year":"2022","journal-title":"IEEE Trans. Mob. Comput."},{"key":"ref_93","doi-asserted-by":"crossref","first-page":"4365","DOI":"10.1080\/10447318.2023.2212228","article-title":"From parking meters to vending machines: A study of usability issues in self-service technologies","volume":"40","author":"Henderson","year":"2023","journal-title":"Int. J. Hum. -Comput. Interact."},{"key":"ref_94","doi-asserted-by":"crossref","unstructured":"Armand, T.P.T., Mozumder, M.A.I., Ali, S., Amaechi, A.O., and Kim, H.C. (2023). Developing a Low-Cost IoT-Based Remote Cardiovascular Patient Monitoring System in Cameroon. Healthcare, 11.","DOI":"10.3390\/healthcare11020199"},{"key":"ref_95","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1007\/s41347-022-00298-8","article-title":"Looking through the lenses of a patient: An empirical study on the factors affecting patients\u2019 intention to use avatar-assisted therapy","volume":"8","author":"Moriuchi","year":"2023","journal-title":"J. Technol. Behav. Sci."},{"key":"ref_96","doi-asserted-by":"crossref","first-page":"e46909","DOI":"10.2196\/46909","article-title":"Clinician and Patient Perspectives on the Use of Passive Mobile Monitoring and Self-Tracking for Patients with Serious Mental Illness: User-Centered Approach","volume":"10","author":"Medich","year":"2023","journal-title":"JMIR Hum. Factors"},{"key":"ref_97","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.iotcps.2023.09.001","article-title":"Internet of things enabled parking management system using long range wide area network for smart city","volume":"4","author":"Jabbar","year":"2024","journal-title":"Internet Things Cyber-Phys. Syst."},{"key":"ref_98","doi-asserted-by":"crossref","first-page":"267","DOI":"10.29379\/jedem.v6i3.247","article-title":"Online participation in higher education decision-making","volume":"6","author":"Bohman","year":"2014","journal-title":"JeDEM-Ejournal Edemocracy Open Gov."},{"key":"ref_99","doi-asserted-by":"crossref","first-page":"3719","DOI":"10.1007\/s11069-022-05540-3","article-title":"CrowdBIG: Crowd-based system for information gathering from the earthquake environment","volume":"114","author":"Bahadori","year":"2022","journal-title":"Nat. Hazards"},{"key":"ref_100","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1007\/s10209-011-0253-9","article-title":"An evaluation of web-based voting usability and accessibility","volume":"11","author":"Fuglerud","year":"2012","journal-title":"Univ. Access Inf. Soc."},{"key":"ref_101","doi-asserted-by":"crossref","first-page":"2025","DOI":"10.1007\/s00779-014-0758-3","article-title":"Providing real-time assistance in disaster relief by leveraging crowdsourcing power","volume":"18","author":"Yang","year":"2014","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_102","doi-asserted-by":"crossref","unstructured":"Brito, J., Vieira, V., and Duran, A. (2015, January 13\u201315). Towards a framework for gamification design on crowdsourcing systems: The GAME approach. Proceedings of the 12th International Conference on Information Technology-New Generations, Las Vegas, NV, USA.","DOI":"10.1109\/ITNG.2015.78"},{"key":"ref_103","unstructured":"Pistolato, A.C., and Brand\u00e3o, W.C. (2016, January 28\u201330). ConnectCity: A collaborative e-government approach to report city incidents. Proceedings of the 15th International Conference WWW\/Internet 2016, Mannheim, Germany."},{"key":"ref_104","doi-asserted-by":"crossref","first-page":"673","DOI":"10.1007\/s10111-016-0383-1","article-title":"Identification of UX dimensions for incident reporting systems with mobile applications in urban contexts: A longitudinal study","volume":"18","author":"Winckler","year":"2016","journal-title":"Cogn. Technol. Work"},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1080\/15230406.2016.1195285","article-title":"Engaging indigenous people as geo-crowdsourcing sensors for ecotourism mapping via mobile data collection: A case study of the Royal Belum State Park","volume":"44","author":"Idris","year":"2017","journal-title":"Cartogr. Geogr. Inf. Sci."},{"key":"ref_106","doi-asserted-by":"crossref","unstructured":"Bousios, A., Gavalas, D., and Lambrinos, L. (2017, January 3\u20136). CityCare: Crowdsourcing daily life issue reports in smart cities. Proceedings of the 2017 IEEE Symposium on Computers and Communications (ISCC), Heraklion, Greece.","DOI":"10.1109\/ISCC.2017.8024540"},{"key":"ref_107","doi-asserted-by":"crossref","first-page":"214","DOI":"10.14207\/ejsd.2019.v8n4p214","article-title":"Developing a collective awareness platform for urban sustainability challenges: Case study of the POWER project","volume":"8","author":"Koroleva","year":"2019","journal-title":"Eur. J. Sustain. Dev."},{"key":"ref_108","doi-asserted-by":"crossref","unstructured":"Falc\u00e3o, A.G.R., Wanderley, P.F., da Silva Leite, T.H., de Souza Baptista, C., de Queiroz, J.E.R., de Oliveira, M.G., and Rocha, J.H. (2019, January 26\u201329). Crowdsourcing urban issues in smart cities: A usability assessment of the Crowd4City system. Proceedings of the 8th International Conference on Electronic Government and the Information Systems Perspective, Linz, Austria.","DOI":"10.1007\/978-3-030-27523-5_11"},{"key":"ref_109","doi-asserted-by":"crossref","unstructured":"Hasim, W., Wibirama, S., and Nugroho, H.A. (2019, January 24\u201325). Redesign of E-participation using user-centered design approach for improving user experience. Proceedings of the 2019 International Conference on Information and Communications Technology (ICOIACT), Yogyakarta, Indonesia.","DOI":"10.1109\/ICOIACT46704.2019.8938545"},{"key":"ref_110","doi-asserted-by":"crossref","unstructured":"Knecht, K., Stefanescu, D.A., and Koenig, R. (2019, January 11\u201313). Citizen Engagement through Design Space Exploration Integrating citizen knowledge and expert design in computational urban planning. Proceedings of the 37th Education and Research in Computer Aided Architectural Design in Europe and XXIII Iberoamerican Society of Digital Graphics, Joint Conference, Porto, Portugal.","DOI":"10.52842\/conf.ecaade.2019.1.785"},{"key":"ref_111","unstructured":"Nguyen, Q.N., Frisiello, A., and Rossi, C. (2019, January 19\u201322). The Design of a Mobile Application for Crowdsourcing in Disaster Risk Reduction. Proceedings of the 16th International Conference on Information Systems for Crisis Response and Management, Val\u00e8ncia, Spain."},{"key":"ref_112","first-page":"122","article-title":"Designing a Human Computation Framework to Enhance Citizen-Government Interaction","volume":"25","author":"Zabaleta","year":"2019","journal-title":"J. Univers. Comput. Sci."},{"key":"ref_113","doi-asserted-by":"crossref","unstructured":"Dioquino, J., Cac, A.R., and Tandingan, D.R. (2019, January 15). Development of Material Recovery Trading System: An Innovative Tool for Community Waste Management Support. Proceedings of the International Conference on Information Technology and Digital Applications 2019, Yogyakarta, Indonesia.","DOI":"10.1088\/1757-899X\/803\/1\/012013"},{"key":"ref_114","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1007\/s12518-019-00284-5","article-title":"Channelling participation into useful representation: Combining digital survey app and collaborative mapping for national slum-upgrading programme","volume":"12","author":"Aditya","year":"2020","journal-title":"Appl. Geomat."},{"key":"ref_115","doi-asserted-by":"crossref","unstructured":"Asfarian, A., Putra, R.P., Panatagama, A.P., Nurhadryani, Y., and Ramadhan, D.A. (2020, January 24). E-Initiative for Food Security: Design of Mobile Crowdfunding Platform to Reduce Food Insecurity in Indonesia. Proceedings of the 8th International Conference on Information and Communication Technology (ICoICT), Yogyakarta, Indonesia.","DOI":"10.1109\/ICoICT49345.2020.9166180"},{"key":"ref_116","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1002\/ajs4.98","article-title":"Strengthening urban community governance through geographical information systems and participation: An evaluation of my Google Map and service coordination","volume":"55","author":"Liu","year":"2020","journal-title":"Aust. J. Soc. Issues"},{"key":"ref_117","doi-asserted-by":"crossref","first-page":"1586","DOI":"10.1080\/1369118X.2020.1716038","article-title":"Find A Loo: An app for sanitation governance","volume":"24","author":"Biswas","year":"2021","journal-title":"Inf. Commun. Soc."},{"key":"ref_118","doi-asserted-by":"crossref","unstructured":"Janoskova, P., Stofkova, K.R., Kovacikova, M., Stofkova, J., and Kovacikova, K. (2021). The concept of a smart city communication in the form of an urban mobile application. Sustainability, 13.","DOI":"10.3390\/su13179703"},{"key":"ref_119","doi-asserted-by":"crossref","unstructured":"Jindal, A., and Chowdhury, A. (2021, January 7\u201310). Designing a donation portal to help underprivileged Indians. Proceedings of the International Conference on Research into Design, Mumbai, India.","DOI":"10.1007\/978-981-16-0041-8_34"},{"key":"ref_120","doi-asserted-by":"crossref","unstructured":"Takenouchi, K., and Choh, I. (2021). Development of a support system for creating disaster prevention maps focusing on road networks and hazardous elements. Vis. Comput. Ind. Biomed. Art, 4.","DOI":"10.1186\/s42492-021-00089-7"},{"key":"ref_121","first-page":"1","article-title":"Integrating online and onsite participation in urban planning: Assessment of a digital participation system","volume":"10","author":"Thoneick","year":"2021","journal-title":"Int. J. E-Plan. Res. (IJEPR)"},{"key":"ref_122","doi-asserted-by":"crossref","unstructured":"Wernbacher, T., Pfeiffer, A., Gebetsroither-Geringer, E., Goels, M., Worster, J., Mei\u00dfner, E., Graf, A., Stollnberger, R., Geyer, R., and Schmidt, R.R. (2021, January 19\u201319). HotCity-A gamified token system for reporting waste heat sources. Proceedings of the International Conference on Intelligent Systems & Networks, Hanoi, Vietnam.","DOI":"10.1007\/978-3-030-93780-5_2"},{"key":"ref_123","doi-asserted-by":"crossref","unstructured":"G\u00f6rg\u00fc, L., O\u2019Grady, M., Mangina, E., and O\u2019Hare, G.M. (2022, January 26\u201329). Participatory risk management in the smart city. Proceedings of the 2022 IEEE International Smart Cities Conference (ISC2), Paphos, Cyprus.","DOI":"10.1109\/ISC255366.2022.9922040"},{"key":"ref_124","first-page":"3063","article-title":"ParmoSense: Scenario-based Participatory Mobile Urban Sensing Platform with User Motivation Engine","volume":"34","author":"Matsuda","year":"2022","journal-title":"Sens. Mater."},{"key":"ref_125","doi-asserted-by":"crossref","first-page":"e12538","DOI":"10.1016\/j.heliyon.2022.e12538","article-title":"Determining factors affecting the perceived usability of air pollution detection mobile application \u201cAirVisual\u201d in Thailand: A structural equation model forest classifier approach","volume":"8","author":"Ong","year":"2022","journal-title":"Heliyon"},{"key":"ref_126","doi-asserted-by":"crossref","unstructured":"Gonzales, A.L.R., Ingalla, E.J.M., Javier, N.A.F., Serrano, E.A., and Rodriguez, R.L. (2022, January 1\u20134). CharitAble: A Software Application for Charity Donation. Proceedings of the 14th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management, Boracay, Philippines.","DOI":"10.1109\/HNICEM57413.2022.10109400"},{"key":"ref_127","doi-asserted-by":"crossref","unstructured":"Ananta, M.T., Rohidah, S., Brata, K.C., and Abidin, Z. (2023, January 24\u201325). Mobile Crowdsourcing App Design: Managing Waste Through Waste Bank in Rural Area of Indonesia. Proceedings of the 8th International Conference on Sustainable Information Engineering and Technology, Bali, Indonesia.","DOI":"10.1145\/3626641.3626685"},{"key":"ref_128","doi-asserted-by":"crossref","first-page":"101877","DOI":"10.1016\/j.giq.2023.101877","article-title":"Trends and challenges of e-government chatbots: Advances in exploring open government data and citizen participation content","volume":"40","author":"Cantador","year":"2023","journal-title":"Gov. Inf. Q."},{"key":"ref_129","doi-asserted-by":"crossref","first-page":"e12274","DOI":"10.1002\/isd2.12274","article-title":"Challenges and barriers faced by older adults to access legislative e-participation platforms in Brazil","volume":"89","author":"Junqueira","year":"2023","journal-title":"Electron. J. Inf. Syst. Dev. Ctries."},{"key":"ref_130","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1080\/24751979.2022.2109502","article-title":"The impact of technological support on citizen searches for missing persons","volume":"6","author":"Lam","year":"2023","journal-title":"Justice Eval. J."},{"key":"ref_131","doi-asserted-by":"crossref","unstructured":"Leoputra, C.D., Satya, D.P., and Al-Ghazali, M.R. (2023, January 10\u201311). Application of User-Centered Design Approach in Developing Interaction Design of In-Kind Donation Feature on a Crowdfunding Platform. Proceedings of the 2023 International Conference on Electrical Engineering and Informatics (ICEEI), Bandung, Indonesia.","DOI":"10.1109\/ICEEI59426.2023.10346892"},{"key":"ref_132","doi-asserted-by":"crossref","first-page":"e14923","DOI":"10.7717\/peerj.14923","article-title":"Usability and acceptance of crowd-based early warning of harmful algal blooms","volume":"11","author":"Manik","year":"2023","journal-title":"PeerJ"},{"key":"ref_133","first-page":"131","article-title":"Users\u2019 evaluation of a digital government collaborative platform (DGCP) in Sri Lanka","volume":"18","author":"Sapraz","year":"2024","journal-title":"Transform. Gov. People Process Policy"},{"key":"ref_134","doi-asserted-by":"crossref","unstructured":"Bastos, D., Fern\u00e1ndez-Caballero, A., Pereira, A., and Rocha, N.P. (2022). Smart city applications to promote citizen participation in city management and governance: A systematic review. Informatics, 9.","DOI":"10.3390\/informatics9040089"},{"key":"ref_135","doi-asserted-by":"crossref","unstructured":"Sapraz, M., and Han, S. (2021). Implicating human values for designing a digital government collaborative platform for environmental issues: A value sensitive design approach. Sustainability, 13.","DOI":"10.3390\/su13116240"},{"key":"ref_136","doi-asserted-by":"crossref","unstructured":"Bajdor, P., and Starostka-Patyk, M. (2021). Smart City: A bibliometric analysis of conceptual dimensions and areas. Energies, 14.","DOI":"10.3390\/en14144288"},{"key":"ref_137","doi-asserted-by":"crossref","unstructured":"Sharif, A., Allam, Z., Feizizadeh, B., and Ghamari, H. (2021). Three decades of research on smart cities: Mapping knowledge structure and trends. Sustainability, 13.","DOI":"10.3390\/su13137140"},{"key":"ref_138","doi-asserted-by":"crossref","first-page":"989","DOI":"10.1080\/01900692.2019.1665065","article-title":"Hands-off or hands-on governance for public innovation? A comparative case study in the EU cohesion policy implementation in Finland","volume":"43","author":"Vento","year":"2020","journal-title":"Int. J. Public Adm."},{"key":"ref_139","doi-asserted-by":"crossref","unstructured":"Almeida, A.F., Rocha, N.P., and Silva, A.G. (2020). Methodological quality of manuscripts reporting on the usability of mobile applications for pain assessment and management: A systematic review. Int. J. Environ. Res. Public Health, 17.","DOI":"10.3390\/ijerph17030785"},{"key":"ref_140","doi-asserted-by":"crossref","first-page":"e22774","DOI":"10.2196\/22774","article-title":"Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults","volume":"8","author":"Silva","year":"2021","journal-title":"JMIR Hum. Factors"}],"container-title":["Digital"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2673-6470\/4\/3\/38\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:49:14Z","timestamp":1760111354000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2673-6470\/4\/3\/38"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,5]]},"references-count":140,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["digital4030038"],"URL":"https:\/\/doi.org\/10.3390\/digital4030038","relation":{},"ISSN":["2673-6470"],"issn-type":[{"value":"2673-6470","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,5]]}}}