{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T18:30:55Z","timestamp":1775068255795,"version":"3.50.1"},"reference-count":51,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2025,1,27]],"date-time":"2025-01-27T00:00:00Z","timestamp":1737936000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"FOCICYT 2022-2023 project of the UTEQ"},{"name":"DIUDA 22436 project of the Universidad de Atacama"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Open-source software has experienced steady growth, driving increased research. However, open-source communities still need standardized processes to ensure software quality, and their characteristics make it challenging to adopt many usability techniques from human\u2013computer interaction directly. Our study aims to adapt and evaluate the feasibility of three usability evaluation techniques\u2014cognitive walkthrough, formal usability testing, and thinking aloud\u2014across three open-source projects (Code::Blocks, Freeplane, and Scribus) from the development team\u2019s perspective. We participated as volunteers in these projects, employing a multiple case study method. We found that usability techniques were not systematically adopted, and procedures specific to open-source projects were lacking. We also identified adverse conditions, such as limited user participation, that hindered adoption. We proposed adaptations to each technique and formalized procedures to address these challenges and apply them in open-source contexts. Additionally, we developed a framework for integrating usability evaluation into OSS projects. To address this, we detailed our framework\u2019s phases, tasks, and artifacts to ensure reusability and adaptability across OSS contexts, providing practical steps for implementation and future validations. In conclusion, usability techniques must be adapted for open-source software, considering the projects\u2019 unique characteristics and philosophy. Although obstacles exist, such as user participation, applying adapted usability techniques in open-source projects is feasible.<\/jats:p>","DOI":"10.3390\/informatics12010012","type":"journal-article","created":{"date-parts":[[2025,1,27]],"date-time":"2025-01-27T06:39:51Z","timestamp":1737959991000},"page":"12","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["How to Improve Usability in Open-Source Software Projects? An Analysis of Evaluation Techniques Through a Multiple Case Study"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4562-6723","authenticated-orcid":false,"given":"Lucrecia","family":"Llerena","sequence":"first","affiliation":[{"name":"Software Engineering Program, Faculty of Engineering Sciences, Quevedo State University, Quevedo 120301, Ecuador"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John W.","family":"Castro","sequence":"additional","affiliation":[{"name":"Departamento de Ingenier\u00eda Inform\u00e1tica y Ciencias de la Computaci\u00f3n, Facultad de Ingenier\u00eda, Universidad de Atacama, Copiap\u00f3 1532297, Chile"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rosa","family":"Llerena","sequence":"additional","affiliation":[{"name":"Economics Program, Faculty of Economics and Financial Social Sciences, Quevedo State University, Quevedo 120301, Ecuador"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nancy","family":"Rodr\u00edguez","sequence":"additional","affiliation":[{"name":"Software Engineering Program, Faculty of Engineering Sciences, Quevedo State University, Quevedo 120301, Ecuador"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,1,27]]},"reference":[{"key":"ref_1","first-page":"3","article-title":"Open source software: A history","volume":"21","author":"Bretthauer","year":"2002","journal-title":"Inf. Technol. Libr."},{"key":"ref_2","first-page":"303","article-title":"Usability in open source software development: Opinions and practice","volume":"35","author":"Andreasen","year":"2006","journal-title":"Inf. Technol. Control"},{"key":"ref_3","unstructured":"Raza, A., Capretz, L.F., and Ahmed, F. (2013, January 10\u201312). Maintenance support in open source software projects. Proceedings of the 8th International Conference on Digital Information Management (ICDIM\u201913), Islamabad, Pakistan."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1016\/j.infsof.2018.10.011","article-title":"Adapting usability techniques for application in open source Software: A multiple case study","volume":"107","author":"Llerena","year":"2019","journal-title":"Inf. Softw. Technol."},{"key":"ref_5","first-page":"7","article-title":"La usabilidad en ingenier\u00eda de software: Definici\u00f3n y caracter\u00edsticas","volume":"2","year":"2011","journal-title":"Innovaci\u00f3n. Rep. De Investig."},{"key":"ref_6","first-page":"228","article-title":"A systematic mapping study of software usability studies","volume":"12","author":"Almazroi","year":"2021","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"ref_7","first-page":"28","article-title":"Automated tools for usability evaluation: A systematic mapping study","volume":"13315","author":"Meiselwitz","year":"2022","journal-title":"Social Computing and Social Media: Design, User Experience and Impact"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1109\/TSE.2022.3150720","article-title":"Using the SOCIO chatbot for UML modelling: A family of experiments","volume":"49","author":"Ren","year":"2023","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_9","unstructured":"Ferr\u00e9, X., and Moreno, A.M. (2004, January 3\u20137). Integraci\u00f3n de la IPO en el proceso de desarrollo de la ingenier\u00eda del software: Propuestas existentes y temas a resolver. Proceedings of the V Congreso Interacci\u00f3n Persona-Ordenador (Interacci\u00f3n\u201904), Lleida, Spain."},{"key":"ref_10","first-page":"259","article-title":"Assessing usability readiness of collaborative projects","volume":"26","year":"2011","journal-title":"Comput. Syst. Sci. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.infsof.2018.09.007","article-title":"A pilot empirical study of applying a usability technique in an open source software project","volume":"106","author":"Llerena","year":"2019","journal-title":"Inf. Softw. Technol."},{"key":"ref_12","unstructured":"Castro, J.W. (2014). Incorporaci\u00f3n de La Usabilidad En El Proceso de Desarrollo Opencas Source Software. [Doctoral Thesis, Escuela Polit\u00e9cnica Superior, Universidad Aut\u00f3noma de Madrid]."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1993). Usability Engineering, Morgan Kaufmann.","DOI":"10.1016\/B978-0-08-052029-2.50009-7"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1109\/MS.2004.1259234","article-title":"Back to the user [open source]","volume":"21","author":"Messerschmitt","year":"2004","journal-title":"IEEE Softw."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"\u0106etin, G., and Mehmet, G. (December, January 30). A measurement based framework for assessment of usability-centricness of open source software projects. Proceedings of the 2008 IEEE International Conference on Signal Image Technology and Internet Based Systems (SITIS\u201908), Bali, Indonesia.","DOI":"10.1109\/SITIS.2008.106"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Terry, M., Kay, M., and Lafreniere, B. (2010, January 10\u201315). Perceptions and practices of usability in the free\/open source software (FOSS) community. Proceedings of the 28th International Conference on Human Factors in Computing Systems (CHI\u201910), Atlanta, GA, USA.","DOI":"10.1145\/1753326.1753476"},{"key":"ref_17","first-page":"58","article-title":"Examining usability work and culture in OSS","volume":"451","author":"Damiani","year":"2015","journal-title":"Open Source Systems: Adoption and Impact"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1007\/978-3-540-73257-0_4","article-title":"An analysis of involvement of HCI experts in distributed software development: Practical issues","volume":"4564","author":"Shuler","year":"2007","journal-title":"Online Communities on Social Computing"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Hedberg, H., Iivari, N., Rajanen, M., and Harjumaa, L. (2007, January 20\u201326). Assuring quality and usability in open source software development. Proceedings of the First International Workshop on Emerging Trends in FLOSS Research and Development (FLOSS\u201907: ICSE Workshops\u201907), Minneapolis, MN, USA.","DOI":"10.1109\/FLOSS.2007.2"},{"key":"ref_20","first-page":"1","article-title":"The usability of open source software","volume":"8","author":"Nichols","year":"2003","journal-title":"First Monday"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ternauciuc, A., and Vasiu, R. (2015, January 17\u201319). Testing usability in Moodle: When and how to do it. Proceedings of the 2015 IEEE 13th International Symposium on Intelligent Systems and Informatics (SISY\u201915), Subotica, Serbia.","DOI":"10.1109\/SISY.2015.7325391"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Assal, H., Chiasson, S., and Biddle, R. (2016, January 24). Cesar: Visual representation of source code vulnerabilities. Proceedings of the 2016 IEEE Symposium on Visualization for Cyber Security (VizSec\u201916), Baltimore, MD, USA.","DOI":"10.1109\/VIZSEC.2016.7739576"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ledesma, A., Al-Musawi, M., and Nieminen, H. (2016). Health figures: An open source JavaScript library for health data visualization. BMC Med. Inform. Decis. Mak., 16.","DOI":"10.1186\/s12911-016-0275-6"},{"key":"ref_24","first-page":"100964","article-title":"An end-to-end model-based approach to support big data analytics development","volume":"58","author":"Khalajzadeh","year":"2020","journal-title":"J. Comput."},{"key":"ref_25","first-page":"75","article-title":"Evaluating an interactive memory analysis tool: Findings from a cognitive walkthrough and a user study","volume":"4","author":"Weninger","year":"2020","journal-title":"ACM Human-Comput. Interact."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Tegos, S., Mavridis, A., and Demetriadis, S. (2020, January 20\u201322). PeerTalk: Enabling agent-enhanced real-time collaboration in MOOCs. Proceedings of the ACM 24th Pan-Hellenic Conference on Informatics (PCI\u201920), Athens, Greece.","DOI":"10.1145\/3437120.3437296"},{"key":"ref_27","unstructured":"Cemellini, B., Thompson, R., Van Oosterom, P., and Vries, M. (2018, January 2\u20134). Usability testing of a web-based 3D Cadastral visualization system. Proceedings of the 6th International FIG Workshop on 3D Cadastres, Delft, The Netherlands."},{"key":"ref_28","unstructured":"Hall, J. (2014). Usability Themes in Open Source Software. [Ph.D. Thesis, University of Minnesota]."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Pietrobon, R., Shah, A., Kuo, P., Harker, M., McCready, M., Butler, C., Martins, H., Moorman, C.T., and Jacobs, D.O. (2006). Duke surgery research central: An open-source web application for the improvement of compliance with research regulation. BMC Med. Inform. Decis. Mak., 6.","DOI":"10.1186\/1472-6947-6-32"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Gujrati, S., and Vasserman, E. (2013, January 18\u201320). The usability of truecrypt, or how I learned to stop whining and fix an interface. Proceedings of the Third ACM Conference on Data and Application Security and Privacy (CODASPY\u201913), San Antonio, TX, USA.","DOI":"10.1145\/2435349.2435360"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Brun, D., Ferreira, S.M., Gouin-Vallerand, C., and George, S. (2016, January 28\u201330). CARTON project: Do-it-yourself approach to turn a smartphone into a smart eyewear. Proceedings of the 14th International Conference on Advances in Mobile Computing and Multi Media (MoMM\u201916), Singapore, Singapore.","DOI":"10.1145\/3007120.3007134"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1002\/cae.21789","article-title":"An easy to use open source authoring tool to create effective and reusable learning objects","volume":"25","author":"Gordillo","year":"2017","journal-title":"Comput. Appl. Eng. Educ."},{"key":"ref_33","unstructured":"Wharton, C., Rieman, J., Lewis, C., and Polson, P. (1994). The Cognitive Walkthrough Method: A Practitioner\u2019s Guide, John Wiley & Sons, Inc."},{"key":"ref_34","unstructured":"Mayhew, D. (1999). The Usability Engineering Lifecycle: A Practitioner\u2019s Handbook for User Interface Design, Morgan Kaufmann. [1st ed.]."},{"key":"ref_35","unstructured":"(2023, September 15). SourceForge. Available online: https:\/\/sourceforge.net\/."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1002\/spip.256","article-title":"Usability processes in open source projects","volume":"11","author":"Nichols","year":"2006","journal-title":"Softw. Process Improv. Pract."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"160","DOI":"10.1016\/j.jce.2006.10.001","article-title":"Intrinsic motivation in open source software development","volume":"35","author":"Bitzer","year":"2007","journal-title":"J. Comp. Econ."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Benson, C., M\u00fcller-Prove, M., and Mzourek, J. (2004, January 24\u201329). Professional usability in open source projects: GNOME, OpenOffice.org, NetBeans. Proceedings of the Extended Abstracts on Human Factors in Computing Systems (CHI EA\u201904: CHI\u201904), Vienna, Austria.","DOI":"10.1145\/985921.985991"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Rajanen, M., and Iivari, N. (2013, January 1\u20134). Open source and human computer interaction philosophies in open source projects\u2014Incompatible or co- existent?. Proceedings of the International Conference on Making Sense of Converging Media (AcademicMindTrek\u201913), Tampere, Finland.","DOI":"10.1145\/2523429.2523463"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Northrop, E.E., and Lipford, H.R. (2014, January 7). Exploring the usability of open source network forensic tools. Proceedings of the 2014 ACM Workshop on Security Information Workers (SIW\u201914), Scottsdale, AZ, USA.","DOI":"10.1145\/2663887.2663903"},{"key":"ref_41","first-page":"3","article-title":"The usability of GNOME","volume":"2014","author":"Hall","year":"2014","journal-title":"Linux J."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Rajanen, M., and Iivari, N. (2015, January 18\u201323). Power, empowerment and open source usability. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI\u201915), Seoul, Republic of Korea.","DOI":"10.1145\/2702123.2702441"},{"key":"ref_43","first-page":"10","article-title":"To do or not to do: Usability in open source development","volume":"71","author":"Nielsen","year":"2007","journal-title":"Interfaces"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Runeson, P., H\u00f6st, M., Rainer, A., and Regnell, B. (2012). Case Study Research in Software Engineering: Guidelines and Examples, John Wiley & Sons.","DOI":"10.1002\/9781118181034"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1007\/s10664-008-9102-8","article-title":"Guidelines for conducting and reporting case study research in software engineering","volume":"14","author":"Runeson","year":"2009","journal-title":"Empir. Softw. Eng."},{"key":"ref_46","unstructured":"Genero Bocco, M.F., Jos\u00e9 Antonio, C.L., and Piattini Velthuis, M.G. (2014). M\u00e9todos de Investigaci\u00f3n En Ingenier\u00eda Del Software, Grupo Editorial Ra-Ma."},{"key":"ref_47","unstructured":"Yin, R.K. (2013). Case Study Research: Design and Methods, SAGE Publications, Inc.. [5th ed.]."},{"key":"ref_48","first-page":"291","article-title":"Harnessing scientific literature reports for pharmacovigilance: Prototype software analytical tool development and usability testing","volume":"8","author":"Sorbello","year":"2017","journal-title":"Appl. Clin. Inform."},{"key":"ref_49","unstructured":"Constantine, L.L., and Lockwood, L.A.D. (1999). Software for Use: A Practical Guide to the Models and Methods of Usage-Centered Design, Addison-Wesley. [1st ed.]."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"597","DOI":"10.2307\/30036550","article-title":"Rigor in information systems positivist case research: Current practices, trends, and recommendations","volume":"27","year":"2003","journal-title":"MIS Q."},{"key":"ref_51","unstructured":"Lewis, C. (1982). Using the \u201cThinking-Aloud\u201d Method in Cognitive Interface Design, IBM T.J. Watson Research Division."}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/12\/1\/12\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T10:37:11Z","timestamp":1759919831000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/12\/1\/12"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,27]]},"references-count":51,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["informatics12010012"],"URL":"https:\/\/doi.org\/10.3390\/informatics12010012","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,27]]}}}