{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T12:21:44Z","timestamp":1783513304653,"version":"3.55.0"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T00:00:00Z","timestamp":1779148800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T00:00:00Z","timestamp":1779148800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005639","name":"Universidade De S\u00e3o Paulo","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005639","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Software Qual J"],"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>In the digital age, the omnipresence of software drives the mass production fueled by artificial intelligence. This dependence requires rigorous methods to assess the quality of the software. Inspired by Proficiency Tests (PT) from other fields, this work proposes an innovative process to evaluate the competence of software testing teams and laboratories. Based on Action Design Research(ADR) and Business Process Management(BPM), the developed solution includes software that automates the process, provides a knowledge base, and generates performance indicators. Furthermore, we explore the use of Large Language Models (LLMs) to extract valuable insights from PT results. Quantitative experiments demonstrated the effectiveness of the proposed process, while a qualitative study validated the solution, demonstrating its feasibility and potential to transform the quality assessment of software.<\/jats:p>","DOI":"10.1007\/s11219-026-09762-7","type":"journal-article","created":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T02:21:50Z","timestamp":1779157310000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Innovating proficiency testing in software: a digital process with LLM-integrated support tool and automated indicators"],"prefix":"10.1007","volume":"34","author":[{"given":"Felipe","family":"Dallilo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simone","family":"Souza","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M\u00e1rcio","family":"Delamaro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,19]]},"reference":[{"key":"9762_CR1","unstructured":"42matters (2025). Google play vs ios app store | store stats for mobile apps. https:\/\/42matters.com\/stats. Provides real-time statistics on the number of apps available on the Google Play Store (over 3 million) and the Apple App Store (over 1.8 million). Accessed 15 April 2025"},{"key":"9762_CR2","unstructured":"AI, G. (2024). Gemini api documentation. https:\/\/ai.google.dev\/gemini-api\/docs\/get-started\/tutorial."},{"key":"9762_CR3","unstructured":"Biolchini, J., Mian, P. G., Natali, A. C. C. & Travassos, G. H. (2005). Systematic review in software engineering. Engineering,\u00a0679, pp. 165\u2013176. http:\/\/www.cin.ufpe.br\/ in1037\/leitura\/systematicReviewSE-COPPE.pdf."},{"key":"9762_CR4","unstructured":"Blog, G. (2024). Grande atualiza\u00e7\u00e3o do gemini: respostas mais r\u00e1pidas com flash 1.5, acesso expandido e muito mais. https:\/\/blog.google\/intl\/pt-br\/produtos\/comunicacao-e-conectividade\/grande-atualizacao-do-gemini-respostas-mais-rapidas-com-flash-15-acesso-expandido-e-muito-mais\/."},{"key":"9762_CR5","doi-asserted-by":"publisher","unstructured":"Chapetta, W. A., das Neves, J. S. & Machado, R. C. S. (2021). Quantitative metrics for performance monitoring of software code analysis accredited testing laboratories. Sensors. https:\/\/doi.org\/10.3390\/s21113660.","DOI":"10.3390\/s21113660"},{"key":"9762_CR6","volume-title":"Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory","author":"J Corbin","year":"2014","unstructured":"Corbin, J., & Strauss, A. (2014). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (4th ed.). Thousand Oaks, CA: SAGE Publications.","edition":"4"},{"key":"9762_CR7","unstructured":"Dalillo, F., De Albuquerque, J. & Fantinato, M. (2014). Making a link between strategy and process model collections: a multi-layered approach."},{"key":"9762_CR8","doi-asserted-by":"crossref","unstructured":"Dallilo, F., Delamaro, M. & Souza, S. (2024). A methodology to support the execution of proficiency tests for software quality assessment. Anais do XXIII Simp\u00f3sio Brasileiro de Qualidade de Software pp. 49\u201359. https:\/\/sol.sbc.org.br\/index.php\/sbqs\/article\/view\/32934.","DOI":"10.1145\/3701625.3701645"},{"key":"9762_CR9","unstructured":"Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805."},{"key":"9762_CR10","unstructured":"Documentation, N. (2012). Nginx documentation. https:\/\/nginx.org\/en\/docs\/."},{"key":"9762_CR11","unstructured":"Documentation, D. (2015). Docker documentation. https:\/\/docs.docker.com\/."},{"key":"9762_CR12","unstructured":"Documentation, D. (2017). Docker compose documentation. https:\/\/docs.docker.com\/compose\/."},{"key":"9762_CR13","doi-asserted-by":"crossref","unstructured":"Dumas, M., La Rosa, M., Mendling, J. & Reijers, H. A. (2013). Fundamentals of business process management Vol. 1 (Springer).","DOI":"10.1007\/978-3-642-33143-5_1"},{"key":"9762_CR14","unstructured":"Ericsson (2023). Ericsson mobility report - november 2023. https:\/\/www.ericsson.com\/en\/reports-and-papers\/mobility-report. Ericsson\u2019s mobility reports offer data on the number of mobile subscriptions worldwide, often exceeding the global population, highlighting the prevalence of software-driven mobile devices"},{"key":"9762_CR15","unstructured":"Freund, J. & R\u00fccker, B. (2013). Camunda. https:\/\/camunda.com\/."},{"key":"9762_CR16","volume":"158","author":"V Garousi","year":"2023","unstructured":"Garousi, V., et al. (2023). Measuring the unmeasurable? a systematic map of software quality metrics for human factors. Information and Software Technology, 158, Article 107185.","journal-title":"Information and Software Technology"},{"key":"9762_CR17","doi-asserted-by":"crossref","unstructured":"Geiwitz, J., Spiker, A. & Fischer, S. (1992). Evaluating maintenance proficiency tests.","DOI":"10.1109\/HFPP.1992.283373"},{"key":"9762_CR18","unstructured":"Habbema, H. (2017). Docker: Compose. distribuindo aplica\u00e7\u00f5es multicontainer. https:\/\/medium.com\/@habbema\/docker-compose-8c16d02585b4."},{"key":"9762_CR19","unstructured":"Hou, X., et al. (2023). Large language models for software engineering: A survey. arXiv preprint arXiv:2308.10620. Accepted for publication in Future Generation Computer Systems."},{"key":"9762_CR20","unstructured":"ISO17025 (2017a). ISO\/IEC 17025:2017 General requirements for the competence of testing and calibration laboratories 3rd edn (International Standard Organization)."},{"key":"9762_CR21","unstructured":"ISO17043 (2017b). ISO\/IEC 17043:2017 Conformity assessment \u2013 General requirements for proficiency testing 1st edn (International Standard Organization)."},{"key":"9762_CR22","doi-asserted-by":"crossref","unstructured":"Jalali S., & Wohlin, C. (2012). Systematic literature studies: Database searches vs. backward snowballing. In Proceedings of the 6th int\u2019l symp.","DOI":"10.1145\/2372251.2372257"},{"key":"9762_CR23","doi-asserted-by":"crossref","unstructured":"Karpukhin, V. et al. (2020). Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.","DOI":"10.18653\/v1\/2020.emnlp-main.550"},{"key":"9762_CR24","unstructured":"Kasurinen, J. (2022). The role of professional competency in modern software testing. Journal of Software: Evolution and Process,\u00a034, Article e2432."},{"key":"9762_CR25","first-page":"9459","volume":"33","author":"P Lewis","year":"2020","unstructured":"Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33, 9459\u20139474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"9762_CR26","doi-asserted-by":"crossref","unstructured":"Liu, Y. & Lapata, M. (2019). Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345.","DOI":"10.18653\/v1\/D19-1387"},{"key":"9762_CR27","first-page":"145","volume":"32","author":"N Maneva","year":"2024","unstructured":"Maneva, N., et al. (2024). Human-centric quality assurance: Bridging the gap between process and competency. Software Quality Journal, 32, 145\u2013168.","journal-title":"Software Quality Journal"},{"key":"9762_CR28","doi-asserted-by":"publisher","first-page":"1670","DOI":"10.1373\/clinchem.2011.168641","volume":"57","author":"WG Miller","year":"2011","unstructured":"Miller, W. G., Jones, G. R., Horowitz, G. L., & Weykamp, C. (2011). Proficiency testing\/external quality assessment: current challenges and future directions. Clinical chemistry, 57, 1670\u20131680.","journal-title":"Clinical chemistry"},{"key":"9762_CR29","doi-asserted-by":"crossref","unstructured":"Pinto, D.L., Aoki, R. & Silva, G. (2009). Statistical analysis of proficiency testing results under elliptical distributions. Computational Statistics & Data Analysis 53, pp. 1427\u20131439. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0167947308005720.","DOI":"10.1016\/j.csda.2008.12.003"},{"key":"9762_CR30","unstructured":"Pressman, R. S. & Maxim, B. R. (2015). Software engineering: a practitioner\u2019s approach 8 edn (McGraw-Hill Education). Another comprehensive software engineering textbook that would cover the diverse categories of software and its importance in various aspects of life and industry."},{"key":"9762_CR31","doi-asserted-by":"crossref","unstructured":"Sinclair, D. & Zairi, M. (1995). Performance measurement as an obstacle to TQM, pp. 42\u201345 (The TQM Magazine).","DOI":"10.1108\/09544789510081117"},{"key":"9762_CR32","unstructured":"Sommerville, I. (2016). Software engineering. Pearson Education. A widely used textbook in software engineering that likely discusses the vast landscape of software types across different platforms and its pervasive role in modern society."},{"key":"9762_CR33","volume-title":"Basics of Qualitative Research: Grounded Theory Procedures and Techniques(SAGE Publications","author":"A Strauss","year":"1990","unstructured":"Strauss, A., & Corbin, J. (1990). Basics of Qualitative Research: Grounded Theory Procedures and Techniques(SAGE Publications. CA: Thousand Oaks."},{"key":"9762_CR34","unstructured":"Turner, S. P. (2018). Docker: Up and Running: Shipping Reliable Containers in Production(O\u2019Reilly Media)."},{"key":"9762_CR35","unstructured":"Vaswani, A. et al. (2017). Attention is all you need. Advances in neural information processing systems 30."},{"key":"9762_CR36","unstructured":"Weske, M. (2007). Business Process Management: Concepts. Architecture (Springer: Languages)."}],"container-title":["Software Quality Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-026-09762-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11219-026-09762-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-026-09762-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T11:54:22Z","timestamp":1783511662000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11219-026-09762-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,19]]},"references-count":36,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,6]]}},"alternative-id":["9762"],"URL":"https:\/\/doi.org\/10.1007\/s11219-026-09762-7","relation":{},"ISSN":["0963-9314","1573-1367"],"issn-type":[{"value":"0963-9314","type":"print"},{"value":"1573-1367","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,19]]},"assertion":[{"value":"15 June 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"There is no Conflict of interest\/Competing interests for this manuscript.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of Interest\/Competing interests"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"25"}}