{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:35:11Z","timestamp":1760060111599,"version":"build-2065373602"},"reference-count":43,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2025,8,5]],"date-time":"2025-08-05T00:00:00Z","timestamp":1754352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Software"],"abstract":"<jats:p>Background: While several test-automation maturity models (e.g., CMMI, TMMi, TAIM) exist, none explicitly integrate ISO 9001-based quality management systems (QMS), leaving a gap for organizations that must align E2E test automation with formal quality assurance. Objective: This study proposes a test-automation maturity model (TAMM) that bridges E2E automation capability with ISO 9001\/ISO 9004 self-assessment principles, and evaluates its reliability and practical impact in industry. Methods: TAMM comprises eight maturity dimensions, 39 requirements, and 429 checklist items. Three independent assessors applied the checklist to three software teams; inter-rater reliability was ensured via consensus review (Cohen\u2019s \u03ba = 0.75). Short-term remediation actions based on the checklist were implemented over six months and re-assessed. Synergy with the organization\u2019s ISO 9001 QMS was analyzed using ISO 9004 self-check scores. Results: Within 6 months of remediation, mean TAMM score rose from 2.75 \u2192 2.85. Inter-rater reliability is filled with Cohen\u2019s \u03ba = 0.75. Conclusions: The proposed TAMM delivers measurable, short-term maturity gains and complements ISO 9001-based QMS without introducing conflicting processes. Practitioners can use the checklist to identify actionable gaps, prioritize remediation, and quantify progress, while researchers may extend TAMM to other domains or automate scoring via repository mining.<\/jats:p>","DOI":"10.3390\/software4030019","type":"journal-article","created":{"date-parts":[[2025,8,5]],"date-time":"2025-08-05T14:23:19Z","timestamp":1754403799000},"page":"19","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Research and Development of Test Automation Maturity Model Building and Assessment Methods for E2E Testing"],"prefix":"10.3390","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9904-8554","authenticated-orcid":false,"given":"Daiju","family":"Kato","sequence":"first","affiliation":[{"name":"Customer Success Division, Nihon Knowledge Co., Ltd., JS Building 9F, 3-19-5 Kotobuki, Taito-ku, Tokyo 111-0042, Japan"}]},{"given":"Ayane","family":"Mogi","sequence":"additional","affiliation":[{"name":"Customer Success Division, Nihon Knowledge Co., Ltd., JS Building 9F, 3-19-5 Kotobuki, Taito-ku, Tokyo 111-0042, Japan"}]},{"given":"Hiroshi","family":"Ishikawa","sequence":"additional","affiliation":[{"name":"Faculty of Systems Design, Tokyo Metropolitan University, 6-6 Asahigaoka, Hino-shi, Tokyo 191-0065, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3072-5785","authenticated-orcid":false,"given":"Yasufumi","family":"Takama","sequence":"additional","affiliation":[{"name":"Faculty of Systems Design, Tokyo Metropolitan University, 6-6 Asahigaoka, Hino-shi, Tokyo 191-0065, Japan"}]}],"member":"1968","published-online":{"date-parts":[[2025,8,5]]},"reference":[{"key":"ref_1","unstructured":"Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., and Jeffries, R. (2024, December 31). Manifesto for Agile Software Development. Available online: https:\/\/agilemanifesto.org\/."},{"key":"ref_2","unstructured":"Allspaw, J., and Hammond, P. (2025, May 07). 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr. Presented at the O'Reilly Velocity Conference, San Francisco, CA, USA, 23 June 2009. Available online: https:\/\/tech-talks.code-maven.com\/ten-plus-deploys-per-day.html."},{"key":"ref_3","unstructured":"(2025, May 07). Crowd Strike Holdings. Available online: https:\/\/www.crowdstrike.com\/."},{"key":"ref_4","unstructured":"(2014). Systems and Software Engineering\u2014Systems and Software Quality Requirements and Evaluation (SQuaRE)\u2014Guide to SQuaRE (Standard No. ISO\/IEC 25000:2014)."},{"key":"ref_5","unstructured":"(2023). Systems and Software Engineering\u2014Systems and Software Quality Requirements and Evaluation (SQuaRE)\u2014Quality-in-Use Model (Standard No. ISO\/IEC 25019:2023)."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"159","DOI":"10.3390\/fi3020159","article-title":"Evolving Web-Based Test Automation into Agile Business Specifications","volume":"3","author":"Mugridge","year":"2011","journal-title":"Future Internet"},{"key":"ref_7","first-page":"69","article-title":"Highly Automated Agile Testing Process: An Industrial Case Study","volume":"10","author":"Kasprzyk","year":"2016","journal-title":"E-Inform. Softw. Eng. J."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"620836","DOI":"10.1155\/2010\/620836","article-title":"Software Test Automation in Practice: Empirical Observations","volume":"2010","author":"Kasurinen","year":"2010","journal-title":"Adv. Softw. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"232","DOI":"10.3390\/digital4010012","article-title":"Quality Control Methods Using Quality Characteristics in Development and Operations","volume":"4","author":"Kato","year":"2024","journal-title":"Digital"},{"key":"ref_10","unstructured":"(2025, May 07). CMMI (Capability Maturity Model Integration). Available online: https:\/\/cmmiinstitute.com\/."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Gibson, D.L., Goldenson, R.D., and Kost, K. (2006). Performance Results of CMMI\u00ae-Based Process Improvement, Software Engineering Institute, Carnegie Mellon University. Available online: https:\/\/insights.sei.cmu.edu\/documents\/764\/2006_005_001_14762.pdf.","DOI":"10.21236\/ADA454687"},{"key":"ref_12","unstructured":"(2024, August 31). TMMi Foundation. TMMi (Test Maturity Model integration). Available online: https:\/\/www.tmmi.org."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.infsof.2017.01.001","article-title":"Software test maturity assessment and test process improvement: A multivocal literature review","volume":"85","author":"Garousi","year":"2017","journal-title":"Inf. Softw. Technol."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Eldh, S., Andersson, K., Ermedahl, A., and Wiklund, K. (April, January 31). Towards a Test Automation Improvement Model (TAIM). Proceedings of the 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Cleveland, OH, USA.","DOI":"10.1109\/ICSTW.2014.38"},{"key":"ref_15","unstructured":"(2015). Quality Management Systems\u2014Requirements (Standard No. ISO 9001:2015)."},{"key":"ref_16","unstructured":"(2015). Information Technology\u2014Process Assessment\u2014Concepts and Terminology (Standard No. ISO\/IEC 33001:2015)."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1109\/32.988497","article-title":"Test case prioritization: A family of empirical studies","volume":"28","author":"Elbaum","year":"2002","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"929","DOI":"10.1109\/32.962562","article-title":"Prioritizing test cases for regression testing","volume":"27","author":"Rothermel","year":"2001","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_19","unstructured":"Kato, D. (2005, January 1\u20132). Improvement of Test Efficiency Through Automation of Regression Testing. Proceedings of the Japan Symposium on Software Testing (JaSST) 2005, Osaka, Japan. (In Japanese)."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"884","DOI":"10.1109\/TSE.2005.117","article-title":"Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software","volume":"31","author":"Memon","year":"2005","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_21","first-page":"38","article-title":"Cost Models and Return on Investment for Automated Regression Testing","volume":"21","author":"Bertolino","year":"1996","journal-title":"ACM SIGSOFT Softw. Eng. Notes"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1007\/s10664-008-9062-z","article-title":"Realizing quality improvement through test driven development: Results and experiences of four industrial teams","volume":"13","author":"Nagappan","year":"2008","journal-title":"Empir. Softw. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.infsof.2016.04.015","article-title":"When and what to automate in software testing? A multi-vocal literature review","volume":"76","author":"Garousi","year":"2016","journal-title":"Inf. Softw. Technol."},{"key":"ref_24","unstructured":"Rafi, D.M., Moses, K., Petersen, K., and M\u00e4ntyl\u00e4, M.V. (2012, January 2\u20133). Benefits and limitations of automated software testing: Systematic literature review and practitioner survey. Proceedings of the 2012 7th International Workshop on Automation of Software Test (AST), Zurich, Switzerland."},{"key":"ref_25","first-page":"17","article-title":"A Survey of Flaky Tests","volume":"31","author":"Parry","year":"2021","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1007\/s11390-023-1935-5","article-title":"An Empirical Study on Automated Test Generation Tools for Java: Effectiveness and Challenges","volume":"39","author":"Liu","year":"2024","journal-title":"J. Comput. Sci. Technol."},{"key":"ref_27","unstructured":"(2024, September 08). Jenkins. Available online: https:\/\/www.jenkins.io\/."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"704","DOI":"10.1134\/S0361768821080193","article-title":"Requirements and GitHub Issues: An Automated Approach for Quality Requirements Classification","volume":"47","year":"2021","journal-title":"Program. Comput. Softw."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"797","DOI":"10.1007\/s11219-017-9383-5","article-title":"The Central Role of Test Automation in Software Quality Assurance","volume":"25","author":"Mariani","year":"2017","journal-title":"Softw. Qual. J."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kato, D., Shimizu, A., and Ishikawa, H. (2022, January 19\u201321). Quality classification for testing work in DevOps. Proceedings of the 14th International Conference on Management of Digital EcoSystems (MEDES '22), Venice, Italy.","DOI":"10.1145\/3508397.3564840"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"546","DOI":"10.1109\/TSE.2009.70","article-title":"Learning a Metric for Code Readability","volume":"36","author":"Buse","year":"2010","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_32","unstructured":"Pressman, R.S., and Maxim, B.R. (2019). Software Engineering: A Practitioner's Approach, McGraw-Hill."},{"key":"ref_33","unstructured":"(2023). Systems and Software Engineering\u2014Systems and Software Quality Requirements and Evaluation (SQuaRE)\u2014Product Quality Model (Standard No. ISO\/IEC 25010:2023)."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wang, Y., M\u00e4ntyl\u00e4, M., Eldh, S., Markkula, S., Wiklund, K., Kairi, T., Raulamo-Jurvanen, P., and Haukinen, A. (2019). A Self-assessment Instrument for Assessing Test Automation Maturity. Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering (EASE \u201919), Copenhagen, Denmark, 14\u201317 April 2019, Association for Computing Machinery.","DOI":"10.1145\/3319008.3319020"},{"key":"ref_35","unstructured":"(2025, May 07). Ranorex. Available online: https:\/\/ranorex.com\/."},{"key":"ref_36","unstructured":"(2025, May 07). Microsoft. Power Automate. Available online: https:\/\/www.microsoft.com\/ja-jp\/power-platform\/products\/power-automate."},{"key":"ref_37","unstructured":"(2025, July 11). Atlassian. Confluence. Available online: https:\/\/www.atlassian.com\/software\/confluence."},{"key":"ref_38","unstructured":"(2025, July 11). Atlassian. Jira. Available online: https:\/\/www.atlassian.com\/software\/jira."},{"key":"ref_39","unstructured":"(2025, May 07). Bitbucket. Available online: https:\/\/bitbucket.org\/product\/."},{"key":"ref_40","unstructured":"(2018). Quality Management\u2014Quality of an Organization\u2014Guidance to Achieve Sustained Success (Standard No. ISO 9004:2018)."},{"key":"ref_41","unstructured":"(2025, May 07). OpenText Functional Testing. Available online: https:\/\/www.opentext.com\/products\/functional-testing."},{"key":"ref_42","unstructured":"(2025, May 07). Pytest. Available online: https:\/\/docs.pytest.org\/en\/stable\/."},{"key":"ref_43","unstructured":"(2024, December 31). Axelos. ITIL (Information Technology Infrastructure Library). Available online: https:\/\/www.axelos.com\/."}],"container-title":["Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2674-113X\/4\/3\/19\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:23:47Z","timestamp":1760034227000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2674-113X\/4\/3\/19"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,5]]},"references-count":43,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["software4030019"],"URL":"https:\/\/doi.org\/10.3390\/software4030019","relation":{},"ISSN":["2674-113X"],"issn-type":[{"type":"electronic","value":"2674-113X"}],"subject":[],"published":{"date-parts":[[2025,8,5]]}}}