{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T14:17:42Z","timestamp":1770905862306,"version":"3.50.1"},"reference-count":32,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,9,27]],"date-time":"2022-09-27T00:00:00Z","timestamp":1664236800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>This paper reports the results of an investigation into system effectiveness\u2019s role in acquiring and sustaining U.S. weapons systems from 1958 to 2022. An understanding is vital because the acquisition system has recently undergone a significant change. In the 64 year period covered by this study there have been many changes to the Department of Defense acquisition and sustainment processes. The investigation used three qualitative methods: a structured literature review, a grounded theory analysis of the structured literature review, and a historiography of the initial grounded theory results. The research identified five epochs, the first two lasting approximately 22 years each. The last epoch is still in its early period. Each epoch corresponds to a change in the acquisition process. There are four conclusions. First, system effectiveness does not serve its original intent and purpose. Second, analysis of source documents provides insight into why system effectiveness plays a diminished role. Third, the original approach to system effectiveness may have relevance for today\u2019s problems and challenges. Finally, an integrated research methodology is valuable for making sense of conflicting information spread over time.<\/jats:p>","DOI":"10.3390\/systems10050169","type":"journal-article","created":{"date-parts":[[2022,9,29]],"date-time":"2022-09-29T01:23:16Z","timestamp":1664414596000},"page":"169","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Investigating the Role of System Effectiveness in the Acquisition and Sustainment of U.S. Defense Systems: 1958 to 2022"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6399-5012","authenticated-orcid":false,"given":"John","family":"Green","sequence":"first","affiliation":[{"name":"Systems Engineering Department, Naval Postgraduate School, Monterey, CA 93943, USA"}]},{"given":"Jerrell","family":"Stracener","sequence":"additional","affiliation":[{"name":"AT&T Center for Virtualization, Southern Methodist University, Dallas, TX 75205, USA"}]}],"member":"1968","published-online":{"date-parts":[[2022,9,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Green, J. (2021). An Investigation of the Role of System Effectiveness in the Acquisition and Sustainment of US Defense Systems: 1958 to 2021. TechRxiv.","DOI":"10.36227\/techrxiv.15138444"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Green, J. (2022, January 11\u201312). An Investigation of the Role of System Effectiveness in the Acquisition and Sustainment of U.S. Defense Systems: 1958 to 2021. Proceedings of the Nineteenth Annual Acquisition Research Symposium, Monterey, CA, USA.","DOI":"10.36227\/techrxiv.15138444"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/TR.1984.6448271","article-title":"Reliability engineering of electronic equipment a historical perspective","volume":"R-33","author":"Coppola","year":"1984","journal-title":"IEEE Trans. Reliab."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Habayeb, A. (1987). Systems Effectiveness, Pergamon Press.","DOI":"10.1016\/B978-0-08-034814-8.50006-7"},{"key":"ref_5","unstructured":"Wasson, C. (2015). System Engineering Analysis, Design, and Development, Wiley. [2nd ed]."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"961","DOI":"10.1111\/j.1559-3584.1967.tb05384.x","article-title":"System Effectiveness in the United States Navy","volume":"79","author":"Aziz","year":"1967","journal-title":"Nav. Eng. J."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1109\/TAES.1967.5408741","article-title":"System Worth, System Effectiveness, Integrated Logistics Support, and Maintainability","volume":"AES-3","author":"Blanchard","year":"1967","journal-title":"IEEE Trans. Aerosp. Electron. Syst."},{"key":"ref_8","unstructured":"Air Force Systems Command (1965). Weapon System Effectiveness Industry Advisory Committee (WSEIAC) Final Report of Task Group VI, Chairman\u2019s Final Report (Integrated Summary), Air Force Systems Command."},{"key":"ref_9","unstructured":"(2022, August 13). Available online: https:\/\/www.dau.edu\/glossary\/Pages\/Glossary.aspx."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1109\/TR.1967.5217480","article-title":"Cost Effectiveness, System Effectiveness, Integrated Logistics Support, and Maintainability","volume":"R-163","author":"Blanchard","year":"1967","journal-title":"IEEE Trans. Reliab."},{"key":"ref_11","unstructured":"Air Force Systems Command (1965). Weapon System Effectiveness Industry Advisory Committee (WSEIAC) Final Report of Task Group II, Prediction\u2014Measurement, Air Force Systems Command."},{"key":"ref_12","unstructured":"Dordick, H.S. (1965). An Introduction to System Effectiveness, The Rand Corporation. RAND Report P-3237."},{"key":"ref_13","unstructured":"Reed, C.M., and Fenwick, A. (2010). A Consistent Multi-User Framework for Assessing System Performance. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1109\/TR.1969.5216965","article-title":"Analytic Justification of the WSEIAC Formula","volume":"R-18","author":"Fowler","year":"1969","journal-title":"IEEE Trans. Reliab."},{"key":"ref_15","unstructured":"(2018). The Operational Availability Handbook, NAVSO P-7001, Assistant Secretary of the Navy."},{"key":"ref_16","unstructured":"(2022, August 13). OPNAVINST 3000.12A, Operational Availability Handbook. 2 September 2003 (Cancelled). Available online: https:\/\/www.wdfxw.net\/doc25688776.htm."},{"key":"ref_17","unstructured":"Department of the Army (1971). Engineering Design Handbook, System Analysis and Cost-Effectiveness, Army Materiel Command\u2019. AMCP-706-191."},{"key":"ref_18","unstructured":"Green, J. (2022, August 13). Establishing System Measures of Effectiveness. Available online: https:\/\/acqnotes.com\/wp-content\/uploads\/2014\/09\/Establishing-System-Measures-of-Effectiveness-by-John-Green.pdf."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1109\/TR.1980.5220844","article-title":"System Effectiveness models: An annotated bibliography","volume":"R-29","author":"Tillman","year":"1980","journal-title":"IEEE Trans. Reliab."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Green, J. (2022). An Investigation of the Role of System Effectiveness in the Acquisition and Sustainment of U.S. Defense Systems: 1958 to 2021. [Ph.D. Thesis, Southern Methodist University].","DOI":"10.36227\/techrxiv.15138444"},{"key":"ref_21","first-page":"52","article-title":"Complex Adaptive Systems of Systems: A Grounded Theory Approach","volume":"17","author":"Johnson","year":"2018","journal-title":"Grounded Theory Rev."},{"key":"ref_22","unstructured":"Johnson, B. (2019). A Framework for Engineered Complex Adaptive Systems of Systems. [Ph.D. Thesis, Naval Postgraduate School]."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Stol, K., Ralph, P., and Fitzgerald, B. (2016, January 14\u201322). Grounded theory in software research: A critical review and guidelines. Proceedings of the International Conference on Software Engineering Proceedings, Austin, TX, USA.","DOI":"10.1145\/2884781.2884833"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Babar, M.A., and Zhang, H. (2009, January 15\u201316). Systematic literature reviews in software engineering: Preliminary results from interviews with researchers. Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement, Lake Buena Vista, FL, USA.","DOI":"10.1109\/ESEM.2009.5314235"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Hoda, R. (2021). Socio-Technical Grounded Theory for Software Engineering. arXiv.","DOI":"10.1109\/ICSE-Companion52605.2021.00139"},{"key":"ref_26","unstructured":"(2022, August 13). Available online: http:\/\/researcharticles.com\/index.php\/data-triangulation-in-qualitative-research\/."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1145\/3356773.3356797","article-title":"The use of grey literature review as evidence for practitioners","volume":"44","author":"Kamel","year":"2019","journal-title":"ACM SIGSOFT Softw. Eng. Notes"},{"key":"ref_28","unstructured":"Wohlin, C. (2014, January 13\u201314). Guidelines for snowballing in systematic literature studies and a replication in software engineering. Proceedings of the EASE\u2019 14, London, UK. Available online: https:\/\/www.overleaf.com\/project\/6313d75eef8761e56a2762fc."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"93","DOI":"10.21061\/see.49","article-title":"New perspectives for implementing grounded theory","volume":"1","author":"McCall","year":"2021","journal-title":"Stud. Eng. Educ."},{"key":"ref_30","unstructured":"Rudwick, B. (1969). Systems Analysis for Effective Planning: Principles and Cases, Wiley."},{"key":"ref_31","unstructured":"Green, J.M., and Stracener, J. (2022, August 13). A Framework for a Defense Systems Effectiveness Modeling and Analysis Capability: Systems Effectiveness Modeling for Acquisition. Available online: https:\/\/dair.nps.edu\/handle\/123456789\/1757."},{"key":"ref_32","unstructured":"Madachy, R., and Green, J. (2019, January 8). Naval Combat System Product Line Architecture Economics. Proceedings of the 16th Annual Acquisition Research Symposium, Monterey, CA, USA."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/10\/5\/169\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:40:30Z","timestamp":1760143230000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/10\/5\/169"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,27]]},"references-count":32,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["systems10050169"],"URL":"https:\/\/doi.org\/10.3390\/systems10050169","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,27]]}}}