{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T23:39:06Z","timestamp":1790638746646,"version":"4.1.0"},"reference-count":171,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2019,2,4]],"date-time":"2019-02-04T00:00:00Z","timestamp":1549238400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2019,11,30]]},"abstract":"<jats:p>Various system metrics have been proposed for measuring the quality of computer-based systems, such as dependability and security metrics for estimating their performance and security characteristics. As computer-based systems grow in complexity with many subsystems or components, measuring their quality in multiple dimensions is a challenging task. In this work, we tackle the problem of measuring the quality of computer-based systems based on the four key attributes of trustworthiness we developed: security, trust, resilience, and agility. In addition to conducting a systematic survey on metrics, measurements, attributes of metrics, and associated ontologies, we propose a system-level trustworthiness metric framework that accommodates four submetrics, called STRAM (&lt;u&gt;S&lt;\/u&gt;ecurity, &lt;u&gt;T&lt;\/u&gt;rust, &lt;u&gt;R&lt;\/u&gt;esilience, and &lt;u&gt;A&lt;\/u&gt;gility &lt;u&gt;M&lt;\/u&gt;etrics). The proposed STRAM framework offers a hierarchical ontology structure where each submetric is defined as a sub-ontology. Moreover, this work proposes developing and incorporating metrics describing key assessment tools, including vulnerability assessment, risk assessment, and red teaming, to provide additional evidence in the measurement and quality of trustworthy systems. We further discuss how assessment tools are related to measuring the quality of computer-based systems and the limitations of the state-of-the-art metrics and measurements. Finally, we suggest future research directions for system-level metrics research toward measuring fundamental attributes of the quality of computer-based systems and improving the current metric and measurement methodologies.<\/jats:p>","DOI":"10.1145\/3277666","type":"journal-article","created":{"date-parts":[[2019,2,5]],"date-time":"2019-02-05T15:40:21Z","timestamp":1549381221000},"page":"1-47","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":92,"title":["STRAM"],"prefix":"10.1145","volume":"51","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0860-8531","authenticated-orcid":false,"given":"Jin-Hee","family":"Cho","sequence":"first","affiliation":[{"name":"Virginia Tech, VA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8034-0942","authenticated-orcid":false,"given":"Shouhuai","family":"Xu","sequence":"additional","affiliation":[{"name":"The University of Texas at San Antonio, TX, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick M.","family":"Hurley","sequence":"additional","affiliation":[{"name":"US Air Force Research Laboratory, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew","family":"Mackay","sequence":"additional","affiliation":[{"name":"UK Defence Science and Technology Laboratory, Salisbury, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Trevor","family":"Benjamin","sequence":"additional","affiliation":[{"name":"UK Defence Science and Technology Laboratory, Salisbury, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark","family":"Beaumont","sequence":"additional","affiliation":[{"name":"Defence Science and Technology Group, Adelaide, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,2,4]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1191\/030913200701540465"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/356876.356879"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2008.920800"},{"key":"e_1_2_2_4_1","volume-title":"focus, and convergence: The future of command and control. The International C2 Journal 1, 1","author":"Alberts D. S.","year":"2007","unstructured":"D. S. Alberts . 2007. Agility , focus, and convergence: The future of command and control. The International C2 Journal 1, 1 ( 2007 ), 1--30. D. S. Alberts. 2007. Agility, focus, and convergence: The future of command and control. The International C2 Journal 1, 1 (2007), 1--30."},{"key":"e_1_2_2_5_1","doi-asserted-by":"crossref","unstructured":"David S. Alberts. 2011. Understanding agility. In The Agility Advantage: A Survival Guide for Complex Enterprises and Endeavors. CCRP 187--196.  David S. Alberts. 2011. Understanding agility. In The Agility Advantage: A Survival Guide for Complex Enterprises and Endeavors. CCRP 187--196.","DOI":"10.21236\/ADA631225"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.21236\/ADA605273"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.5711\/1082598318121"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00126465"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2009.54"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2014.2310221"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2015.30"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2011.18"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2004.2"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/319709.319710"},{"key":"e_1_2_2_15_1","unstructured":"Nicholas J. Bahr. 2014. Definitions and concepts. In System Safety Engineering and Risk Assessment: A Practical Approach. Taylors and Francis 5--9.  Nicholas J. Bahr. 2014. Definitions and concepts. In System Safety Engineering and Risk Assessment: A Practical Approach. Taylors and Francis 5--9."},{"key":"e_1_2_2_16_1","volume-title":"Network Science","author":"Barab\u00e1si Albert-Laszl\u00f3","unstructured":"Albert-Laszl\u00f3 Barab\u00e1si . 2016. Network robustness . In Network Science . Cambridge University Press , 270--319. Albert-Laszl\u00f3 Barab\u00e1si. 2016. Network robustness. In Network Science. Cambridge University Press, 270--319."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csi.2010.12.002"},{"key":"e_1_2_2_18_1","volume-title":"Proceedings of SPIE 9119","author":"Blasch E.","year":"2014","unstructured":"E. Blasch . 2014 . Trust metrics in information fusion . In Proceedings of SPIE 9119 . Machine Intelligence and Bio-inspired Computation : Theory and Applications VIII, 91190L. E. Blasch. 2014. Trust metrics in information fusion. In Proceedings of SPIE 9119. Machine Intelligence and Bio-inspired Computation: Theory and Applications VIII, 91190L."},{"key":"e_1_2_2_19_1","volume-title":"Proceedings of RAID\u201914","author":"Boggs Nathaniel","unstructured":"Nathaniel Boggs , Senyao Du , and Salvatore J. Stolfo . {n. d.}. Measuring drive-by download defense in depth . In Proceedings of RAID\u201914 . 172--191. Nathaniel Boggs, Senyao Du, and Salvatore J. Stolfo. {n. d.}. Measuring drive-by download defense in depth. In Proceedings of RAID\u201914. 172--191."},{"key":"e_1_2_2_20_1","volume-title":"Proceedings of the 5th Layered Assurance Workshop (LAW\u201911)","author":"Boggs Nathaniel Gordon","year":"2011","unstructured":"Nathaniel Gordon Boggs and Salvatore Stolfo . 2011 . ALDR: A new metric for measuring effective layering of defenses . Proceedings of the 5th Layered Assurance Workshop (LAW\u201911) (2011). Nathaniel Gordon Boggs and Salvatore Stolfo. 2011. ALDR: A new metric for measuring effective layering of defenses. Proceedings of the 5th Layered Assurance Workshop (LAW\u201911) (2011)."},{"key":"e_1_2_2_21_1","doi-asserted-by":"crossref","unstructured":"R. B\u00f6hme and F. C. Freiling. 2008. Metrics and measurements. In Dependability Metrics. Vol. 4909. Springer-Verlag Lecture Notes in Computer Science 7--13.  R. B\u00f6hme and F. C. Freiling. 2008. Metrics and measurements. In Dependability Metrics. Vol. 4909. Springer-Verlag Lecture Notes in Computer Science 7--13.","DOI":"10.1007\/978-3-540-68947-8_2"},{"key":"e_1_2_2_22_1","volume-title":"Construction of Engineering Ontologies","author":"Borst W. N.","unstructured":"W. N. Borst . 1997. Construction of Engineering Ontologies . Centre for Telematica and Information Technology , University of Tweente, Eschede. W. N. Borst. 1997. Construction of Engineering Ontologies. Centre for Telematica and Information Technology, University of Tweente, Eschede."},{"key":"e_1_2_2_23_1","volume-title":"NATO CCDCOE Cyber Red Teaming: Organisational, technical and legal implications in a military context","author":"Brangetto Pascal","year":"2015","unstructured":"Pascal Brangetto , Emin Caliskan , and Henry Roigas . 2015. NATO CCDCOE Cyber Red Teaming: Organisational, technical and legal implications in a military context ( 2015 ). https:\/\/ccdcoe.org\/sites\/default\/files\/multimedia\/pdf\/Cyber_Red_Team.pdf. Pascal Brangetto, Emin Caliskan, and Henry Roigas. 2015. NATO CCDCOE Cyber Red Teaming: Organisational, technical and legal implications in a military context (2015). https:\/\/ccdcoe.org\/sites\/default\/files\/multimedia\/pdf\/Cyber_Red_Team.pdf."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/1245581.1245586"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2010.01492.x"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2010.01563.x"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/35.1.3"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/876661.876669"},{"key":"e_1_2_2_30_1","volume-title":"Proceedings of the IEEE Military Communications Conference.","author":"Chan K.","unstructured":"K. Chan , J. H. Cho , T. Trout , J. Wampler , A. Toth , and B. Rivera . 2015. Trustd: Trust daemon experimental testbed for network emulation . In Proceedings of the IEEE Military Communications Conference. K. Chan, J. H. Cho, T. Trout, J. Wampler, A. Toth, and B. Rivera. 2015. Trustd: Trust daemon experimental testbed for network emulation. In Proceedings of the IEEE Military Communications Conference."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/1234689.1234697"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190619.3190639"},{"key":"e_1_2_2_33_1","volume-title":"Spatiotemporal patterns and predictability of cyberattacks. PLoS One 10, 5 (5","author":"Chen Yu-Zhong","year":"2015","unstructured":"Yu-Zhong Chen , Zi-Gang Huang , Shouhuai Xu , and Ying-Cheng Lai . 2015. Spatiotemporal patterns and predictability of cyberattacks. PLoS One 10, 5 (5 2015 ), e0124472. Yu-Zhong Chen, Zi-Gang Huang, Shouhuai Xu, and Ying-Cheng Lai. 2015. Spatiotemporal patterns and predictability of cyberattacks. PLoS One 10, 5 (5 2015), e0124472."},{"key":"e_1_2_2_34_1","volume-title":"Secure Programming with Static Analysis","author":"Chess Brian","unstructured":"Brian Chess and Jacob West . 2007. Secure Programming with Static Analysis ( first ed.). Addison-Wesley Professional . Brian Chess and Jacob West. 2007. Secure Programming with Static Analysis (first ed.). Addison-Wesley Professional."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/SURV.2011.092110.00088"},{"key":"e_1_2_2_36_1","doi-asserted-by":"crossref","DOI":"10.1109\/TCYB.2014.2335744","article-title":"Tradeoffs between trust and survivability for mission effectiveness in tactical networks","volume":"45","author":"Cho Jin-Hee","year":"2015","unstructured":"Jin-Hee Cho . 2015 . Tradeoffs between trust and survivability for mission effectiveness in tactical networks . IEEE Transactions on Cybernetics 45 , 4 (2015). Jin-Hee Cho. 2015. Tradeoffs between trust and survivability for mission effectiveness in tactical networks. IEEE Transactions on Cybernetics 45, 4 (2015).","journal-title":"IEEE Transactions on Cybernetics"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/GLOCOM.2016.7842168"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/COGSIMA.2016.7497779"},{"key":"e_1_2_2_39_1","volume-title":"PROVEST: Provenance-based trust model for delay tolerant networks","author":"Cho Jin-Hee","year":"2016","unstructured":"Jin-Hee Cho and Ing-Ray Chen . 2016 . PROVEST: Provenance-based trust model for delay tolerant networks . IEEE Transactions on Dependable and Secure Computing ( 2016). Jin-Hee Cho and Ing-Ray Chen. 2016. PROVEST: Provenance-based trust model for delay tolerant networks. IEEE Transactions on Dependable and Secure Computing (2016)."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/MNET.2009.4804331"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.92.4.909"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.1090.0236"},{"key":"e_1_2_2_43_1","volume-title":"Review of the Department of Homeland Security\u2019s Approach to Risk Analysis","author":"National Research Concil. 2010.","unstructured":"National Research Concil. 2010. Review of the Department of Homeland Security\u2019s Approach to Risk Analysis . The National Academies Press . National Research Concil. 2010. Review of the Department of Homeland Security\u2019s Approach to Risk Analysis. The National Academies Press."},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2005.00615.x"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2009.01209.x"},{"key":"e_1_2_2_46_1","volume-title":"Proceedings of the 2014 Symposium and Bootcamp on the Science of Security (HotSoS\u201914)","author":"Da Gaofeng","unstructured":"Gaofeng Da , Maochao Xu , and Shouhuai Xu. {n. d.}. A new approach to modeling and analyzing security of networked systems . In Proceedings of the 2014 Symposium and Bootcamp on the Science of Security (HotSoS\u201914) . 6. Gaofeng Da, Maochao Xu, and Shouhuai Xu. {n. d.}. A new approach to modeling and analyzing security of networked systems. In Proceedings of the 2014 Symposium and Bootcamp on the Science of Security (HotSoS\u201914). 6."},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2012.71"},{"key":"e_1_2_2_48_1","volume-title":"Proceedings of the 13th USENIX Security Symposium","author":"Davis Darren","unstructured":"Darren Davis , Fabian Monrose , and Michael K. Reiter . 2004. On user choice in graphical password schemes . In Proceedings of the 13th USENIX Security Symposium . San Diego, CA. Darren Davis, Fabian Monrose, and Michael K. Reiter. 2004. On user choice in graphical password schemes. In Proceedings of the 13th USENIX Security Symposium. San Diego, CA."},{"key":"e_1_2_2_49_1","first-page":"1","article-title":"Measuring the agility of networked military forces","volume":"9","author":"Dekker A. H.","year":"2006","unstructured":"A. H. Dekker . 2006 . Measuring the agility of networked military forces . Journal of Battlefield Technology 9 , 1 (March 2006), 1--6. A. H. Dekker. 2006. Measuring the agility of networked military forces. Journal of Battlefield Technology 9, 1 (March 2006), 1--6.","journal-title":"Journal of Battlefield Technology"},{"key":"e_1_2_2_50_1","volume-title":"A theory of information warfare. Information Warfare and Security","author":"Denning Dorothy E.","unstructured":"Dorothy E. Denning . 1999. A theory of information warfare. Information Warfare and Security . ACM Press , 21--42. Dorothy E. Denning. 1999. A theory of information warfare. Information Warfare and Security. ACM Press, 21--42."},{"key":"e_1_2_2_51_1","unstructured":"Department of Homeland Security. 2015. National Critical Infrastructure Security and Resilience Research and Development Plan. http:\/\/publish.illinois.edu\/ciri-new-theme\/files\/2016\/09\/National-CISR-RD-Plan_Nov-2015.pdf.  Department of Homeland Security. 2015. National Critical Infrastructure Security and Resilience Research and Development Plan. http:\/\/publish.illinois.edu\/ciri-new-theme\/files\/2016\/09\/National-CISR-RD-Plan_Nov-2015.pdf."},{"key":"e_1_2_2_52_1","volume-title":"trustworthiness, and the F scale. 61 (08","author":"Deutsch Morton","year":"1960","unstructured":"Morton Deutsch . 1960. Trust , trustworthiness, and the F scale. 61 (08 1960 ), 138--40. Morton Deutsch. 1960. Trust, trustworthiness, and the F scale. 61 (08 1960), 138--40."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/UBICOMM.2007.43"},{"key":"e_1_2_2_54_1","volume-title":"Towards a security ontology","author":"Donner Marc","year":"2003","unstructured":"Marc Donner . 2003. Towards a security ontology . IEEE Security 8 Privacy 1, 3 ( 2003 ), 6--7. Marc Donner. 2003. Towards a security ontology. IEEE Security 8 Privacy 1, 3 (2003), 6--7."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.5555\/1884848.1884858"},{"key":"e_1_2_2_56_1","volume-title":"Response Ability: The Language, Structure, and Culture of the Agile Enterprise","author":"Dove Rick","year":"2001","unstructured":"Rick Dove . 2001 . Response Ability: The Language, Structure, and Culture of the Agile Enterprise . John Wiley 8 Sons. Rick Dove. 2001. Response Ability: The Language, Structure, and Culture of the Agile Enterprise. John Wiley 8 Sons."},{"key":"e_1_2_2_57_1","volume-title":"Proceedings of the Conference on Systems Engineering Research (CSER\u201905)","author":"Dove Rick","year":"2005","unstructured":"Rick Dove . 2005 . Fundamental principles for agile systems engineering . In Proceedings of the Conference on Systems Engineering Research (CSER\u201905) . Rick Dove. 2005. Fundamental principles for agile systems engineering. In Proceedings of the Conference on Systems Engineering Research (CSER\u201905)."},{"key":"e_1_2_2_58_1","volume-title":"The Closed World: Computers and the Politics of Discourse in Cold War America","author":"Edwards N.","unstructured":"N. Edwards , Paul. 1996. The Closed World: Computers and the Politics of Discourse in Cold War America . MIT Press . N. Edwards, Paul. 1996. The Closed World: Computers and the Politics of Discourse in Cold War America. MIT Press."},{"key":"e_1_2_2_59_1","unstructured":"ENISA. 2011. Ontology and taxonomies of resilience. https:\/\/www.enisa.europa.eu\/publications\/ontology_taxonomies.  ENISA. 2011. Ontology and taxonomies of resilience. https:\/\/www.enisa.europa.eu\/publications\/ontology_taxonomies."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/365024.365037"},{"key":"e_1_2_2_61_1","volume-title":"Resilience thinking: Integrating resilience, adaptability and transformability","author":"Walker Stephen R.","year":"2010","unstructured":"Stephen R. Walker , Brian Scheffer , Marten Chapin , Terry Rockstrom , Johan Folke , and Carl Carpenter . 2010. Resilience thinking: Integrating resilience, adaptability and transformability . Ecology and Society 15, 4 ( 2010 ). Stephen R. Walker, Brian Scheffer, Marten Chapin, Terry Rockstrom, Johan Folke, and Carl Carpenter. 2010. Resilience thinking: Integrating resilience, adaptability and transformability. Ecology and Society 15, 4 (2010)."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.07.458"},{"key":"e_1_2_2_63_1","volume-title":"Information Assurance Tools Report -- Vulnerability Assessment","author":"Goertzel Karen Mercedes","unstructured":"Karen Mercedes Goertzel and Theodore Winograd . 2011. Information Assurance Tools Report -- Vulnerability Assessment ( sixth ed.). Technical Report. IATAC. Karen Mercedes Goertzel and Theodore Winograd. 2011. Information Assurance Tools Report -- Vulnerability Assessment (sixth ed.). Technical Report. IATAC."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJMSO.2006.008770"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2015.8"},{"key":"e_1_2_2_66_1","volume-title":"Formal ontology in information systems","author":"Guarino N.","unstructured":"N. Guarino . 1998. Formal ontology in information systems . IOS Press , 3--15. N. Guarino. 1998. Formal ontology in information systems. IOS Press, 3--15."},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/289.291"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4379(91)90005-T"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2009.01216.x"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2009.01216.x"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600176.2600180"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2007.77"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.142"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature08932"},{"key":"e_1_2_2_75_1","unstructured":"Kirk Hayes. 2016. Penetration test vs. red team assessment: The age old debate of pirates vs. ninjas continues. https:\/\/community.rapid7.com\/community\/infosec\/blog\/2016\/06\/23\/penetration-testing-vs-red-teaming-the-age-old-debate-of-pirates-vs-ninja-continues.  Kirk Hayes. 2016. Penetration test vs. red team assessment: The age old debate of pirates vs. ninjas continues. https:\/\/community.rapid7.com\/community\/infosec\/blog\/2016\/06\/23\/penetration-testing-vs-red-teaming-the-age-old-debate-of-pirates-vs-ninja-continues."},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.es.04.110173.000245"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.es.04.110173.000245"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2015.2443790"},{"key":"e_1_2_2_79_1","volume-title":"Proceedings of the Workshop on Performance Metrics for Intelligent Systems (PerMIS'04)","author":"Huang Hui-Min","unstructured":"Hui-Min Huang , Elena Messina , and James S. Albus . 2004. Toward a generic model for autonomy levels for unmanned systems (ALFUS) . In Proceedings of the Workshop on Performance Metrics for Intelligent Systems (PerMIS'04) . NIST Special Publication 1014. Hui-Min Huang, Elena Messina, and James S. Albus. 2004. Toward a generic model for autonomy levels for unmanned systems (ALFUS). In Proceedings of the Workshop on Performance Metrics for Intelligent Systems (PerMIS'04). NIST Special Publication 1014."},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/44.476633"},{"key":"e_1_2_2_81_1","volume-title":"Software and Systems Division. Retrieved","author":"Laboratory Information Technology","year":"2016","unstructured":"Information Technology Laboratory , Software and Systems Division. Retrieved 2 Dec. 2016 . Metrics and Measures . https:\/\/samate.nist.gov\/index.php\/Metrics_and_Measures.html. Information Technology Laboratory, Software and Systems Division. Retrieved 2 Dec. 2016. Metrics and Measures. https:\/\/samate.nist.gov\/index.php\/Metrics_and_Measures.html."},{"key":"e_1_2_2_82_1","unstructured":"ISO\/IEC 1998. 9241-14 Ergonomic Requirements for Office Work with Visual Display Terminals (VDT)s-Part 14 Menu Dialogues. ISO\/IEC. ISO\/IEC 9241-14: 1998(E).  ISO\/IEC 1998. 9241-14 Ergonomic Requirements for Office Work with Visual Display Terminals (VDT)s-Part 14 Menu Dialogues. ISO\/IEC. ISO\/IEC 9241-14: 1998(E)."},{"key":"e_1_2_2_83_1","volume-title":"Routledge Encyclopedia of Philosophy","author":"Ivanhoe Philip J.","unstructured":"Philip J. Ivanhoe . 1998. Xin (trustworthiness). Routledge Encyclopedia of Philosophy . Taylor and Francis . https:\/\/www.rep.routledge.com\/articles\/thematic\/xin-trustworthiness\/v-1. Philip J. Ivanhoe. 1998. Xin (trustworthiness). Routledge Encyclopedia of Philosophy. Taylor and Francis. https:\/\/www.rep.routledge.com\/articles\/thematic\/xin-trustworthiness\/v-1."},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1198\/016214502760301264"},{"key":"e_1_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.5555\/820756.821829"},{"key":"e_1_2_2_86_1","volume-title":"Agile Manufacturing: Forging New Frontiers","author":"Kidd P. T.","year":"1994","unstructured":"P. T. Kidd . 1994 . Agile Manufacturing: Forging New Frontiers . Addison-Wesley . P. T. Kidd. 1994. Agile Manufacturing: Forging New Frontiers. Addison-Wesley."},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866835.1866852"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-016-0560-y"},{"key":"e_1_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2008.75"},{"key":"e_1_2_2_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/2991079.2991102"},{"key":"e_1_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-02786-9_13"},{"key":"e_1_2_2_93_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSYST.2009.2017397"},{"key":"e_1_2_2_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663474.2663476"},{"key":"e_1_2_2_95_1","volume-title":"Software Security: Building Security In","author":"McGraw Gary","year":"2006","unstructured":"Gary McGraw . 2006 . Software Security: Building Security In . Addison-Wesley Professional . Gary McGraw. 2006. Software Security: Building Security In. Addison-Wesley Professional."},{"key":"e_1_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csi.2006.04.002"},{"key":"e_1_2_2_97_1","volume-title":"Proceedings of the International Conference on Cyber Conflict (CyCon).","author":"Mireles J. D.","unstructured":"J. D. Mireles , J. H. Cho , and S. Xu . 2016. Extracting attack narratives from traffic datasets . In Proceedings of the International Conference on Cyber Conflict (CyCon). J. D. Mireles, J. H. Cho, and S. Xu. 2016. Extracting attack narratives from traffic datasets. In Proceedings of the International Conference on Cyber Conflict (CyCon)."},{"key":"e_1_2_2_98_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-08509-8_7"},{"key":"e_1_2_2_99_1","doi-asserted-by":"crossref","unstructured":"Mohammadi et al. 2014. Trustworthiness attributes and metrics for engineering trusted Internet-based software systems. In Cloud Computing and Services Science. Vol. 453. Springer International Publishing Switzerland 19--35.  Mohammadi et al. 2014. Trustworthiness attributes and metrics for engineering trusted Internet-based software systems. In Cloud Computing and Services Science. Vol. 453. Springer International Publishing Switzerland 19--35.","DOI":"10.1007\/978-3-319-11561-0_2"},{"key":"e_1_2_2_100_1","first-page":"56","article-title":"Analyzing malware detection eciency with multiple anti-malware programs","volume":"1","author":"Morales Jose Andre","year":"2012","unstructured":"Jose Andre Morales , Shouhuai Xu , and Ravi Sandhu . 2012 . Analyzing malware detection eciency with multiple anti-malware programs . ASE Science Journal 1 , 2 (2012), 56 -- 66 . Jose Andre Morales, Shouhuai Xu, and Ravi Sandhu. 2012. Analyzing malware detection eciency with multiple anti-malware programs. ASE Science Journal 1, 2 (2012), 56--66.","journal-title":"ASE Science Journal"},{"key":"e_1_2_2_101_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2004.11"},{"key":"e_1_2_2_102_1","volume-title":"Usability Engineering","author":"Nielsen J.","unstructured":"J. Nielsen . 1993. Usability Engineering . Academic Press . J. Nielsen. 1993. Usability Engineering. Academic Press."},{"key":"e_1_2_2_103_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2012.01885.x"},{"key":"e_1_2_2_104_1","doi-asserted-by":"publisher","DOI":"10.1145\/1626195.1626209"},{"key":"e_1_2_2_105_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSC.2008.46"},{"key":"e_1_2_2_106_1","doi-asserted-by":"publisher","DOI":"10.1145\/3005714"},{"key":"e_1_2_2_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/3005714"},{"key":"e_1_2_2_108_1","first-page":"1","article-title":"Modeling and predicting extreme cyber attack rates via marked point processes","volume":"0","author":"Peng Chen","year":"2016","unstructured":"Chen Peng , Maochao Xu , Shouhuai Xu , and Taizhong Hu . 2016 . Modeling and predicting extreme cyber attack rates via marked point processes . Journal of Applied Statistics 0 , 0 (2016), 1 -- 30 . Chen Peng, Maochao Xu, Shouhuai Xu, and Taizhong Hu. 2016. Modeling and predicting extreme cyber attack rates via marked point processes. Journal of Applied Statistics 0, 0 (2016), 1--30.","journal-title":"Journal of Applied Statistics"},{"key":"e_1_2_2_109_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1355770X06003020"},{"key":"e_1_2_2_110_1","volume-title":"Is there a security problem in computing? In Security in Computing","author":"Pfleeger Charles P.","unstructured":"Charles P. Pfleeger . 2006. Is there a security problem in computing? In Security in Computing . Prentice Hall . Charles P. Pfleeger. 2006. Is there a security problem in computing? In Security in Computing. Prentice Hall."},{"key":"e_1_2_2_111_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2010.60"},{"key":"e_1_2_2_112_1","volume-title":"The complexity and stability of ecosystems. Nature 307, 5945","author":"Pimm S.","year":"1984","unstructured":"S. Pimm . 1984. The complexity and stability of ecosystems. Nature 307, 5945 ( 1984 ), 321--326. S. Pimm. 1984. The complexity and stability of ecosystems. Nature 307, 5945 (1984), 321--326."},{"key":"e_1_2_2_113_1","doi-asserted-by":"publisher","DOI":"10.1145\/505248.506010"},{"key":"e_1_2_2_114_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13042-011-0012-5"},{"key":"e_1_2_2_116_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMST.2017.2745505"},{"key":"e_1_2_2_117_1","unstructured":"M. Rausand and A. Hoyland. 2009. System Reliability Theory: Models and Statistical Methods. John Wiley 8 Sons.  M. Rausand and A. Hoyland. 2009. System Reliability Theory: Models and Statistical Methods. John Wiley 8 Sons."},{"key":"e_1_2_2_118_1","unstructured":"RedTeams. March 22 2013. Difference between penetration testing and red team exercises. http:\/\/redteams.net\/blog\/2013\/difference-between-penetration-testing-and-red-team-exercises.  RedTeams. March 22 2013. Difference between penetration testing and red team exercises. http:\/\/redteams.net\/blog\/2013\/difference-between-penetration-testing-and-red-team-exercises."},{"key":"e_1_2_2_119_1","unstructured":"ReliaSoft Corporation. 2003. Reliability HotWire: The eMagazine for the reliability professional. http:\/\/www.weibull.com\/hotwire\/issue26\/relbasics26.htm.  ReliaSoft Corporation. 2003. Reliability HotWire: The eMagazine for the reliability professional. http:\/\/www.weibull.com\/hotwire\/issue26\/relbasics26.htm."},{"key":"e_1_2_2_120_1","volume-title":"Advances in Human Factors, Business Management, Training and Education, Part XVII.","author":"Salo Markku","unstructured":"Markku Salo , Evangelos Markopoulos , Hannu Vanharanta , and Jussi Ilari Kantola . 2016. Degree of agility with an ontology-based application . In Advances in Human Factors, Business Management, Training and Education, Part XVII. Vol. 498 . Markku Salo, Evangelos Markopoulos, Hannu Vanharanta, and Jussi Ilari Kantola. 2016. Degree of agility with an ontology-based application. In Advances in Human Factors, Business Management, Training and Education, Part XVII. Vol. 498."},{"key":"e_1_2_2_121_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2007.00957.x"},{"key":"e_1_2_2_122_1","volume-title":"The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty","author":"Savage Sam L.","unstructured":"Sam L. Savage . 2009. The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty . Wiley . Sam L. Savage. 2009. The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty. Wiley."},{"key":"e_1_2_2_123_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2013.6698898"},{"key":"e_1_2_2_124_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1009440108"},{"key":"e_1_2_2_125_1","doi-asserted-by":"crossref","unstructured":"F. Schneider (Ed.). 1999. Trust in Cyberspace. National Academy Press.  F. Schneider (Ed.). 1999. Trust in Cyberspace. National Academy Press.","DOI":"10.1016\/S1353-4858(00)80006-9"},{"key":"e_1_2_2_126_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2015.51"},{"key":"e_1_2_2_127_1","volume-title":"Proceedings of the 2013 International Conference on Software Engineering (ICSE\u201913)","author":"Shar Lwin Khin","unstructured":"Lwin Khin Shar , Hee Beng Kuan Tan , and Lionel C. Briand . 2013. Mining SQL injection and cross site scripting vulnerabilities using hybrid program analysis . In Proceedings of the 2013 International Conference on Software Engineering (ICSE\u201913) . 642--651. Lwin Khin Shar, Hee Beng Kuan Tan, and Lionel C. Briand. 2013. Mining SQL injection and cross site scripting vulnerabilities using hybrid program analysis. In Proceedings of the 2013 International Conference on Software Engineering (ICSE\u201913). 642--651."},{"key":"e_1_2_2_128_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ergon.2007.01.007"},{"key":"e_1_2_2_129_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0378-7206(03)00079-X"},{"key":"e_1_2_2_130_1","doi-asserted-by":"publisher","DOI":"10.1145\/1414004.1414065"},{"key":"e_1_2_2_131_1","doi-asserted-by":"publisher","DOI":"10.1145\/1103780.1103784"},{"key":"e_1_2_2_132_1","doi-asserted-by":"publisher","DOI":"10.1109\/MAHC.2015.30"},{"key":"e_1_2_2_133_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2006.3"},{"key":"e_1_2_2_134_1","unstructured":"Stanford Center for Biomedical Informatics Research. 2015. WebProt\u00e9g\u00e9. http:\/\/protege.stanford.edu\/.  Stanford Center for Biomedical Informatics Research. 2015. WebProt\u00e9g\u00e9. http:\/\/protege.stanford.edu\/."},{"key":"e_1_2_2_135_1","volume-title":"Strom","author":"Stech Frank J.","year":"2016","unstructured":"Frank J. Stech , Kristin E. Heckman , and Blake E . Strom . 2016 . Integrating cyber-D&D into adversary modeling for active cyber defense. In Cyber Deception. Springer , 1--24. Frank J. Stech, Kristin E. Heckman, and Blake E. Strom. 2016. Integrating cyber-D&D into adversary modeling for active cyber defense. In Cyber Deception. Springer, 1--24."},{"key":"e_1_2_2_136_1","volume-title":"The Web Application Hacker\u2019s Handbook: Discovering and Exploiting Security Flaws","author":"Stuttard Dafydd","unstructured":"Dafydd Stuttard and Marcus Pinto . 2007. The Web Application Hacker\u2019s Handbook: Discovering and Exploiting Security Flaws . John Wiley 8 Sons. Dafydd Stuttard and Marcus Pinto. 2007. The Web Application Hacker\u2019s Handbook: Discovering and Exploiting Security Flaws. John Wiley 8 Sons."},{"key":"e_1_2_2_137_1","doi-asserted-by":"publisher","DOI":"10.1109\/WAINA.2008.39"},{"key":"e_1_2_2_138_1","unstructured":"The Forum of Incident Response and Security Teams. 2015. The Common Vulnerability Scoring System (CVSS). https:\/\/www.first.org\/cvss.  The Forum of Incident Response and Security Teams. 2015. The Common Vulnerability Scoring System (CVSS). https:\/\/www.first.org\/cvss."},{"key":"e_1_2_2_139_1","doi-asserted-by":"publisher","DOI":"10.1109\/AINA.2006.329"},{"key":"e_1_2_2_140_1","unstructured":"TTCP. 2014. The Technical Cooperation Program. http:\/\/www.acq.osd.mil\/ttcp\/index.html.  TTCP. 2014. The Technical Cooperation Program. http:\/\/www.acq.osd.mil\/ttcp\/index.html."},{"key":"e_1_2_2_141_1","unstructured":"U.S. DoD. 2015. U.S. Department of Defense Cybersecurity Test and Evaluation Guidebook Version 1.0. http:\/\/www.dote.osd.mil\/docs\/TempGuide3\/Cybersecurity_TE_Guidebook_July1_2015_v1_0.pdf.  U.S. DoD. 2015. U.S. Department of Defense Cybersecurity Test and Evaluation Guidebook Version 1.0. http:\/\/www.dote.osd.mil\/docs\/TempGuide3\/Cybersecurity_TE_Guidebook_July1_2015_v1_0.pdf."},{"key":"e_1_2_2_142_1","volume-title":"Proceedings of the IEEE\/IFIP International Conference on Dependable Systems Networks. 566--571","author":"Vieira M.","unstructured":"M. Vieira , N. Antunes , and H. Madeira . 2009. Using web security scanners to detect vulnerabilities in web services . In Proceedings of the IEEE\/IFIP International Conference on Dependable Systems Networks. 566--571 . M. Vieira, N. Antunes, and H. Madeira. 2009. Using web security scanners to detect vulnerabilities in web services. In Proceedings of the IEEE\/IFIP International Conference on Dependable Systems Networks. 566--571."},{"key":"e_1_2_2_143_1","doi-asserted-by":"publisher","DOI":"10.1007\/11537878_18"},{"key":"e_1_2_2_144_1","volume-title":"Vlacheas et al","author":"Panagiotis","year":"2011","unstructured":"Panagiotis T. Vlacheas et al . 2011 . Ontology and Taxonomies of Resilience. Technical Report. European Network and Information Security Agency . Panagiotis T. Vlacheas et al. 2011. Ontology and Taxonomies of Resilience. Technical Report. European Network and Information Security Agency."},{"key":"e_1_2_2_145_1","doi-asserted-by":"crossref","unstructured":"B. Walker C. S. Hollings S. R. Carpenter and A. Kinzig. 2004. Resilience adaptability and transformability in social-ecological systems. Ecology and Society 9 2 (2004) Article 5.  B. Walker C. S. Hollings S. R. Carpenter and A. Kinzig. 2004. Resilience adaptability and transformability in social-ecological systems. Ecology and Society 9 2 (2004) Article 5.","DOI":"10.5751\/ES-00650-090205"},{"key":"e_1_2_2_146_1","volume-title":"HyperCheck: A Hardware-Assisted Integrity Monitor","author":"Wang Jiang","unstructured":"Jiang Wang , Angelos Stavrou , and Anup Ghosh . 2010. HyperCheck: A Hardware-Assisted Integrity Monitor . Springer Berlin Heidelberg , 158--177. Jiang Wang, Angelos Stavrou, and Anup Ghosh. 2010. HyperCheck: A Hardware-Assisted Integrity Monitor. Springer Berlin Heidelberg, 158--177."},{"key":"e_1_2_2_147_1","volume-title":"Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops). IEEE, 481--486","author":"Wang Xinlei","year":"2013","unstructured":"Xinlei Wang , Jin-Hee Cho , Kevin Chan , MoonJeong Chang , Ananthram Swami , and Prasant Mohapatra . 2013 . Trust and independence aware decision fusion in distributed networks . In Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops). IEEE, 481--486 . Xinlei Wang, Jin-Hee Cho, Kevin Chan, MoonJeong Chang, Ananthram Swami, and Prasant Mohapatra. 2013. Trust and independence aware decision fusion in distributed networks. In Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops). IEEE, 481--486."},{"key":"e_1_2_2_148_1","doi-asserted-by":"publisher","DOI":"10.5555\/648253.752541"},{"key":"e_1_2_2_149_1","volume-title":"Proceedings of the DARPA Information Survivability Conference and Exposition","volume":"2","author":"Wood B. J.","unstructured":"B. J. Wood and R. A. Duggan . 2000. Red teaming of advanced information assurance concepts . In Proceedings of the DARPA Information Survivability Conference and Exposition , Vol. 2 . 112--118. B. J. Wood and R. A. Duggan. 2000. Red teaming of advanced information assurance concepts. In Proceedings of the DARPA Information Survivability Conference and Exposition, Vol. 2. 112--118."},{"key":"e_1_2_2_150_1","volume-title":"Automation and Human Performance: Theory and Applications","author":"Woods David D.","unstructured":"David D. Woods . 1996. Decomposing automation: Apparent simplicity, real complexity . In Automation and Human Performance: Theory and Applications . CRC Press . David D. Woods. 1996. Decomposing automation: Apparent simplicity, real complexity. In Automation and Human Performance: Theory and Applications. CRC Press."},{"key":"e_1_2_2_151_1","volume-title":"Woods and Erik Hollnagel","author":"David","year":"2006","unstructured":"David D. Woods and Erik Hollnagel . 2006 . Resilience Engineering : Concepts and Precepts. CRC Press . David D. Woods and Erik Hollnagel. 2006. Resilience Engineering: Concepts and Precepts. CRC Press."},{"key":"e_1_2_2_152_1","doi-asserted-by":"publisher","DOI":"10.1080\/15427951.2014.902407"},{"key":"e_1_2_2_153_1","volume-title":"A vine copula model for predicting the effectiveness of cyber defense early-warning. Technometrics 0, ja","author":"Xu Maochao","year":"2017","unstructured":"Maochao Xu , Lei Hua , and Shouhuai Xu. 2017. A vine copula model for predicting the effectiveness of cyber defense early-warning. Technometrics 0, ja ( 2017 ), 0--0. arXiv:http:\/\/dx.doi.org\/10.1080\/00401706.2016.1256841 10.1080\/00401706.2016.1256841 Maochao Xu, Lei Hua, and Shouhuai Xu. 2017. A vine copula model for predicting the effectiveness of cyber defense early-warning. Technometrics 0, ja (2017), 0--0. arXiv:http:\/\/dx.doi.org\/10.1080\/00401706.2016.1256841"},{"key":"e_1_2_2_154_1","doi-asserted-by":"publisher","DOI":"10.1080\/15427951.2012.654480"},{"key":"e_1_2_2_155_1","doi-asserted-by":"publisher","DOI":"10.5555\/1370616.1370617"},{"key":"e_1_2_2_156_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.850156"},{"key":"e_1_2_2_157_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600176.2600190"},{"key":"e_1_2_2_158_1","unstructured":"Shouhuai Xu. 2014b. Cybersecurity dynamics: With application to formulating a cyber defense C2 framework. Presentation at Workshop on Cyber Security: From Tactics to Strategy and Back.  Shouhuai Xu. 2014b. Cybersecurity dynamics: With application to formulating a cyber defense C2 framework. Presentation at Workshop on Cyber Security: From Tactics to Strategy and Back."},{"key":"e_1_2_2_159_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600176.2600189"},{"key":"e_1_2_2_160_1","doi-asserted-by":"publisher","DOI":"10.1080\/15427951.2013.830583"},{"key":"e_1_2_2_161_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348832.2348835"},{"key":"e_1_2_2_162_1","doi-asserted-by":"publisher","DOI":"10.1145\/2555613"},{"key":"e_1_2_2_163_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2011.33"},{"key":"e_1_2_2_164_1","volume-title":"Proceedings of 11th International Conference on Web-Age Information Management (WAIM\u201910)","author":"Xu Shouhuai","unstructured":"Shouhuai Xu , Haifeng Qian , Fengying Wang , Zhenxin Zhan , Elisa Bertino , and Ravi S. Sandhu . 2010. Trustworthy information: Concepts and mechanisms . In Proceedings of 11th International Conference on Web-Age Information Management (WAIM\u201910) . 398--404. Shouhuai Xu, Haifeng Qian, Fengying Wang, Zhenxin Zhan, Elisa Bertino, and Ravi S. Sandhu. 2010. Trustworthy information: Concepts and mechanisms. In Proceedings of 11th International Conference on Web-Age Information Management (WAIM\u201910). 398--404."},{"key":"e_1_2_2_165_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02056-8_11"},{"key":"e_1_2_2_166_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-03549-4_12"},{"key":"e_1_2_2_167_1","doi-asserted-by":"crossref","unstructured":"Y. Yusuf M. Sarhadi and A. Gunasekaran. 1999. Agile manufacturing: The drivers concepts and attributes. International Journal of Production Economics 62 (1999).  Y. Yusuf M. Sarhadi and A. Gunasekaran. 1999. Agile manufacturing: The drivers concepts and attributes. International Journal of Production Economics 62 (1999).","DOI":"10.1016\/S0925-5273(98)00219-9"},{"key":"e_1_2_2_168_1","doi-asserted-by":"publisher","DOI":"10.1109\/AIHAS.1990.93914"},{"key":"e_1_2_2_169_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2013.2279800"},{"key":"e_1_2_2_170_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-27998-5_7"},{"key":"e_1_2_2_171_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2015.2422261"},{"key":"e_1_2_2_172_1","doi-asserted-by":"publisher","DOI":"10.1145\/2746194.2746196"},{"key":"e_1_2_2_173_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNSE.2017.2734904"},{"key":"e_1_2_2_174_1","volume-title":"Resilience: Why Thins Bounce Back. Simon 8","author":"Zolli A.","year":"2012","unstructured":"A. Zolli and A. Healy . 2012 . Robust yet fragile. Resilience: Why Thins Bounce Back. Simon 8 Schuster , 25--30. A. Zolli and A. Healy. 2012. Robust yet fragile. Resilience: Why Thins Bounce Back. Simon 8 Schuster, 25--30."}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3277666","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3277666","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:44:26Z","timestamp":1750193066000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3277666"}},"subtitle":["Measuring the Trustworthiness of Computer-Based Systems"],"short-title":[],"issued":{"date-parts":[[2019,2,4]]},"references-count":171,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2019,11,30]]}},"alternative-id":["10.1145\/3277666"],"URL":"https:\/\/doi.org\/10.1145\/3277666","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,2,4]]},"assertion":[{"value":"2017-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-02-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}