{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T05:22:29Z","timestamp":1791436949771,"version":"4.3.3"},"reference-count":89,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,4,27]],"date-time":"2023-04-27T00:00:00Z","timestamp":1682553600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>\n            Producing secure software is challenging. The poor usability of security Application Programming Interfaces (APIs) makes this even harder. Many recommendations have been proposed to support developers by improving the usability of cryptography libraries\u2014rooted in wider\n            <jats:italic>best practice<\/jats:italic>\n            guidance in software engineering and API design. In this SLR, we systematize knowledge regarding these recommendations. We identify and analyze 65 papers, offering 883 recommendations. Through thematic analysis, we identify seven core ways to improve usability of APIs. Most of the recommendations focus on helping API developers to\n            <jats:italic>construct<\/jats:italic>\n            and\n            <jats:italic>structure<\/jats:italic>\n            their code and make it more usable and easier for programmers to\n            <jats:italic>understand<\/jats:italic>\n            . There is less focus, however, on\n            <jats:italic>documentation<\/jats:italic>\n            ,\n            <jats:italic>writing requirements<\/jats:italic>\n            ,\n            <jats:italic>code quality assessment<\/jats:italic>\n            , and the impact of\n            <jats:italic>organizational software development practices<\/jats:italic>\n            . By tracing and analyzing paper ancestry, we map how this knowledge becomes validated and translated over time. We find that very few API usability recommendations are empirically validated, and that recommendations specific to usable security APIs lag even further behind.\n          <\/jats:p>","DOI":"10.1145\/3561383","type":"journal-article","created":{"date-parts":[[2022,9,15]],"date-time":"2022-09-15T09:53:45Z","timestamp":1663235625000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["SLR: From Saltzer and Schroeder to 2021\u202647 Years of Research on the Development and Validation of Security API Recommendations"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8055-3750","authenticated-orcid":false,"given":"Nikhil","family":"Patnaik","sequence":"first","affiliation":[{"name":"University of Bristol, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2207-6834","authenticated-orcid":false,"given":"Andrew","family":"Dwyer","sequence":"additional","affiliation":[{"name":"University of Durham, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6146-9852","authenticated-orcid":false,"given":"Joseph","family":"Hallett","sequence":"additional","affiliation":[{"name":"University of Bristol, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0109-1341","authenticated-orcid":false,"given":"Awais","family":"Rashid","sequence":"additional","affiliation":[{"name":"University of Bristol, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,4,27]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.52"},{"key":"e_1_3_1_3_2","first-page":"03\u2013CP","article-title":"Achieving saturation in thematic analysis: Development and refinement of a codebook","volume":"3","author":"Ando Hikari","year":"2014","unstructured":"Hikari Ando, Rosanna Cousins, and Carolyn Young. 2014. Achieving saturation in thematic analysis: Development and refinement of a codebook. Comprehens. Psychol. 3 (2014), 03\u2013CP.","journal-title":"Comprehens. Psychol."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvlc.2004.08.003"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2005.103"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.5555\/556709"},{"key":"e_1_3_1_7_2","first-page":"281","volume-title":"Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918)","author":"Assal Hala","year":"2018","unstructured":"Hala Assal and Sonia Chiasson. 2018. Security in the software development lifecycle. In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918). USENIX, 281\u2013296."},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2008.4639084"},{"key":"e_1_3_1_9_2","first-page":"207","article-title":"Introduction to modern cryptography","volume":"207","author":"Bellare Mihir","year":"2005","unstructured":"Mihir Bellare and Phillip Rogaway. 2005. Introduction to modern cryptography. UCSD CSE 207 (2005), 207.","journal-title":"UCSD CSE"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33481-8_9"},{"key":"e_1_3_1_11_2","volume-title":"Effective Java","author":"Bloch Joshua","year":"2001","unstructured":"Joshua Bloch. 2001. Effective Java. Pearson Education."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/1176617.1176622"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1137627.1137631"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.68"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2014.15"},{"key":"e_1_3_1_16_2","unstructured":"BSIMM. 2009. Retrieved from https:\/\/www.bsimm.com."},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/146802.146834"},{"key":"e_1_3_1_18_2","first-page":"359","volume-title":"Proceedings of the 1st Joint Conference of EASE PPIG (PPIG\u201903)","author":"Clarke Steven","year":"2003","unstructured":"Steven Clarke and Curtis Becker. 2003. Using the cognitive dimensions framework to evaluate the usability of a class library. In Proceedings of the 1st Joint Conference of EASE PPIG (PPIG\u201903). EASE, 359\u2013366."},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_1_20_2","volume-title":"Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries","author":"Cwalina Krzysztof","year":"2008","unstructured":"Krzysztof Cwalina and Brad Abrams. 2008. Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries. Pearson Education."},{"key":"e_1_3_1_21_2","volume-title":"Proceedings of EclipseCon","author":"Rivi\u00e8res Jim des","year":"2004","unstructured":"Jim des Rivi\u00e8res. 2004. Eclipse APIs: Lines in the sand. In Proceedings of EclipseCon."},{"key":"e_1_3_1_22_2","unstructured":"Anthony Desnos et\u00a0al. 2011. Androguard. https:\/\/buildmedia.readthedocs.org\/media\/pdf\/androguard\/v3.1.0-rc2\/androguard.pdf."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/2508859.2516693"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/2382196.2382205"},{"key":"e_1_3_1_25_2","volume-title":"Refactoring: Improving the Design of Existing Code","author":"Fowler Martin","year":"2018","unstructured":"Martin Fowler. 2018. Refactoring: Improving the Design of Existing Code. Addison-Wesley Professional."},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.5555\/646151.679366"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/2382196.2382204"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2017.24"},{"key":"e_1_3_1_29_2","volume-title":"Inside Java 2 Platform Security: Architecture, API Design, and Implementation (2nd ed.)","author":"Gong Li","year":"2003","unstructured":"Li Gong and Gary Ellison. 2003. Inside Java 2 Platform Security: Architecture, API Design, and Implementation (2nd ed.). Pearson Education."},{"key":"e_1_3_1_30_2","first-page":"265","volume-title":"Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918)","author":"Gorski Peter Leo","year":"2018","unstructured":"Peter Leo Gorski, Luigi Lo Iacono, Dominik Wermke, Christian Stransky, Sebastian M\u00f6ller, Yasemin Acar, and Sascha Fahl. 2018. Developers deserve security warnings, too: On the effect of integrated security advice on cryptographic API misuse. In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918). USENIX, 265\u2013281."},{"key":"e_1_3_1_31_2","volume-title":"The Java Language Specification","author":"Gosling James","year":"2000","unstructured":"James Gosling, Bill Joy, Guy Steele, and Gilad Bracha. 2000. The Java Language Specification. Addison-Wesley Professional."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2016.111"},{"key":"e_1_3_1_33_2","first-page":"443","article-title":"Cognitive dimensions of notations","volume":"5","author":"Green Thomas R. G.","year":"1989","unstructured":"Thomas R. G. Green. 1989. Cognitive dimensions of notations. People Comput. V 5 (1989), 443\u2013460.","journal-title":"People Comput. V"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1006\/jvlc.1996.0009"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-34347-6_10"},{"key":"e_1_3_1_36_2","unstructured":"Peter Gutmann. 1995. Cryptlib security toolkit: User\u2019s guide and manual. https:\/\/cryptlib.com\/downloads\/manual.pdf."},{"key":"e_1_3_1_37_2","first-page":"1","volume-title":"Proceedings of the USENIX Security Symposium","author":"Gutmann Peter","year":"1999","unstructured":"Peter Gutmann. 1999. The design of a cryptographic security architecture. In Proceedings of the USENIX Security Symposium. USENIX, 1\u201316."},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.5555\/647253.720291"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2007.70754"},{"key":"e_1_3_1_40_2","first-page":"205","volume-title":"Proceedings of the 21st USENIX Security Symposium (USENIX Security\u201912)","author":"Heninger Nadia","year":"2012","unstructured":"Nadia Heninger, Zakir Durumeric, Eric Wustrow, and J. Alex Halderman. 2012. Mining your Ps and Qs: Detection of widespread weak keys in network devices. In Proceedings of the 21st USENIX Security Symposium (USENIX Security\u201912). USENIX, 205\u2013220."},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/1255421.1255422"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.52776"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/CMPSAC.1989.65148"},{"key":"e_1_3_1_44_2","unstructured":"Mathieu Jacques. 2004. API usability: Guidelines to improve your code ease of use. Retrieved from http:\/\/www.codeproject.com\/KB\/usability\/APIUsabilityArticle.aspx."},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/2602649.2602816"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/1166324.1166347"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jvlc.2004.08.003"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2004.47"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.2307\/2529786"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSAC.2004.41"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/52.676963"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.21236\/ADA443493"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2006.109"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2018.00039"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180201"},{"key":"e_1_3_1_56_2","unstructured":"Microsoft. 2002. Security Development Lifecycle. Retrieved from https:\/\/www.microsoft.com\/en-us\/sdl."},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/QRS.2018.00028"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/77481.77486"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/2896587"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884790"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/191666.191729"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.1996.8740869"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97281"},{"issue":"4","key":"e_1_3_1_64_2","first-page":"109","article-title":"Introducing API design principles in CS2","volume":"24","author":"Ni\u00f1o Jaime","year":"2009","unstructured":"Jaime Ni\u00f1o. 2009. Introducing API design principles in CS2. J. Comput. Small Coll 24, 4 (2009), 109\u2013116.","journal-title":"J. Comput. Small Coll"},{"key":"e_1_3_1_65_2","first-page":"315","volume-title":"Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918)","author":"Oliveira Daniela Seabra","year":"2018","unstructured":"Daniela Seabra Oliveira, Tian Lin, Muhammad Sajidur Rahman, Rad Akefirad, Donovan Ellis, Eliany Perez, Rahul Bobhate, Lois A. DeLong, Justin Cappos, and Yuriy Brun. 2018. API Blindspots: Why experienced developers write vulnerable code. In Proceedings of the 14th Symposium on Usable Privacy and Security (SOUPS\u201918). USENIX, 315\u2013328."},{"key":"e_1_3_1_66_2","first-page":"609","volume-title":"Proceedings of the 26th USENIX Security Symposium (USENIX Security\u201917)","author":"O\u2019Neill Mark","year":"2017","unstructured":"Mark O\u2019Neill, Scott Heidbrink, Scott Ruoti, Jordan Whitehead, Dan Bunker, Luke Dickinson, Travis Hendershot, Joshua Reynolds, Kent Seamons, and Daniel Zappala. 2017. Trustbase: An architecture to repair and strengthen certificate-based authentication. In Proceedings of the 26th USENIX Security Symposium (USENIX Security\u201917). USENIX, 609\u2013624."},{"key":"e_1_3_1_67_2","first-page":"799","volume-title":"Proceedings of the 27th USENIX Security Symposium (USENIX Security\u201918)","author":"O\u2019Neill Mark","year":"2018","unstructured":"Mark O\u2019Neill, Scott Heidbrink, Jordan Whitehead, Tanner Perdue, Luke Dickinson, Torstein Collett, Nick Bonner, Kent Seamons, and Daniel Zappala. 2018. The secure socket API: TLS as an operating system service. In Proceedings of the 27th USENIX Security Symposium (USENIX Security\u201918). USENIX, 799\u2013816."},{"key":"e_1_3_1_68_2","unstructured":"OWASP. 2012. OWASP Developer Guide. Retrieved from https:\/\/www.owasp.org\/index.php\/Category:OWASP_Guide_Project."},{"key":"e_1_3_1_69_2","unstructured":"OWASP. 2012. OWASP SAMM Project. Retrieved from www.owasp.org\/index.php\/OWASP_SAMM_Project."},{"key":"e_1_3_1_70_2","volume-title":"Usability Issues in the Design of Novice Programming Systems","author":"Pane John F.","year":"1996","unstructured":"John F. Pane and Brad A. Myers. 1996. Usability Issues in the Design of Novice Programming Systems. Technical Report. Carnegie-Mellon University."},{"key":"e_1_3_1_71_2","first-page":"245","volume-title":"Proceedings of the 15th Symposium on Usable Privacy and Security (SOUPS\u201919)","author":"Patnaik Nikhil","year":"2019","unstructured":"Nikhil Patnaik, Joseph Hallett, and Awais Rashid. 2019. Usability smells: An analysis of developers\u2019 struggle with crypto libraries. In Proceedings of the 15th Symposium on Usable Privacy and Security (SOUPS\u201919). USENIX, 245\u2013257."},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15327051hci0502&3_3"},{"key":"e_1_3_1_73_2","first-page":"367","article-title":"Coding and analysing qualitative data","volume":"3","author":"Rivas Carol","year":"2012","unstructured":"Carol Rivas. 2012. Coding and analysing qualitative data. Research. Soc. Cult. 3 (2012), 367\u2013392.","journal-title":"Research. Soc. Cult."},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2009.193"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-010-9150-8"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/361011.361067"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/PROC.1975.9939"},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.256942"},{"key":"e_1_3_1_79_2","volume-title":"Enforceable Security Policies","author":"Schneider Fred B.","year":"1999","unstructured":"Fred B. Schneider. 1999. Enforceable Security Policies. Technical Report. Cornell University."},{"key":"e_1_3_1_80_2","unstructured":"R. Seacord. 2018. Top 10 secure coding practices. Retrieved from https:\/\/www.securecoding.cert.org\/confluence\/display\/seccode\/Top+10+Secure+Coding+Practices."},{"key":"e_1_3_1_81_2","volume-title":"The CLASP Application Security Process","author":"Secure Software, Inc","year":"2005","unstructured":"Secure Software, Inc 2005. The CLASP Application Security Process. Secure Software, Inc."},{"key":"e_1_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00766-004-0194-4"},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.1108\/09685220010353178"},{"issue":"11","key":"e_1_3_1_84_2","first-page":"242","article-title":"Designing the star user interface","volume":"7","author":"Smith David C.","year":"1982","unstructured":"David C. Smith, Charles Irby, Ralph Kimball, and Bill Verplank. 1982. Designing the star user interface. BYTE 7, 11 (April1982), 242\u2013282.","journal-title":"BYTE"},{"key":"e_1_3_1_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.85"},{"key":"e_1_3_1_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2007.36"},{"key":"e_1_3_1_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2008.19"},{"key":"e_1_3_1_88_2","first-page":"1","volume-title":"Proceedings of the 29th USENIX Security Symposium (USENIX Security\u201920)","author":"Votipka Daniel","year":"2020","unstructured":"Daniel Votipka, Kelsey R. Fulton, James Parker, Matthew Hou, Michelle L. Mazurek, and Michael Hicks. 2020. Understanding security mistakes developers make: Qualitative analysis from build it, break it, fix it. In Proceedings of the 29th USENIX Security Symposium (USENIX Security\u201920), USENIX, 1\u201318."},{"key":"e_1_3_1_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"},{"issue":"4","key":"e_1_3_1_90_2","first-page":"255","article-title":"What makes APIs difficult to use","volume":"8","author":"Zibran Minhaz","year":"2008","unstructured":"Minhaz Zibran. 2008. What makes APIs difficult to use. Int. J. Comput. Sci. Netw. Secur. 8, 4 (2008), 255\u2013261.","journal-title":"Int. J. Comput. Sci. Netw. Secur."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3561383","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3561383","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:35Z","timestamp":1750186835000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3561383"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,27]]},"references-count":89,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3561383"],"URL":"https:\/\/doi.org\/10.1145\/3561383","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,27]]},"assertion":[{"value":"2021-12-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-15","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}