{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T21:37:54Z","timestamp":1783460274000,"version":"3.55.0"},"reference-count":73,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>\n            While the automated detection of cryptographic API misuses has progressed significantly, its precision diminishes for intricate targets due to the reliance on manually defined patterns. Large Language Models (LLMs) offer a promising context-aware understanding to address this shortcoming, yet the stochastic nature and the hallucination issue pose challenges to their applications in precise security analysis. This paper presents the first systematic study to explore LLMs\u2019 application in cryptographic API misuse detection. Our findings are noteworthy: The instability of directly applying LLMs results in over half of the initial reports being false positives. Despite this, the reliability of LLM-based detection could be significantly enhanced by aligning detection scopes with realistic scenarios and employing a novel\n            <jats:italic toggle=\"yes\">code &amp; analysis<\/jats:italic>\n            validation technique, achieving a nearly 90% detection recall. This improvement substantially surpasses traditional methods and leads to the discovery of previously unknown vulnerabilities in established benchmarks. Nevertheless, we identify recurring failure patterns that illustrate current LLMs\u2019 blind spots, including cryptographic knowledge deficiencies and code semantics misinterpretations. Leveraging these findings, we deploy an LLM-based detection system and uncover 63 new vulnerabilities (47 confirmed, 7 already fixed) in open-source Java and Python repositories, including prominent projects like Apache.\n          <\/jats:p>","DOI":"10.1145\/3728875","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"113-136","source":"Crossref","is-referenced-by-count":7,"title":["Beyond Static Pattern Matching? Rethinking Automatic Cryptographic API Misuse Detection in the Era of LLMs"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-7986-8519","authenticated-orcid":false,"given":"Yifan","family":"Xia","sequence":"first","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3905-3456","authenticated-orcid":false,"given":"Zichen","family":"Xie","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7793-7633","authenticated-orcid":false,"given":"Peiyu","family":"Liu","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4763-7354","authenticated-orcid":false,"given":"Kangjie","family":"Lu","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6021-1358","authenticated-orcid":false,"given":"Yan","family":"Liu","sequence":"additional","affiliation":[{"name":"Ant Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1936-2840","authenticated-orcid":false,"given":"Wenhai","family":"Wang","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4268-372X","authenticated-orcid":false,"given":"Shouling","family":"Ji","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2017. List of Rainbow Tables. http:\/\/project- rainbowcrack.com\/table.htm"},{"key":"e_1_2_1_2_1","unstructured":"2024. Apache\u00ae Druid a high performance real-time analytics database. https:\/\/druid.apache.org\/"},{"key":"e_1_2_1_3_1","unstructured":"2024. GitHub Advisory Database. https:\/\/github.com\/advisories"},{"key":"e_1_2_1_4_1","unstructured":"2024. Project planning for developers. https:\/\/github.com\/features\/issues"},{"key":"e_1_2_1_5_1","unstructured":"2024. Scala. https:\/\/www.scala-lang.org\/"},{"key":"e_1_2_1_6_1","unstructured":"2024. Spark. Unified engine for large-scale data analytics. https:\/\/spark.apache.org\/"},{"key":"e_1_2_1_7_1","unstructured":"2024. SpotBugs: Find bugs in Java Programs. https:\/\/spotbugs.github.io\/"},{"key":"e_1_2_1_8_1","volume-title":"Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat.","author":"Achiam Josh","year":"2023","unstructured":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","unstructured":"Sharmin Afrose Sazzadur Rahaman and Danfeng Yao. 2019. CryptoAPI-Bench: A Comprehensive Benchmark on Java Cryptographic API Misuses. In 2019 IEEE Cybersecurity Development (SecDev). 49\u201361. https:\/\/doi.org\/10.1109\/SecDev.2019.00017 10.1109\/SecDev.2019.00017","DOI":"10.1109\/SecDev.2019.00017"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3154717"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP46214.2022.9833582"},{"key":"e_1_2_1_12_1","volume-title":"Computer Security \u2013 ESORICS","author":"Baek Heewon","year":"2024","unstructured":"Heewon Baek, Minwook Lee, and Hyoungshick Kim. 2024. CryptoLLM: Harnessing the\u00a0Power of\u00a0LLMs to\u00a0Detect Cryptographic API Misuse. In Computer Security \u2013 ESORICS 2024, Joaquin Garcia-Alfaro, Rafa\u0142 Kozik, Micha\u0142 Chora\u015b, and Sokratis Katsikas (Eds.). Springer Nature Switzerland, Cham. 353\u2013373. isbn:978-3-031-70879-4"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2016.23418"},{"key":"e_1_2_1_14_1","volume-title":"Yuanzhi Li, and Scott Lundberg.","author":"Bubeck S\u00e9bastien","year":"2023","unstructured":"S\u00e9bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, and Scott Lundberg. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00065"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","unstructured":"Yikang Chen Yibo Liu Ka Wu Duc Le and Sze Chau. 2024. Towards Precise Reporting of Cryptographic Misuses. https:\/\/doi.org\/10.14722\/ndss.2024.241032 10.14722\/ndss.2024.241032","DOI":"10.14722\/ndss.2024.241032"},{"key":"e_1_2_1_17_1","unstructured":"CryptoAPIBench. 2019. Impractical cryptographic operations example: Load URLs as Keystore. https:\/\/github.com\/CryptoGuardOSS\/cryptoapi-bench\/blob\/master\/src\/main\/java\/org\/cryptoapi\/bench\/predictablekeystorepassword\/PredictableKeyStorePasswordABHCase1.java#L26"},{"key":"e_1_2_1_18_1","unstructured":"CryptoAPIBench. 2019. Incomplete cryptographic logic example: Unused Key Specification. https:\/\/github.com\/CryptoGuardOSS\/cryptoapi-bench\/blob\/master\/src\/main\/java\/org\/cryptoapi\/bench\/predictablepbepassword\/PredictablePBEPasswordABHCase1.java#L25"},{"key":"e_1_2_1_19_1","unstructured":"CryptoAPIBench. 2019. Unintended cryptographic misuse example: Insufficient Keysize. https:\/\/github.com\/CryptoGuardOSS\/cryptoapi-bench\/blob\/master\/src\/main\/java\/org\/cryptoapi\/bench\/predictablepbepassword\/PredictablePBEPasswordABHCase1.java#L24"},{"key":"e_1_2_1_20_1","unstructured":"CryptoGPT-Online-Appendix. 2023. Comparison among SATs and LLM-based Detection.. https:\/\/cryptogpt-online.github.io\/cmp_sat.html"},{"key":"e_1_2_1_21_1","unstructured":"CryptoGPT-Online-Appendix. 2023. Context-specific Dummy Condition Statement in MASC Benchmark.. https:\/\/cryptogpt-online.github.io\/code.html##Example6"},{"key":"e_1_2_1_22_1","unstructured":"CryptoGPT-Online-Appendix. 2023. Context-specific Secure SHA-1 Usage.. https:\/\/cryptogpt-online.github.io\/code.html##Example7"},{"key":"e_1_2_1_23_1","unstructured":"CryptoGPT-Online-Appendix. 2023. Implicit Usage of Insecure Parameters Case in MASC Benchmark.. https:\/\/cryptogpt-online.github.io\/code.html##Example5"},{"key":"e_1_2_1_24_1","unstructured":"CryptoGPT-Online-Appendix. 2023. Insecure Parameters Passed through Method Chaining Case in MASC Benchmark.. https:\/\/cryptogpt-online.github.io\/code.html##Example3"},{"key":"e_1_2_1_25_1","unstructured":"CryptoGPT-Online-Appendix. 2023. Insecure Parameters Replacement Case in MASC Benchmark.. https:\/\/cryptogpt-online.github.io\/code.html##Example4"},{"key":"e_1_2_1_26_1","unstructured":"CryptoGPT-Online-Appendix. 2023. LLMs\u2019 Common Misunderstanding categories.. https:\/\/cryptogpt-online.github.io\/misunder.html"},{"key":"e_1_2_1_27_1","unstructured":"CryptoGPT-Online-Appendix. 2023. Prompt design and LLM implementation.. https:\/\/cryptogpt-online.github.io\/index.html"},{"key":"e_1_2_1_28_1","unstructured":"CryptoGPT-Online-Appendix. 2023. User Study Extension.. https:\/\/cryptogpt-online.github.io\/user.html"},{"key":"e_1_2_1_29_1","unstructured":"CryptoGPT-Online-Appendix. 2023. The Vulnerable Customized Encryption in Apache Projects.. https:\/\/cryptogpt-online.github.io\/code.html##Example1"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508859.2516693"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00373"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2382196.2382205"},{"key":"e_1_2_1_33_1","unstructured":"Chongzhou Fang Ning Miao Shaurya Srivastav Jialin Liu Ruoyu Zhang Ruijie Fang Asmita Asmita Ryan Tsang Najmeh Nazari and Han Wang. 2023. Large language models for code analysis: Do llms really do their job? arXiv preprint arXiv:2310.12357."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3377182"},{"key":"e_1_2_1_35_1","volume-title":"25th USENIX Security Symposium (USENIX Security 16)","author":"Garman Christina","year":"2016","unstructured":"Christina Garman, Matthew Green, Gabriel Kaptchuk, Ian Miers, and Michael Rushanan. 2016. Dancing on the lip of the volcano: Chosen ciphertext attacks on apple $iMessage$. In 25th USENIX Security Symposium (USENIX Security 16). 655\u2013672."},{"key":"e_1_2_1_36_1","unstructured":"Google. 2023. Gemini. https:\/\/gemini.google.com\/app"},{"key":"e_1_2_1_37_1","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Y Wu and YK Li. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming\u2013The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695513"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2017.8115707"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2948910"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3649828"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3564625.3567989"},{"key":"e_1_2_1_43_1","volume-title":"Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Processing Systems. https:\/\/openreview.net\/forum?id=1qvx610Cu7","author":"Liu Jiawei","year":"2023","unstructured":"Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Thirty-seventh Conference on Neural Information Processing Systems. https:\/\/openreview.net\/forum?id=1qvx610Cu7"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/3698900.3698946"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180201"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3579856.3582830"},{"key":"e_1_2_1_47_1","unstructured":"NIST.. 2017. SP 800-63B-3 \u2013 Digital Identity Guidelines Authentication and Lifecycle Management.."},{"key":"e_1_2_1_48_1","unstructured":"OpenAI. 2023. Chatgpt. https:\/\/chat.openai.com\/"},{"key":"e_1_2_1_49_1","unstructured":"Oracle. 2021. Java cryptography architecture. https:\/\/docs.oracle.com\/javase\/9\/security\/java-cryptography-architecture-jca-reference-guide.htm"},{"key":"e_1_2_1_50_1","unstructured":"Oracle. 2024. Classes and interfaces for cryptographic operations. https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/javax\/crypto\/package-summary.html"},{"key":"e_1_2_1_51_1","volume-title":"Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35 (2022), 27730\u201327744."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.trustnlp-1.14"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP46215.2023.10179420"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.5555\/3618408.3619552"},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 1639\u20131650","author":"Garc\u00eda Cesar Pereida","year":"2016","unstructured":"Cesar Pereida Garc\u00eda, Billy Bob Brumley, and Yuval Yarom. 2016. Make sure DSA signing exponentiations really are constant-time. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 1639\u20131650."},{"key":"e_1_2_1_56_1","volume-title":"2021 IEEE Symposium on Security and Privacy (SP). 1972\u20131989","author":"Piccolboni Luca","year":"2021","unstructured":"Luca Piccolboni, Giuseppe Di Guglielmo, Luca P Carloni, and Simha Sethumadhavan. 2021. Crylogger: Detecting crypto misuses dynamically. In 2021 IEEE Symposium on Security and Privacy (SP). 1972\u20131989."},{"key":"e_1_2_1_57_1","unstructured":"PyCryptodome. 2024. Cryptographic library for Python. https:\/\/pypi.org\/project\/pycryptodome\/"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3319535.3345659"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","unstructured":"Sazzadur Rahaman and Danfeng Daphne Yao. 2017. Program Analysis of Cryptographic Implementations for Security. In 2017 IEEE Cybersecurity Development (SecDev). 61\u201368. https:\/\/doi.org\/10.1109\/SecDev.2017.23 10.1109\/SecDev.2017.23","DOI":"10.1109\/SecDev.2017.23"},{"key":"e_1_2_1_60_1","volume-title":"Yossi Adi, Jingyu Liu, Tal Remez, and J\u00e9r\u00e9my Rapin.","author":"Roziere Baptiste","year":"2023","unstructured":"Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, and J\u00e9r\u00e9my Rapin. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950."},{"key":"e_1_2_1_61_1","volume-title":"The 19th International Conference on Mining Software Repositories.","author":"Schlichtig Michael","year":"2022","unstructured":"Michael Schlichtig, Anna-Katharina Wickert, Stefan Kr\u00fcger, Eric Bodden, and Mira Mezini. 2022. CamBench\u2013Cryptographic API Misuse Detection Tool Benchmark Suite. The 19th International Conference on Mining Software Repositories."},{"key":"e_1_2_1_62_1","doi-asserted-by":"crossref","unstructured":"Marc Stevens Elie Bursztein Pierre Karpman Ange Albertini and Yarik Markov. 2017. The first collision for full SHA-1. Cryptology ePrint Archive Paper 2017\/190. https:\/\/eprint.iacr.org\/2017\/190","DOI":"10.1007\/978-3-319-63688-7_19"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2302.13971"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978406"},{"key":"e_1_2_1_65_1","volume-title":"\u0141 ukasz Kaiser, and Illia Polosukhin","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30 (2017)."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-emnlp.217"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3475716.3484195"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00129"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680323"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASE49443.2020.00029"},{"key":"e_1_2_1_71_1","volume-title":"CryptoREX: Large-scale Analysis of Cryptographic Misuse in IoT Devices. In 22nd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2019","author":"Zhang Li","year":"2019","unstructured":"Li Zhang, Jiongyi Chen, Wenrui Diao, Shanqing Guo, Jian Weng, and Kehuan Zhang. 2019. CryptoREX: Large-scale Analysis of Cryptographic Misuse in IoT Devices. In 22nd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2019). USENIX Association, Chaoyang District, Beijing. 151\u2013164. isbn:978-1-939133-07-6 https:\/\/www.usenix.org\/conference\/raid2019\/presentation\/zhang-li"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3150302"},{"key":"e_1_2_1_73_1","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang and Zican Dong. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223."}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728875","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:54:26Z","timestamp":1752684866000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728875"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":73,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728875"],"URL":"https:\/\/doi.org\/10.1145\/3728875","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}