{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T23:46:42Z","timestamp":1779925602239,"version":"3.53.1"},"publisher-location":"Cham","reference-count":45,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031855924","type":"print"},{"value":"9783031855931","type":"electronic"}],"license":[{"start":{"date-parts":[[2025,1,1]],"date-time":"2025-01-01T00:00:00Z","timestamp":1735689600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T00:00:00Z","timestamp":1741824000000},"content-version":"vor","delay-in-days":71,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>The increasing frequency of attacks on Android applications coupled with the recent popularity of large language models (LLMs) necessitates a comprehensive understanding of the capabilities of the latter in identifying potential vulnerabilities, which is key to mitigate the overall risk. To this end, the work at hand compares the ability of nine state-of-the-art LLMs to detect Android code vulnerabilities listed in the latest Open Worldwide Application Security Project (OWASP) Mobile Top 10. Each LLM was evaluated against an open dataset of over 100 vulnerable code samples, assessing each model\u2019s ability to identify key vulnerabilities. Our analysis reveals the strengths and weaknesses of each LLM, identifying important factors that contribute to their performance. Additionally, we offer insights into context augmentation with retrieval-augmented generation (RAG) for detecting Android code vulnerabilities, which in turn may propel secure application development. Finally, while the reported findings regarding code vulnerability analysis show promise, they also reveal significant discrepancies among the different LLMs.\n<\/jats:p>","DOI":"10.1007\/978-3-031-85593-1_9","type":"book-chapter","created":{"date-parts":[[2025,3,12]],"date-time":"2025-03-12T19:23:56Z","timestamp":1741807436000},"page":"139-154","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Assessing the\u00a0Effectiveness of\u00a0LLMs in\u00a0Android Application Vulnerability Analysis"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4233-5998","authenticated-orcid":false,"given":"Vasileios","family":"Kouliaridis","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0142-7503","authenticated-orcid":false,"given":"Georgios","family":"Karopoulos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6348-5031","authenticated-orcid":false,"given":"Georgios","family":"Kambourakis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,3,13]]},"reference":[{"key":"9_CR1","unstructured":"Lookout: Mobile Threat Landscape Report: 2023 in Review (2023). https:\/\/www.lookout.com\/threat-intelligence\/report\/mobile-landscape-threat-report. Accessed 20 Aug 2024"},{"key":"9_CR2","unstructured":"OWASP: Mobile Top 10 2024: Final Release Updates (2024). https:\/\/owasp.org\/www-project-mobile-top-10\/. Accessed 20 Aug 2024"},{"issue":"8","key":"9_CR3","doi-asserted-by":"publisher","first-page":"457","DOI":"10.3390\/info14080457","volume":"14","author":"V Kouliaridis","year":"2023","unstructured":"Kouliaridis, V., Karopoulos, G., Kambourakis, G.: Assessing the security and privacy of android official id wallet apps. Information 14(8), 457 (2023)","journal-title":"Information"},{"issue":"1","key":"9_CR4","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1017\/S1351324916000334","volume":"23","author":"Kenneth Ward Church","year":"2017","unstructured":"Kenneth Ward Church: Word2vec. Nat. Lang. Eng. 23(1), 155\u2013162 (2017)","journal-title":"Nat. Lang. Eng."},{"key":"9_CR5","unstructured":"Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding (2019)"},{"key":"9_CR6","unstructured":"Solaiman, I., et al.: Release strategies and the social impacts of language models (2019)"},{"key":"9_CR7","unstructured":"OpenAI: OpenAI (2024). https:\/\/openai.com\/. Accessed 20 Aug 2024"},{"key":"9_CR8","unstructured":"Brown, T.B., et\u00a0al.: Language models are few-shot learners (2020)"},{"key":"9_CR9","doi-asserted-by":"crossref","unstructured":"Wan, Y., Zhao, W., Zhang, H., Sui, Y., Xu, G., Jin, H.: What do they capture? A structural analysis of pre-trained language models for source code. In: Proceedings of the 44th International Conference on Software Engineering, ICSE 2022, pp. 2377\u20132388. Association for Computing Machinery, New York (2022)","DOI":"10.1145\/3510003.3510050"},{"key":"9_CR10","unstructured":"Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by chatGPT really correct? Rigorous evaluation of large language models for code generation (2023)"},{"key":"9_CR11","unstructured":"Bearer. https:\/\/github.com\/Bearer\/bearer. Accessed 20 Aug 2024"},{"key":"9_CR12","unstructured":"MobSFscan. https:\/\/github.com\/MobSF\/mobsfscan. Accessed 20 Aug 2024"},{"key":"9_CR13","doi-asserted-by":"publisher","first-page":"3421","DOI":"10.1007\/s10586-023-04124-5","volume":"26","author":"M Al-Hawawreh","year":"2023","unstructured":"Al-Hawawreh, M., Aljuhani, A., Jararweh, Y.: ChatGPT for cybersecurity: practical applications, challenges, and future directions. Clust. Comput. 26, 3421\u20133436 (2023)","journal-title":"Clust. Comput."},{"issue":"2","key":"9_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.hcc.2024.100211","volume":"4","author":"Y Yao","year":"2024","unstructured":"Yao, Y., Duan, J., Kaidi, X., Cai, Y., Sun, Z., Zhang, Y.: A survey on large language model (LLM) security and privacy: the good, the bad, and the ugly. High-Confid. Comput. 4(2), 100211 (2024)","journal-title":"High-Confid. Comput."},{"key":"9_CR15","doi-asserted-by":"publisher","first-page":"80218","DOI":"10.1109\/ACCESS.2023.3300381","volume":"11","author":"M Gupta","year":"2023","unstructured":"Gupta, M., Akiri, C., Aryal, K., Parker, E., Praharaj, L.: From chatGPT to threatGPT: impact of generative AI in cybersecurity and privacy. IEEE Access 11, 80218\u201380245 (2023)","journal-title":"IEEE Access"},{"key":"9_CR16","doi-asserted-by":"crossref","unstructured":"Motlagh, F.N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., Meinel, C.: Large language models in cybersecurity: State-of-the-art (2024)","DOI":"10.5220\/0013377600003899"},{"key":"9_CR17","doi-asserted-by":"crossref","unstructured":"Thapa, C., Jang, S.I., Ahmed, M.E., Camtepe, S., Pieprzyk, J., Nepal, S.: Transformer-based language models for software vulnerability detection. In: Proceedings of the 38th Annual Computer Security Applications Conference, ACSAC 2022, pp. 481\u2013496. Association for Computing Machinery, New York (2022)","DOI":"10.1145\/3564625.3567985"},{"key":"9_CR18","doi-asserted-by":"crossref","unstructured":"Liu, Z., Liao, Q., Gu, W., Gao, C.: Software vulnerability detection with GPT and in-context learning. In: 2023 8th International Conference on Data Science in Cyberspace (DSC), pp. 229\u2013236 (2023)","DOI":"10.1109\/DSC59305.2023.00041"},{"key":"9_CR19","doi-asserted-by":"crossref","unstructured":"Purba, M.D., Ghosh, A., Radford, B.J., Chu, B.: Software vulnerability detection using large language models. In: 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW), pp. 112\u2013119 (2023)","DOI":"10.1109\/ISSREW60843.2023.00058"},{"key":"9_CR20","unstructured":"Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., Dolan-Gavitt, B.: Lost at C: a user study on the security implications of large language model code assistants. In: 32nd USENIX Security Symposium (USENIX Security 2023), Anaheim, CA, August 2023, pp. 2205\u20132222. USENIX Association (2023)"},{"key":"9_CR21","unstructured":"Liu, P., et al.: Harnessing the power of LLM to support binary taint analysis (2023)"},{"key":"9_CR22","unstructured":"Wang, J., Huang, Z., Liu, H., Yang, N., Xiao, Y.: DefectHunter: a novel LLM-driven boosted-conformer-based code vulnerability detection mechanism (2023)"},{"key":"9_CR23","unstructured":"Cheshkov, A., Zadorozhny, P., Levichev, R.: Evaluation of chatGPT model for vulnerability detection (2023)"},{"key":"9_CR24","doi-asserted-by":"crossref","unstructured":"Noever, D.: Can large language models find and fix vulnerable software? (2023)","DOI":"10.5121\/ijaia.2023.14301"},{"key":"9_CR25","unstructured":"Sun, Y., et al.: LLM4Vuln: a unified evaluation framework for decoupling and enhancing LLMs\u2019 vulnerability reasoning (2024)"},{"key":"9_CR26","doi-asserted-by":"crossref","unstructured":"Hu, S.,\u00a0Huang, T.,\u00a0Ilhan, F.,\u00a0Tekin, S.,\u00a0Liu, L.: Large language model-powered smart contract vulnerability detection: new perspectives. In: 2023 5th IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), Los Alamitos, CA, USA, November 2023, pp. 297\u2013306. IEEE Computer Society (2023)","DOI":"10.1109\/TPS-ISA58951.2023.00044"},{"key":"9_CR27","doi-asserted-by":"crossref","unstructured":"Senanayake, J., Kalutarage, H., Al-Kadri, M.O., Piras, L., Petrovski, A.: Labelled vulnerability dataset on android source code (LVDAndro) to develop AI-based code vulnerability detection models. In: Proceedings of the 20th International Conference on Security and Cryptography - SECRYPT, pp. 659\u2013666. INSTICC. SciTePress (2023)","DOI":"10.5220\/0012060400003555"},{"key":"9_CR28","unstructured":"Vulcorpus-2024. https:\/\/github.com\/billkoul\/vulcorpus-2024. Accessed 20 Aug 2024"},{"key":"9_CR29","unstructured":"Android developers - manifest permissions. https:\/\/developer.android.com\/reference\/android\/Manifest.permission#ACCESS_FINE_LOCATION. Accessed 20 Aug 2024"},{"key":"9_CR30","unstructured":"Android developers - mediastore. https:\/\/developer.android.com\/reference\/android\/provider\/MediaStore#ACTION_IMAGE_CAPTURE. Accessed 20 Aug 2024"},{"key":"9_CR31","unstructured":"Android developers - intent. https:\/\/developer.android.com\/reference\/android\/content\/Intent#ACTION_OPEN_DOCUMENT. Accessed 20 Aug 2024"},{"key":"9_CR32","unstructured":"What we know about the XZ utils backdoor that almost infected the world. https:\/\/arstechnica.com\/security\/2024\/04\/what-we-know-about-the-xz-utils-backdoor-that-almost-infected-the-world\/. Accessed 17 Apr 2024"},{"key":"9_CR33","unstructured":"OpenAI: GPT-4 is OpenAI\u2019s most advanced system, producing safer and more useful responses (2024). Accessed 20 Aug 2024"},{"key":"9_CR34","unstructured":"OpenAI: GPT-4 technical report (2024)"},{"key":"9_CR35","unstructured":"Touvron, H., et\u00a0al.: Llama 2: open foundation and fine-tuned chat models (2023)"},{"key":"9_CR36","unstructured":"Tunstall, L., et al.: Zephyr: direct distillation of LM alignment (2023)"},{"key":"9_CR37","unstructured":"Nous Hermes 2 mixtral 8X7B DPO. https:\/\/huggingface.co\/NousResearch\/Nous-Hermes-2-Mixtral-8x7B-DPO. Accessed 20 Aug 2024"},{"key":"9_CR38","unstructured":"Lian, W., et al.: MistralOrca: Mistral-7B model instruct-tuned on filtered openOrcaV1 GPT-4 dataset (2023). Accessed 20 Aug 2024"},{"key":"9_CR39","unstructured":"Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., Awadallah, A.: Orca: progressive learning from complex explanation traces of GPT-4 (2023)"},{"key":"9_CR40","unstructured":"Longpre, S., et al.: The flan collection: designing data and methods for effective instruction tuning (2023)"},{"key":"9_CR41","unstructured":"Code llama. https:\/\/ai.meta.com\/blog\/code-llama-large-language-model-coding\/. Accessed 20 Aug 2024"},{"key":"9_CR42","unstructured":"Llamaindex. https:\/\/docs.llamaindex.ai\/en\/stable\/. Accessed 20 Aug 2024"},{"key":"9_CR43","unstructured":"BAAI\/bge-small-en-v1.5. https:\/\/huggingface.co\/BAAI\/bge-small-en-v1.5. Accessed 20 Aug 2024"},{"key":"9_CR44","unstructured":"Security guidelines. https:\/\/developer.android.com\/privacy-and-security\/security-tips. Accessed 20 Aug 2024"},{"key":"9_CR45","unstructured":"MobSF. https:\/\/github.com\/MobSF\/Mobile-Security-Framework-MobSF. Accessed 20 Aug 2024"}],"container-title":["Lecture Notes in Computer Science","Attacks and Defenses for the Internet-of-Things"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-85593-1_9","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,12]],"date-time":"2025-03-12T19:24:09Z","timestamp":1741807449000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-85593-1_9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025]]},"ISBN":["9783031855924","9783031855931"],"references-count":45,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-85593-1_9","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025]]},"assertion":[{"value":"13 March 2025","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"ADIoT","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Attacks and Defenses for Internet-of-Things","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Hanghzou","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"China","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"14 December 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"15 December 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"7","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"adiot2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/adiot.compute.dtu.dk\/2024\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}