{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T03:46:09Z","timestamp":1782877569171,"version":"3.54.5"},"reference-count":78,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA1","license":[{"start":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T00:00:00Z","timestamp":1714348800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2024,4,29]]},"abstract":"<jats:p>While deep neural networks provide state-of-the-art solutions to a wide range of programming language tasks, their effectiveness in dealing with foundational program analysis tasks remains under explored. In this paper, we present an empirical study that evaluates four prominent models of code (i.e., CuBERT, CodeBERT, GGNN, and Graph Sandwiches) in two such foundational tasks: (1) alias prediction, in which models predict whether two pointers must alias, may alias or must not alias; and (2) equivalence prediction, in which models predict whether or not two programs are semantically equivalent. At the core of this study is CodeSem, a dataset built upon the source code of real-world flagship software (e.g., Linux Kernel, GCC, MySQL) and manually validated for the two prediction tasks.  \nResults show that all models are accurate in both prediction tasks, especially CuBERT with an accuracy of 89% and 84% in alias prediction and equivalence prediction, respectively. We also conduct a comprehensive, in-depth analysis of the results of all models in both tasks, concluding that deep learning models are generally capable of performing foundational tasks in program analysis even though in specific cases their weaknesses are also evident.<\/jats:p>\n          <jats:p>Our code and evaluation data are publicly available at https:\/\/github.com\/CodeSemDataset\/CodeSem.<\/jats:p>","DOI":"10.1145\/3649829","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T17:53:50Z","timestamp":1714413230000},"page":"500-528","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Evaluating the Effectiveness of Deep Learning Models for Foundational Program Analysis Tasks"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-0312-0571","authenticated-orcid":false,"given":"Qian","family":"Chen","sequence":"first","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9194-0656","authenticated-orcid":false,"given":"Chenyang","family":"Yu","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-7233-9272","authenticated-orcid":false,"given":"Ruyan","family":"Liu","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2848-6108","authenticated-orcid":false,"given":"Chi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7216-6929","authenticated-orcid":false,"given":"Yu","family":"Wang","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0844-5023","authenticated-orcid":false,"given":"Ke","family":"Wang","sequence":"additional","affiliation":[{"name":"Visa Research, Palo Alto, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1628-9796","authenticated-orcid":false,"given":"Ting","family":"Su","sequence":"additional","affiliation":[{"name":"East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4794-1652","authenticated-orcid":false,"given":"Linzhang","family":"Wang","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,4,29]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1711.00740"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290353"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/199448.199466"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409757"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3088515.3088517"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553380"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549123"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2940179"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1406.1078"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3314221.3314596"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3037697.3037754"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2003.10555"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-71237-6_7"},{"key":"e_1_2_2_14_1","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1). 4171\u20134186. https:\/\/aclweb.org\/anthology\/papers\/N\/N19\/N19-1423\/","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1). 4171\u20134186. https:\/\/aclweb.org\/anthology\/papers\/N\/N19\/N19-1423\/"},{"key":"e_1_2_2_15_1","volume-title":"International Conference on Learning Representations (ICLR \u201919)","author":"Dinella Elizabeth","year":"2019","unstructured":"Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. 2019. Hoppity: Learning Graph Transformations to Detect and Fix Bugs in Programs. In International Conference on Learning Representations (ICLR \u201919)."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642987"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2002.08155"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/24039.24041"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1146238.1146254"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/SANER50967.2021.00053"},{"key":"e_1_2_2_21_1","volume-title":"28th USENIX Security Symposium (USENIX Security 19)","author":"Guo Wenbo","year":"2019","unstructured":"Wenbo Guo, Dongliang Mu, Xinyu Xing, Min Du, and Dawn Song. 2019. $DEEPVSA$: Facilitating Value-set Analysis with Deep Learning for Postmortem Program Analysis. In 28th USENIX Security Symposium (USENIX Security 19). 1787\u20131804."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-94144-8_22"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480881.1480911"},{"key":"e_1_2_2_24_1","volume-title":"International conference on learning representations.","author":"Hellendoorn Vincent J","year":"2019","unstructured":"Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. 2019. Global relational models of source code. In International conference on learning representations."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/93542.93574"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/239912.239913"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","unstructured":"Hamel Husain Ho-Hsiang Wu Tiferet Gazit Miltiadis Allamanis and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 https:\/\/doi.org\/10.48550\/arXiv.1909.09436","DOI":"10.48550\/arXiv.1909.09436"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2007.30"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1572272.1572283"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1019480"},{"key":"e_1_2_2_31_1","volume-title":"Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"5121","author":"Kanade Aditya","year":"2020","unstructured":"Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and Evaluating Contextual Embedding of Source Code. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). PMLR, 5110\u20135121."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178372.3179519"},{"key":"e_1_2_2_33_1","volume-title":"Reformer: The Efficient Transformer. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=rkgNKkHtvB","author":"Kitaev Nikita","year":"2020","unstructured":"Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=rkgNKkHtvB"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3271065"},{"key":"e_1_2_2_35_1","volume-title":"The BSD conference. 5.","author":"Lattner Chris","year":"2008","unstructured":"Chris Lattner. 2008. LLVM and Clang: Next generation compiler technology. In The BSD conference. 5."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/CGO.2004.1281665"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00040"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2102.04664"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","unstructured":"Ruben Martinez-Cantin. 2014. BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization Experimental Design and Bandits. arXiv preprint arXiv:1405.7430 https:\/\/doi.org\/10.48550\/arXiv.1405.7430","DOI":"10.48550\/arXiv.1405.7430"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.10139"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/349299.349314"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3487062"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3385412.3385962"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-49430-8_3"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","unstructured":"Ruchir Puri David S Kung Geert Janssen Wei Zhang Giacomo Domeniconi Vladimir Zolotov Julian Dolby Jie Chen Mihir Choudhury and Lindsey Decker. 2021. CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655 https:\/\/doi.org\/10.48550\/arXiv.2105.12655","DOI":"10.48550\/arXiv.2105.12655"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-22110-1_55"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2009.02.007"},{"key":"e_1_2_2_49_1","volume-title":"McClelland","author":"Rumelhart David E.","year":"1987","unstructured":"David E. Rumelhart and James L. McClelland. 1987. Learning Internal Representations by Error Propagation. 318\u2013362."},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884877"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2451116.2451150"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462183"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2509136.2509509"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3192366.3192418"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/3327757.3327873"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1133255.1134027"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36946-9_8"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","unstructured":"Eric Stoltz Michael P Gerlek and Michael Wolfe. 1994. Extended SSA with factored use-def chains to support optimization and parallelism. In HICSS (2). 43\u201353. https:\/\/doi.org\/10.1109\/HICSS.1994.323280 10.1109\/HICSS.1994.323280","DOI":"10.1109\/HICSS.1994.323280"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510160"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2014.77"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480881.1480915"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3062341.3062359"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-37057-1_15"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","unstructured":"Mircea Trofin Yundi Qian Eugene Brevdo Zinan Lin Krzysztof Choromanski and David Li. 2021. Mlgo: a machine learning guided compiler optimizations framework. arXiv preprint arXiv:2101.04808 https:\/\/doi.org\/10.48550\/arXiv.2101.04808","DOI":"10.48550\/arXiv.2101.04808"},{"key":"e_1_2_2_66_1","volume-title":"\u0141 ukasz Kaiser, and Illia Polosukhin","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998\u20136008."},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1905.11445"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3385412.3385999"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180179"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3428205"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1905.12265"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/207110.207111"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001420.2001440"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/301631.301647"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/2351676.2351725"},{"key":"e_1_2_2_76_1","unstructured":"Micha\u0142 Zalewski. 2016. American Fuzzy Lop-Whitepaper. https:\/\/lcamtuf. coredump. cx\/afl\/technical_details. txt.."},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462159"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/1328438.1328464"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649829","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3649829","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:06Z","timestamp":1750287246000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649829"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,29]]},"references-count":78,"journal-issue":{"issue":"OOPSLA1","published-print":{"date-parts":[[2024,4,29]]}},"alternative-id":["10.1145\/3649829"],"URL":"https:\/\/doi.org\/10.1145\/3649829","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,29]]},"assertion":[{"value":"2024-04-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}