{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T14:02:25Z","timestamp":1785420145261,"version":"3.56.0"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>\n                    Static analyzers are playing crucial roles in helping find programming mistakes and security vulnerabilities. The correctness of their analysis results is crucial for the usability in practice. Otherwise, the potential defects in these analyzers (e.g., implementation errors, improper design choices) could affect the soundness (leading to false negatives) and precision (leading to false positives). However, finding the defects in off-the-shelf static analyzers is challenging because these analyzers usually lack clear and complete specifications, and the results of different analyzers may differ. To this end, this paper designs two novel types of automated oracles to find defects in static analyzers with randomly generated programs. The first oracle is constructed by using dynamic program executions and the second one leverages the inferred static analysis results. We applied these two oracles on three state-of-the-art static analyzers: Clang Static Analyzer (CSA), GCC Static Analyzer (GSA), and Pinpoint. We found 38 unique defects in these analyzers, 28 of which have been confirmed or fixed by the developers. We conducted a case study on these found defects followed by several insights and lessons learned for improving and better understanding static analyzers. We have made all the artifacts publicly available at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/Geoffrey1014\/SA_Bugs\">https:\/\/github.com\/Geoffrey1014\/SA_Bugs<\/jats:ext-link>\n                    for replication and benefit the community.\n                  <\/jats:p>","DOI":"10.1145\/3660781","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"1656-1678","source":"Crossref","is-referenced-by-count":6,"title":["Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-2086-7034","authenticated-orcid":false,"given":"Weigang","family":"He","sequence":"first","affiliation":[{"name":"East China Normal University, Shanghai, China"},{"name":"University of Technology Sydney, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5799-5876","authenticated-orcid":false,"given":"Peng","family":"Di","sequence":"additional","affiliation":[{"name":"Ant Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-6010-893X","authenticated-orcid":false,"given":"Mengli","family":"Ming","sequence":"additional","affiliation":[{"name":"East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7285-289X","authenticated-orcid":false,"given":"Chengyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"ETH Zurich, Zurich, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1628-9796","authenticated-orcid":false,"given":"Ting","family":"Su","sequence":"additional","affiliation":[{"name":"East China Normal University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7223-7843","authenticated-orcid":false,"given":"Shijie","family":"Li","sequence":"additional","affiliation":[{"name":"Ant Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9510-6574","authenticated-orcid":false,"given":"Yulei","family":"Sui","sequence":"additional","affiliation":[{"name":"University of New South Wales, Sydney, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3088515.3088521"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","unstructured":"Alexandra Bugariu Valentin W\u00fcstholz Maria Christakis and Peter M\u00fcller. 2018. Automatically Testing Implementations of Numerical Abstract Domains. In Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering (ASE). 768\u2013778. https:\/\/doi.org\/10.1145\/3238147.3240464 10.1145\/3238147.3240464","DOI":"10.1145\/3238147.3240464"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","unstructured":"Cristian Cadar and Alastair F. Donaldson. 2016. Analysing the Program Analyser. In Proceedings of the 38th International Conference on Software Engineering Companion. 765\u2013768. https:\/\/doi.org\/10.1145\/2889160.2889206 10.1145\/2889160.2889206","DOI":"10.1145\/2889160.2889206"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2408776.2408795"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-68446-4_14"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639220"},{"key":"e_1_3_1_10_2","unstructured":"Clang Static Analyzer. 2023. https:\/\/clang-analyzer.llvm.org\/ Last accessed on 2023-04-10."},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","unstructured":"Pascal Cuoq Benjamin Monate Anne Pacalet Virgile Prevosto John Regehr Boris Yakobowski and Xuejun Yang. 2012. Testing Static Analyzers with Randomly Generated Programs. In NASA Formal Methods. 120\u2013125. https:\/\/doi.org\/10.1007\/978-3-642-28891-3_12 10.1007\/978-3-642-28891-3_12","DOI":"10.1007\/978-3-642-28891-3_12"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","unstructured":"Leonardo De Moura and Nikolaj Bj\u00f8rner. 2008. Z3: An Efficient SMT Solver. In Proceedings of the Theory and Practice of Software 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems. 337\u2013340. https:\/\/doi.org\/10.5555\/1792734.1792766 10.5555\/1792734.1792766","DOI":"10.5555\/1792734.1792766"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598130"},{"key":"e_1_3_1_14_2","unstructured":"GCC Static Analyzer. 2023. https:\/\/gcc.gnu.org\/wiki\/StaticAnalyzer Last accessed on 2023-04-10."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3329667"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.14232\/actacyb.282831"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","unstructured":"Timotej Kapus and Cristian Cadar. 2017. Automatic testing of symbolic execution engines via program generation and differential testing. In 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE). 590\u2013600. https:\/\/doi.org\/10.1109\/ASE.2017.8115669 10.1109\/ASE.2017.8115669","DOI":"10.1109\/ASE.2017.8115669"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330553"},{"key":"e_1_3_1_19_2","unstructured":"Chris Lattner. 2008. LLVM and Clang: Next generation compiler technology. In The BSD conference. 1\u201320."},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","unstructured":"Vu Le Mehrdad Afshari and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). 216\u2013226. https:\/\/doi.org\/10.1145\/2594291.2594334 10.1145\/2594291.2594334","DOI":"10.1145\/2594291.2594334"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/1538788.1538814"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","unstructured":"Zhenmin Li Lin Tan Xuanhui Wang Shan Lu Yuanyuan Zhou and Chengxiang Zhai. 2006. Have things changed now?: an empirical study of bug characteristics in modern open source software. In Proceedings of the 1st Workshop on Architectural and System Support for Improving Software Dependability (ASID). 25\u201333. https:\/\/doi.org\/10.1145\/1181309.1181314 10.1145\/1181309.1181314","DOI":"10.1145\/1181309.1181314"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","unstructured":"Stephan Lipp Sebastian Banescu and Alexander Pretschner. 2022. An Empirical Study on the Effectiveness of Static C Code Analyzers for Vulnerability Detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 544\u2013555. https:\/\/doi.org\/10.1145\/3533767.3534380 10.1145\/3533767.3534380","DOI":"10.1145\/3533767.3534380"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","unstructured":"Jiangchao Liu Jierui Liu Peng Di Diyu Wu Hengjie Zheng Alex Liu and Jingling Xue. 2023. Hybrid Inlining: A Compositional and Context Sensitive Static Analysis Framework. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 114\u2013126. https:\/\/doi.org\/10.48550\/arXiv.2210.14436 10.48550\/arXiv.2210.14436","DOI":"10.48550\/arXiv.2210.14436"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2644805"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2006.114"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213847"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","unstructured":"Jan Midtgaard and Anders M\u00f8ller. 2015. QuickChecking Static Analysis Properties. In 8th IEEE International Conference on Software Testing Verification and Validation ICST 2015 Graz Austria April 13-17 2015. 1\u201310. https:\/\/doi.org\/10.1109\/ICST.2015.7102603 10.1109\/ICST.2015.7102603","DOI":"10.1109\/ICST.2015.7102603"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","unstructured":"Mikhail R. Gadelha Enrico Steffinlongo Lucas C. Cordeiro Bernd Fischer and Denis Nicole. 2019. SMT-Based Refutation of Spurious Bug Reports in the Clang Static Analyzer. In 2019 IEEE\/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 11\u201314. https:\/\/doi.org\/10.1109\/ICSE-Companion.2019.00026 10.1109\/ICSE-Companion.2019.00026","DOI":"10.1109\/ICSE-Companion.2019.00026"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","unstructured":"John Regehr Yang Chen Pascal Cuoq Eric Eide Chucky Ellison and Xuejun Yang. 2012. Test-Case Reduction for C Compiler Bugs. In Conference on Programming Language Design and Implementation (PLDI). 335\u2013346. https:\/\/doi.org\/10.1145\/2254064.2254104 10.1145\/2254064.2254104","DOI":"10.1145\/2254064.2254104"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3188720"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","unstructured":"Qingkai Shi Xiao Xiao Rongxin Wu Jinguo Zhou Gang Fan and Charles Zhang. 2018. Pinpoint: Fast and Precise Sparse Value Flow Analysis for Million Lines of Code. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). 693\u2013706. https:\/\/doi.org\/10.1145\/3296979.3192418 10.1145\/3296979.3192418","DOI":"10.1145\/3296979.3192418"},{"key":"e_1_3_1_33_2","unstructured":"Donna Spencer. 2009. Card sorting: Designing usable categories. Rosenfeld Media."},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","unstructured":"Yulei Sui Peng Di and Jingling Xue. 2016. Sparse Flow-Sensitive Pointer Analysis for Multithreaded Programs. In Proceedings of the 2016 International Symposium on Code Generation and Optimization (CGO). 160\u2013170. https:\/\/doi.org\/10.1145\/2854038.2854043 10.1145\/2854038.2854043","DOI":"10.1145\/2854038.2854043"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","unstructured":"Yulei Sui and Jingling Xue. 2016. SVF: interprocedural static value-flow analysis in LLVM. In Proceedings of the 25th international conference on compiler construction. 265\u2013266. https:\/\/doi.org\/10.1145\/2892208.2892235 10.1145\/2892208.2892235","DOI":"10.1145\/2892208.2892235"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.14232\/actacyb.283176"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9258-8"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","unstructured":"Jubi Taneja Zhengyang Liu and John Regehr. 2020. Testing static analyses for precision and soundness. In Proceedings of the 18th ACM\/IEEE International Symposium on Code Generation and Optimization. 81\u201393. https:\/\/doi.org\/10.1145\/3368826.3377927 10.1145\/3368826.3377927","DOI":"10.1145\/3368826.3377927"},{"key":"e_1_3_1_39_2","unstructured":"The Clang Team. 2024. Available Checkers for the Clang Static Analyzer. https:\/\/clang.llvm.org\/docs\/analyzer\/checkers.html. Accessed: 2024-02-20."},{"key":"e_1_3_1_40_2","unstructured":"The Clang Team. 2024. Debug Checkers for the Clang Static Analyzer. https:\/\/clang.llvm.org\/docs\/analyzer\/developer-docs\/DebugChecks.html. Accessed: 2024-02-20."},{"key":"e_1_3_1_41_2","unstructured":"The Frama-C Team. 2024. Frama-C home page. https:\/\/frama-c.com\/. Accessed: 2024-02-20."},{"key":"e_1_3_1_42_2","unstructured":"The GCC Team. 2024. Gcov \u2013 a Test Coverage Program. https:\/\/gcc.gnu.org\/onlinedocs\/gcc\/Gcov.html. Accessed: 2024-02-20."},{"key":"e_1_3_1_43_2","unstructured":"The GCC Team. 2024. Options That Control GCC Static Analysis. https:\/\/gcc.gnu.org\/onlinedocs\/gcc\/Static-Analyzer-Options.html. Accessed: 2024-02-20."},{"key":"e_1_3_1_44_2","unstructured":"The GCC Team. 2024. Special Functions for Debugging the GCC Static Analyzer. https:\/\/gcc.gnu.org\/onlinedocs\/ gccint\/Debugging-the-Analyzer.html. Accessed: 2024-02-20."},{"key":"e_1_3_1_45_2","unstructured":"The Clang Team. 2019. Clang 9 documentation: LibTooling Retrieved Feb. 2019 from https:\/\/clang.llvm.org\/docs\/LibTooling.html."},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","unstructured":"Junjie Wang Yuchao Huang Song Wang and Qing Wang. 2022. Find bugs in static bug finders. In Proceedings of the 30th IEEE\/ACM International Conference on Program Comprehension (ICPC). 516\u2013527. https:\/\/doi.org\/10.1145\/3377811.3380380 10.1145\/3377811.3380380","DOI":"10.1145\/3377811.3380380"},{"key":"e_1_3_1_47_2","unstructured":"Weigang He Peng Di Mengli Ming Chengyu Zhang Ting Su Shijie Li Yulei Sui. 2024. Supplementary Material. Retrieved 2024-05-03 from https:\/\/github.com\/Geoffrey1014\/SA_Bugs\/blob\/master\/figures\/Supplementary-Materials.pdf."},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","unstructured":"Jingyue Wu Gang Hu Yang Tang and Junfeng Yang. 2013. Effective dynamic detection of alias analysis errors. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE). 279\u2013289. https:\/\/doi.org\/10.1145\/2491411.2491439 10.1145\/2491411.2491439","DOI":"10.1145\/2491411.2491439"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","unstructured":"Xuejun Yang Yang Chen Eric Eide and John Regehr. 2011. Finding and Understanding Bugs in C Compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). 283\u2013294. https:\/\/doi.org\/10.1145\/1993498.1993532 10.1145\/1993498.1993532","DOI":"10.1145\/1993498.1993532"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","unstructured":"Chengyu Zhang Ting Su Yichen Yan Fuyuan Zhang Geguang Pu and Zhendong Su. 2019. Finding and understanding bugs in software model checkers. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE). 763\u2013773. https:\/\/doi.org\/10.1145\/3338906.3338932 10.1145\/3338906.3338932","DOI":"10.1145\/3338906.3338932"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616272"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","unstructured":"Zexin Zhong Jiangchao Liu Diyu Wu Peng Di Yulei Sui and Alex X. Liu. 2022. Field-Based Static Taint Analysis for Industrial Microservices. In 2022 IEEE\/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 149\u2013150. https:\/\/doi.org\/10.1145\/3510457.3513075 10.1145\/3510457.3513075","DOI":"10.1145\/3510457.3513075"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","unstructured":"Zexin Zhong Jiangchao Liu Diyu Wu Peng Di Yulei Sui Alex X. Liu and John C. S. Lui. 2023. Scalable Compositional Static Taint Analysis for Sensitive Data Tracing on Industrial Micro-Services. In 2023 IEEE\/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 110\u2013121. https:\/\/doi.org\/10.1109\/ICSESEIP58684.2023.00015 10.1109\/ICSESEIP58684.2023.00015","DOI":"10.1109\/ICSESEIP58684.2023.00015"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660781","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3660781","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T07:56:22Z","timestamp":1770191782000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660781"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":52,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3660781"],"URL":"https:\/\/doi.org\/10.1145\/3660781","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}