{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T23:16:46Z","timestamp":1776122206740,"version":"3.50.1"},"reference-count":58,"publisher":"Association for Computing Machinery (ACM)","issue":"PLDI","license":[{"start":{"date-parts":[[2023,6,6]],"date-time":"2023-06-06T00:00:00Z","timestamp":1686009600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Research Foundation of Korea","award":["2022R1A2C200366011 and 2021R1A5A1021944"],"award-info":[{"award-number":["2022R1A2C200366011 and 2021R1A5A1021944"]}]},{"name":"Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government","award":["2022-0-00460 and 2023-2020-0-01819"],"award-info":[{"award-number":["2022-0-00460 and 2023-2020-0-01819"]}]},{"name":"Samsung Electronics Co., Ltd","award":["G01210570"],"award-info":[{"award-number":["G01210570"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2023,6,6]]},"abstract":"<jats:p>The conformance testing of programming language implementations is crucial to support correct and consistent execution environments. Because manually maintaining conformance tests for real-world programming languages is cumbersome and labor-intensive, researchers have presented various ways to make conformance tests effective and efficient. One such approach is to use graph coverage, one of the most widely-used coverage criteria, to generate tests that reach different parts of a mechanized language specification. Since mechanized specifications use functions or inductive definitions to describe the semantics of language features, traditional graph coverage criteria for software work as they are. However, they may not produce high-quality conformance tests because language implementations often have specialized execution paths for different features, even when their semantics descriptions use the same functions. Traditional graph coverage may not distinguish test requirements of such language features, which degrades the quality of conformance testing. Similarly, it may not distinguish test requirements of different parts of the same language feature when their semantics descriptions use the same functions.<\/jats:p>\n          <jats:p>We present feature-sensitive (FS) coverage as a novel coverage criterion to generate high-quality conformance tests for language implementations. It is a general extension of graph coverage, refining conventional test requirements using the innermost enclosing language features. We also introduce feature-call-path-sensitive (FCPS) coverage, a variant of FS coverage, and extend both coverage criteria using the \ud835\udc58-limiting approach. To evaluate the effectiveness of the new coverage criteria for language implementations, we apply them to a mechanized specification of JavaScript. We extend JEST, the state-of-the-art JavaScript conformance test synthesizer using coverage-guided mutational fuzzing, with various FS and FCPS coverage criteria. For the latest JavaScript language specification (ES13, 2022), our tool automatically synthesizes 237,981 conformance tests in 50 hours with five coverage criteria. We evaluated the conformance of eight mainstream JavaScript implementations (four engines and four transpilers) with the synthesized conformance tests and discovered bugs in all of them. The tool detected 143 distinct conformance bugs (42 in engines and 101 in transpilers), 85 of which were confirmed by the developers and 83 of which were newly discovered bugs.<\/jats:p>","DOI":"10.1145\/3591240","type":"journal-article","created":{"date-parts":[[2023,6,6]],"date-time":"2023-06-06T20:06:24Z","timestamp":1686081984000},"page":"493-515","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Feature-Sensitive Coverage for Conformance Testing of Programming Language Implementations"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8387-1984","authenticated-orcid":false,"given":"Jihyeok","family":"Park","sequence":"first","affiliation":[{"name":"Korea University, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5766-2035","authenticated-orcid":false,"given":"Dongjun","family":"Youn","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7846-8282","authenticated-orcid":false,"given":"Kanguk","family":"Lee","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0019-9772","authenticated-orcid":false,"given":"Sukyoung","family":"Ryu","sequence":"additional","affiliation":[{"name":"KAIST, South Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,6,6]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Introduction to Software Testing","author":"Ammann Paul","unstructured":"Paul Ammann and Jeff Offutt . 2008. Introduction to Software Testing . Cambridge University Press . isbn:0-52188-038-1 Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing. Cambridge University Press. isbn:0-52188-038-1"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2017.33"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/FormaliSE.2017.10"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3548606.3560624"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10817-009-9148-3"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290357"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2676726.2676982"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3184558.3185969"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00127"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1049\/sej.1994.0025"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213848"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2021.24224"},{"key":"e_1_2_1_13_1","volume-title":"ECMA-262","author":"International ECMA","year":"2022","unstructured":"ECMA International . 2022. ECMA-262 , 13 th edition, ECMAScript \u00ae 2022 Language Specification . https:\/\/262.ecma-international.org\/13.0\/ ECMA International. 2022. ECMA-262, 13th edition, ECMAScript \u00ae 2022 Language Specification. https:\/\/262.ecma-international.org\/13.0\/","edition":"13"},{"key":"e_1_2_1_14_1","unstructured":"ECMA International. 2022. Test262: A conformanece test suite for the latest draft of ECMAScript. https:\/\/github.com\/tc39\/test262 \t\t\t\t  ECMA International. 2022. Test262: A conformanece test suite for the latest draft of ECMAScript. https:\/\/github.com\/tc39\/test262"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3158138"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290379"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3371111"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23263"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460120.3484823"},{"key":"e_1_2_1_20_1","first-page":"92","article-title":"A Data Flow Analysis Approach to Program Testing","volume":"8","author":"Herman PM","year":"1976","unstructured":"PM Herman . 1976 . A Data Flow Analysis Approach to Program Testing . Australian Computer Journal , 8 , 3 (1976), 92 \u2013 96 . PM Herman. 1976. A Data Flow Analysis Approach to Program Testing. Australian Computer Journal, 8, 3 (1976), 92\u201396.","journal-title":"Australian Computer Journal"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-03237-0_17"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635904"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551357.3551381"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550270"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of 19th International Workshop on Foundations of Object-Oriented Languages (FOOL).","author":"Lee Hongki","year":"2012","unstructured":"Hongki Lee , Sooncheol Won , Joonho Jin , Junhee Cho , and Sukyoung Ryu . 2012 . SAFE: Formal Specification and Implementation of a Scalable Analysis Framework for ECMAScript . In Proceedings of 19th International Workshop on Foundations of Object-Oriented Languages (FOOL). Hongki Lee, Sooncheol Won, Joonho Jin, Junhee Cho, and Sukyoung Ryu. 2012. SAFE: Formal Specification and Implementation of a Scalable Analysis Framework for ECMAScript. In Proceedings of 19th International Workshop on Foundations of Object-Oriented Languages (FOOL)."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of 29th USENIX Security Symposium. 2613\u20132630","author":"Han Suyoung","year":"2020","unstructured":"Lee, Suyoung and Han , HyungSeok and Cha , Sang Kil and Son, Sooel. 2020 . Montage: A Neural Network Language Model-Guided JavaScript Engine Fuzzer . In Proceedings of 29th USENIX Security Symposium. 2613\u20132630 . Lee, Suyoung and Han, HyungSeok and Cha, Sang Kil and Son, Sooel. 2020. Montage: A Neural Network Language Model-Guided JavaScript Engine Fuzzer. In Proceedings of 29th USENIX Security Symposium. 2613\u20132630."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236037"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2009.30"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_2_1_30_1","first-page":"100","article-title":"Differential Testing for Software","volume":"10","author":"McKeeman William M","year":"1998","unstructured":"William M McKeeman . 1998 . Differential Testing for Software . Digital Technical Journal , 10 , 1 (1998), 100 \u2013 107 . William M McKeeman. 1998. Differential Testing for Software. Digital Technical Journal, 10, 1 (1998), 100\u2013107.","journal-title":"Digital Technical Journal"},{"key":"e_1_2_1_31_1","unstructured":"Michal Zalewski. 2007. American Fuzzy Lop. https:\/\/lcamtuf.coredump.cx\/afl\/ \t\t\t\t  Michal Zalewski. 2007. American Fuzzy Lop. https:\/\/lcamtuf.coredump.cx\/afl\/"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of International Conference on Machine Learning (ICML).","author":"Odena Augustus","year":"2019","unstructured":"Augustus Odena , Catherine Olsson , David Andersen , and Ian Goodfellow . 2019 . TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing . In Proceedings of International Conference on Machine Learning (ICML). Augustus Odena, Catherine Olsson, David Andersen, and Ian Goodfellow. 2019. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In Proceedings of International Conference on Machine Learning (ICML)."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998392.2998397"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78739-6_1"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2737924.2737991"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549097"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678781"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00015"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416632"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-C.2017.4"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7787547"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP40000.2020.00067"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2103656.2103718"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jlap.2010.03.012"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499370.2462168"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0956796809990293"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2022.3171220"},{"key":"e_1_2_1_50_1","volume-title":"Practical Model-based Testing: A Tools Approach","author":"Utting Mark","unstructured":"Mark Utting and Bruno Legeard . 2010. Practical Model-based Testing: A Tools Approach . Elsevier . Mark Utting and Bruno Legeard. 2010. Practical Model-based Testing: A Tools Approach. Elsevier."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-45744-4_29"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00081"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.1997.630875"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_2_1_55_1","unstructured":"XMLTestSuite. 2011. Extensible Markup Language (XML) Conformance Test Suites. https:\/\/www.w3.org\/XML\/Test\/ \t\t\t\t  XMLTestSuite. 2011. Extensible Markup Language (XML) Conformance Test Suites. https:\/\/www.w3.org\/XML\/Test\/"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993498.1993532"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454054"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.430"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3591240","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3591240","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:47:47Z","timestamp":1750178867000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3591240"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,6]]},"references-count":58,"journal-issue":{"issue":"PLDI","published-print":{"date-parts":[[2023,6,6]]}},"alternative-id":["10.1145\/3591240"],"URL":"https:\/\/doi.org\/10.1145\/3591240","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,6]]},"assertion":[{"value":"2023-06-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}