{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T22:36:45Z","timestamp":1787524605728,"version":"build-2736575974"},"publisher-location":"New York, NY, USA","reference-count":60,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,5,23]],"date-time":"2022-05-23T00:00:00Z","timestamp":1653264000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["851720"],"award-info":[{"award-number":["851720"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,5,23]]},"DOI":"10.1145\/3524842.3528440","type":"proceedings-article","created":{"date-parts":[[2022,10,18]],"date-time":"2022-10-18T00:08:36Z","timestamp":1666051716000},"page":"167-178","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["To what extent do deep learning-based code recommenders generate predictions by cloning code from the training set?"],"prefix":"10.1145","author":[{"given":"Matteo","family":"Ciniselli","sequence":"first","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luca","family":"Pascarella","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriele","family":"Bavota","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,10,17]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Milton Allamanis. [n.d.]. CodeSearchNet Deduplication Algorithm. https:\/\/github.com\/github\/CodeSearchNet\/blob\/master\/src\/dataextraction\/dedup_split.py.  Milton Allamanis. [n.d.]. CodeSearchNet Deduplication Algorithm. https:\/\/github.com\/github\/CodeSearchNet\/blob\/master\/src\/dataextraction\/dedup_split.py."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359591.3359735"},{"key":"e_1_3_2_1_3_1","volume-title":"International Conference on Machine Learning. PMLR, 245--256","author":"Alon Uri","year":"2020","unstructured":"Uri Alon , Roy Sadaka , Omer Levy , and Eran Yahav . 2020 . Structural language models of code . In International Conference on Machine Learning. PMLR, 245--256 . Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. 2020. Structural language models of code. In International Conference on Machine Learning. PMLR, 245--256."},{"key":"e_1_3_2_1_4_1","unstructured":"Anon. 2022. Replication Package https:\/\/github.com\/DLCloning\/DLClone.git.  Anon. 2022. Replication Package https:\/\/github.com\/DLCloning\/DLClone.git."},{"key":"e_1_3_2_1_5_1","volume-title":"Sequence Model Design for Code Completion in the Modern IDE. arXiv preprint arXiv:2004.05249","author":"Aye Gareth Ari","year":"2020","unstructured":"Gareth Ari Aye and Gail E Kaiser . 2020. Sequence Model Design for Code Completion in the Modern IDE. arXiv preprint arXiv:2004.05249 ( 2020 ). Gareth Ari Aye and Gail E Kaiser. 2020. Sequence Model Design for Code Completion in the Modern IDE. arXiv preprint arXiv:2004.05249 (2020)."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.1995.514697"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.1998.738528"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1595696.1595728"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.15514\/SYRCOSE-2008-2-22"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3074309"},{"key":"e_1_3_2_1_11_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)."},{"key":"e_1_3_2_1_12_1","volume-title":"Massimiliano Di Penta, and Gabriele Bavota","author":"Ciniselli Matteo","year":"2021","unstructured":"Matteo Ciniselli , Nathan Cooper , Luca Pascarella , Antonio Mastropaolo , Emad Aghajani , Denys Poshyvanyk , Massimiliano Di Penta, and Gabriele Bavota . 2021 . Replication Package https:\/\/github.com\/mciniselli\/T5_Replication_Package.git. Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Antonio Mastropaolo, Emad Aghajani, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2021. Replication Package https:\/\/github.com\/mciniselli\/T5_Replication_Package.git."},{"key":"e_1_3_2_1_13_1","volume-title":"Massimiliano Di Penta, and Gabriele Bavota","author":"Ciniselli Matteo","year":"2022","unstructured":"Matteo Ciniselli , Nathan Cooper , Luca Pascarella , Antonio Mastropaolo , Emad Aghajani , Denys Poshyvanyk , Massimiliano Di Penta, and Gabriele Bavota . 2022 . An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering ( 2022). Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Antonio Mastropaolo, Emad Aghajani, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2022. An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering (2022)."},{"key":"e_1_3_2_1_14_1","volume-title":"How Much Does Tokenization Affect Neural Machine Translation? arXiv preprint arXiv:1812.08621","author":"Domingo Miguel","year":"2018","unstructured":"Miguel Domingo , Mercedes Garc\u0131a-Mart\u0131nez , Alexandre Helle , Francisco Casacuberta , and Manuel Herranz . 2018. How Much Does Tokenization Affect Neural Machine Translation? arXiv preprint arXiv:1812.08621 ( 2018 ). Miguel Domingo, Mercedes Garc\u0131a-Mart\u0131nez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2018. How Much Does Tokenization Affect Neural Machine Translation? arXiv preprint arXiv:1812.08621 (2018)."},{"key":"e_1_3_2_1_15_1","article-title":"A New Algorithm for Data Compression","volume":"12","author":"Gage Philip","year":"1994","unstructured":"Philip Gage . 1994 . A New Algorithm for Data Compression . C Users J. 12 , 2 (1994), 23?38. Philip Gage. 1994. A New Algorithm for Data Compression. C Users J. 12, 2 (1994), 23?38.","journal-title":"C Users J."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491956.2462192"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2009.64"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-011-0083-2"},{"key":"e_1_3_2_1_19_1","unstructured":"Simon Harris. [n.d.]. Simian https:\/\/www.harukizaemon.com\/simian.  Simon Harris. [n.d.]. Simian https:\/\/www.harukizaemon.com\/simian."},{"key":"e_1_3_2_1_20_1","volume-title":"Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE","author":"Vincent","year":"2017","unstructured":"Vincent J. Hellendoorn and Premkumar Devanbu. 2017. Are Deep Neural Networks the Best Choice for Modeling Source Code? . In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE 2017 ). 763?773. Vincent J. Hellendoorn and Premkumar Devanbu. 2017. Are Deep Neural Networks the Best Choice for Modeling Source Code?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE 2017). 763?773."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00101"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2009.39"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSMR.2011.12"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227135"},{"key":"e_1_3_2_1_25_1","volume-title":"CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. CoRR abs\/1909.09436","author":"Husain Hamel","year":"2019","unstructured":"Hamel Husain , Ho-Hsiang Wu , Tiferet Gazit , Miltiadis Allamanis , and Marc Brockschmidt . 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. CoRR abs\/1909.09436 ( 2019 ). http:\/\/arxiv.org\/abs\/1909.09436 Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. CoRR abs\/1909.09436 (2019). http:\/\/arxiv.org\/abs\/1909.09436"},{"key":"e_1_3_2_1_26_1","unstructured":"IntelliJ. [n.d.]. IntelliJ https:\/\/www.jetbrains.com\/idea\/.  IntelliJ. [n.d.]. IntelliJ https:\/\/www.jetbrains.com\/idea\/."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1287624.1287634"},{"key":"e_1_3_2_1_28_1","unstructured":"T. Kamiya. [n.d.]. Simian http:\/\/www.ccfinder.net.  T. Kamiya. [n.d.]. Simian http:\/\/www.ccfinder.net."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2002.1019480"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9076-6"},{"key":"e_1_3_2_1_31_1","volume-title":"Sutton","author":"Karampatsis Rafael-Michael","year":"2019","unstructured":"Rafael-Michael Karampatsis and Charles A . Sutton . 2019 . Maybe Deep Neural Networks are the Best Choice for Modeling Source Code. CoRR abs\/1903.05734 (2019). http:\/\/arxiv.org\/abs\/1903.05734 Rafael-Michael Karampatsis and Charles A. Sutton. 2019. Maybe Deep Neural Networks are the Best Choice for Modeling Source Code. CoRR abs\/1903.05734 (2019). http:\/\/arxiv.org\/abs\/1903.05734"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380342"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2009.36"},{"key":"e_1_3_2_1_34_1","volume-title":"Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE'04. IEEE, 83--92","author":"Kim Miryung","year":"2004","unstructured":"Miryung Kim , Lawrence Bergman , Tessa Lau , and David Notkin . 2004 . An ethnographic study of copy and paste programming practices in OOPL . In Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE'04. IEEE, 83--92 . Miryung Kim, Lawrence Bergman, Tessa Lau, and David Notkin. 2004. An ethnographic study of copy and paste programming practices in OOPL. In Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE'04. IEEE, 83--92."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00026"},{"key":"e_1_3_2_1_36_1","volume-title":"International static analysis symposium","author":"Komondoor Raghavan","unstructured":"Raghavan Komondoor and Susan Horwitz . 2001. Using slicing to identify duplication in source code . In International static analysis symposium . Springer , 40--56. Raghavan Komondoor and Susan Horwitz. 2001. Using slicing to identify duplication in source code. In International static analysis symposium. Springer, 40--56."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2001.957835"},{"key":"e_1_3_2_1_38_1","volume-title":"SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. CoRR abs\/1808.06226","author":"Kudo Taku","year":"2018","unstructured":"Taku Kudo and John Richardson . 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. CoRR abs\/1808.06226 ( 2018 ). Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. CoRR abs\/1808.06226 (2018)."},{"key":"e_1_3_2_1_39_1","volume-title":"Code completion with neural attention and pointer networks. arXiv preprint arXiv:1711.09573","author":"Li Jian","year":"2017","unstructured":"Jian Li , Yue Wang , Michael R Lyu , and Irwin King . 2017. Code completion with neural attention and pointer networks. arXiv preprint arXiv:1711.09573 ( 2017 ). Jian Li, Yue Wang, Michael R Lyu, and Irwin King. 2017. Code completion with neural attention and pointer networks. arXiv preprint arXiv:1711.09573 (2017)."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2006.28"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416591"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133908"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00041"},{"key":"e_1_3_2_1_44_1","volume-title":"Character n-gram tokenization for European language text retrieval. Information retrieval 7, 1","author":"McNamee Paul","year":"2004","unstructured":"Paul McNamee and James Mayfield . 2004. Character n-gram tokenization for European language text retrieval. Information retrieval 7, 1 ( 2004 ), 73--97. Paul McNamee and James Mayfield. 2004. Character n-gram tokenization for European language text retrieval. Information retrieval 7, 1 (2004), 73--97."},{"key":"e_1_3_2_1_45_1","unstructured":"PMD. [n.d.]. Cpd http:\/\/pmd.sourceforge.net\/.  PMD. [n.d.]. Cpd http:\/\/pmd.sourceforge.net\/."},{"key":"e_1_3_2_1_46_1","volume-title":"Liu","author":"Raffel Colin","year":"2019","unstructured":"Colin Raffel , Noam Shazeer , Adam Roberts , Katherine Lee , Sharan Narang , Michael Matena , Yanqi Zhou , Wei Li , and Peter J . Liu . 2019 . Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer . arXiv:1910.10683 [cs.LG] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs.LG]"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2013.01.008"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-010-0064-x"},{"key":"e_1_3_2_1_49_1","first-page":"64","article-title":"A survey on software clone detection research","volume":"541","author":"Roy Chanchal Kumar","year":"2007","unstructured":"Chanchal Kumar Roy and James R Cordy . 2007 . A survey on software clone detection research . Queen's School of Computing TR 541 , 115 (2007), 64 -- 68 . Chanchal Kumar Roy and James R Cordy. 2007. A survey on software clone detection research. Queen's School of Computing TR 541, 115 (2007), 64--68.","journal-title":"Queen's School of Computing TR"},{"key":"e_1_3_2_1_50_1","volume-title":"NICAD: Accurate detection of near-miss intentional clones using flexible pretty-printing and code normalization. In 2008 16th iEEE international conference on program comprehension","author":"Roy Chanchal K","year":"2008","unstructured":"Chanchal K Roy and James R Cordy . 2008 . NICAD: Accurate detection of near-miss intentional clones using flexible pretty-printing and code normalization. In 2008 16th iEEE international conference on program comprehension . IEEE , 172--181. Chanchal K Roy and James R Cordy. 2008. NICAD: Accurate detection of near-miss intentional clones using flexible pretty-printing and code normalization. In 2008 16th iEEE international conference on program comprehension. IEEE, 172--181."},{"key":"e_1_3_2_1_51_1","volume-title":"Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909","author":"Sennrich Rico","year":"2015","unstructured":"Rico Sennrich , Barry Haddow , and Alexandra Birch . 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 ( 2015 ). Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 (2015)."},{"key":"e_1_3_2_1_52_1","unstructured":"SimScan. [n.d.]. Simian http:\/\/www.blue-edge.bg\/download.html.  SimScan. [n.d.]. Simian http:\/\/www.blue-edge.bg\/download.html."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417058"},{"key":"e_1_3_2_1_54_1","volume-title":"Fast and Memory-Efficient Neural Code Completion. In 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 329--340","author":"Svyatkovskiy Alexey","year":"2021","unstructured":"Alexey Svyatkovskiy , Sebastian Lee , Anna Hadjitofi , Maik Riechert , Juliana Vicente Franco , and Miltiadis Allamanis . 2021 . Fast and Memory-Efficient Neural Code Completion. In 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 329--340 . Alexey Svyatkovskiy, Sebastian Lee, Anna Hadjitofi, Maik Riechert, Juliana Vicente Franco, and Miltiadis Allamanis. 2021. Fast and Memory-Efficient Neural Code Completion. In 2021 IEEE\/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 329--340."},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635875"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340544"},{"key":"e_1_3_2_1_57_1","volume-title":"Towards Automating Code Review Activities. In 43rd IEEE\/ACM International Conference on Software Engineering, ICSE 2021","author":"Tufano Rosalia","year":"2021","unstructured":"Rosalia Tufano , Luca Pascarella , Michele Tufano , Denys Poshyvanyk , and Gabriele Bavota . 2021 . Towards Automating Code Review Activities. In 43rd IEEE\/ACM International Conference on Software Engineering, ICSE 2021 , Madrid, Spain, 22- -30 May 2021. IEEE, 163--174. Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. 2021. Towards Automating Code Review Activities. In 43rd IEEE\/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22--30 May 2021. IEEE, 163--174."},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.5555\/2820518.2820559"},{"key":"e_1_3_2_1_59_1","unstructured":"Terry Yin. [n.d.]. Lizard https:\/\/pypi.org\/project\/lizard\/.  Terry Yin. [n.d.]. Lizard https:\/\/pypi.org\/project\/lizard\/."},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1972.10481251"}],"event":{"name":"MSR '22: 19th International Conference on Mining Software Repositories","location":"Pittsburgh Pennsylvania","acronym":"MSR '22","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS"]},"container-title":["Proceedings of the 19th International Conference on Mining Software Repositories"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3524842.3528440","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3524842.3528440","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:35Z","timestamp":1750183775000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3524842.3528440"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,23]]},"references-count":60,"alternative-id":["10.1145\/3524842.3528440","10.1145\/3524842"],"URL":"https:\/\/doi.org\/10.1145\/3524842.3528440","relation":{},"subject":[],"published":{"date-parts":[[2022,5,23]]},"assertion":[{"value":"2022-10-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}