{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T13:43:37Z","timestamp":1770299017122,"version":"3.49.0"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T00:00:00Z","timestamp":1717372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62272221, 62172202, 62172205, 62302222"],"award-info":[{"award-number":["62272221, 62172202, 62172205, 62302222"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Cooperation Fund of Nanjing University-Huawei Novel Software Technology Lab","award":["TC20230202021-2023-10"],"award-info":[{"award-number":["TC20230202021-2023-10"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>In recent years, dynamic languages such as Python have become popular due to their flexibility and productivity. The lack of static typing makes programs face the challenges of fixing type errors, early bug detection, and code understanding. To alleviate these issues, PEP 484 introduced optional type annotations for Python in 2014, but unfortunately, a large number of programs are still not annotated by developers. Annotation generation tools can utilize type inference techniques. However, several important aspects of type annotation generation are overlooked by existing works, such as in-depth effectiveness analysis, potential improvement exploration, and practicality evaluation. And it is unclear how far we have been and how far we can go.<\/jats:p>\n          <jats:p>In this paper, we set out to comprehensively investigate the effectiveness of type inference tools for generating type annotations, applying three categories of state-of-the-art tools on a carefully-cleaned dataset. First, we use a comprehensive set of metrics and categories, finding that existing tools have different effectiveness and cannot achieve both high accuracy and high coverage. Then, we summarize six patterns to present the limitations in type annotation generation. Next, we implement a simple but effective tool to demonstrate that existing tools can be improved in practice. Finally, we conduct a controlled experiment showing that existing tools can reduce the time spent annotating types and determine more precise types, but cannot reduce subjective difficulty. Our findings point out the limitations and improvement directions in type annotation generation, which can inspire future work.<\/jats:p>","DOI":"10.1145\/3652153","type":"journal-article","created":{"date-parts":[[2024,3,11]],"date-time":"2024-03-11T12:18:30Z","timestamp":1710159510000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Generating Python Type Annotations from Type Inference: How Far Are We?"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5678-9061","authenticated-orcid":false,"given":"Yimeng","family":"Guo","sequence":"first","affiliation":[{"name":"Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3475-6979","authenticated-orcid":false,"given":"Zhifei","family":"Chen","sequence":"additional","affiliation":[{"name":"Nanjing University of Science and Technology, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2352-2226","authenticated-orcid":false,"given":"Lin","family":"Chen","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1884-7700","authenticated-orcid":false,"given":"Wenjie","family":"Xu","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2282-7175","authenticated-orcid":false,"given":"Yanhui","family":"Li","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4645-2526","authenticated-orcid":false,"given":"Yuming","family":"Zhou","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7743-1296","authenticated-orcid":false,"given":"Baowen","family":"Xu","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}]}],"member":"320","published-online":{"date-parts":[[2024,6,3]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3385412.3385997"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/1925844.1926437"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/11531142_19"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528454"},{"key":"e_1_3_2_6_2","volume-title":"Localized Type Inference of Atomic Types in Python","author":"Cannon Brett","year":"2005","unstructured":"Brett Cannon. 2005. Localized Type Inference of Atomic Types in Python. California Polytechnic State University."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3022671.2984017"},{"key":"e_1_3_2_8_2","unstructured":"Collin Winter and Tony Lownds. 2006. PEP 3107 - Function Annotations. https:\/\/peps.python.org\/pep-3107\/"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.47"},{"key":"e_1_3_2_10_2","article-title":"PYInfer: Deep learning semantic type inference for Python variables","author":"Cui Siwei","year":"2021","unstructured":"Siwei Cui, Gang Zhao, Zeyu Dai, Luochao Wang, Ruihong Huang, and Jeff Huang. 2021. PYInfer: Deep learning semantic type inference for Python variables. arXiv preprint arXiv:2106.14316 (2021).","journal-title":"arXiv preprint arXiv:2106.14316"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236042"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549114"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1529282.1529700"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.75"},{"key":"e_1_3_2_15_2","unstructured":"Google. 2018. Pytype. https:\/\/github.com\/google\/pytype"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9289-1"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-96142-2_2"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236051"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-03237-0_17"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473135"},{"key":"e_1_3_2_21_2","unstructured":"Jetbrains. 2020. Python developer survey conducted by Jetbrains and Python software foundation. https:\/\/www.jetbrains.com\/lp\/python-developers-survey-2020\/"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678947"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3082068"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316447003000308"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383458"},{"key":"e_1_3_2_26_2","unstructured":"Jukka Lehtosalo. 2019. PEP 589 \u2013 TypedDict: Type hints for dictionaries with a fixed set of keys. https:\/\/www.python.org\/dev\/peps\/pep-0589\/"},{"key":"e_1_3_2_27_2","unstructured":"Python. 2014. mypy. https:\/\/mypy-lang.org\/"},{"key":"e_1_3_2_28_2","unstructured":"Ivan Levkivskyi Jukka Lehtosalo and \u0141ukasz Langa. 2017. PEP 544 \u2013 Protocols: Structural subtyping (static duck typing). https:\/\/www.python.org\/dev\/peps\/pep-0544\/"},{"key":"e_1_3_2_29_2","unstructured":"Magnus Madsen. 2015. Static analysis of dynamic languages. https:\/\/pure.au.dk\/ws\/files\/85299449\/Thesis.pdf (2015)."},{"issue":"1","key":"e_1_3_2_30_2","first-page":"1","article-title":"A static type inference for Python","volume":"5","author":"Maia Eva","year":"2012","unstructured":"Eva Maia, Nelma Moreira, and Rog\u00e9rio Reis. 2012. A static type inference for Python. Proc. of DYLA 5, 1 (2012), 1.","journal-title":"Proc. of DYLA"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00045"},{"key":"e_1_3_2_32_2","unstructured":"Microsoft. 2019. Pyright. https:\/\/github.com\/microsoft\/pyright"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00079"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510124"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3290331"},{"key":"e_1_3_2_36_2","unstructured":"GitHub Octoverse. 2022. The 2022 state of open source software. https:\/\/octoverse.github.com\/"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3439775"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238173"},{"key":"e_1_3_2_39_2","first-page":"1","volume-title":"European Conference on Object-Oriented Programming (ECOOP), Scripts to Programs Workshop, STOP","volume":"15","author":"Ortin Francisco","year":"2015","unstructured":"Francisco Ortin, Jose Baltasar Garcia Perez-Schofield, and Jose Manuel Redondo. 2015. Towards a static type checker for Python. In European Conference on Object-Oriented Programming (ECOOP), Scripts to Programs Workshop, STOP, Vol. 15. 1\u20132."},{"key":"e_1_3_2_40_2","article-title":"OptTyper: Probabilistic type inference by optimising logical and natural constraints","author":"Pandi Irene Vlassi","year":"2020","unstructured":"Irene Vlassi Pandi, Earl T. Barr, Andrew D. Gordon, and Charles Sutton. 2020. OptTyper: Probabilistic type inference by optimising logical and natural constraints. arXiv preprint arXiv:2004.00348 (2020).","journal-title":"arXiv preprint arXiv:2004.00348"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510144"},{"key":"e_1_3_2_42_2","volume-title":"Leveraging Program Analysis for Type Inference","author":"Pavlinovic Zvonimir","year":"2019","unstructured":"Zvonimir Pavlinovic. 2019. Leveraging Program Analysis for Type Inference. Ph. D. Dissertation. New York University."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510038"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409715"},{"key":"e_1_3_2_45_2","unstructured":"Facebook. 2017. mypy. https:\/\/pyre-check.org\/"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2008.15"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635922"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/2775051.2677009"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/2480362.2480655"},{"key":"e_1_3_2_50_2","volume-title":"Starkiller: A Static Type Inferencer and Compiler for Python","author":"Salib Michael","year":"2004","unstructured":"Michael Salib. 2004. Starkiller: A Static Type Inferencer and Compiler for Python. Ph. D. Dissertation. Massachusetts Institute of Technology."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884815"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/2517208.2517215"},{"key":"e_1_3_2_53_2","volume-title":"Introduction to Information Retrieval","author":"Sch\u00fctze Hinrich","year":"2008","unstructured":"Hinrich Sch\u00fctze, Christopher D. Manning, and Prabhakar Raghavan. 2008. Introduction to Information Retrieval. Vol. 39. Cambridge University Press Cambridge."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-73589-2_2"},{"key":"e_1_3_2_55_2","unstructured":"IEEE Spectrum. 2022. Top Programming Languages 2022. https:\/\/spectrum.ieee.org\/top-programming-languages"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2047849.2047861"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3561150"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635873"},{"key":"e_1_3_2_59_2","unstructured":"Guido van Rossum Jukka Lehtosalo and \u0141ukasz Langa. 2014. PEP 484 \u2013 Type Hints. https:\/\/www.python.org\/dev\/peps\/pep-0484\/"},{"key":"e_1_3_2_60_2","unstructured":"Guido van Rossum and Ivan Levkivskyi. 2014. PEP 483 \u2013 The Theory of Type Hints. https:\/\/www.python.org\/dev\/peps\/pep-0483\/"},{"key":"e_1_3_2_61_2","unstructured":"Guido van van Rossum. 2004. Adding Optional Static Typing to Python. https:\/\/www.artima.com\/weblogs\/viewpost.jsp?thread=85551"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/2661088.2661101"},{"key":"e_1_3_2_63_2","unstructured":"Yin Wang. 2014. Pysonar2. https:\/\/github.com\/yinwang0\/pysonar2"},{"key":"e_1_3_2_64_2","article-title":"LambdaNet: Probabilistic type inference using graph neural networks","author":"Wei Jiayi","year":"2020","unstructured":"Jiayi Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. 2020. LambdaNet: Probabilistic type inference using graph neural networks. arXiv preprint arXiv:2005.02161 (2020).","journal-title":"arXiv preprint arXiv:2005.02161"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950357"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950343"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00170"},{"key":"e_1_3_2_68_2","unstructured":"\u0141ukasz Langa. 2019. PEP 585 \u2013 Type hinting generics in standard collections. https:\/\/www.python.org\/dev\/peps\/pep-0585\/"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3652153","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3652153","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:14Z","timestamp":1750291394000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3652153"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,3]]},"references-count":67,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3652153"],"URL":"https:\/\/doi.org\/10.1145\/3652153","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,3]]},"assertion":[{"value":"2023-02-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-22","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}