{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T11:34:15Z","timestamp":1791459255387,"version":"4.3.4"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T00:00:00Z","timestamp":1700524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T00:00:00Z","timestamp":1700524800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001786","name":"The University of Adelaide","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001786","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2024,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Outdated documentation is a pervasive problem in software development, preventing effective use of software, and misleading users and developers alike. We posit that one possible reason why documentation becomes out of sync so easily is that developers are unaware of when their source code modifications render the documentation obsolete. Ensuring that the documentation is always in sync with the source code takes considerable effort, especially for large codebases. To address this situation, we propose an approach that can automatically detect code element references that survive in the documentation after all source code instances have been deleted. In this work, we analysed over 3,000 GitHub projects and found that most projects contain at least one outdated code element reference at some point in their history. We submitted GitHub issues to real-world projects containing outdated references detected by our approach, some of which have already led to documentation fixes. As an initiative toward keeping documentation in software repositories up-to-date, we have made our implementation available for developers to scan their GitHub projects for outdated code element references.<\/jats:p>","DOI":"10.1007\/s10664-023-10397-6","type":"journal-article","created":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T14:02:58Z","timestamp":1700575378000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Detecting outdated code element references in software repository documentation"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1503-8576","authenticated-orcid":false,"given":"Wen Siang","family":"Tan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Markus","family":"Wagner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoph","family":"Treude","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,11,21]]},"reference":[{"key":"10397_CR1","doi-asserted-by":"crossref","unstructured":"Aghajani E, Nagy C, Vega-M\u00e1rquez OL, Linares-V\u00e1squez M, Moreno L, Bavota G, Lanza M (2019) Software documentation issues unveiled. In: Proceedings of the International Conference on Software Engineering, pp 1199\u20131210","DOI":"10.1109\/ICSE.2019.00122"},{"key":"10397_CR2","unstructured":"Aldaeej A (2021) Towards effective technical debt decision making in software startups: A multiple case study of web and mobile app startups. PhD thesis, University of Maryland, Baltimore County"},{"key":"10397_CR3","doi-asserted-by":"crossref","unstructured":"Bacchelli A, Lanza M, Robbes R (2010) Linking e-mails and source code artifacts. In: Proceedings of the 32nd ACM\/IEEE International conference on software engineering vol 1, pp 375\u2013384","DOI":"10.1145\/1806799.1806855"},{"key":"10397_CR4","doi-asserted-by":"crossref","unstructured":"Dagenais B, Robillard MP (2012) Recovering traceability links between an api and its learning resources. In: 2012 34th international conference on software engineering (icse), IEEE, pp 47\u201357","DOI":"10.1109\/ICSE.2012.6227207"},{"issue":"11","key":"10397_CR5","doi-asserted-by":"publisher","first-page":"1126","DOI":"10.1109\/TSE.2014.2347969","volume":"40","author":"B Dagenais","year":"2014","unstructured":"Dagenais B, Robillard MP (2014) Using traceability links to recommend adaptive changes for documentation evolution. IEEE Trans Software Eng 40(11):1126\u20131146","journal-title":"IEEE Trans Software Eng"},{"key":"10397_CR6","doi-asserted-by":"crossref","unstructured":"Forward A, Lethbridge TC (2002) The relevance of software documentation, tools and technologies: a survey. In: Proceedings of the symposium on document engineering, pp 26\u201333","DOI":"10.1145\/585058.585065"},{"issue":"1","key":"10397_CR7","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1023\/B:LIDA.0000048322.42751.ca","volume":"10","author":"M Kajko-Mattsson","year":"2005","unstructured":"Kajko-Mattsson M (2005) A survey of documentation practice within corrective maintenance. Empir Softw Eng 10(1):31\u201355","journal-title":"Empir Softw Eng"},{"issue":"6","key":"10397_CR8","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/MS.2012.167","volume":"29","author":"P Kruchten","year":"2012","unstructured":"Kruchten P, Nord RL, Ozkaya I (2012) Technical debt: From metaphor to theory and practice. IEEE Softw 29(6):18\u201321","journal-title":"IEEE Softw"},{"key":"10397_CR9","unstructured":"Lee S, Wu R, Cheung SC, Kang S (2019) Automatic detection and update suggestion for outdated API names in documentation. IEEE Transactions on Software Engineering"},{"issue":"6","key":"10397_CR10","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1109\/MS.2003.1241364","volume":"20","author":"TC Lethbridge","year":"2003","unstructured":"Lethbridge TC, Singer J, Forward A (2003) How software engineers use documentation: The state of the practice. IEEE Softw 20(6):35\u201339","journal-title":"IEEE Softw"},{"issue":"2","key":"10397_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10664-020-09917-5","volume":"26","author":"J Liu","year":"2021","unstructured":"Liu J, Huang Q, Xia X, Shihab E, Lo D, Li S (2021) An exploratory study on the introduction and removal of different types of technical debt in deep learning frameworks. Empir Softw Eng 26(2):1\u201336","journal-title":"Empir Softw Eng"},{"issue":"3","key":"10397_CR12","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 22(3):276\u2013282","journal-title":"Biochem Med (Zagreb)"},{"key":"10397_CR13","doi-asserted-by":"crossref","unstructured":"Mendes TS, de\u00a0F\u00a0Farias MA, Mendon\u00e7a M, Soares HF, Kalinowski M, Sp\u00ednola RO (2016) Impacts of agile requirements documentation debt on software projects: a retrospective study. In: Proceedings of the 31st annual ACM symposium on applied computing, pp 1290\u20131295","DOI":"10.1145\/2851613.2851761"},{"key":"10397_CR14","doi-asserted-by":"crossref","unstructured":"Panthaplackel S, Nie P, Gligoric M, Li JJ, Mooney RJ (2020) Learning to update natural language comments based on code changes. arXiv preprint arXiv:2004.12169","DOI":"10.18653\/v1\/2020.acl-main.168"},{"key":"10397_CR15","doi-asserted-by":"crossref","unstructured":"Parnas DL (1994) Software aging. In: Proceedings of international conference on software engineering, pp 279\u2013287","DOI":"10.1109\/ICSE.1994.296790"},{"key":"10397_CR16","doi-asserted-by":"crossref","unstructured":"Petrosyan G, Robillard MP, De\u00a0Mori R (2015) Discovering information explaining api types using text classification. In: 2015 IEEE\/ACM 37th IEEE international conference on software engineering, IEEE, vol\u00a01, pp 869\u2013879","DOI":"10.1109\/ICSE.2015.97"},{"issue":"3","key":"10397_CR17","doi-asserted-by":"publisher","first-page":"1296","DOI":"10.1007\/s10664-018-9660-3","volume":"24","author":"GAA Prana","year":"2019","unstructured":"Prana GAA, Treude C, Thung F, Atapattu T, Lo D (2019) Categorizing the content of github readme files. Empir Softw Eng 24(3):1296\u20131327","journal-title":"Empir Softw Eng"},{"key":"10397_CR18","doi-asserted-by":"crossref","unstructured":"Ratol IK, Robillard MP (2017) Detecting fragile comments. In: Proceedings of the international conference on automated software engineering, pp 112\u2013122","DOI":"10.1109\/ASE.2017.8115624"},{"key":"10397_CR19","doi-asserted-by":"crossref","unstructured":"Rigby PC, Robillard MP (2013) Discovering essential code elements in informal documentation. In: 2013 35th international conference on software engineering (ICSE), IEEE, pp 832\u2013841","DOI":"10.1109\/ICSE.2013.6606629"},{"key":"10397_CR20","doi-asserted-by":"crossref","unstructured":"Rios N, Mendes L, Cerdeiral C, Magalh\u00e3es APF, Perez B, Correal D, Astudillo H, Seaman C, Izurieta C, Santos G, et\u00a0al. (2020) Hearing the voice of software practitioners on causes, effects, and practices to deal with documentation debt. In: International working conference on requirements engineering: foundation for software quality, Springer, pp 55\u201370","DOI":"10.1007\/978-3-030-44429-7_4"},{"key":"10397_CR21","doi-asserted-by":"crossref","unstructured":"Robillard MP, Marcus A, Treude C, Bavota G, Chaparro O, Ernst N, Gerosa MA, Godfrey M, Lanza M, Linares-V\u00e1squez M, et\u00a0al. (2017) On-demand developer documentation. In: 2017 IEEE International conference on software maintenance and evolution (ICSME), IEEE, pp 479\u2013483","DOI":"10.1109\/ICSME.2017.17"},{"issue":"9","key":"10397_CR22","doi-asserted-by":"publisher","first-page":"e1007296","DOI":"10.1371\/journal.pcbi.1007296","volume":"15","author":"D Sholler","year":"2019","unstructured":"Sholler D, Steinmacher I, Ford D, Averick M, Hoye M, Wilson G (2019) Ten simple rules for helping newcomers become contributors to open projects. PLoS Comput Biol 15(9):e1007296","journal-title":"PLoS Comput Biol"},{"key":"10397_CR23","doi-asserted-by":"crossref","unstructured":"de\u00a0Souza SCB, Anquetil N, de\u00a0Oliveira KM (2005) A study of the documentation essential to software maintenance. In: Proceedings of the international conference on design of communication: documenting & designing for pervasive information, pp 68\u201375","DOI":"10.1145\/1085313.1085331"},{"key":"10397_CR24","doi-asserted-by":"crossref","unstructured":"Steinmacher I, Treude C, Gerosa MA (2018) Let me in: Guidelines for the successful onboarding of newcomers to open source projects. IEEE Software 36(4):41\u201349","DOI":"10.1109\/MS.2018.110162131"},{"key":"10397_CR25","doi-asserted-by":"crossref","unstructured":"Tan SH, Marinov D, Tan L, Leavens GT (2012) @tcomment: Testing Javadoc comments to detect comment-code inconsistencies. In: Proceedings of the international conference on software testing, verification and validation, pp 260\u2013269","DOI":"10.1109\/ICST.2012.106"},{"issue":"6","key":"10397_CR26","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1109\/TSE.2014.2387172","volume":"41","author":"C Treude","year":"2014","unstructured":"Treude C, Robillard MP, Dagenais B (2014) Extracting development tasks to navigate software documentation. IEEE Trans Software Eng 41(6):565\u2013581","journal-title":"IEEE Trans Software Eng"},{"issue":"4","key":"10397_CR27","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1109\/MS.2014.80","volume":"32","author":"G Uddin","year":"2015","unstructured":"Uddin G, Robillard MP (2015) How API documentation fails. IEEE Software 32(4):68\u201375","journal-title":"IEEE Software"},{"key":"10397_CR28","doi-asserted-by":"crossref","unstructured":"Wen F, Nagy C, Bavota G, Lanza M (2019) A large-scale empirical study on code-comment inconsistencies. In: Proceedings of the international conference on program comprehension, pp 53\u201364","DOI":"10.1109\/ICPC.2019.00019"},{"key":"10397_CR29","doi-asserted-by":"crossref","unstructured":"Zhong H, Su Z (2013) Detecting API documentation errors. In: Proceedings of the international conference on object oriented programming systems languages & applications, pp 803\u2013816","DOI":"10.1145\/2544173.2509523"},{"issue":"9","key":"10397_CR30","doi-asserted-by":"publisher","first-page":"1004","DOI":"10.1109\/TSE.2018.2872971","volume":"46","author":"Y Zhou","year":"2020","unstructured":"Zhou Y, Wang C, Yan X, Chen T, Panichella S, Gall HC (2020) Automatic detection and repair recommendation of directive defects in Java API documentation. IEEE Trans Software Eng 46(9):1004\u20131023","journal-title":"IEEE Trans Software Eng"},{"key":"10397_CR31","doi-asserted-by":"publisher","unstructured":"Zlotnick F (2017) Github open source survey 2017. https:\/\/doi.org\/10.5281\/zenodo.806811","DOI":"10.5281\/zenodo.806811"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10397-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-023-10397-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-023-10397-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,27]],"date-time":"2024-03-27T13:24:21Z","timestamp":1711545861000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-023-10397-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,21]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1]]}},"alternative-id":["10397"],"URL":"https:\/\/doi.org\/10.1007\/s10664-023-10397-6","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,21]]},"assertion":[{"value":"25 September 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 November 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Christoph Treude is a member of the Empirical Software Engineering Editorial Board. The authors have no other conflict of interest to declare.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}],"article-number":"5"}}