{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T10:57:01Z","timestamp":1785495421754,"version":"3.56.0"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,8,24]],"date-time":"2022-08-24T00:00:00Z","timestamp":1661299200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,24]],"date-time":"2022-08-24T00:00:00Z","timestamp":1661299200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002428","name":"Austrian Science Fund","doi-asserted-by":"publisher","award":["P 32653-N"],"award-info":[{"award-number":["P 32653-N"]}],"id":[{"id":"10.13039\/501100002428","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002428","name":"Austrian Science Fund","doi-asserted-by":"publisher","award":["P 32653-N"],"award-info":[{"award-number":["P 32653-N"]}],"id":[{"id":"10.13039\/501100002428","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Austrian Science Fund"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Autom Softw Eng"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Textual documents produced in the software engineering process are a popular target for natural language processing (NLP) and information retrieval (IR) approaches. However, issue tickets often contain artifacts such as code snippets, log outputs and stack traces. These artifacts not only inflate the issue ticket sizes, but also can this noise constitute a real problem for some NLP approaches, and therefore has to be removed in the pre-processing of some approaches. In this paper, we present a machine learning based approach to classify textual content into natural language and non-natural language artifacts at line level. We show how data from GitHub issue trackers can be used for automated training set generation, and present a custom preprocessing approach for the task of artifact removal. The training sets are automatically created from Markdown annotated issue tickets and project documentation files. We use these generated training sets to train a Markdown agnostic model that is able to classify un-annotated content. We evaluate our approach on issue tickets from projects written in C++, Java, JavaScript, PHP, and Python. Our approach achieves ROC-AUC scores between 0.92 and 0.96 for language-specific models. A multi-language model trained on the issue tickets of all languages achieves ROC-AUC scores between 0.92 and 0.95. The provided models are intended to be used as noise reduction pre-processing steps for NLP and IR approaches working on issue tickets.<\/jats:p>","DOI":"10.1007\/s10515-022-00350-0","type":"journal-article","created":{"date-parts":[[2022,8,24]],"date-time":"2022-08-24T08:04:04Z","timestamp":1661328244000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Detecting non-natural language artifacts for de-noising bug reports"],"prefix":"10.1007","volume":"29","author":[{"given":"Thomas","family":"Hirsch","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Birgit","family":"Hofer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,8,24]]},"reference":[{"key":"350_CR1","doi-asserted-by":"publisher","unstructured":"Bacchelli, A., Sasso, T.D., D\u2019Ambros, M., Lanza, M.: Content classification of development emails. In: 34th International Conference on Software Engineering (ICSE), pp. 375\u2013385 (2012). https:\/\/doi.org\/10.1109\/ICSE.2012.6227177","DOI":"10.1109\/ICSE.2012.6227177"},{"key":"350_CR2","doi-asserted-by":"publisher","unstructured":"Bacchelli, A., Cleve, A., Lanza, M., Mocci, A.: Extracting structured data from natural language documents with island parsing. In: 26th International Conference on Automated Software Engineering (ASE), pp. 476\u2013479 (2011). https:\/\/doi.org\/10.1109\/ASE.2011.6100103","DOI":"10.1109\/ASE.2011.6100103"},{"key":"350_CR3","unstructured":"Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval (1999)"},{"key":"350_CR4","doi-asserted-by":"publisher","unstructured":"Bettenburg, N., Zimmermann, T., Premraj, R., Kim, S.: Extracting structural information from bug reports. In: International Conference on Software Engineering (ICSE), pp. 27\u201330 (2008). https:\/\/doi.org\/10.1145\/1370750.1370757","DOI":"10.1145\/1370750.1370757"},{"key":"350_CR5","doi-asserted-by":"publisher","unstructured":"Bishop, C.M.: Pattern recognition and machine learning (2006). https:\/\/doi.org\/10.1007\/978-0-387-45528-0","DOI":"10.1007\/978-0-387-45528-0"},{"key":"350_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/J.INFSOF.2019.05.012","volume":"114","author":"F Calefato","year":"2019","unstructured":"Calefato, F., Lanubile, F., Vasilescu, B.: A large-scale, in-depth analysis of developers\u2019 personalities in the Apache ecosystem. Inf. Softw. Technol. 114, 1\u201320 (2019). https:\/\/doi.org\/10.1016\/J.INFSOF.2019.05.012","journal-title":"Inf. Softw. Technol."},{"key":"350_CR7","doi-asserted-by":"publisher","unstructured":"Chawla, I., Singh, S.K.: An automated approach for bug categorization using fuzzy logic. In: 8th India Software Engineering Conference (ISEC 2015), pp. 90\u201399 (2015). https:\/\/doi.org\/10.1145\/2723742.2723751","DOI":"10.1145\/2723742.2723751"},{"issue":"1","key":"350_CR8","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1), 37\u201346 (1960). https:\/\/doi.org\/10.1177\/001316446002000104","journal-title":"Educ. Psychol. Meas."},{"key":"350_CR9","doi-asserted-by":"publisher","unstructured":"Devaiya, D., Anvik, J., Bheree, M., Yeasmin Omee, F.: Evaluating a Tool for Creating Bug Report Assignment Recommenders. In: 33rd International Conference on Software Engineering and Knowledge Engineering (2021). https:\/\/doi.org\/10.18293\/SEKE2021-163","DOI":"10.18293\/SEKE2021-163"},{"key":"350_CR10","doi-asserted-by":"publisher","unstructured":"Goseva-Popstojanova, K., Tyo, J.: Identification of Security related Bug Reports via Text Mining using Supervised and Unsupervised Classification. In: International Conference on Software Quality, Reliability and Security (QRS), pp. 344\u2013355 (2018). https:\/\/doi.org\/10.1109\/QRS.2018.00047","DOI":"10.1109\/QRS.2018.00047"},{"key":"350_CR11","doi-asserted-by":"publisher","unstructured":"Hirsch, T., Hofer, B.: Using textual bug reports to predict the fault category of software bugs. Array 15 (2022). https:\/\/doi.org\/10.1016\/J.ARRAY.2022.100189","DOI":"10.1016\/J.ARRAY.2022.100189"},{"key":"350_CR12","doi-asserted-by":"publisher","unstructured":"Hirsch, T., Hofer, B.: Identifying non-natural language artifacts in bug reports. In: 2nd International Workshop on Software Engineering Automation: A Natural Language Perspective (NLP-SEA) - 36th IEEE\/ACM International Conference on Automated Software Engineering Workshops (ASEW), pp. 191\u2013197 (2021). https:\/\/doi.org\/10.1109\/ASEW52652.2021.00046","DOI":"10.1109\/ASEW52652.2021.00046"},{"key":"350_CR13","doi-asserted-by":"publisher","unstructured":"Hirsch, T., Hofer, B.: artifact_detection - A tool for NLP tasks on textual bug reports. Zenodo (2022). https:\/\/doi.org\/10.5281\/zenodo.6393129","DOI":"10.5281\/zenodo.6393129"},{"key":"350_CR14","doi-asserted-by":"crossref","unstructured":"James, G., Witten, D., Hastie, T., Tibshirani, R.: An introduction to statistical learning (2013). https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-1-0716-1418-1.pdf","DOI":"10.1007\/978-1-4614-7138-7"},{"key":"350_CR15","unstructured":"Jauhiainen, T., Lind\u00e9n, K., Jauhiainen, H.: Evaluation of language identification methods using 285 languages. In: 21st Nordic Conference on computational linguistics, pp. 183\u2013191 (2017)"},{"key":"350_CR16","doi-asserted-by":"publisher","first-page":"200749","DOI":"10.1109\/ACCESS.2020.3033045","volume":"8","author":"A Kukkar","year":"2020","unstructured":"Kukkar, A., Mohana, R., Kumar, Y., Nayyar, A., Bilal, M., Kwak, K.S.: Duplicate bug report detection and classification system based on deep learning technique. IEEE Access 8, 200749\u2013200763 (2020). https:\/\/doi.org\/10.1109\/ACCESS.2020.3033045","journal-title":"IEEE Access"},{"issue":"13","key":"350_CR17","doi-asserted-by":"publisher","first-page":"2964","DOI":"10.3390\/S19132964","volume":"19","author":"A Kukkar","year":"2019","unstructured":"Kukkar, A., Mohana, R., Nayyar, A., Kim, J., Kang, B.-G., Chilamkurti, N.: A novel deep-learning-based bug severity classification technique using convolutional neural networks and random forest with boosting. Sensors 19(13), 2964 (2019). https:\/\/doi.org\/10.3390\/S19132964","journal-title":"Sensors"},{"key":"350_CR18","doi-asserted-by":"publisher","unstructured":"Kumar, L., Dastidar, T.G., Murthy Neti, L.B., Satapathy, S.M., Misra, S., Kocher, V., Padmanabhuni, S.: Deep-Learning Approach with DeepXplore for Software Defect Severity Level Prediction. In: International Conference on computational science and its applications (ICCSA 2021), pp. 398\u2013410 (2021). https:\/\/doi.org\/10.1007\/978-3-030-87007-2_28","DOI":"10.1007\/978-3-030-87007-2_28"},{"issue":"1","key":"350_CR19","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159 (1977). https:\/\/doi.org\/10.2307\/2529310","journal-title":"Biometrics"},{"key":"350_CR20","doi-asserted-by":"publisher","unstructured":"Mani, S., Sankaran, A., Aralikatte, R.: DeepTriage: Exploring the effectiveness of deep learning for bug triaging. In: India Joint International Conference on Data Science and Management of Data (CoDS-COMAD), pp. 171\u2013179 (2019). https:\/\/doi.org\/10.1145\/3297001.3297023","DOI":"10.1145\/3297001.3297023"},{"key":"350_CR21","doi-asserted-by":"crossref","unstructured":"M\u00e4ntyl\u00e4, M., Calefato, F., Claes, M.: Natural Language or Not (NLoN) - A Package for Software Engineering Text Analysis Pipeline. In: IEEE\/ACM 15th International Conference on Mining Software Repositories (MSR), pp. 387\u2013391 (2018)","DOI":"10.1145\/3196398.3196444"},{"key":"350_CR22","doi-asserted-by":"publisher","unstructured":"Ortu, M., Destefanis, G., Swift, S., Marchesi, M.: Measuring high and low priority defects on traditional and mobile open source software. In: 7th International Workshop on Emerging Trends in Software Metrics (WETSoM), pp. 1\u20137 (2016). https:\/\/doi.org\/10.1145\/2897695.2897696","DOI":"10.1145\/2897695.2897696"},{"key":"350_CR23","doi-asserted-by":"publisher","unstructured":"Ponzanelli, L., Mocci, A., Lanza, M.: StORMeD: Stack overflow ready made data. In: International Working Conference on Mining Software Repositories (MSR), vol. 2015-Augus, pp. 474\u2013477 (2015). https:\/\/doi.org\/10.1109\/MSR.2015.67","DOI":"10.1109\/MSR.2015.67"},{"key":"350_CR24","doi-asserted-by":"publisher","unstructured":"Ray, B., Posnett, D., Filkov, V., Devanbu, P.: A large scale study of programming languages and code quality in GitHub. In: ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE\u201914), pp. 155\u2013165 (2014). https:\/\/doi.org\/10.1145\/2635868.2635922","DOI":"10.1145\/2635868.2635922"},{"key":"350_CR25","doi-asserted-by":"publisher","unstructured":"Rigby, P.C., Robillard, M.P.: Discovering essential code elements in informal documentation. In: 35th International Conference on Software Engineering (ICSE 2013), pp. 832\u2013841 (2013). https:\/\/doi.org\/10.1109\/ICSE.2013.6606629","DOI":"10.1109\/ICSE.2013.6606629"},{"key":"350_CR26","doi-asserted-by":"publisher","unstructured":"Saha, R.K., Lease, M., Khurshid, S., Perry, D.E.: Improving bug localization using structured information retrieval. In: 28th International Conference on Automated Software Engineering (ASE), pp. 345\u2013355 (2013). https:\/\/doi.org\/10.1109\/ASE.2013.6693093","DOI":"10.1109\/ASE.2013.6693093"},{"issue":"6","key":"350_CR27","doi-asserted-by":"publisher","first-page":"5255","DOI":"10.1007\/S10664-020-09882-Z","volume":"25","author":"M Soltani","year":"2020","unstructured":"Soltani, M., Hermans, F., B\u00e4ck, T.: The significance of bug report elements. Empir. Softw. Eng. 25(6), 5255\u20135294 (2020). https:\/\/doi.org\/10.1007\/S10664-020-09882-Z","journal-title":"Empir. Softw. Eng."},{"issue":"6","key":"350_CR28","doi-asserted-by":"publisher","first-page":"1665","DOI":"10.1007\/s10664-013-9258-8","volume":"19","author":"L Tan","year":"2014","unstructured":"Tan, L., Liu, C., Li, Z., Wang, X., Zhou, Y., Zhai, C.: Bug characteristics in open source software. Emp. Softw. Eng. 19(6), 1665\u20131705 (2014). https:\/\/doi.org\/10.1007\/s10664-013-9258-8","journal-title":"Emp. Softw. Eng."},{"key":"350_CR29","doi-asserted-by":"publisher","unstructured":"Thung, F., Lo, D., Jiang, L.: Automatic defect categorization. In: working conference on reverse engineering (WCRE), pp. 205\u2013214 (2012). https:\/\/doi.org\/10.1109\/WCRE.2012.30","DOI":"10.1109\/WCRE.2012.30"},{"key":"350_CR30","doi-asserted-by":"publisher","unstructured":"Ye, D., Xing, Z., Foo, C.Y., Li, J., Kapre, N.: Learning to extract API mentions from informal natural language discussions. In: IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 389\u2013399 (2017). https:\/\/doi.org\/10.1109\/ICSME.2016.11","DOI":"10.1109\/ICSME.2016.11"},{"key":"350_CR31","doi-asserted-by":"publisher","unstructured":"Ye, X., Shen, H., Ma, X., Bunescu, R., Liu, C.: From Word Embeddings To Document Similarities for Improved Information Retrieval in Software Engineering. In: 38th International Conference on Software Engineering (ICSE), pp. 404\u2013415 (2016). https:\/\/doi.org\/10.1145\/2884781.2884862","DOI":"10.1145\/2884781.2884862"},{"issue":"6","key":"350_CR32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/S10664-021-10010-8","volume":"26","author":"C Zhou","year":"2021","unstructured":"Zhou, C., Li, B., Sun, X., Bo, L.: Why and what happened? Aiding bug comprehension with automated category and causal link identification. Emp. Softw. Eng. 26(6), 1\u201336 (2021). https:\/\/doi.org\/10.1007\/S10664-021-10010-8","journal-title":"Emp. Softw. Eng."},{"key":"350_CR33","doi-asserted-by":"publisher","unstructured":"Zhou, J., Zhang, H., Lo, D.: Where should the bugs be fixed? More accurate information retrieval-based bug localization based on bug reports. In: International Conference on Software Engineering (ICSE), pp. 14\u201324 (2012). https:\/\/doi.org\/10.1109\/ICSE.2012.6227210","DOI":"10.1109\/ICSE.2012.6227210"}],"container-title":["Automated Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-022-00350-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10515-022-00350-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-022-00350-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,28]],"date-time":"2022-10-28T11:23:22Z","timestamp":1666956202000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10515-022-00350-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,24]]},"references-count":33,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["350"],"URL":"https:\/\/doi.org\/10.1007\/s10515-022-00350-0","relation":{},"ISSN":["0928-8910","1573-7535"],"issn-type":[{"value":"0928-8910","type":"print"},{"value":"1573-7535","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,24]]},"assertion":[{"value":"31 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 August 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 August 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The work described in this paper has been funded by the Austrian Science Fund (FWF): P 32653-N (Automated Debugging in Use). The authors have no other relevant financial or non-financial interests to disclose. The authors have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"We carefully conducted all experiments and honestly reported the results. Since there are neither humans nor animals involved in the experiments, the study has not been approved by any ethics committee.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}],"article-number":"52"}}