{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T01:16:38Z","timestamp":1771550198232,"version":"3.50.1"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2019,11,16]],"date-time":"2019-11-16T00:00:00Z","timestamp":1573862400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"German Academic Exchange Service (DAAD) STIBET Matching Funds"},{"DOI":"10.13039\/501100001659","name":"German Research Foundation","doi-asserted-by":"crossref","award":["LE 3382\/2-1, LE 3382\/2-3, SA 465\/9-1, SA 465\/49-3"],"award-info":[{"award-number":["LE 3382\/2-1, LE 3382\/2-3, SA 465\/9-1, SA 465\/49-3"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Data and Information Quality"],"published-print":{"date-parts":[[2020,3,31]]},"abstract":"<jats:p>Researchers use\u00a0systematic literature reviews (SLRs) to synthesize existing evidence regarding a research topic. While being an important means to condense knowledge, conducting an SLR requires a large amount of time and effort. Consequently, researchers have proposed semi-automatic techniques to support different stages of the review process. Two of the most time-consuming tasks are (1) to select primary studies and (2) to assess their quality. In this article, we report an SLR in which we identify, discuss, and synthesize existing techniques of the software-engineering domain that aim to semi-automate these two tasks. Instead of solely providing statistics, we discuss these techniques in detail and compare them, aiming to improve our understanding of supported and unsupported activities. To this end, we identified eight primary studies that report unique techniques that have been published between 2007 and 2016. Most of these techniques rely on text mining and can be beneficial for researchers, but an independent validation using real\u00a0SLRs is missing for most of them. Moreover, the results indicate the necessity of developing more reliable techniques, providing access to their implementations, and extending their scope to further activities to facilitate the selection and quality assessment of primary studies.<\/jats:p>","DOI":"10.1145\/3356901","type":"journal-article","created":{"date-parts":[[2019,11,18]],"date-time":"2019-11-18T13:01:53Z","timestamp":1574082113000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Automated Selection and Quality Assessment of Primary Studies"],"prefix":"10.1145","volume":"12","author":[{"given":"Yusra","family":"Shakeel","sequence":"first","affiliation":[{"name":"Otto von Guericke University 8 METOP GmbH, Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jacob","family":"Kr\u00fcger","sequence":"additional","affiliation":[{"name":"Otto von Guericke University 8 Harz University of Applied Sciences, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ivonne Von","family":"Nostitz-Wallwitz","sequence":"additional","affiliation":[{"name":"Otto von Guericke University 8 METOP GmbH, Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gunter","family":"Saake","sequence":"additional","affiliation":[{"name":"Otto von Guericke University, Magdeburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Thomas","family":"Leich","sequence":"additional","affiliation":[{"name":"Harz University of Applied Sciences 8 METOP GmbH, Weringerode 8 Magdeburg,, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,11,16]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"3","article-title":"Applying information retrieval techniques to detect duplicates and to rank references in the preliminary phases of systematic literature reviews","volume":"18","author":"Abilio Ramon","year":"2015","journal-title":"CLEI Electronic Journal"},{"key":"e_1_2_1_2_1","volume-title":"The Semantic Web","author":"Auer S\u00f6ren"},{"key":"e_1_2_1_3_1","volume-title":"International Symposium on Empirical Software Engineering and Measurement (ESEM\u201909)","author":"Muhammad","year":"2009"},{"key":"e_1_2_1_4_1","first-page":"528","article-title":"The goal question metrics approach","volume":"1","author":"Basili Victor R.","year":"1994","journal-title":"Encyclopedia of Software Engineering"},{"key":"e_1_2_1_5_1","volume-title":"Systematic Approaches to a Successful Literature Review","author":"Booth Andrew"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICGSE.2014.11"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978915.1978919"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2011.04.004"},{"key":"e_1_2_1_9_1","unstructured":"Gabriel C. Durand Anusha Janardhana Marcus Pinnecke Yusra Shakeel Jacob Kr\u00fcger Thomas Leich and Gunter Saake. 2018. Exploring large scholarly networks with Hermes. In Extending Database Technology (EDBT\u201918). ACM. DOI:https:\/\/doi.org\/10.5441\/002\/edbt.2018.76  Gabriel C. Durand Anusha Janardhana Marcus Pinnecke Yusra Shakeel Jacob Kr\u00fcger Thomas Leich and Gunter Saake. 2018. Exploring large scholarly networks with Hermes. In Extending Database Technology (EDBT\u201918). ACM. DOI:https:\/\/doi.org\/10.5441\/002\/edbt.2018.76"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.01.006"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1414004.1414034"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2915970.2916013"},{"key":"e_1_2_1_13_1","volume-title":"International Symposium on Empirical Software Engineering and Measurement (ESEM\u201911)","author":"Felizardo Katia R.","year":"2011"},{"key":"e_1_2_1_14_1","volume-title":"International Workshop on Replication in Empirical Software Engineering Research (RESER\u201913)","author":"Felizardo Katia R.","year":"2013"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.10.011"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460999.2461025"},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/meet.14504301185","article-title":"Citation analysis: A comparison of Google scholar, Scopus, and Web of Science","volume":"43","author":"Kiduk Yang","year":"2007","journal-title":"Proceedings of the American Society for Information Science and Technology"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"e_1_2_1_20_1","volume-title":"Evidence-Based Software Engineering and Systematic Reviews","author":"Kitchenham Barbara A."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2004.1317449"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Jacob Kr\u00fcger Christian Lausberger Ivonne von Nostitz-Wallwitz Gunter Saake and Thomas Leich. 2019. Search. Review. Repeat? An empirical study of threats to replicating SLR searches. Empirical Software Engineering. 51 pages. DOI:https:\/\/doi.org\/10.1007\/s10664-019-09763-0  Jacob Kr\u00fcger Christian Lausberger Ivonne von Nostitz-Wallwitz Gunter Saake and Thomas Leich. 2019. Search. Review. Repeat? An empirical study of threats to replicating SLR searches. Empirical Software Engineering. 51 pages. DOI:https:\/\/doi.org\/10.1007\/s10664-019-09763-0","DOI":"10.1007\/s10664-019-09763-0"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.08.010"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2007.21"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2013.32"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2915970.2916000"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9342-8"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851581.2892334"},{"key":"e_1_2_1_30_1","first-page":"195","article-title":"Semantic enrichment for recommendation of primary studies in a systematic literature review","volume":"32","author":"Rizzo Giuseppe","year":"2015","journal-title":"Digital Scholarship in the Humanities"},{"key":"e_1_2_1_31_1","volume-title":"International Conference on Software Engineering 8 Knowledge Engineering (SEKE\u201912)","author":"Jefferson"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5220\/0006368706530658"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194747.3194748"},{"key":"e_1_2_1_34_1","volume-title":"International Conference on Business Information Systems (BIS\u201918)","author":"Shakeel Yusra","year":"2018"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1049\/ic.2011.0004"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.5120\/14937-3507"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1049\/ic.2012.0001"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017160.2017162"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.lisr.2006.06.002"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2745802.2745815"}],"container-title":["Journal of Data and Information Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3356901","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3356901","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:37Z","timestamp":1750203877000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3356901"}},"subtitle":["A Systematic Literature Review"],"short-title":[],"issued":{"date-parts":[[2019,11,16]]},"references-count":39,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,3,31]]}},"alternative-id":["10.1145\/3356901"],"URL":"https:\/\/doi.org\/10.1145\/3356901","relation":{},"ISSN":["1936-1955","1936-1963"],"issn-type":[{"value":"1936-1955","type":"print"},{"value":"1936-1963","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,16]]},"assertion":[{"value":"2018-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-11-16","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}