{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T13:23:03Z","timestamp":1777555383716,"version":"3.51.4"},"reference-count":69,"publisher":"SAGE Publications","issue":"1-2","license":[{"start":{"date-parts":[[2019,8,19]],"date-time":"2019-08-19T00:00:00Z","timestamp":1566172800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Data Science"],"published-print":{"date-parts":[[2019,11,25]]},"abstract":"<jats:sec>\n                    <jats:title>Context:<\/jats:title>\n                    <jats:p>Systematic Reviews (SRs) are means for collecting and synthesizing evidence from the identification and analysis of relevant studies from multiple sources. To this aim, they use a well-defined methodology meant to mitigate the risks of biases and ensure repeatability for later updates. SRs, however, involve significant effort.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Goal:<\/jats:title>\n                    <jats:p>The goal of this paper is to introduce a novel methodology that reduces the amount of manual tedious tasks involved in SRs while taking advantage of the value provided by human expertise.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Method:<\/jats:title>\n                    <jats:p>Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been applied in the Software Engineering sub-area of Software Architecture and evaluated by human annotators.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>The result is a novel Expert-Driven Automatic Methodology, EDAM, for assisting researchers in performing SRs. EDAM combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand and knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture against six senior researchers. As a result, we found that the performance of the senior researchers in classifying papers was not statistically significantly different from EDAM.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions:<\/jats:title>\n                    <jats:p>Thanks to automation of the less-creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3233\/ds-190019","type":"journal-article","created":{"date-parts":[[2019,8,20]],"date-time":"2019-08-20T10:48:48Z","timestamp":1566298128000},"page":"311-340","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":11,"title":["Reducing the effort for systematic reviews in software engineering"],"prefix":"10.1177","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6557-3131","authenticated-orcid":false,"given":"Francesco","family":"Osborne","sequence":"first","affiliation":[{"name":"Knowledge Media Institute, The Open University, UK. E-mail:\u00a0"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6365-6515","authenticated-orcid":false,"given":"Henry","family":"Muccini","sequence":"additional","affiliation":[{"name":"DISIM Department, University of L\u2019Aquila, Italy. E-mail:\u00a0"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2234-0845","authenticated-orcid":false,"given":"Patricia","family":"Lago","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands. E-mail:\u00a0"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0015-1952","authenticated-orcid":false,"given":"Enrico","family":"Motta","sequence":"additional","affiliation":[{"name":"Knowledge Media Institute, The Open University, UK. E-mail:\u00a0"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2019,8,19]]},"reference":[{"key":"ref001","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.06.007"},{"key":"ref002","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2015.060121"},{"key":"ref003","doi-asserted-by":"publisher","DOI":"10.1097\/01.NAJ.0000446779.99522.f6"},{"key":"ref004","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30284-8_21"},{"key":"ref005","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2011.10.006"},{"key":"ref006","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2008.03.004"},{"key":"ref007","unstructured":"S.Bhatia, J.H.Lau and T.Baldwin, Automatic labelling of topics with neural embeddings, 2016, arXiv preprint arXiv:1612.05340."},{"key":"ref008","first-page":"993","volume":"3","author":"Blei D.M.","year":"2003","journal-title":"Journal of machine Learning research"},{"key":"ref009","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-49004-5_4"},{"key":"ref010","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-789X.2011.00865.x"},{"key":"ref011","doi-asserted-by":"publisher","DOI":"10.1007\/11428817_21"},{"key":"ref012","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-012-9227-7"},{"key":"ref013","doi-asserted-by":"publisher","DOI":"10.1016\/j.aei.2006.11.006"},{"key":"ref014","doi-asserted-by":"publisher","DOI":"10.1145\/2961111.2962632"},{"key":"ref015","doi-asserted-by":"publisher","DOI":"10.1145\/2362499.2362501"},{"key":"ref016","doi-asserted-by":"publisher","DOI":"10.1145\/2961111.2962630"},{"key":"ref017","unstructured":"B.Filipovic, B.Van Lindschoten, G.Procaccianti and P.Lago, Systematic Literature Study on Sustainable Software, VU Technical Report, 2, 2017. https:\/\/research.vu.nl\/ws\/portalfiles\/portal\/58444907\/Technical_Report_SLR.pdf."},{"key":"ref018","doi-asserted-by":"publisher","DOI":"10.3233\/SW-160240"},{"key":"ref019","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46547-0_16"},{"key":"ref020","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.10.011"},{"issue":"2004","key":"ref021","first-page":"1","volume":"33","author":"Kitchenham B.","year":"2004","journal-title":"Keele, UK, Keele University"},{"key":"ref022","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2013.07.010"},{"key":"ref023","unstructured":"B.A.Kitchenham and S.Charters, Guidelines for performing systematic literature reviews in software engineering, 2007. https:\/\/edisciplinas.usp.br\/pluginfile.php\/4108896\/mod_resource\/content\/2\/slrPCS5012_highlighted.pdf."},{"key":"ref024","doi-asserted-by":"publisher","DOI":"10.1045\/november2012-knoth"},{"key":"ref025","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-38288-8_33"},{"key":"ref026","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9495-8"},{"key":"ref027","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"ref028","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687577"},{"key":"ref029","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339754"},{"key":"ref030","doi-asserted-by":"crossref","unstructured":"A.Mannocci, F.Osborne and E.Motta, The evolution of IJHCS and CHI: A quantitative analysis,\n                      International Journal of Human-Computer Studies\n                      (2019). arXiv:1908.04088.","DOI":"10.1016\/j.ijhcs.2019.05.009"},{"key":"ref031","doi-asserted-by":"publisher","DOI":"10.1145\/2745802.2745827"},{"key":"ref032","doi-asserted-by":"publisher","DOI":"10.1145\/2063518.2063519"},{"key":"ref033","doi-asserted-by":"publisher","DOI":"10.1145\/2961111.2962619"},{"key":"ref034","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2017.30"},{"key":"ref035","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9342-8"},{"key":"ref036","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35176-1_26"},{"key":"ref037","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-25007-6_24"},{"key":"ref038","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41335-3_29"},{"key":"ref039","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46547-0_33"},{"key":"ref040","doi-asserted-by":"publisher","DOI":"10.1108\/JD-12-2013-0166"},{"key":"ref041","doi-asserted-by":"publisher","DOI":"10.14236\/ewic\/EASE2008.8"},{"key":"ref042","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.03.007"},{"key":"ref043","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-49004-5_31"},{"key":"ref044","unstructured":"J.Ramoset al., Using tf-idf to determine word relevance in document queries, in: Proceedings of the First Instructional Conference on Machine Learning, Vol.\u00a0242, Piscataway, NJ, 2003, pp.\u00a0133\u2013142. https:\/\/dl.acm.org\/citation.cfm?id=2227115.2227123."},{"key":"ref045","doi-asserted-by":"publisher","DOI":"10.1016\/j.rser.2015.04.065"},{"key":"ref046","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpr.2012.02.004"},{"key":"ref047","unstructured":"G.Rizzo and R.Troncy, Nerd: A framework for unifying named entity recognition and disambiguation extraction tools, in: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics, 2012, pp.\u00a073\u201376. https:\/\/dl.acm.org\/citation.cfm?id=2380936."},{"key":"ref048","unstructured":"R.Ros, E.Bjarnason and P.Runeson, A machine learning approach for semi-automated search and selection in literature studies, in: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE\u201917, ACM, 2017, pp.\u00a0118\u2013127. ISBN 978-1-4503-4804-1. https:\/\/dl.acm.org\/citation.cfm?id=3084243."},{"key":"ref049","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13325"},{"key":"ref050","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35176-1_32"},{"key":"ref051","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30796-7_31"},{"key":"ref052","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.119"},{"key":"ref053","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30760-8_26"},{"key":"ref054","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-00668-6_12"},{"key":"ref055","doi-asserted-by":"publisher","DOI":"10.21954\/ou.rd.7204814.v1"},{"key":"ref056","doi-asserted-by":"publisher","DOI":"10.1145\/312624.312679"},{"key":"ref057","unstructured":"J.M.Schultz and M.Liberman, Topic detection and tracking using idf-weighted cosine coefficient, in: Proceedings of the DARPA Broadcast News Workshop, Morgan Kaufmann, San Francisco, 1999, pp.\u00a0189\u2013192. http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.591.7692&rep=rep1&type=pdf."},{"key":"ref058","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2742839"},{"key":"ref059","doi-asserted-by":"publisher","DOI":"10.1049\/ic.2012.0022"},{"key":"ref060","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.09.019"},{"key":"ref061","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-43839-8"},{"key":"ref062","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.18"},{"issue":"3","key":"ref063","first-page":"243","volume":"3","author":"Wohlgenannt G.","year":"2012","journal-title":"Journal of Information and Data Management"},{"key":"ref064","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2013.02.002"},{"key":"ref065","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2013.04.076"},{"key":"ref066","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"ref067","doi-asserted-by":"publisher","DOI":"10.23919\/sustainit.2017.8379798"},{"key":"ref068","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.09.008"},{"key":"ref069","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2010.12.010"}],"container-title":["Data Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/DS-190019","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.3233\/DS-190019","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.3233\/DS-190019","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T18:10:02Z","timestamp":1777399802000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.3233\/DS-190019"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8,19]]},"references-count":69,"journal-issue":{"issue":"1-2","published-print":{"date-parts":[[2019,11,25]]}},"alternative-id":["10.3233\/DS-190019"],"URL":"https:\/\/doi.org\/10.3233\/ds-190019","relation":{},"ISSN":["2451-8484","2451-8492"],"issn-type":[{"value":"2451-8484","type":"print"},{"value":"2451-8492","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,8,19]]}}}