{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T19:30:38Z","timestamp":1768419038279,"version":"3.49.0"},"reference-count":185,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2020,3,5]],"date-time":"2020-03-05T00:00:00Z","timestamp":1583366400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,3,5]],"date-time":"2020-03-05T00:00:00Z","timestamp":1583366400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Queen's University of Belfast"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2020,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Software engineering (SE) research should be relevant to industrial practice. There have been regular discussions in the SE community on this issue since the 1980\u2019s, led by pioneers such as Robert Glass. As we recently passed the milestone of \u201c50 years of software engineering\u201d, some recent positive efforts have been made in this direction, e.g., establishing \u201cindustrial\u201d tracks in several SE conferences. However, many researchers and practitioners believe that we, as a community, are still struggling with research relevance and utility. The goal of this paper is to synthesize the evidence and experience-based opinions shared on this topic so far in the SE community, and to encourage the community to further reflect and act on the research relevance. For this purpose, we have conducted a Multi-vocal Literature Review (MLR) of 54 systematically-selected sources (papers and non peer-reviewed articles). Instead of relying on and considering the individual opinions on research relevance, mentioned in each of the sources, the MLR aims to synthesize and provide the \u201cholistic\u201d view on the topic. The highlights of our MLR findings are as follows. The top three root causes of low relevance, discussed in the community, are: (1) Researchers having simplistic views (or wrong assumptions) about SE in practice; (2) Lack of connection with industry; and (3) Wrong identification of research problems. The top three suggestions for improving research relevance are: (1) Using appropriate research approaches such as action-research; (2) Choosing relevant (practical) research problems; and (3) Collaborating with industry. By synthesizing all the discussions on this important topic so far, this paper aims to encourage further discussions and actions in the community to increase our collective efforts to improve the research relevance. Furthermore, we raise the need for empirically-grounded and rigorous studies on the relevance problem in SE research, as carried out in other fields such as management science.<\/jats:p>","DOI":"10.1007\/s10664-020-09803-0","type":"journal-article","created":{"date-parts":[[2020,3,5]],"date-time":"2020-03-05T06:03:44Z","timestamp":1583388224000},"page":"1687-1754","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":42,"title":["Practical relevance of software engineering research: synthesizing the community\u2019s voice"],"prefix":"10.1007","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6590-7576","authenticated-orcid":false,"given":"Vahid","family":"Garousi","sequence":"first","affiliation":[]},{"given":"Markus","family":"Borg","sequence":"additional","affiliation":[]},{"given":"Markku","family":"Oivo","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,3,5]]},"reference":[{"issue":"8","key":"9803_CR1","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1145\/1278201.1278226","volume":"50","author":"RL Glass","year":"2007","unstructured":"Glass RL (2007) A deja-vu look at software engineering researchers who care about practice. Communication of ACM 50(8):21\u201323. https:\/\/doi.org\/10.1145\/1278201.1278226","journal-title":"Communication of ACM"},{"issue":"3","key":"9803_CR2","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1007\/s10664-010-9146-4","volume":"16","author":"M Ivarsson","year":"2011","unstructured":"Ivarsson M, Gorschek T (2011) A method for evaluating rigor and industrial relevance of technology evaluations. Empir Softw Eng 16(3):365\u2013395. https:\/\/doi.org\/10.1007\/s10664-010-9146-4","journal-title":"Empir Softw Eng"},{"key":"9803_CR3","unstructured":"Mahaux M, Mavin A (2013) A new paradigm for applied requirements engineering research,\" in IEEE International Requirements Engineering Conference, pp. 353\u2013356"},{"key":"9803_CR4","unstructured":"Ko AJ (2017) A three-year participant observation of software startup software evolution,\" in Proceedings of International Conference on Software Engineering: Software Engineering in Practice Track, pp. 3\u201312"},{"issue":"10","key":"9803_CR5","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/2.796106","volume":"32","author":"SL Pfleeger","year":"1999","unstructured":"Pfleeger SL (1999) Albert Einstein and empirical software engineering. IEEE Computer 32(10):32\u201338","journal-title":"IEEE Computer"},{"key":"9803_CR6","doi-asserted-by":"publisher","unstructured":"Arcuri A (2017) An experience report on applying software testing academic results in industry: we need usable automated test generation. Empir Softw Eng. https:\/\/doi.org\/10.1007\/s10664-017-9570-9","DOI":"10.1007\/s10664-017-9570-9"},{"key":"9803_CR7","doi-asserted-by":"crossref","unstructured":"Parnin C, Orso A (2011) Are automated debugging techniques actually helping programmers?, in Proceedings of the International Symposium on Software Testing and Analysis, 2011, 2001445, pp. 199\u2013209, doi: 10.1145\/2001420.2001445","DOI":"10.1145\/2001420.2001445"},{"key":"9803_CR8","unstructured":"Riehle D (2019) Challenges to making software engineering research relevant to industry,\" http:\/\/dirkriehle.com\/2016\/11\/17\/challenges-to-making-software-engineering-research-relevant-to-industry\/\n, Last accessed: Feb. 1, 2019"},{"issue":"4","key":"9803_CR9","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1109\/MS.2012.86","volume":"29","author":"L Briand","year":"2012","unstructured":"Briand L (2012) Embracing the engineering side of software engineering. IEEE Softw 29(4):96\u201396. https:\/\/doi.org\/10.1109\/MS.2012.86","journal-title":"IEEE Softw"},{"issue":"6","key":"9803_CR10","doi-asserted-by":"crossref","first-page":"579","DOI":"10.1093\/comjnl\/35.6.579","volume":"35","author":"LM Barroca","year":"1992","unstructured":"Barroca LM, McDermid JA (1992) Formal methods: use and relevance for the development of safety-critical systems. Comput J 35(6):579\u2013599","journal-title":"Comput J"},{"key":"9803_CR11","unstructured":"Ko AJ (2019) \"Grand challenges in program comprehension and software repository mining: interdisciplinarity and research relevance,\" https:\/\/medium.com\/bits-and-behavior\/grand-challenges-in-program-comprehension-and-software-repository-mining-my-keynote-on-7bf2cfc182a5\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR12","unstructured":"Miller S (1997) \"How Can Requirements Engineering Research Become Requirements Engineering Practice?,\" in A panel in the Int. Conf. on Requirements Engineering (RE), p. 260"},{"key":"9803_CR13","unstructured":"Aranda J (2019) \"How do practitioners perceive software engineering research?,\" https:\/\/catenary.wordpress.com\/2011\/05\/19\/how-do-practitioners-perceive-software-engineering-research\/\n, 2011, Last accessed: Feb. 1, 2019"},{"key":"9803_CR14","doi-asserted-by":"crossref","unstructured":"Carver JC, Dieste O, Kraft NA, Lo D, Zimmermann T (2016) \"how practitioners perceive the relevance of ESEM research,\" presented at the proceedings of the ACM\/IEEE international symposium on empirical software engineering and measurement","DOI":"10.1145\/2961111.2962597"},{"key":"9803_CR15","doi-asserted-by":"crossref","unstructured":"Lo D, Nagappan N, Zimmermann T (2015) \"how practitioners perceive the relevance of software engineering research,\" presented at the proceedings of joint meeting on foundations of software engineering","DOI":"10.1145\/2786805.2786809"},{"key":"9803_CR16","unstructured":"Murphy G (2019) \"Impactful SE Research: Some Do's and More Don'ts,\" https:\/\/www.slideshare.net\/murphygc\/icse2017-doctoralsymposium\n, Last accessed: Feb. 1, 2019"},{"issue":"2","key":"9803_CR17","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1109\/MS.2009.40","volume":"26","author":"RL Glass","year":"2009","unstructured":"Glass RL (2009) Making research more relevant while not diminishing its rigor. IEEE Softw 26(2):96\u201396. https:\/\/doi.org\/10.1109\/MS.2009.40","journal-title":"IEEE Softw"},{"issue":"4","key":"9803_CR18","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1109\/MC.2014.92","volume":"47","author":"S Beecham","year":"2014","unstructured":"Beecham S, Leary PO, Baker S, Richardson I, Noll J (2014) Making software engineering research relevant. IEEE Computer 47(4):80\u201383. https:\/\/doi.org\/10.1109\/MC.2014.92","journal-title":"IEEE Computer"},{"key":"9803_CR19","unstructured":"Zeller A (2019) \"On impact in Software Engineering Research,\" ICSE 2018 New Faculty Symposium,\nhttps:\/\/www.slideshare.net\/andreas.zeller\/on-impact-in-software-engineering-research-icse-2018-new-faculty-symposium\n, Last accessed: Feb. 1"},{"key":"9803_CR20","doi-asserted-by":"crossref","unstructured":"Floyd C (1985) \"On the relevance of formal methods to software development,\" in International Joint Conference on Theory and Practice of Software Development: Springer, pp. 1\u201311","DOI":"10.1007\/3-540-15199-0_1"},{"key":"9803_CR21","unstructured":"Tan A, Tang A (2019) \"On the Worthiness of Software Engineering Research,\" Technical report,\nhttp:\/\/shidler.hawaii.edu\/sites\/shidler.hawaii.edu\/files\/users\/kazman\/se_research_worthiness.pdf\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR22","doi-asserted-by":"crossref","unstructured":"Ghaisas S (2014) \"Practicing what we preach,\" IEEE software, no. 1, pp. 88\u201392","DOI":"10.1109\/MS.2014.10"},{"issue":"1","key":"9803_CR23","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1109\/MC.2010.22","volume":"43","author":"DL Parnas","year":"2010","unstructured":"Parnas DL (2010) Really rethinking'formal methods'. Computer 43(1):28\u201334","journal-title":"Computer"},{"issue":"3","key":"9803_CR24","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1007\/s007660200008","volume":"7","author":"H Kaindl","year":"2002","unstructured":"Kaindl H et al (2002) Requirements engineering and technology transfer: obstacles, incentives and improvement agenda. Requir Eng 7(3):113\u2013123","journal-title":"Requir Eng"},{"issue":"2","key":"9803_CR25","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/s007660200007","volume":"7","author":"AM Davis","year":"2002","unstructured":"Davis AM, Hickey AM (2002) Requirements researchers: do we practice what we preach? Requir Eng 7(2):107\u2013111","journal-title":"Requir Eng"},{"key":"9803_CR26","unstructured":"Obrenovic Z (2018) Research and practice: essays of a researcher-practitioner. Amazon Digital Services"},{"key":"9803_CR27","unstructured":"Xie T (2019) \"Research methodology on pursuing impact-driven research,\" https:\/\/www.slideshare.net\/taoxiease\/isec18-tutorial-research-methodology-on-pursuing-impactdriven-research\n, Last accessed: Feb. 1, 2019"},{"issue":"10","key":"9803_CR28","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1109\/MC.2011.268","volume":"44","author":"D Parnas","year":"2011","unstructured":"Parnas D (2011) Software engineering-missing in action: a personal perspective. Computer 44(10):54\u201358","journal-title":"Computer"},{"issue":"5","key":"9803_CR29","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1109\/MS.2018.290110216","volume":"35","author":"V Basili","year":"2018","unstructured":"Basili V, Briand L, Bianculli D, Nejati S, Pastore F, Sabetzadeh M (2018) Software engineering research and industry: a symbiotic relationship to Foster impact. IEEE Softw 35(5):44\u201349","journal-title":"IEEE Softw"},{"issue":"5","key":"9803_CR30","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1109\/52.232392","volume":"10","author":"C Potts","year":"1993","unstructured":"Potts C (1993) Software-engineering research revisited. IEEE Softw 10(5):19\u201328. https:\/\/doi.org\/10.1109\/52.232392","journal-title":"IEEE Softw"},{"key":"9803_CR31","unstructured":"Wohlin C (2013a) \"Software Engineering Research under the Lamppost,\" in International Joint Conference on Software Technologies, J. Cordeiro, D. A. Marca, and M. van Sinderen, Eds., pp. IS-11. [Online]. Available: http:\/\/dblp.uni-trier.de\/db\/conf\/icsoft\/icsoft2013.html#Wohlin13. [Online]. Available:\nhttp:\/\/dblp.uni-trier.de\/db\/conf\/icsoft\/icsoft2013.html#Wohlin13"},{"issue":"4","key":"9803_CR32","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1109\/MS.2006.113","volume":"23","author":"RL Glass","year":"2006","unstructured":"Glass RL, Collard R, Bertolino A, Bach J, Kaner C (2006) Software testing and industry needs. IEEE Softw 23(4):55\u201357","journal-title":"IEEE Softw"},{"issue":"3","key":"9803_CR33","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1145\/279437.279464","volume":"23","author":"DL Parnas","year":"1998","unstructured":"Parnas DL (1998) Successful software engineering research. SIGSOFT Softw. Eng. Notes 23(3):64\u201368. https:\/\/doi.org\/10.1145\/279437.279464","journal-title":"SIGSOFT Softw. Eng. Notes"},{"issue":"1","key":"9803_CR34","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1145\/157397.157399","volume":"18","author":"WF Tichy","year":"1993","unstructured":"Tichy WF, Habermann N, Prechelt L (1993) Summary of the Dagstuhl workshop on future directions in software engineering: 1992. SIGSOFT Softw. Eng. Notes 18(1):35\u201348. https:\/\/doi.org\/10.1145\/157397.157399","journal-title":"SIGSOFT Softw. Eng. Notes"},{"key":"9803_CR35","unstructured":"Sommerville I (2019) \"The (ir)relevance of academic software engineering research,\" http:\/\/iansommerville.com\/systems-software-and-technology\/the-irrelevance-of-academic-software-engineering-research\/\n, Last accessed: Feb. 1, 2019"},{"issue":"6","key":"9803_CR36","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1109\/MS.2015.107","volume":"32","author":"WG Griswold","year":"2015","unstructured":"Griswold WG, Opdyke WF (2015) The birth of refactoring: a retrospective on the nature of high-impact software engineering research. IEEE Softw 32(6):30\u201338","journal-title":"IEEE Softw"},{"key":"9803_CR37","unstructured":"Ko AJ (2015) \"The black hole of software engineering research,\" https:\/\/blogs.uw.edu\/ajko\/2015\/10\/05\/the-black-hole-of-software-engineering-research\/\n, Last accessed: Feb. 1, 2019"},{"issue":"5","key":"9803_CR38","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1109\/MS.2017.3571562","volume":"34","author":"L Briand","year":"2017","unstructured":"Briand L, Bianculli D, Nejati S, Pastore F, Sabetzadeh M (2017a) The case for context-driven software engineering research: generalizability is overrated. IEEE Softw 34(5):72\u201375. https:\/\/doi.org\/10.1109\/MS.2017.3571562","journal-title":"IEEE Softw"},{"key":"9803_CR39","unstructured":"Meyer B (2010) \"The other impediment to software engineering research,\" https:\/\/bertrandmeyer.com\/2010\/04\/25\/the-other-impediment-to-software-engineering-research\/\n, Last accessed: Feb. 1, 2019"},{"issue":"11","key":"9803_CR40","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1145\/240455.240458","volume":"39","author":"RL Glass","year":"1996","unstructured":"Glass RL (1996) The relationship between theory and practice in software engineering. Commun ACM 39(11):11\u201313. https:\/\/doi.org\/10.1145\/240455.240458","journal-title":"Commun ACM"},{"issue":"6","key":"9803_CR41","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1109\/52.329400","volume":"11","author":"RL Glass","year":"1994","unstructured":"Glass RL (1994) The software-research crisis. IEEE Softw 11(6):42\u201347. https:\/\/doi.org\/10.1109\/52.329400","journal-title":"IEEE Softw"},{"key":"9803_CR42","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1016\/j.scico.2018.01.002","volume":"156","author":"P Ralph","year":"2018","unstructured":"Ralph P (2018) The two paradigms of software development research. Science of Computer Programming 156:68\u201389. https:\/\/doi.org\/10.1016\/j.scico.2018.01.002","journal-title":"Science of Computer Programming"},{"key":"9803_CR43","unstructured":"Jones D (2019a) \"The world view of research in software engineering,\" http:\/\/shape-of-code.coding-guidelines.com\/2018\/09\/26\/the-world-view-of-research-in-software-engineering\/\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR44","unstructured":"Awasthy R, Flint S, Sankaranarayana R (2016) \"Towards improved Adoption: Effectiveness of Research Tools in the Real World,\" in International Workshop on Quantitative Approaches to Software Quality, pp. 20\u201326"},{"key":"9803_CR45","doi-asserted-by":"crossref","unstructured":"Beecham S, Richardson I, Sommerville I, O\u2019Leary P, Baker S, and Noll J (2018) \"triangulating research dissemination methods: a three-pronged approach to closing the research\u2013practice divide,\" in Software Technology: 10 Years of Innovation in IEEE Computer: John Wiley & Sons","DOI":"10.1002\/9781119174240.ch4"},{"key":"9803_CR46","unstructured":"Wilson G, Aranda J (2019) \"Two Solitudes Illustrated,\" http:\/\/third-bit.com\/2012\/12\/06\/two-solitudes-illustrated.html\n, 2012, Last accessed: Feb. 1, 2019"},{"key":"9803_CR47","unstructured":"Jones D (2019b) \"Unappreciated bubble research,\" http:\/\/shape-of-code.coding-guidelines.com\/2017\/06\/07\/unappreciated-bubble-research\/\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR48","doi-asserted-by":"crossref","unstructured":"Briand LC (2011) \"Useful software engineering research - leading a double-agent life,\" in IEEE International Conference on Software Maintenance, 25\u201330 Sept. 2011pp. 2\u20132, doi: 10.1109\/ICSM.2011.6080766","DOI":"10.1109\/ICSM.2011.6080766"},{"key":"9803_CR49","unstructured":"Jones D (2019c) \"Waiting for the funerals: culture in software engineering research,\" http:\/\/shape-of-code.coding-guidelines.com\/2018\/11\/22\/waiting-for-the-funerals-culture-in-software-engineering-research\/\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR50","doi-asserted-by":"crossref","unstructured":"Ivanov V, Rogers A, Succi G, Yi J, Zorin V (2017) \"What do software engineers care about? gaps between research and practice,\" in Proceedings of the Joint Meeting on Foundations of Software Engineering, pp. 890\u2013895","DOI":"10.1145\/3106237.3117778"},{"issue":"6","key":"9803_CR51","doi-asserted-by":"crossref","first-page":"869","DOI":"10.1109\/TSE.2012.74","volume":"39","author":"I Malavolta","year":"2013","unstructured":"Malavolta I, Lago P, Muccini H, Pelliccione P, Tang A (2013) What industry needs from architectural languages: a survey. IEEE Trans Softw Eng 39(6):869\u2013891","journal-title":"IEEE Trans Softw Eng"},{"key":"9803_CR52","doi-asserted-by":"crossref","unstructured":"Garousi V, Felderer M, Kuhrmann M, and Herkilo\u011flu K (2017a) \"what industry wants from academia in software testing? Hearing practitioners\u2019 opinions,\" in International Conference on Evaluation and Assessment in Software Engineering, Karlskrona, Sweden, pp. 65\u201369","DOI":"10.1145\/3084226.3084264"},{"key":"9803_CR53","unstructured":"ICST (2018), \"Panel: When are Software Testing Research Contributions, Real Contributions?,\" https:\/\/www.youtube.com\/watch?v=I8yz0HLWeQ8\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR54","doi-asserted-by":"crossref","unstructured":"Beecham S, OLeary P, Richardson I, Baker S, and Noll J (2013) \"Who Are We Doing Global Software Engineering Research For?,\" in IEEE International Conference on Global Software Engineering, 26\u201329 Aug. 2013, pp. 41\u201350, doi: 10.1109\/ICGSE.2013.14","DOI":"10.1109\/ICGSE.2013.14"},{"key":"9803_CR55","doi-asserted-by":"crossref","unstructured":"Kieser A, Nicolai A, Seidl D (2015) \"The Practical Relevance of Management Research: Turning the Debate on Relevance into a Rigorous Scientific Research Program,\" The Academy of Management Annals, vol. 9, no. 1, pp. 143\u2013233, 2015\/01\/01, doi: 10.1080\/19416520.2015.1011853","DOI":"10.1080\/19416520.2015.1011853"},{"issue":"4","key":"9803_CR56","doi-asserted-by":"crossref","first-page":"3","DOI":"10.2307\/41165601","volume":"6","author":"M Haire","year":"1964","unstructured":"Haire M (1964) The social sciences and management practices. Calif Manag Rev 6(4):3\u201310","journal-title":"Calif Manag Rev"},{"issue":"12","key":"9803_CR57","doi-asserted-by":"crossref","first-page":"1326","DOI":"10.1145\/214956.214961","volume":"28","author":"DL Parnas","year":"1985","unstructured":"Parnas DL (1985) Software aspects of strategic defense systems. Commun ACM 28(12):1326\u20131335","journal-title":"Commun ACM"},{"key":"9803_CR58","unstructured":"Wilson G (2019) \"It Will Never Work in Theory,\" http:\/\/neverworkintheory.org\/about.html\n, Last accessed: Feb. 1, 2019"},{"issue":"5","key":"9803_CR59","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1109\/MS.2018.3571228","volume":"35","author":"C Ebert","year":"2018","unstructured":"Ebert C (2018) 50 years of software engineering: Progress and perils. IEEE Softw 35(5):94\u2013101. https:\/\/doi.org\/10.1109\/MS.2018.3571228","journal-title":"IEEE Softw"},{"key":"9803_CR60","unstructured":"Heleta S (2019) \"Academics can change the world \u2013 if they stop talking only to their peers,\" https:\/\/theconversation.com\/academics-can-change-the-world-if-they-stop-talking-only-to-their-peers-55713\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR61","unstructured":"Rohrer JE, Phillips KA, Morrison KR, Andersen RM, Aday LA (2000) \"Rigor at the expense of relevance equals rigidity: Where to next in the study of medical care utilization?,\" Health Services Research, Note vol. 34, no. 6, pp. 1307\u20131314. [Online]. Available:\nhttps:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-0033973269&partnerID=40&md5=db4b08c4b67ad1b90a795094e749cd21"},{"key":"9803_CR62","unstructured":"Slawson DC, Shaughnessy AF, Barry H (2001) \"Which should come first: Rigor or relevance?,\" Journal of Family Practice, Review vol. 50, no. 3, pp. 209\u2013210. [Online]. Available:\nhttps:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-0035103397&partnerID=40&md5=e93426c916c60d70a56bacd109e3a26f"},{"key":"9803_CR63","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1108\/09696470410538288","volume":"11","author":"D Andriessen","year":"2004","unstructured":"Andriessen D (2004) \"reconciling the rigor-relevance dilemma in intellectual capital research,\" The Learning Organization. Article 11:393\u2013401. https:\/\/doi.org\/10.1108\/09696470410538288","journal-title":"Article"},{"key":"9803_CR64","unstructured":"Desouza KC, El Sawy OA, Galliers RD, Loebbecke C, and Watson RT (2005) \"Information systems research that really matters: Beyond the is rigor versus relevance debate,\" in International Conference on Information Systems, pp. 957\u2013959"},{"issue":"2","key":"9803_CR65","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1111\/j.1745-493X.2008.00056.x","volume":"44","author":"BB Flynn","year":"2008","unstructured":"Flynn BB (2008) Having it all: rigor versus relevance in supply chain management research. Journal of Supply Chain Management 44(2):63\u201367. https:\/\/doi.org\/10.1111\/j.1745-493X.2008.00056.x","journal-title":"Journal of Supply Chain Management"},{"key":"9803_CR66","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.infsof.2018.09.006","volume":"106","author":"V Garousi","year":"2019","unstructured":"Garousi V, Felderer M, M\u00e4ntyl\u00e4 MV (2019) Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Inf Softw Technol 106:101\u2013121","journal-title":"Inf Softw Technol"},{"key":"9803_CR67","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.infsof.2016.04.015","volume":"76","author":"V Garousi","year":"2016","unstructured":"Garousi V, M\u00e4ntyl\u00e4 MV (2016a) When and what to automate in software testing? A multivocal literature review. Inf Softw Technol 76:92\u2013117","journal-title":"Inf Softw Technol"},{"key":"9803_CR68","doi-asserted-by":"crossref","unstructured":"M\u00e4ntyl\u00e4 MV, Smolander K (2016) \"Gamification of Software Testing - An MLR,\" in International Conference on Product-Focused Software Process Improvement, pp. 611\u2013614","DOI":"10.1007\/978-3-319-49094-6_46"},{"key":"9803_CR69","doi-asserted-by":"publisher","unstructured":"Lwakatare LE, Kuvaja P, Oivo M (2016) \"Relationship of DevOps to Agile, Lean and Continuous Deployment: A Multivocal Literature Review Study,\" in Proceedings of International Conference on Product-Focused Software Process Improvement, pp. 399\u2013415, doi: https:\/\/doi.org\/10.1007\/978-3-319-49094-6_27","DOI":"10.1007\/978-3-319-49094-6_27"},{"key":"9803_CR70","doi-asserted-by":"publisher","unstructured":"Franca BBND, Jeronimo JH, and Travassos GH (2016) \"Characterizing DevOps by Hearing Multiple Voices,\" in Proceedings of the Brazilian Symposium on Software Engineering, 2973845, pp. 53\u201362, doi: https:\/\/doi.org\/10.1145\/2973839.2973845","DOI":"10.1145\/2973839.2973845"},{"key":"9803_CR71","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.infsof.2017.01.001","volume":"85","author":"V Garousi","year":"2017","unstructured":"Garousi V, Felderer M, Hacalo\u011flu T (2017b) Software test maturity assessment and test process improvement: a multivocal literature review. Inf Softw Technol 85:16\u201342","journal-title":"Inf Softw Technol"},{"key":"9803_CR72","doi-asserted-by":"crossref","unstructured":"Myrbakken H, Colomo-Palacios R (2017) \"DevSecOps: A Multivocal Literature Review,\" in Conf. on Software Process Improvement and Capability Determination, A. Mas, A. Mesquida, R. V. O'Connor, T. Rout, and A. Dorling, Eds., 2017, pp. 17\u201329","DOI":"10.1007\/978-3-319-67383-7_2"},{"key":"9803_CR73","doi-asserted-by":"crossref","unstructured":"Garousi V, Felderer M, M\u00e4ntyl\u00e4 MV (2016a) \"the need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature,\" in International Conference on Evaluation and Assessment in Software Engineering, Limmerick, Ireland, pp. 171\u2013176","DOI":"10.1145\/2915970.2916008"},{"issue":"3","key":"9803_CR74","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1177\/1534484317707562","volume":"16","author":"O Tkachenko","year":"2017","unstructured":"Tkachenko O, Hahn H-J, Peterson SL (2017) Research\u2013practice gap in applied fields: an integrative literature review. Hum Resour Dev Rev 16(3):235\u2013262","journal-title":"Hum Resour Dev Rev"},{"issue":"2","key":"9803_CR75","doi-asserted-by":"crossref","first-page":"166","DOI":"10.3917\/mana.202.0166","volume":"20","author":"G Carton","year":"2017","unstructured":"Carton G, Mouricou P (2017) Is management research relevant? A systematic analysis of the rigor-relevance debate in top-tier journals (1994\u20132013). M@n@gement 20(2):166\u2013203","journal-title":"M@n@gement"},{"key":"9803_CR76","doi-asserted-by":"crossref","unstructured":"Moeini M, Rahrovani Y, Chan YE (2019) \"A review of the practical relevance of IS strategy scholarly research,\" J Strateg Inf Syst","DOI":"10.1016\/j.jsis.2018.12.003"},{"key":"9803_CR77","unstructured":"Brings J, Daun M, Brinckmann S, Keller K, Weyer T (2018) \"Approaches, success factors, and barriers for technology transfer in software engineering\u2014Results of a systematic literature review,\" Journal of Software: Evolution and Process, vol. 30, no. 11, p. e1981"},{"issue":"3","key":"9803_CR78","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1093\/rev\/7.3.159","volume":"7","author":"S Hemlin","year":"1998","unstructured":"Hemlin S (1998) Utility evaluation of academic research: six basic propositions. Research Evaluation 7(3):159\u2013165. https:\/\/doi.org\/10.1093\/rev\/7.3.159","journal-title":"Research Evaluation"},{"issue":"3","key":"9803_CR79","first-page":"231","volume":"12","author":"H Averch","year":"1975","unstructured":"Averch H (1975) Notes on improving research utility. Inquiry 12(3):231\u2013234","journal-title":"Inquiry"},{"issue":"1","key":"9803_CR80","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1016\/j.jss.2010.09.036","volume":"84","author":"WW Eric","year":"2011","unstructured":"Eric WW, Tse TH, Glass RL, Basili VR, Chen TY (2011) An assessment of systems and software engineering scholars and institutions (2003\u20132007 and 2004\u20132008). J Syst Softw 84(1):162\u2013168","journal-title":"J Syst Softw"},{"key":"9803_CR81","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.cosrev.2015.12.002","volume":"19","author":"V Garousi","year":"2016","unstructured":"Garousi V, M\u00e4ntyl\u00e4 MV (2016b) Citations, research topics and active countries in software engineering: a bibliometrics study. Elsevier Computer Science Review 19:56\u201377","journal-title":"Elsevier Computer Science Review"},{"issue":"3","key":"9803_CR82","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1109\/MC.2008.85","volume":"41","author":"LJ Osterweil","year":"2008","unstructured":"Osterweil LJ, Ghezzi C, Kramer J, Wolf AL (2008) Determining the impact of software engineering research on practice. IEEE Computer 41(3):39\u201349. https:\/\/doi.org\/10.1109\/MC.2008.85","journal-title":"IEEE Computer"},{"issue":"1","key":"9803_CR83","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1145\/1226816.1226829","volume":"32","author":"W Emmerich","year":"2007","unstructured":"Emmerich W, Aoyama M, Sventek J (2007) The impact of research on middleware technology. SIGSOFT Softw Eng Notes 32(1):21\u201346. https:\/\/doi.org\/10.1145\/1226816.1226829","journal-title":"SIGSOFT Softw Eng Notes"},{"issue":"6","key":"9803_CR84","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1145\/1449603.1449609","volume":"33","author":"D Rombach","year":"2008","unstructured":"Rombach D, Ciolkowski M, Jeffery R, Laitenberger O, McGarry F, Shull F (2008) Impact of research on practice in the field of inspections, reviews and walkthroughs: learning from successful industrial uses. SIGSOFT Softw. Eng. Notes 33(6):26\u201335. https:\/\/doi.org\/10.1145\/1449603.1449609","journal-title":"SIGSOFT Softw. Eng. Notes"},{"key":"9803_CR85","doi-asserted-by":"crossref","unstructured":"Wohlin C, Runeson P, H\u00f6st M, Ohlsson MC, Regnell B, Wessl\u00e9n A (2012a) Experimentation in software engineering. Springer Science & Business Media","DOI":"10.1007\/978-3-642-29044-2"},{"issue":"4","key":"9803_CR86","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1007\/s00766-006-0037-6","volume":"11","author":"RJ Wieringa","year":"2006","unstructured":"Wieringa RJ, Heerkens JM (2006) The methodological soundness of requirements engineering papers: a conceptual framework and two case studies. Requir Eng 11(4):295\u2013307","journal-title":"Requir Eng"},{"key":"9803_CR87","first-page":"49","volume":"27","author":"PG Keen","year":"1991","unstructured":"Keen PG (1991) Relevance and rigor in information systems research: improving quality, confidence, cohesion and impact. Information systems research: Contemporary approaches and emergent traditions 27:49","journal-title":"Information systems research: Contemporary approaches and emergent traditions"},{"issue":"4","key":"9803_CR88","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1348\/096317901167451","volume":"74","author":"N Anderson","year":"2010","unstructured":"Anderson N, Herriot P, Hodgkinson Gerard P (2010) The practitioner-researcher divide in industrial, work and organizational (IWO) psychology: where are we now, and where do we go from here? J Occup Organ Psychol 74(4):391\u2013411. https:\/\/doi.org\/10.1348\/096317901167451","journal-title":"J Occup Organ Psychol"},{"key":"9803_CR89","unstructured":"Coplien J (2019) \"Academic Relevance,\" http:\/\/wiki.c2.com\/?AcademicRelevance\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR90","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1016\/j.mar.2016.07.001","volume":"34","author":"A Rautiainen","year":"2017","unstructured":"Rautiainen A, Sippola K, M\u00e4tt\u00f6 T (2017) Perspectives on relevance: the relevance test in the constructive research approach. Manag Account Res 34:19\u201329","journal-title":"Manag Account Res"},{"issue":"2","key":"9803_CR91","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1007\/s10664-008-9102-8","volume":"14","author":"P Runeson","year":"2009","unstructured":"Runeson P, H\u00f6st M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131\u2013164","journal-title":"Empir Softw Eng"},{"key":"9803_CR92","unstructured":"Kitchenham B, Charters S (2007a) \"guidelines for performing systematic literature reviews in software engineering,\" Technical report, School of Computer Science, Keele University, EBSE-2007-01"},{"key":"9803_CR93","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.infsof.2015.03.007","volume":"64","author":"K Petersen","year":"2015","unstructured":"Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64:1\u201318. https:\/\/doi.org\/10.1016\/j.infsof.2015.03.007","journal-title":"Information and Software Technology"},{"key":"9803_CR94","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.infsof.2015.08.004","volume":"68","author":"D Fucci","year":"2015","unstructured":"Fucci D, Turhan B, Juristo N, Dieste O, Tosun-Misirli A, Oivo M (2015) Towards an operationalization of test-driven development skills: an industrial empirical study. Inf Softw Technol 68:82\u201397","journal-title":"Inf Softw Technol"},{"key":"9803_CR95","doi-asserted-by":"crossref","unstructured":"Bhat T and Nagappan N (2006) \"Evaluating the efficacy of test-driven development: industrial case studies,\" in Proceedings of the 2006 ACM\/IEEE international symposium on Empirical software engineering: ACM, pp. 356\u2013363","DOI":"10.1145\/1159733.1159787"},{"issue":"6","key":"9803_CR96","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1109\/TSE.2005.72","volume":"31","author":"T Zimmermann","year":"2005","unstructured":"Zimmermann T, Zeller A, Weissgerber P, Diehl S (2005) Mining version histories to guide software changes. IEEE Trans Softw Eng 31(6):429\u2013445","journal-title":"IEEE Trans Softw Eng"},{"key":"9803_CR97","doi-asserted-by":"crossref","unstructured":"Johnson B, Zimmermann T, Bird C (2019) The effect of work environments on productivity and satisfaction of software engineers. IEEE Trans Softw Eng","DOI":"10.1109\/TSE.2019.2903053"},{"issue":"1","key":"9803_CR98","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1109\/TSE.2017.2768368","volume":"45","author":"E Kalliamvakou","year":"2019","unstructured":"Kalliamvakou E, Bird C, Zimmermann T, Begel A, DeLine R, German DM (2019) What makes a great manager of software engineers? IEEE Trans Softw Eng 45(1):87\u2013106","journal-title":"IEEE Trans Softw Eng"},{"key":"9803_CR99","doi-asserted-by":"crossref","unstructured":"Alshahwan N et al. (2018) \"Deploying Search Based Software Engineering with Sapienz at Facebook,\" in International Symposium on Search Based Software Engineering: Springer, pp. 3\u201345","DOI":"10.1007\/978-3-319-99241-9_1"},{"key":"9803_CR100","unstructured":"Shepherd DC and Robinson BP (2017) \"combined code searching and automatic code navigation,\" Google Patents, US9727635B2"},{"key":"9803_CR101","doi-asserted-by":"crossref","unstructured":"Benbasat I and Zmud RW (1999) \"Empirical research in information systems: the practice of relevance,\" MIS Q, pp. 3\u201316","DOI":"10.2307\/249403"},{"key":"9803_CR102","doi-asserted-by":"crossref","unstructured":"Toffel MW (2016) \"Enhancing the Practical Relevance of Research,\" Production and Operations Management, pp. n\/a-n\/a, doi: 10.1111\/poms.12558","DOI":"10.1111\/poms.12558"},{"key":"9803_CR103","doi-asserted-by":"crossref","unstructured":"Naitou K et al. (2018) \"Toward introducing automated program repair techniques to industrial software development,\" in Proceedings of the Conference on Program Comprehension, pp. 332\u2013335","DOI":"10.1145\/3196321.3196358"},{"key":"9803_CR104","doi-asserted-by":"crossref","unstructured":"Petersen K, Wohlin C (2009a) \"Context in industrial software engineering research,\" in International Symposium on Empirical Software Engineering and Measurement: IEEE, pp. 401\u2013404","DOI":"10.1109\/ESEM.2009.5316010"},{"key":"9803_CR105","unstructured":"Anderson V and Johnson L (1997) Systems thinking basics: from concepts to causal loops. Pegasus Communications"},{"key":"9803_CR106","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1016\/j.infsof.2016.07.006","volume":"79","author":"V Garousi","year":"2016","unstructured":"Garousi V, Petersen K, \u00d6zkan B (2016b) Challenges and best practices in industry-academia collaborations in software engineering: a systematic literature review. Inf Softw Technol 79:106\u2013127","journal-title":"Inf Softw Technol"},{"key":"9803_CR107","unstructured":"H. Simon (1976) Administrative Behavior, 3rd edition ed. Simon & Schuster"},{"key":"9803_CR108","doi-asserted-by":"crossref","unstructured":"Garousi V, \u00d6zkan R, and Betin-Can A (2018) \"Multi-objective regression test selection in practice: An empirical study in the defense software industry,\" Information and Software Technology, In press, vol. 103, pp. 40\u201354, doi: 10.1016\/j.infsof.2018.06.007","DOI":"10.1016\/j.infsof.2018.06.007"},{"issue":"5","key":"9803_CR109","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/MS.2017.3571562","volume":"34","author":"L Briand","year":"2017","unstructured":"Briand L, Bianculli D, Nejati S, Pastore F, Sabetzadeh M (2017b) The case for context-driven software engineering research. IEEE Softw 34(5):72\u201375","journal-title":"IEEE Softw"},{"issue":"2","key":"9803_CR110","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1109\/MS.2011.92","volume":"29","author":"C Wohlin","year":"2012","unstructured":"Wohlin C et al (2012b) The success factors powering industry-academia collaboration. IEEE Softw 29(2):67\u201373. https:\/\/doi.org\/10.1109\/MS.2011.92","journal-title":"IEEE Softw"},{"key":"9803_CR111","doi-asserted-by":"crossref","unstructured":"Garousi V, Felderer M, Fernandes JM, Pfahl D, and Mantyla MV (2017c) \"industry-academia collaborations in software engineering: an empirical analysis of challenges, patterns and anti-patterns in research projects,\" in Proceedings of International Conference on Evaluation and Assessment in Software Engineering, Karlskrona, Sweden, pp. 224\u2013229","DOI":"10.1145\/3084226.3084279"},{"key":"9803_CR112","unstructured":"Wohlin C (2013b) \"software engineering research under the lamppost,\" in Proceedings of the International Joint Conference on Software Technologies"},{"key":"9803_CR113","doi-asserted-by":"crossref","unstructured":"Yamashita A (2015) \"Integration of SE Research and Industry: Reflections, Theories and Illustrative Example,\" in IEEE\/ACM International Workshop on Software Engineering Research and Industrial Practice, 17\u201317 May 2015, pp. 11\u201317, doi: 10.1109\/SERIP.2015.11","DOI":"10.1109\/SERIP.2015.11"},{"key":"9803_CR114","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.infsof.2017.10.013","volume":"95","author":"T Mikkonen","year":"2018","unstructured":"Mikkonen T, Lassenius C, M\u00e4nnist\u00f6 T, Oivo M, J\u00e4rvinen J (2018) Continuous and collaborative technology transfer: Software engineering research with real-time industry impact. Information and Software Technology 95:34\u201345. https:\/\/doi.org\/10.1016\/j.infsof.2017.10.013","journal-title":"Information and Software Technology"},{"key":"9803_CR115","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1007\/978-1-84800-044-5_11","volume-title":"Guide to advanced empirical software engineering","author":"S Easterbrook","year":"2008","unstructured":"Easterbrook S, Singer J, Storey M-A, Damian D (2008) Selecting empirical methods for software engineering research. In: Shull F, Singer J, Sj\u00f8berg DIK (eds) Guide to advanced empirical software engineering. Springer London, London, pp 285\u2013311"},{"key":"9803_CR116","doi-asserted-by":"crossref","unstructured":"Runeson P, Host M, Rainer A, Regnell B (2012) Case study research in software engineering: guidelines and examples. John Wiley & Sons","DOI":"10.1002\/9781118181034"},{"key":"9803_CR117","doi-asserted-by":"crossref","first-page":"2004","DOI":"10.4018\/978-1-60566-026-4.ch315","volume-title":"Encyclopedia of information science and technology","author":"SC Srivastava","year":"2009","unstructured":"Srivastava SC, Teo TS (2009) Information systems research relevance. In: Encyclopedia of information science and technology. IGI Global, Second Edition, pp 2004\u20132009"},{"key":"9803_CR118","doi-asserted-by":"crossref","unstructured":"Westfall R (1999) \"An IS research relevancy manifesto,\" Communications of the AIS, vol. 2, no. 2es, p. 5","DOI":"10.17705\/1CAIS.00214"},{"key":"9803_CR119","doi-asserted-by":"publisher","unstructured":"Hamet J and Michel S (2018) \"Rigor, relevance, and the knowledge \u201cmarket\u201d,\" European Business Review, Article vol. 30, no. 2, pp. 183\u2013201, doi: https:\/\/doi.org\/10.1108\/EBR-01-2017-0025","DOI":"10.1108\/EBR-01-2017-0025"},{"key":"9803_CR120","doi-asserted-by":"crossref","unstructured":"Lee AS (1999) Rigor and relevance in MIS research: beyond the approach of positivism alone. MIS Q:29\u201333","DOI":"10.2307\/249407"},{"key":"9803_CR121","unstructured":"Moody DL (2000) \"Building links between IS research and professional practice: improving the relevance and impact of IS research,\" in Proceedings of the international conference on Information systems, pp. 351\u2013360"},{"key":"9803_CR122","doi-asserted-by":"crossref","unstructured":"Biswas AK, Tortajada C (2009) Impacts of megaconferences on the water sector (no. 333.91 I34). Springer","DOI":"10.1007\/978-3-540-37224-0"},{"issue":"2","key":"9803_CR123","doi-asserted-by":"crossref","first-page":"237","DOI":"10.3233\/IDA-2008-12206","volume":"12","author":"M Pechenizkiy","year":"2008","unstructured":"Pechenizkiy M, Puuronen S, Tsymbal A (2008) Towards more relevance-oriented data mining research. Intelligent Data Analysis 12(2):237\u2013249","journal-title":"Intelligent Data Analysis"},{"issue":"4","key":"9803_CR124","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1145\/1806491.1806494","volume":"17","author":"DA Norman","year":"2010","unstructured":"Norman DA (2010) The research-practice gap: the need for translational developers. Interactions 17(4):9\u201312. https:\/\/doi.org\/10.1145\/1806491.1806494","journal-title":"Interactions"},{"key":"9803_CR125","doi-asserted-by":"publisher","unstructured":"Vizecky K and El-Gayar O (2011) \"Increasing research relevance in DSS: Looking forward by reflecting on 40 years of progress,\" in Proceedings of the Annual Hawaii International Conference on System Sciences, doi: https:\/\/doi.org\/10.1109\/HICSS.2011.239. [Online]. Available:\nhttps:\/\/www.scopus.com\/inward\/record.uri?eid=2-s2.0-79952960683&doi=10.1109%2fHICSS.2011.239&partnerID=40&md5=ded84df51e0e2999ed21dab36ce59740","DOI":"10.1109\/HICSS.2011.239"},{"key":"9803_CR126","doi-asserted-by":"crossref","unstructured":"Fox S, Groesser SN (2016) \"Reframing the relevance of research to practice,\" European Management Journal, vol. 34, no. 5, pp. 457\u2013465, 2016\/10\/01\/, doi: 10.1016\/j.emj.2016.07.005","DOI":"10.1016\/j.emj.2016.07.005"},{"key":"9803_CR127","unstructured":"Sperber D, Wilson D (2004) Relevance theory. Handbook of Pragmatics Oxford: Blackwell:607\u2013632"},{"issue":"3","key":"9803_CR128","doi-asserted-by":"crossref","first-page":"534","DOI":"10.1111\/j.1467-6486.2009.00832.x","volume":"46","author":"GP Hodgkinson","year":"2009","unstructured":"Hodgkinson GP, Rousseau DM (2009) Bridging the rigour\u2013relevance gap in management research: It's already happening! J Manag Stud 46(3):534\u2013546","journal-title":"J Manag Stud"},{"issue":"3","key":"9803_CR129","doi-asserted-by":"crossref","first-page":"516","DOI":"10.1111\/j.1467-6486.2009.00831.x","volume":"46","author":"A Kieser","year":"2009","unstructured":"Kieser A, Leiner L (2009) Why the rigour\u2013relevance gap in management research is unbridgeable. J Manag Stud 46(3):516\u2013533","journal-title":"J Manag Stud"},{"key":"9803_CR130","unstructured":"Estabrooks CA (1999) \"Mapping the research utilization field in nursing,\" Canadian Journal of Nursing Research Archive, vol. 31, no. 1"},{"issue":"9\u201310","key":"9803_CR131","doi-asserted-by":"crossref","first-page":"833","DOI":"10.1016\/j.infsof.2008.01.006","volume":"50","author":"T Dyb\u00e5","year":"2008","unstructured":"Dyb\u00e5 T, Dings\u00f8yr T (2008) Empirical studies of agile software development: a systematic review. Inf Softw Technol 50(9\u201310):833\u2013859","journal-title":"Inf Softw Technol"},{"issue":"10","key":"9803_CR132","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1016\/j.infsof.2014.04.014","volume":"56","author":"N Paternoster","year":"2014","unstructured":"Paternoster N, Giardino C, Unterkalmsteiner M, Gorschek T, Abrahamsson P (2014) Software development in startup companies: a systematic mapping study. Inf Softw Technol 56(10):1200\u20131218","journal-title":"Inf Softw Technol"},{"issue":"4","key":"9803_CR133","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1016\/j.infsof.2014.01.002","volume":"56","author":"H Munir","year":"2014","unstructured":"Munir H, Moayyed M, Petersen K (2014) Considering rigor and relevance when evaluating test driven development: a systematic review. Inf Softw Technol 56(4):375\u2013394","journal-title":"Inf Softw Technol"},{"key":"9803_CR134","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1016\/j.jss.2014.01.010","volume":"91","author":"S Do\u011fan","year":"2014","unstructured":"Do\u011fan S, Betin-Can A, Garousi V (2014) Web application testing: a systematic literature review. J Syst Softw 91:174\u2013201","journal-title":"J Syst Softw"},{"issue":"5","key":"9803_CR135","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/MS.2017.3641116","volume":"34","author":"V Garousi","year":"2017","unstructured":"Garousi V, Felderer M (2017) Worlds apart: industrial and academic focus areas in software testing. IEEE Softw 34(5):38\u201345","journal-title":"IEEE Softw"},{"key":"9803_CR136","unstructured":"Evans Data Corporation (2019) \"Global Developer Population and Demographic Study 2018 Vol. 1,\" https:\/\/evansdata.com\/reports\/viewRelease.php?reportID=9\n, Last accessed: Feb. 1, 2019"},{"issue":"5","key":"9803_CR137","doi-asserted-by":"crossref","first-page":"1354","DOI":"10.1016\/j.jss.2012.12.051","volume":"86","author":"V Garousi","year":"2013","unstructured":"Garousi V, Zhi J (2013) A survey of software testing practices in Canada. J Syst Softw 86(5):1354\u20131376","journal-title":"J Syst Softw"},{"key":"9803_CR138","unstructured":"King S (2019) \"From academia to Silicon Valley \u2014 and back,\" http:\/\/blogs.nature.com\/naturejobs\/2018\/04\/30\/from-academia-to-silicon-valley-and-back\/\n, 2018, Last accessed: Feb. 1, 2019"},{"key":"9803_CR139","doi-asserted-by":"crossref","unstructured":"Boehm BW, Evangelist M, Gruhn V, Kramer J, and Edward J Miller F (2000) \"why don't we get more (self?) respect: the positive impact of software engineering research upon practice,\" presented at the proceedings of international conference on software engineering","DOI":"10.1145\/337180.343191"},{"key":"9803_CR140","unstructured":"ICSE 2011, \"what industry wants from research,\" http:\/\/2011.icse-conferences.org\/content\/research-industry-panel\n, 2011, Last accessed: Feb. 1, 2019"},{"key":"9803_CR141","unstructured":"FSE 2018, \"Panel: The state of software engineering research,\" http:\/\/www.cs.ucdavis.edu\/fse2016\/program\/panel\/\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR142","unstructured":"Glass RL (2006) Software Creativity 2.0. developer.* Books"},{"key":"9803_CR143","unstructured":"Glass RL and Hunt A (2006) Software Conflict 2.0: The Art And Science of Software Engineering. Developer.* Books"},{"key":"9803_CR144","doi-asserted-by":"crossref","unstructured":"Tom E, Aurum A, and Vidgen R (2013) \"An exploration of technical debt,\" Journal of Systems and Software, vol. 86, no. 6, pp. 1498\u20131516, doi: 10.1016\/j.jss.2012.12.052","DOI":"10.1016\/j.jss.2012.12.052"},{"key":"9803_CR145","doi-asserted-by":"crossref","unstructured":"Garousi V, K\u00fc\u00e7\u00fck B (2018) \"Smells in software test code: A survey of knowledge in industry and academia,\" Journal of Systems and Software, vol. 138, pp. 52\u201381, doi: 10.1016\/j.jss.2017.12.013","DOI":"10.1016\/j.jss.2017.12.013"},{"key":"9803_CR146","doi-asserted-by":"crossref","unstructured":"Calder\u00f3n A, Ruiz M, and O'Connor RV (2018) \"A multivocal literature review on serious games for software process standards education,\" Computer Standards & Interfaces, vol. 57, pp. 36\u201348, 2018, doi: 10.1016\/j.csi.2017.11.003","DOI":"10.1016\/j.csi.2017.11.003"},{"key":"9803_CR147","unstructured":"Kitchenham B and Charters S (2007b) \"Guidelines for performing systematic literature reviews in software engineering,\" in evidence-based software engineering,\" Evidence-Based Software Engineering"},{"key":"9803_CR148","doi-asserted-by":"crossref","unstructured":"Hopewell S, McDonald S, Clarke M, and Egger M (2007) \"Grey literature in meta-analyses of randomized trials of health care interventions,\" Cochrane Database Systematic Reviews","DOI":"10.1002\/14651858.MR000010.pub3"},{"key":"9803_CR149","unstructured":"Ogawa RT and Malen B (1991) \"Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method,\" Review of Educational Research, vol. 61, no. 3, pp. 265\u2013286. [Online]. Available:\nhttp:\/\/www.jstor.org\/stable\/1170630"},{"key":"9803_CR150","doi-asserted-by":"crossref","unstructured":"Petersen K, Feldt R, Mujtaba S, and Mattsson M (2008) \"systematic mapping studies in software engineering,\" presented at the international conference on evaluation and assessment in software engineering (EASE)","DOI":"10.14236\/ewic\/EASE2008.8"},{"key":"9803_CR151","unstructured":"Rafi DM, Moses KRK, Petersen K, and Mantyla MV (2012) \"Benefits and limitations of automated software testing- Systematic literature review and practitioner survey,\" in International Workshop on Automation of Software Test, pp. 36\u201342"},{"issue":"10","key":"9803_CR152","doi-asserted-by":"crossref","first-page":"1679","DOI":"10.1016\/j.infsof.2013.03.004","volume":"55","author":"I Banerjee","year":"2013","unstructured":"Banerjee I, Nguyen B, Garousi V, Memon A (2013) Graphical user Interface (GUI) testing: systematic mapping and repository. Inf Softw Technol 55(10):1679\u20131694","journal-title":"Inf Softw Technol"},{"key":"9803_CR153","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.infsof.2014.06.009","volume":"58","author":"V Garousi","year":"2015","unstructured":"Garousi V, Amannejad Y, Betin-Can A (2015) Software test-code engineering: a systematic mapping. J Inf Softw Technol 58:123\u2013147","journal-title":"J Inf Softw Technol"},{"key":"9803_CR154","doi-asserted-by":"crossref","unstructured":"Wohlin C (2014) \"guidelines for snowballing in systematic literature studies and a replication in software engineering,\" presented at the proceedings of the 18th international conference on evaluation and assessment in software engineering, London, England, United Kingdom","DOI":"10.1145\/2601248.2601268"},{"key":"9803_CR155","doi-asserted-by":"publisher","unstructured":"Franch X et al. (2017) \"How do Practitioners Perceive the Relevance of Requirements Engineering Research? An Ongoing Study,\" in IEEE International Requirements Engineering Conference (RE), 4\u20138 Sept. 2017, pp. 382\u2013387, doi: https:\/\/doi.org\/10.1109\/RE.2017.17","DOI":"10.1109\/RE.2017.17"},{"key":"9803_CR156","unstructured":"Tyndall J (2019) \"AACODS checklist,\" Archived at the Flinders Academic Commons:\nhttps:\/\/dspace.flinders.edu.au\/jspui\/bitstream\/2328\/3326\/4\/AACODS_Checklist.pdf\n, Last accessed: Feb. 2019"},{"issue":"2\u20133","key":"9803_CR157","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/S0164-1212(99)00085-0","volume":"49","author":"C Wohlin","year":"1999","unstructured":"Wohlin C, Regnell B (1999) Strategies for industrial relevance in software engineering education. J Syst Softw 49(2\u20133):125\u2013134","journal-title":"J Syst Softw"},{"key":"9803_CR158","unstructured":"Lethbridge TC (1999) \"the relevance of education to software practitioners: data from the 1998 survey,\" School of Information Technology and Engineering, University of Ottowa, Ottowa (Canada), Computer Science Technical Report TR-99-05"},{"key":"9803_CR159","unstructured":"Miles MB, Huberman AM, and Saldana J (2014) Qualitative data analysis: a methods sourcebook, Third Edition ed. SAGE Publications Inc"},{"key":"9803_CR160","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1016\/j.jss.2014.03.082","volume":"95","author":"T Gorschek","year":"2014","unstructured":"Gorschek T, Tempero E, Angelis L (2014) On the use of software design models in software development practice: an empirical investigation. J Syst Softw 95:176\u2013193","journal-title":"J Syst Softw"},{"key":"9803_CR161","unstructured":"Glegg G (1969) The Design of Design. Cambridge University Press"},{"key":"9803_CR162","unstructured":"Washington Post (2019) \"Academia is a cult,\" https:\/\/www.washingtonpost.com\/outlook\/academia-is-a-cult\/2018\/10\/31\/eea787a0-bd08-11e8-b7d2-0773aa1e33da_story.html\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR163","unstructured":"Felderer M and Beer A (2015) \"Mutual knowledge transfer between industry and academia to improve testing withdefect taxonomies,\" in German software-engineering and management conference, pp. 238\u2013242"},{"key":"9803_CR164","unstructured":"Crick J (2019) \"Why Simple is So Complex: The complexity of simplicity in software engineering,\" https:\/\/itnext.io\/why-simple-is-so-complex-362bc835b763\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR165","unstructured":"Lant M (2019) \"Occam\u2019s Razor And The Art Of Software Design,\" http:\/\/michaellant.com\/2010\/08\/10\/occams-razor-and-the-art-of-software-design\/\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR166","unstructured":"Biswas AK and Kirchherr J (2015) \"Prof, no one is reading you,\" Singapore Press Holdings,\nhttps:\/\/www.straitstimes.com\/opinion\/prof-no-one-is-reading-you\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR167","unstructured":"Mulkay M (2014) Science and the sociology of knowledge (RLE social theory). Routledge"},{"issue":"2","key":"9803_CR168","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1080\/02601370701219509","volume":"26","author":"S Gravett","year":"2007","unstructured":"Gravett S, Petersen N (2007) \u2018You just try to find your own way\u2019: the experience of newcomers to academia. Int J Lifelong Educ 26(2):193\u2013207","journal-title":"Int J Lifelong Educ"},{"key":"9803_CR169","doi-asserted-by":"crossref","unstructured":"Garousi V and Herkilo\u011flu K (2016) \"Selecting the right topics for industry-academia collaborations in software testing: an experience report,\" in IEEE International Conference on Software Testing, Verification, and Validation, pp. 213\u2013222","DOI":"10.1109\/ICST.2016.7"},{"key":"9803_CR170","doi-asserted-by":"crossref","unstructured":"Misirli AT, Erdogmus H, Juristo N, and Dieste O (2014) \"topic selection in industry experiments,\" presented at the proceedings of the international workshop on conducting empirical studies in industry, Hyderabad, India","DOI":"10.1145\/2593690.2593691"},{"key":"9803_CR171","doi-asserted-by":"publisher","unstructured":"Petersen K and Wohlin C (2009b) \"Context in industrial software engineering research,\" in Proceedings of International Symposium on Empirical Software Engineering and Measurement, 2009, 1671293, pp. 401\u2013404, doi: https:\/\/doi.org\/10.1109\/esem.2009.5316010","DOI":"10.1109\/esem.2009.5316010"},{"key":"9803_CR172","unstructured":"Zimmermann J (2019) \"Cost and Benefit of Model-Based Development of Automotive Software - Results of a Global Study,\" https:\/\/uk.mathworks.com\/videos\/cost-and-benefit-of-model-based-development-of-automotive-software-results-of-a-global-study-92771.html\n, Last accessed: Feb. 1, 2019"},{"key":"9803_CR173","doi-asserted-by":"crossref","unstructured":"Biffl S, Aurum A, Boehm B, Erdogmus H, and Gr\u00fcnbacher P (2006) Value-based software engineering. Springer","DOI":"10.1007\/3-540-29263-2"},{"key":"9803_CR174","doi-asserted-by":"crossref","unstructured":"Boehm BW (1984) \"Software engineering economics,\" IEEE transactions on Software Engineering, no. 1, pp. 4\u201321","DOI":"10.1109\/TSE.1984.5010193"},{"key":"9803_CR175","doi-asserted-by":"crossref","unstructured":"Jolly SA, Garousi V, and Eskandar MM (2012) \"Automated Unit Testing of a SCADA Control Software: An Industrial Case Study based on Action Research,\" in IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 400\u2013409","DOI":"10.1109\/ICST.2012.120"},{"key":"9803_CR176","doi-asserted-by":"crossref","unstructured":"Xie J, Sreenivasan S, Korniss G, Zhang W, Lim C, Szymanski BK (2011) Social consensus through the influence of committed minorities. Physical Review E 84(1)","DOI":"10.1103\/PhysRevE.84.011130"},{"key":"9803_CR177","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1016\/j.jss.2014.09.042","volume":"99","author":"J Zhi","year":"2015","unstructured":"Zhi J, Garousi V, Sun B, Garousi G, Shahnewaz S, Ruhe G (2015) Cost, benefits and quality of software development documentation: a systematic mapping. J Syst Softw 99:175\u2013198","journal-title":"J Syst Softw"},{"key":"9803_CR178","doi-asserted-by":"crossref","unstructured":"H\u00e4ser F, Felderer M, and Breu R (2014) \"software paradigms, assessment types and non-functional requirements in model-based integration testing: a systematic literature review,\" presented at the proceedings of the international conference on evaluation and assessment in software engineering","DOI":"10.1145\/2601248.2601257"},{"key":"9803_CR179","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1580","volume-title":"Model-based security testing: a taxonomy and systematic classification","author":"M Felderer","year":"2015","unstructured":"Felderer M, Zech P, Breu R, B\u00fcchler M, Pretschner A (2015) Model-based security testing: a taxonomy and systematic classification. Software Testing, Verification and Reliability. https:\/\/doi.org\/10.1002\/stvr.1580"},{"key":"9803_CR180","doi-asserted-by":"crossref","unstructured":"Mohacsi S, Felderer M, and Beer A (2015) \"Estimating the Cost and Benefit of Model-Based Testing: A Decision Support Procedure for the Application of Model-Based Testing in Industry,\" in 2015 41st Euromicro Conference on Software Engineering and Advanced Applications, 26\u201328 Aug. 2015, pp. 382\u2013389, doi: 10.1109\/SEAA.2015.18","DOI":"10.1109\/SEAA.2015.18"},{"issue":"4","key":"9803_CR181","doi-asserted-by":"crossref","first-page":"7346","DOI":"10.1016\/j.eswa.2008.10.027","volume":"36","author":"C Catal","year":"2009","unstructured":"Catal C, Diri B (2009) A systematic review of software fault prediction studies. Expert Syst Appl 36(4):7346\u20137354","journal-title":"Expert Syst Appl"},{"key":"9803_CR182","doi-asserted-by":"crossref","unstructured":"Tarhan A, Giray G (2017) \"On the use of ontologies in software process assessment: a systematic literature review,\" in Proceedings of the International Conference on Evaluation and Assessment in Software Engineering, 2017, pp. 2\u201311","DOI":"10.1145\/3084226.3084261"},{"issue":"2","key":"9803_CR183","first-page":"131","volume":"11","author":"G Giray","year":"2018","unstructured":"Giray G, Tuzun E (2018) A systematic mapping study on the current status of Total cost of ownership for information systems. International Journal of Informatics Technologies 11(2):131\u2013145","journal-title":"International Journal of Informatics Technologies"},{"issue":"6","key":"9803_CR184","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1111\/j.1467-6486.2004.00462.x","volume":"41","author":"AT Nicolai","year":"2004","unstructured":"Nicolai AT (2004) The bridge to the \u2018real world\u2019: applied science or a \u2018schizophrenic tour de force\u2019? J Manag Stud 41(6):951\u2013976","journal-title":"J Manag Stud"},{"issue":"2","key":"9803_CR185","doi-asserted-by":"crossref","first-page":"340","DOI":"10.5465\/3069460","volume":"44","author":"SL Rynes","year":"2001","unstructured":"Rynes SL, Bartunek JM, Daft RL (2001) Across the great divide: knowledge creation and transfer between practitioners and academics. Acad Manag J 44(2):340\u2013355","journal-title":"Acad Manag J"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09803-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10664-020-09803-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09803-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,5]],"date-time":"2021-03-05T00:36:03Z","timestamp":1614904563000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10664-020-09803-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,5]]},"references-count":185,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,5]]}},"alternative-id":["9803"],"URL":"https:\/\/doi.org\/10.1007\/s10664-020-09803-0","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,5]]},"assertion":[{"value":"5 March 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}