{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T19:07:25Z","timestamp":1771614445801,"version":"3.50.1"},"reference-count":114,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T00:00:00Z","timestamp":1660608000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T00:00:00Z","timestamp":1660608000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003170","name":"KK-stiftelsen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003170","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100006359","name":"Blekinge Institute of Technology","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100006359","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Context:<\/jats:title>\n                <jats:p>Software companies must become better at delivering software to remain relevant in the market. Continuous integration and delivery practices promise to streamline software deliveries to end-users by implementing an automated software development and delivery pipeline. However, implementing or retrofitting an organization with such a pipeline is a substantial investment, while the reporting on benefits and their relevance in specific contexts\/domains are vague.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Aim:<\/jats:title>\n                <jats:p>In this study, we explore continuous software engineering practices from an investment-benefit perspective. We identify what benefits can be attained by adopting continuous practices, what the associated investments and risks are, and analyze what parameters determine their relevance.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Method:<\/jats:title>\n                <jats:p>We perform a multiple case study to understand state-of-practice, organizational aims, and challenges in adopting continuous software engineering practices. We compare state-of-practice with state-of-the-art to validate the best practices and identify relevant gaps for further investigation.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results:<\/jats:title>\n                <jats:p>We found that companies start the CI\/CD adoption by automating and streamlining the internal development process with clear and immediate benefits. However, upgrading customers to continuous deliveries is a major obstacle due to existing agreements and customer push-back. Renegotiating existing agreements comes with a risk of losing customers and disrupting the whole organization.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions:<\/jats:title>\n                <jats:p>We conclude that the benefits of CI\/CD are overstated in literature without considering the contextual and domain complexities rendering some benefits infeasible. We identify the need to understand the customer and organizational perspectives further and understand the contextual requirements towards the CI\/CD.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s10664-022-10191-w","type":"journal-article","created":{"date-parts":[[2022,8,16]],"date-time":"2022-08-16T05:02:32Z","timestamp":1660626152000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Towards cost-benefit evaluation for continuous software engineering activities"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1987-2234","authenticated-orcid":false,"given":"Eriks","family":"Klotins","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tony","family":"Gorschek","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katarina","family":"Sundelin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Erik","family":"Falk","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,8,16]]},"reference":[{"key":"10191_CR1","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1016\/j.infsof.2018.08.006","volume":"105","author":"H Alahyari","year":"2019","unstructured":"Alahyari H., Gorschek T., Svensson R. B. (2019) An exploratory study of waste in software development organizations using agile or lean approaches: A multiple case study at 14 organizations. Information and Software Technology 105:78\u201394","journal-title":"Information and Software Technology"},{"key":"10191_CR2","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1016\/j.jss.2016.12.007","volume":"125","author":"H Alahyari","year":"2017","unstructured":"Alahyari H., Svensson R. B., Gorschek T. (2017) A study of value in agile software development organizations. Journal of Systems and Software 125:271\u2013288","journal-title":"Journal of Systems and Software"},{"key":"10191_CR3","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1016\/j.infsof.2019.05.008","volume":"113","author":"J Anderson","year":"2019","unstructured":"Anderson J., Azizi M., Salem S., Do H. (2019) On the use of usage patterns from telemetry data for test case prioritization. Information and Software Technology 113:110\u2013130","journal-title":"Information and Software Technology"},{"issue":"3","key":"10191_CR4","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1109\/MS.2016.64","volume":"33","author":"A Balalaie","year":"2016","unstructured":"Balalaie A., Heydarnoori A., Jamshidi P. (2016) Microservices architecture enables devops: Migration to a cloud-native architecture. Ieee Software 33 (3):42\u201352","journal-title":"Ieee Software"},{"issue":"2","key":"10191_CR5","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1145\/638750.638776","volume":"28","author":"B Boehm","year":"2003","unstructured":"Boehm B. (2003) Value-based software engineering. ACM SIGSOFT Software Engineering Notes 28(2):4","journal-title":"ACM SIGSOFT Software Engineering Notes"},{"key":"10191_CR6","doi-asserted-by":"crossref","unstructured":"Bosch J. (2012) Building products as innovation experiment systems. In: International Conference of Software Business, Springer, pp 27\u201339","DOI":"10.1007\/978-3-642-30746-1_3"},{"key":"10191_CR7","doi-asserted-by":"crossref","unstructured":"Carriere J., Kazman R., Ozkaya I. (2010) A cost-benefit framework for making architectural decisions in a business context. In: 2010 ACM\/IEEE 32nd International Conference on Software Engineering, IEEE. vol. 2, pp 149\u2013157","DOI":"10.1145\/1810295.1810317"},{"key":"10191_CR8","doi-asserted-by":"crossref","unstructured":"Chandra D. G., Borah M. D. (2012) Cost benefit analysis of cloud computing in education. In: 2012 International Conference on Computing, Communication and Applications, IEEE, pp 1\u20136","DOI":"10.1109\/ICCCA.2012.6179142"},{"issue":"2","key":"10191_CR9","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1109\/MS.2015.27","volume":"32","author":"L Chen","year":"2015","unstructured":"Chen L. (2015) Continuous delivery: Huge benefits, but challenges too. IEEE Softw 32(2):50\u201354","journal-title":"IEEE Softw"},{"key":"10191_CR10","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1016\/j.jss.2017.02.013","volume":"128","author":"L Chen","year":"2017","unstructured":"Chen L. (2017) Continuous delivery: Overcoming adoption challenges. J Syst Softw 128:72\u201386","journal-title":"J Syst Softw"},{"key":"10191_CR11","doi-asserted-by":"crossref","unstructured":"Chen L. (2018) Microservices: architecting for continuous delivery and devops. In: 2018 IEEE International conference on software architecture (ICSA), IEEE, pp 39\u2013397","DOI":"10.1109\/ICSA.2018.00013"},{"key":"10191_CR12","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/j.infsof.2014.07.009","volume":"57","author":"GG Claps","year":"2015","unstructured":"Claps G. G., Svensson R. B., Aurum A. (2015) On the journey to continuous deployment: Technical and social challenges along the way. Information and Software Technology 57:21\u201331","journal-title":"Information and Software Technology"},{"issue":"4","key":"10191_CR13","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1080\/10686967.2001.11918977","volume":"8","author":"RE Cole","year":"2001","unstructured":"Cole R. E. (2001) From continuous improvement to continuous innovation. Qual Manag J 8(4):7\u201321","journal-title":"Qual Manag J"},{"key":"10191_CR14","doi-asserted-by":"crossref","unstructured":"D\u00e4nnart S., Constante F. M., Beckers K. (2019) An assessment model for continuous security compliance in large scale agile environments. In: International Conference on Advanced Information Systems Engineering, Springer, pp 529\u2013544","DOI":"10.1007\/978-3-030-21290-2_33"},{"issue":"5","key":"10191_CR15","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1002\/smr.337","volume":"18","author":"C Del Rosso","year":"2006","unstructured":"Del Rosso C. (2006) Continuous evolution through software architecture evaluation: a case study. Journal of Software Maintenance and Evolution: Research and Practice 18(5):351\u2013383","journal-title":"Journal of Software Maintenance and Evolution: Research and Practice"},{"key":"10191_CR16","doi-asserted-by":"crossref","unstructured":"Dr\u00e8ze J., Stern N. (1987) The theory of cost-benefit analysis. In: Handbook of public economics, Elsevier. vol 2, pp 909\u2013989","DOI":"10.1016\/S1573-4420(87)80009-5"},{"issue":"2007","key":"10191_CR17","first-page":"2007","volume":"6","author":"A Dubey","year":"2007","unstructured":"Dubey A., Wagle D. (2007) Delivering software as a service. McKinsey Q 6(2007):2007","journal-title":"McKinsey Q"},{"key":"10191_CR18","doi-asserted-by":"crossref","unstructured":"Ehlers J., van Hoorn A., Waller J., Hasselbring W. (2011) Self-adaptive software system monitoring for performance anomaly localization. In: Proceedings of the 8th ACM international conference on Autonomic computing, pp 197\u2013200","DOI":"10.1145\/1998582.1998628"},{"key":"10191_CR19","doi-asserted-by":"crossref","unstructured":"Elazhary O., Werner C., Li Z. S., Lowlind D., Ernst N. A., Storey M.-A. (2021) Uncovering the benefits and challenges of continuous integration practices. IEEE Trans Softw Eng","DOI":"10.1109\/TSE.2021.3064953"},{"key":"10191_CR20","doi-asserted-by":"crossref","unstructured":"Fabijan A., Dmitriev P., Olsson H. H., Bosch J. (2017) The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale. In: 2017 IEEE\/ACM 39th International Conference on Software Engineering (ICSE), IEEE, pp 770\u2013780","DOI":"10.1109\/ICSE.2017.76"},{"key":"10191_CR21","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1016\/j.jss.2016.03.034","volume":"123","author":"F Fagerholm","year":"2017","unstructured":"Fagerholm F., Guinea A. S., M\u00e4enp\u00e4\u00e4 H., M\u00fcnch J. (2017) The right model for continuous experimentation. J Syst Softw 123:292\u2013305","journal-title":"J Syst Softw"},{"key":"10191_CR22","doi-asserted-by":"publisher","first-page":"50","DOI":"10.5815\/ijitcs.2017.03.06","volume":"3","author":"A Fatima","year":"2017","unstructured":"Fatima A., Nazir N., Khan M. G. (2017) Data cleaning in data warehouse: A survey of data pre-processing techniques and tools. IJ Information Technology and Computer Science 3:50\u201361","journal-title":"IJ Information Technology and Computer Science"},{"issue":"4","key":"10191_CR23","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1109\/MIC.2013.25","volume":"17","author":"DG Feitelson","year":"2013","unstructured":"Feitelson D. G., Frachtenberg E., Beck K. L. (2013) Development and deployment at facebook. IEEE Internet Computing 17(4):8\u201317","journal-title":"IEEE Internet Computing"},{"key":"10191_CR24","doi-asserted-by":"crossref","unstructured":"Felidr\u00e9 W., Furtado L., da Costa D. A., Cartaxo B., Pinto G. (2019) Continuous integration theater. In: 2019 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, pp 1\u201310","DOI":"10.1109\/ESEM.2019.8870152"},{"key":"10191_CR25","doi-asserted-by":"crossref","unstructured":"Fitzgerald B., Stol K.-J. (2014) Continuous software engineering and beyond: trends and challenges. In: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering, pp 1\u20139","DOI":"10.1145\/2593812.2593813"},{"key":"10191_CR26","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1016\/j.jss.2015.06.063","volume":"123","author":"B Fitzgerald","year":"2017","unstructured":"Fitzgerald B., Stol K.-J. (2017) Continuous software engineering: A roadmap and agenda. J Syst Softw 123:176\u2013189","journal-title":"J Syst Softw"},{"key":"10191_CR27","doi-asserted-by":"crossref","unstructured":"Fitzgerald B., Stol K.-J., O\u2019Sullivan R., O\u2019Brien D. (2013) Scaling agile methods to regulated environments: An industry case study. In: 2013 35th International Conference on Software Engineering (ICSE), IEEE, pp 863\u2013872","DOI":"10.1109\/ICSE.2013.6606635"},{"issue":"8","key":"10191_CR28","first-page":"28","volume":"9","author":"M Fowler","year":"2001","unstructured":"Fowler M., Highsmith J., et al. (2001) The agile manifesto. Software Development 9(8):28\u201335","journal-title":"Software Development"},{"issue":"3","key":"10191_CR29","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1002\/1520-6858(200033)3:3<163::AID-SYS5>3.0.CO;2-T","volume":"3","author":"M Frank","year":"2000","unstructured":"Frank M. (2000) Engineering systems thinking and systems thinking. Syst Eng 3(3):163\u2013168","journal-title":"Syst Eng"},{"issue":"4","key":"10191_CR30","doi-asserted-by":"publisher","first-page":"444","DOI":"10.1016\/j.aos.2009.10.003","volume":"35","author":"N Frow","year":"2010","unstructured":"Frow N., Marginson D., Ogden S. (2010) \u201ccontinuous\u201d? budgeting: Reconciling budget flexibility with budgetary control. Acc Organ Soc 35(4):444\u2013461","journal-title":"Acc Organ Soc"},{"issue":"9","key":"10191_CR31","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1145\/285070.285080","volume":"41","author":"SR Gardner","year":"1998","unstructured":"Gardner S. R. (1998) Building the data warehouse. Commun ACM 41(9):52\u201360","journal-title":"Commun ACM"},{"issue":"3","key":"10191_CR32","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1109\/MS.2016.30","volume":"33","author":"V Garousi","year":"2016","unstructured":"Garousi V., Felderer M. (2016) Developing, verifying, and maintaining high-quality automated test scripts. IEEE Softw 33(3):68\u201375","journal-title":"IEEE Softw"},{"key":"10191_CR33","doi-asserted-by":"crossref","unstructured":"Gefen D., Karahanna E., Straub D. W. (2003) Trust and tam in online shopping: An integrated model, MIS quarterly pp 51\u201390","DOI":"10.2307\/30036519"},{"key":"10191_CR34","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1016\/j.jss.2016.11.027","volume":"125","author":"N Genc-Nayebi","year":"2017","unstructured":"Genc-Nayebi N., Abran A. (2017) A systematic literature review: Opinion mining studies from mobile app store user reviews. J Syst Softw 125:207\u2013219","journal-title":"J Syst Softw"},{"issue":"6","key":"10191_CR35","doi-asserted-by":"publisher","first-page":"585","DOI":"10.1109\/TSE.2015.2509970","volume":"42","author":"C Giardino","year":"2015","unstructured":"Giardino C., Paternoster N., Unterkalmsteiner M., Gorschek T., Abrahamsson P. (2015) Software development in startup companies: the greenfield startup model. IEEE Trans Softw Eng 42(6):585\u2013604","journal-title":"IEEE Trans Softw Eng"},{"issue":"4","key":"10191_CR36","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1509\/jmkg.2005.69.4.210","volume":"69","author":"A Gustafsson","year":"2005","unstructured":"Gustafsson A., Johnson M. D., Roos I. (2005) The effects of customer satisfaction, relationship commitment dimensions, and triggers on customer retention. Journal of marketing 69(4):210\u2013218","journal-title":"Journal of marketing"},{"key":"10191_CR37","doi-asserted-by":"crossref","unstructured":"Guzman E., Ibrahim M., Glinz M. (2017) Prioritizing user feedback from twitter: A survey report. In: 2017 IEEE\/ACM 4th International Workshop on CrowdSourcing in Software Engineering (CSI-SE), IEEE, pp 21\u201324","DOI":"10.1109\/CSI-SE.2017.4"},{"issue":"5","key":"10191_CR38","doi-asserted-by":"publisher","first-page":"1159","DOI":"10.1111\/joms.12639","volume":"58","author":"A Hanelt","year":"2021","unstructured":"Hanelt A., Bohnsack R., Marz D., Antunes Marante C. (2021) A systematic review of the literature on digital transformation: insights and implications for strategy and organizational change. J Manag Stud 58(5):1159\u20131197","journal-title":"J Manag Stud"},{"key":"10191_CR39","doi-asserted-by":"crossref","unstructured":"Hasselbring W., Steinacker G. (2017) Microservice architectures for scalability, agility and reliability in e-commerce. In: 2017 IEEE International Conference on Software Architecture Workshops (ICSAW), IEEE, pp 243\u2013246","DOI":"10.1109\/ICSAW.2017.11"},{"key":"10191_CR40","doi-asserted-by":"crossref","unstructured":"Hazzan O., Dubinsky Y. (2009) Agile software engineering. Springer Science & Business Media","DOI":"10.1007\/978-1-84800-198-5"},{"key":"10191_CR41","doi-asserted-by":"crossref","unstructured":"Hilton M., Nelson N., Tunnell T., Marinov D., Dig D. (2017) Trade-offs in continuous integration: assurance, security, and flexibility. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp 197\u2013207","DOI":"10.1145\/3106237.3106270"},{"key":"10191_CR42","doi-asserted-by":"crossref","unstructured":"Hilton M., Tunnell T., Huang K., Marinov D., Dig D. (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: 2016 31st IEEE\/ACM International Conference on Automated Software Engineering (ASE), IEEE, pp 426\u2013437","DOI":"10.1145\/2970276.2970358"},{"key":"10191_CR43","doi-asserted-by":"crossref","unstructured":"Huang X., Zhang H., Zhou X., Babar M. A., Yang S. (2018) Synthesizing qualitative research in software engineering: A critical review. In: Proceedings of the 40th International Conference on Software Engineering, pp 1207\u20131218","DOI":"10.1145\/3180155.3180235"},{"key":"10191_CR44","unstructured":"Humble J., Kim G. (2018) Accelerate: The science of lean software and devops: Building and scaling high performing technology organizations. IT Revolution"},{"issue":"8","key":"10191_CR45","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1109\/MC.2018.3191268","volume":"51","author":"J Isaak","year":"2018","unstructured":"Isaak J., Hanna M. J. (2018) User data privacy: Facebook, cambridge analytica, and privacy protection. Computer 51(8):56\u201359","journal-title":"Computer"},{"issue":"6","key":"10191_CR46","first-page":"643","volume":"24","author":"S Jalali","year":"2012","unstructured":"Jalali S., Wohlin C. (2012) Global software engineering and agile practices: a systematic review. Journal of software: Evolution and Process 24(6):643\u2013659","journal-title":"Journal of software: Evolution and Process"},{"key":"10191_CR47","doi-asserted-by":"publisher","first-page":"48","DOI":"10.1016\/j.infsof.2016.10.006","volume":"84","author":"J Jiang","year":"2017","unstructured":"Jiang J., Yang Y., He J., Blanc X., Zhang L. (2017) Who should comment on this pull request? analyzing attributes for more accurate commenter recommendation in pull-based development. Inf Softw Technol 84:48\u201362","journal-title":"Inf Softw Technol"},{"issue":"4","key":"10191_CR48","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1109\/MS.2005.95","volume":"22","author":"PM Johnson","year":"2005","unstructured":"Johnson P. M., Kou H., Paulding M., Zhang Q., Kagawa A., Yamashita T. (2005) Improving software development management through software project telemetry. IEEE software 22(4):76\u201385","journal-title":"IEEE software"},{"key":"10191_CR49","doi-asserted-by":"crossref","unstructured":"Kasurinen J., Taipale O., Smolander K. (2010) Software test automation in practice: empirical observations. Advances in Software Engineering (2010)","DOI":"10.1155\/2010\/620836"},{"issue":"7","key":"10191_CR50","first-page":"711","volume":"25","author":"M Khurum","year":"2013","unstructured":"Khurum M., Gorschek T., Wilson M. (2013) The software value map\u2014an exhaustive collection of value aspects for the development of software intensive products. Journal of software: Evolution and Process 25(7):711\u2013741","journal-title":"Journal of software: Evolution and Process"},{"key":"10191_CR51","doi-asserted-by":"crossref","unstructured":"Kim S., Park S., Yun J., Lee Y. (2008) Automated continuous integration of component-based software: An industrial experience. In: 2008 23rd IEEE\/ACM International Conference on Automated Software Engineering, IEEE, pp 423\u2013426","DOI":"10.1109\/ASE.2008.64"},{"issue":"1","key":"10191_CR52","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1145\/356715.356718","volume":"10","author":"JL King","year":"1978","unstructured":"King J. L., Schrems E. L. (1978) Cost-benefit analysis in information systems development and operation. ACM Computing Surveys (CSUR) 10(1):19\u201334","journal-title":"ACM Computing Surveys (CSUR)"},{"issue":"2004","key":"10191_CR53","first-page":"1","volume":"33","author":"B Kitchenham","year":"2004","unstructured":"Kitchenham B. (2004) Procedures for performing systematic reviews. Keele, UK, Keele University 33(2004):1\u201326","journal-title":"Keele, UK, Keele University"},{"key":"10191_CR54","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.procs.2016.03.003","volume":"79","author":"D Kumar","year":"2016","unstructured":"Kumar D., Mishra K.K. (2016) The impacts of test automation on software\u2019s cost, quality and time to market. Procedia Computer Science 79:8\u201315","journal-title":"Procedia Computer Science"},{"key":"10191_CR55","doi-asserted-by":"crossref","unstructured":"Kuula S., Haapasalo H. (2017) Continuous and co-creative business model creation. In: Service business model innovation in healthcare and hospital management, Springer, pp 249\u2013268","DOI":"10.1007\/978-3-319-46412-1_14"},{"key":"10191_CR56","doi-asserted-by":"crossref","unstructured":"Lacoste F. J. (2009) Killing the gatekeeper: Introducing a continuous integration system. In: 2009 agile conference, IEEE, pp 387\u2013392","DOI":"10.1109\/AGILE.2009.35"},{"key":"10191_CR57","doi-asserted-by":"crossref","unstructured":"Lam W., Godefroid P., Nath S., Santhiar A., Thummalapenta S. (2019) Root causing flaky tests in a large-scale industrial setting. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp 101\u2013111","DOI":"10.1145\/3293882.3330570"},{"key":"10191_CR58","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1016\/j.infsof.2016.10.001","volume":"82","author":"E Laukkanen","year":"2017","unstructured":"Laukkanen E., Itkonen J., Lassenius C. (2017) Problems, causes and solutions when adopting continuous delivery-a systematic literature review. Inf Softw Technol 82:55\u201379","journal-title":"Inf Softw Technol"},{"key":"10191_CR59","doi-asserted-by":"crossref","unstructured":"Laukkanen E., Paasivaara M., Arvonen T. (2015) Stakeholder perceptions of the adoption of continuous integration\u2013a case study. In: 2015 agile conference, IEEE, pp 11\u201320","DOI":"10.1109\/Agile.2015.15"},{"issue":"2","key":"10191_CR60","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1007\/s00766-009-0078-8","volume":"14","author":"L Lehtola","year":"2009","unstructured":"Lehtola L., Kauppinen M., V\u00e4h\u00e4niitty J., Komssi M. (2009) Linking business and requirements engineering: is solution planning a missing activity in software product companies?. Requirements engineering 14(2):113\u2013128","journal-title":"Requirements engineering"},{"key":"10191_CR61","doi-asserted-by":"crossref","unstructured":"Letier E., Stefan D., Barr E. T. (2014) Uncertainty, risk, and information value in software requirements and architecture. In: Proceedings of the 36th International Conference on Software Engineering, pp 883\u2013894","DOI":"10.1145\/2568225.2568239"},{"issue":"10","key":"10191_CR62","doi-asserted-by":"publisher","first-page":"1726","DOI":"10.1016\/j.jss.2010.04.077","volume":"83","author":"Y Li","year":"2010","unstructured":"Li Y., Chang K.-C., Chen H.-G., Jiang J. J. (2010) Software development team flexibility antecedents. J Syst Softw 83(10):1726\u20131734","journal-title":"J Syst Softw"},{"issue":"1","key":"10191_CR63","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1109\/EMR.2018.2810099","volume":"46","author":"C Lin","year":"2018","unstructured":"Lin C. (2018) Data driven product management. IEEE Eng Manag Rev 46(1):16\u201318","journal-title":"IEEE Eng Manag Rev"},{"issue":"3","key":"10191_CR64","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1016\/j.jsis.2015.08.002","volume":"24","author":"C Loebbecke","year":"2015","unstructured":"Loebbecke C., Picot A. (2015) Reflections on societal and business model transformation arising from digitization and big data analytics: A research agenda. The Journal of Strategic Information Systems 24(3):149\u2013157","journal-title":"The Journal of Strategic Information Systems"},{"key":"10191_CR65","doi-asserted-by":"crossref","unstructured":"Lohan G. (2013) A brief history of budgeting: reflections on beyond budgeting, its link to performance management and its appropriateness for software development. In: International Conference on Lean Enterprise Software and Systems, Springer, pp 81\u2013105","DOI":"10.1007\/978-3-642-44930-7_6"},{"key":"10191_CR66","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1016\/j.infsof.2016.09.001","volume":"80","author":"S M\u00e4kinen","year":"2016","unstructured":"M\u00e4kinen S., Lepp\u00e4nen M., Kilamo T., Mattila A.-L., Laukkanen E., Pagels M., M\u00e4nnist\u00f6 T. (2016) Improving the delivery cycle: A multiple-case study of the toolchains in finnish software intensive enterprises. Inf Softw Technol 80:175\u2013194","journal-title":"Inf Softw Technol"},{"issue":"6","key":"10191_CR67","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1080\/00036846.2016.1200188","volume":"49","author":"P Maresova","year":"2017","unstructured":"Maresova P., Sobeslav V., Krejcar O. (2017) Cost\u2013benefit analysis\u2013evaluation model of cloud computing deployment for use in companies. Appl Econ 49 (6):521\u2013533","journal-title":"Appl Econ"},{"key":"10191_CR68","unstructured":"Martin R. C. (2002) Agile software development: principles, patterns, and practices. Prentice Hall"},{"key":"10191_CR69","doi-asserted-by":"crossref","unstructured":"McHugh M., McCaffery F., Fitzgerald B., Stol K.-J., Casey V., Coady G. (2013) Balancing agility and discipline in a medical device software organisation. In: International Conference on Software Process Improvement and Capability Determination, Springer, pp 199\u2013210","DOI":"10.1007\/978-3-642-38833-0_18"},{"key":"10191_CR70","doi-asserted-by":"crossref","unstructured":"Memon A., Gao Z., Nguyen B., Dhanda S., Nickell E., Siemborski R., Micco J. (2017) Taming google-scale continuous testing. In: 2017 IEEE\/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), IEEE, pp 233\u2013242","DOI":"10.1109\/ICSE-SEIP.2017.16"},{"issue":"3","key":"10191_CR71","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1109\/MS.2014.58","volume":"31","author":"M Meyer","year":"2014","unstructured":"Meyer M. (2014) Continuous integration and its tools. IEEE software 31(3):14\u201316","journal-title":"IEEE software"},{"key":"10191_CR72","doi-asserted-by":"crossref","unstructured":"Moyon F., Beckers K., Klepper S., Lachberger P., Bruegge B. (2018) Towards continuous security compliance in agile software development at scale. In: 2018 IEEE\/ACM 4th International Workshop on Rapid Continuous Software Engineering (RCoSE), IEEE, pp 31\u201334","DOI":"10.1145\/3194760.3194767"},{"key":"10191_CR73","doi-asserted-by":"crossref","unstructured":"Neely S., Stolt S. (2013) Continuous delivery? easy! just change everything (well, maybe it is not that easy). In: 2013 Agile Conference, IEEE, pp 121\u2013128","DOI":"10.1109\/AGILE.2013.17"},{"issue":"11","key":"10191_CR74","first-page":"e1866","volume":"29","author":"RV O\u2019Connor","year":"2017","unstructured":"O\u2019Connor R. V., Elger P., Clarke P. M. (2017) Continuous software engineering-a microservices architecture perspective. Journal of Software: Evolution and Process 29(11):e1866","journal-title":"Journal of Software: Evolution and Process"},{"key":"10191_CR75","doi-asserted-by":"crossref","unstructured":"Olsson H. H., Alahyari H., Bosch J. (2012) Climbing the\u201d stairway to heaven\u201d\u2013a mulitiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In: 2012 38th euromicro conference on software engineering and advanced applications, IEEE, pp 392\u2013399","DOI":"10.1109\/SEAA.2012.54"},{"key":"10191_CR76","doi-asserted-by":"crossref","unstructured":"Olsson T., Wnuk K. (2018) Qreme\u2013quality requirements management model for supporting decision-making. In: International Working Conference on Requirements Engineering: Foundation for Software Quality, Springer, pp 173\u2013188","DOI":"10.1007\/978-3-319-77243-1_11"},{"issue":"03","key":"10191_CR77","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1142\/S0218194019500153","volume":"29","author":"RAB Ouriques","year":"2019","unstructured":"Ouriques R. A. B., Wnuk K., Gorschek T., Svensson R. B. (2019) Knowledge management strategies and processes in agile software development: a systematic literature review. International journal of software engineering and knowledge engineering 29(03):345\u2013380","journal-title":"International journal of software engineering and knowledge engineering"},{"key":"10191_CR78","doi-asserted-by":"crossref","unstructured":"Petersen K., Feldt R., Mujtaba S., Mattsson M. (2008) Systematic mapping studies in software engineering. In: 12th International Conference on Evaluation and Assessment in Software Engineering (EASE) 12, pp 1\u201310","DOI":"10.14236\/ewic\/EASE2008.8"},{"issue":"12","key":"10191_CR79","first-page":"2223","volume":"48","author":"G Pinto","year":"2018","unstructured":"Pinto G., Castor F., Bonifacio R., Rebou\u00e7as M. (2018) Work practices and challenges in continuous integration: A survey with travis ci users. Software: Practice and Experience 48(12):2223\u20132236","journal-title":"Software: Practice and Experience"},{"issue":"2011","key":"10191_CR80","first-page":"1","volume":"18","author":"M Poppendieck","year":"2011","unstructured":"Poppendieck M., et al. (2011) Principles of lean thinking. IT Management Select 18(2011):1\u20137","journal-title":"IT Management Select"},{"key":"10191_CR81","unstructured":"Poppendieck M., Poppendieck T. (2003) Lean software development: An agile toolkit: An agile toolkit. Addison-Wesley"},{"issue":"1","key":"10191_CR82","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1089\/big.2013.1508","volume":"1","author":"F Provost","year":"2013","unstructured":"Provost F., Fawcett T. (2013) Data science and its relationship to big data and data-driven decision making. Big data 1(1):51\u201359","journal-title":"Big data"},{"issue":"4","key":"10191_CR83","first-page":"3","volume":"23","author":"E Rahm","year":"2000","unstructured":"Rahm E., Do H. H. (2000) Data cleaning: Problems and current approaches. IEEE Data Eng. Bull. 23(4):3\u201313","journal-title":"IEEE Data Eng. Bull."},{"key":"10191_CR84","doi-asserted-by":"crossref","unstructured":"Raulamo-Jurvanen P., M\u00e4ntyl\u00e4 M., Garousi V. (2017) Choosing the right test automation tool: a grey literature review of practitioner sources. In: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, pp 21\u201330","DOI":"10.1145\/3084226.3084252"},{"key":"10191_CR85","doi-asserted-by":"crossref","unstructured":"Riaz M., Sulayman M., Naqvi H. (2009) Architectural decay during continuous software evolution and impact of \u2018design for change\u2019on software architecture. In: International Conference on Advanced Software Engineering and Its Applications, Springer, pp 119\u2013126","DOI":"10.1007\/978-3-642-10619-4_15"},{"key":"10191_CR86","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1016\/j.jss.2015.12.015","volume":"123","author":"P Rodr\u00edguez","year":"2017","unstructured":"Rodr\u00edguez P., Haghighatkhah A., Lwakatare L. E., Teppola S., Suomalainen T., Eskeli J., Karvonen T., Kuvaja P., Verner J. M., Oivo M. (2017) Continuous deployment of software intensive products and services: A systematic mapping study. J Syst Softw 123:263\u2013291","journal-title":"J Syst Softw"},{"key":"10191_CR87","doi-asserted-by":"crossref","unstructured":"Rodr\u00edguez P., Markkula J., Oivo M., Garbajosa J. (2012) Analyzing the drivers of the combination of lean and agile in software development companies. In: International Conference on Product Focused Software Process Improvement, Springer, pp 145\u2013159","DOI":"10.1007\/978-3-642-31063-8_12"},{"key":"10191_CR88","doi-asserted-by":"crossref","unstructured":"Rogers R. O. (2004) Scaling continuous integration. In: International conference on extreme programming and agile processes in software engineering, Springer, pp 68\u201376","DOI":"10.1007\/978-3-540-24853-8_8"},{"issue":"2","key":"10191_CR89","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1057\/ejis.2010.17","volume":"19","author":"NC Romano Jr","year":"2010","unstructured":"Romano Jr N. C., Pick J. B., Roztocki N. (2010) A motivational model for technology-supported cross-organizational and cross-border collaboration. Eur J Inf Syst 19(2):117\u2013133","journal-title":"Eur J Inf Syst"},{"key":"10191_CR90","doi-asserted-by":"crossref","unstructured":"Runeson P., Host M., Rainer A., Regnell B. (2012) Case study research in software engineering: Guidelines and examples. John Wiley & Sons","DOI":"10.1002\/9781118181034"},{"issue":"4","key":"10191_CR91","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1509\/jmkg.2005.69.4.252","volume":"69","author":"L Ryals","year":"2005","unstructured":"Ryals L. (2005) Making customer relationship management work: the measurement and profitable management of customer relationships. Journal of marketing 69(4):252\u2013261","journal-title":"Journal of marketing"},{"key":"10191_CR92","volume-title":"Cost-benefit analysis: a handbook, vol 182","author":"PG Sassone","year":"1978","unstructured":"Sassone P. G., Schaffer W. A. (1978) Cost-benefit analysis: a handbook, vol 182. Academic Press, New York"},{"key":"10191_CR93","doi-asserted-by":"crossref","unstructured":"Senapathi M., Buchan J., Osman H. (2018) Devops capabilities, practices, and challenges: insights from a case study. In: Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018, pp 57\u201367","DOI":"10.1145\/3210459.3210465"},{"key":"10191_CR94","doi-asserted-by":"crossref","unstructured":"Serrat O. (2017) Bridging organizational silos. In: Knowledge Solutions. Springer, pp 711\u2013716","DOI":"10.1007\/978-981-10-0983-9_77"},{"key":"10191_CR95","doi-asserted-by":"crossref","unstructured":"Shahin M., Babar M. A., Zahedi M., Zhu L. (2017) Beyond continuous delivery: an empirical investigation of continuous deployment challenges. In: 2017 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, pp 111\u2013120","DOI":"10.1109\/ESEM.2017.18"},{"key":"10191_CR96","doi-asserted-by":"publisher","first-page":"3909","DOI":"10.1109\/ACCESS.2017.2685629","volume":"5","author":"M Shahin","year":"2017","unstructured":"Shahin M., Babar M. A., Zhu L. (2017) Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5:3909\u20133943","journal-title":"IEEE Access"},{"issue":"3","key":"10191_CR97","doi-asserted-by":"publisher","first-page":"1061","DOI":"10.1007\/s10664-018-9651-4","volume":"24","author":"M Shahin","year":"2019","unstructured":"Shahin M., Zahedi M., Babar M. A., Zhu L. (2019) An empirical study of architecting for continuous delivery and deployment. Empir Softw Eng 24(3):1061\u20131108","journal-title":"Empir Softw Eng"},{"key":"10191_CR98","doi-asserted-by":"crossref","unstructured":"Shamshiri S., Just R., Rojas J. M., Fraser G., McMinn P., Arcuri A. (2015) Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t). In: 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE), IEEE, pp 201\u2013211","DOI":"10.1109\/ASE.2015.86"},{"issue":"3","key":"10191_CR99","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1007\/s11334-007-0026-z","volume":"3","author":"A Sidky","year":"2007","unstructured":"Sidky A., Arthur J., Bohner S. (2007) A disciplined approach to adopting agile practices: the agile adoption framework. Innovations in systems and software engineering 3(3):203\u2013216","journal-title":"Innovations in systems and software engineering"},{"key":"10191_CR100","doi-asserted-by":"crossref","unstructured":"Stolberg S. (2009) Enabling agile testing through continuous integration. In: 2009 agile conference, IEEE, pp 369\u2013374","DOI":"10.1109\/AGILE.2009.16"},{"issue":"1","key":"10191_CR101","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1109\/MS.2017.4541034","volume":"35","author":"D Sturtevant","year":"2017","unstructured":"Sturtevant D. (2017) Modular architectures make you agile in the long run. IEEE Softw 35(1):104\u2013108","journal-title":"IEEE Softw"},{"key":"10191_CR102","doi-asserted-by":"crossref","unstructured":"Sundelin A., Gonzalez-Huerta J., Wnuk K. (2018) Test-driving fintech product development: An experience report. In: International Conference on Product-Focused Software Process Improvement, Springer, pp 219\u2013226","DOI":"10.1007\/978-3-030-03673-7_16"},{"key":"10191_CR103","doi-asserted-by":"crossref","unstructured":"Sundelin A., Gonzalez-Huerta J., Wnuk K. (2020) The hidden cost of backward compatibility: when deprecation turns into technical debt-an experience report. In: Proceedings of the 3rd International Conference on Technical Debt, pp 67\u201376","DOI":"10.1145\/3387906.3388629"},{"issue":"2","key":"10191_CR104","doi-asserted-by":"publisher","first-page":"205","DOI":"10.2753\/MIS0742-1222260209","volume":"26","author":"A Susarla","year":"2009","unstructured":"Susarla A., Barua A., Whinston A. B. (2009) A transaction cost perspective of the\u201d software as a service\u201d business model. J Manag Inf Syst 26(2):205\u2013240","journal-title":"J Manag Inf Syst"},{"key":"10191_CR105","doi-asserted-by":"crossref","unstructured":"T\u00f3masd\u00f3ttir K. F., Aniche M., van Deursen A. (2017) Why and how javascript developers use linters. In: 2017 32nd IEEE\/ACM International Conference on Automated Software Engineering (ASE), IEEE, pp 578\u2013589","DOI":"10.1109\/ASE.2017.8115668"},{"key":"10191_CR106","unstructured":"van Hoorn A., Rohr M., Hasselbring W., Waller J., Ehlers J., Frey S., Kieselhorst D. (2009) Continuous monitoring of software services: Design and application of the kieker framework"},{"key":"10191_CR107","doi-asserted-by":"crossref","unstructured":"Vasilescu B., Yu Y., Wang H., Devanbu P., Filkov V. (2015) Quality and productivity outcomes relating to continuous integration in github. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp 805\u2013816","DOI":"10.1145\/2786805.2786850"},{"issue":"8","key":"10191_CR108","doi-asserted-by":"publisher","first-page":"e1639","DOI":"10.1002\/stvr.1639","volume":"27","author":"K Wiklund","year":"2017","unstructured":"Wiklund K., Eldh S., Sundmark D., Lundqvist K. (2017) Impediments for software test automation: A systematic literature review. Software Testing, Verification and Reliability 27(8):e1639","journal-title":"Software Testing, Verification and Reliability"},{"key":"10191_CR109","doi-asserted-by":"crossref","unstructured":"Williams L., Kudrjavets G., Nagappan N. (2009) On the effectiveness of unit test automation at microsoft. In: 2009 20th International Symposium on Software Reliability Engineering, IEEE, pp 81\u201389","DOI":"10.1109\/ISSRE.2009.32"},{"key":"10191_CR110","doi-asserted-by":"crossref","unstructured":"Williams L., Maximilien E. M., Vouk M. (2003) Test-driven development as a defect-reduction practice. In: 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003., IEEE, pp 34\u201345","DOI":"10.1109\/ISSRE.2003.1251029"},{"key":"10191_CR111","doi-asserted-by":"crossref","unstructured":"Yaman S. G., Sauvola T., Riungu-Kalliosaari L., Hokkanen L., Kuvaja P., Oivo M., M\u00e4nnist\u00f6 T. (2016) Customer involvement in continuous deployment: a systematic literature review. In: International Working Conference on Requirements Engineering: Foundation for Software Quality, Springer, pp 249\u2013265","DOI":"10.1007\/978-3-319-30282-9_18"},{"key":"10191_CR112","doi-asserted-by":"publisher","first-page":"106199","DOI":"10.1016\/j.infsof.2019.106199","volume":"117","author":"L Yu","year":"2020","unstructured":"Yu L., Al\u00e9groth E., Chatzipetrou P., Gorschek T. (2020) Utilising ci environment for efficient and effective testing of nfrs. Inf Softw Technol 117:106199","journal-title":"Inf Softw Technol"},{"key":"10191_CR113","doi-asserted-by":"crossref","unstructured":"Zhang D. (2018) Big data security and privacy protection. In: 8th International Conference on Management and Computer Science (ICMCS 2018), Atlantis Press, pp 275\u2013278","DOI":"10.2991\/icmcs-18.2018.56"},{"issue":"3","key":"10191_CR114","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1109\/MS.2016.81","volume":"33","author":"L Zhu","year":"2016","unstructured":"Zhu L., Bass L., Champlin-Scharff G. (2016) Devops and its practices. IEEE Softw 33(3):32\u201334","journal-title":"IEEE Softw"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10191-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10191-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10191-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T13:11:29Z","timestamp":1665753089000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10191-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,16]]},"references-count":114,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["10191"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10191-w","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,16]]},"assertion":[{"value":"17 May 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 August 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"157"}}