{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T17:36:12Z","timestamp":1778693772411,"version":"3.51.4"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T00:00:00Z","timestamp":1646092800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T00:00:00Z","timestamp":1646092800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010669","name":"H2020 LEIT Information and Communication Technologies","doi-asserted-by":"publisher","award":["780351"],"award-info":[{"award-number":["780351"]}],"id":[{"id":"10.13039\/100010669","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010669","name":"H2020 LEIT Information and Communication Technologies","doi-asserted-by":"publisher","award":["871525"],"award-info":[{"award-number":["871525"]}],"id":[{"id":"10.13039\/100010669","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001665","name":"Agence Nationale de la Recherche","doi-asserted-by":"crossref","award":["ANR-19-CE25-0003-01"],"award-info":[{"award-number":["ANR-19-CE25-0003-01"]}],"id":[{"id":"10.13039\/501100001665","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Computing"],"published-print":{"date-parts":[[2024,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>A self-adaptive system can automatically maintain its quality requirements in the presence of dynamic environment changes. Developing a self-adaptive system may be difficult due to design time uncertainty; e.g., anticipating all potential environment changes at design time is in most cases infeasible. To realize self-adaptive systems in the presence of design time uncertainty, online machine learning, i.e., machine learning at runtime, is increasingly used. In particular, online reinforcement learning is proposed, which learns suitable adaptation actions through interactions with the environment at runtime. To learn about its environment, online reinforcement learning has to select actions that were not selected before, which is known as exploration. How exploration happens impacts the performance of the learning process. We focus on two problems related to how adaptation actions are explored. First, existing solutions randomly explore adaptation actions and thus may exhibit slow learning if there are many possible adaptation actions. Second, they are unaware of system evolution, and thus may explore new adaptation actions introduced during evolution rather late. We propose novel exploration strategies that use feature models (from software product line engineering) to guide exploration in the presence of many adaptation actions and system evolution. Experimental results for two realistic self-adaptive systems indicate an average speed-up of the learning process of 33.7% in the presence of many adaptation actions, and of 50.6% in the presence of evolution.<\/jats:p>","DOI":"10.1007\/s00607-022-01052-x","type":"journal-article","created":{"date-parts":[[2022,3,1]],"date-time":"2022-03-01T09:02:32Z","timestamp":1646125352000},"page":"1251-1272","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Realizing self-adaptive systems via online reinforcement learning and feature-model-guided exploration"],"prefix":"10.1007","volume":"106","author":[{"given":"Andreas","family":"Metzger","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cl\u00e9ment","family":"Quinton","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5741-2709","authenticated-orcid":false,"given":"Zolt\u00e1n \u00c1d\u00e1m","family":"Mann","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luciano","family":"Baresi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Klaus","family":"Pohl","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,3,1]]},"reference":[{"key":"1052_CR1","doi-asserted-by":"crossref","unstructured":"Acher M, Heymans P, Collet P, Quinton C, Lahire P, Merle P (2012) Feature model differences. In: Proceedings of the 24th International Conference on Advanced Information Systems Engineering, CAiSE\u201912, pp. 629\u2013645","DOI":"10.1007\/978-3-642-31095-9_41"},{"key":"1052_CR2","doi-asserted-by":"crossref","unstructured":"Arabnejad H, Pahl C, Jamshidi P, Estrada G (2017) A comparison of reinforcement learning techniques for fuzzy cloud auto-scaling. In: 17th Intl Symposium on Cluster, Cloud and Grid Computing, CCGRID 2017, pp. 64\u201373","DOI":"10.1109\/CCGRID.2017.15"},{"issue":"12","key":"1052_CR3","doi-asserted-by":"publisher","first-page":"1656","DOI":"10.1002\/cpe.2864","volume":"25","author":"E Barrett","year":"2013","unstructured":"Barrett E, Howley E, Duggan J (2013) Applying reinforcement learning towards automating resource allocation and application scalability in the cloud. Concurrency and Computation: Practice and Experience 25(12):1656\u20131674","journal-title":"Concurrency and Computation: Practice and Experience"},{"issue":"4","key":"1052_CR4","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1109\/TPDS.2012.174","volume":"24","author":"X Bu","year":"2013","unstructured":"Bu X, Rao J, Xu C (2013) Coordinated self-configuration of virtual machines and appliances using a model-free learning approach. IEEE Trans. Parallel Distrib. Syst. 24(4):681\u2013690","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"issue":"4","key":"1052_CR5","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1007\/s10515-015-0185-3","volume":"23","author":"J B\u00fcrdek","year":"2016","unstructured":"B\u00fcrdek J, Kehrer T, Lochau M, Reuling D, Kelter U, Sch\u00fcrr A (2016) Reasoning about product-line evolution using complex feature model differences. Automated Softw. Eng. 23(4):687\u2013733","journal-title":"Automated Softw. Eng."},{"key":"1052_CR6","doi-asserted-by":"crossref","unstructured":"Calinescu R, Mirandola R, Perez-Palacin D, Weyns D (2020) Understanding uncertainty in self-adaptive systems. In: IEEE International Conference on Autonomic Computing and Self-Organizing Systems, ACSOS 2020, Washington, DC, USA, August 17-21, 2020, pp. 242\u2013251. IEEE","DOI":"10.1109\/ACSOS49614.2020.00047"},{"key":"1052_CR7","doi-asserted-by":"crossref","unstructured":"Caporuscio M, D\u2019Angelo M, Grassi V, Mirandola R (2016) Reinforcement learning techniques for decentralized self-adaptive service assembly. In: 5th Eur. Conference on Service-Oriented and Cloud Computing, ESOCC\u201916, vol. 9846, pp. 53\u201368","DOI":"10.1007\/978-3-319-44482-6_4"},{"key":"1052_CR8","doi-asserted-by":"crossref","unstructured":"Chen B, Peng X, Yu Y, Nuseibeh B, Zhao W (2014) Self-adaptation through incremental generative model transformations at runtime. In: 36th Intl Conf on Softw. Eng., ICSE \u201914, pp. 676\u2013687","DOI":"10.1145\/2568225.2568310"},{"key":"1052_CR9","unstructured":"De\u00a0Lemos R, et\u00a0al. (2013) Software Engineering for Self-Adaptive Systems: A Second Research Roadmap. In: Softw. Eng. for Self-Adaptive Systems II, LNCS, vol. 7475, pp. 1\u201332. Springer"},{"key":"1052_CR10","unstructured":"Dulac-Arnold G, Evans R, Sunehag P, Coppin B (2015) Reinforcement learning in large discrete action spaces. CoRR arXiv:1512.07679"},{"key":"1052_CR11","unstructured":"Dutreilh X, Kirgizov S, Melekhova O, Malenfant J, Rivierre N, Truck I (2011) Using reinforcement learning for autonomic resource allocation in clouds: towards a fully automated workflow. In: 7th Intl Conf. on Autonomic and Autonomous Systems, ICAS\u201911, pp. 67\u201374"},{"issue":"11","key":"1052_CR12","doi-asserted-by":"publisher","first-page":"1467","DOI":"10.1109\/TSE.2013.37","volume":"39","author":"N Esfahani","year":"2013","unstructured":"Esfahani N, Elkhodary A, Malek S (2013) A Learning-Based Framework for Engineering Feature-Oriented Self-Adaptive Software Systems. IEEE Trans. Softw. Eng. 39(11):1467\u20131493","journal-title":"IEEE Trans. Softw. Eng."},{"key":"1052_CR13","doi-asserted-by":"crossref","unstructured":"Filho RVR, Porter B (2017) Defining emergent software using continuous self-assembly, perception, and learning. TAAS 12(3), 16:1\u201316:25","DOI":"10.1145\/3092691"},{"issue":"5","key":"1052_CR14","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1007\/s00607-018-0646-1","volume":"101","author":"JA Galindo","year":"2019","unstructured":"Galindo JA, Benavides D, Trinidad P, Guti\u00e9rrez-Fern\u00e1ndez AM, Ruiz-Cort\u00e9s A (2019) Automated analysis of feature models: Quo vadis? Computing 101(5):387\u2013433","journal-title":"Computing"},{"key":"1052_CR15","doi-asserted-by":"crossref","unstructured":"Ghezzi C (2017) Of software and change. Journal of Software: Evolution and Process 29(9)","DOI":"10.1002\/smr.1888"},{"issue":"10","key":"1052_CR16","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/MC.2012.332","volume":"45","author":"M Hinchey","year":"2012","unstructured":"Hinchey M, Park S, Schmid K (2012) Building dynamic software product lines. IEEE Computer 45(10):22\u201326","journal-title":"IEEE Computer"},{"key":"1052_CR17","doi-asserted-by":"crossref","unstructured":"de\u00a0la Iglesia DG, Weyns D (2015) MAPE-K formal templates to rigorously design behaviors for self-adaptive systems. TAAS 10(3), 15:1\u201315:31","DOI":"10.1145\/2724719"},{"key":"1052_CR18","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1613\/jair.301","volume":"4","author":"LP Kaelbling","year":"1996","unstructured":"Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: A survey. J. Artif. Intell. Res. 4:237\u2013285","journal-title":"J. Artif. Intell. Res."},{"issue":"1","key":"1052_CR19","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1109\/MC.2003.1160055","volume":"36","author":"JO Kephart","year":"2003","unstructured":"Kephart JO, Chess DM (2003) The vision of autonomic computing. IEEE Computer 36(1):41\u201350","journal-title":"IEEE Computer"},{"key":"1052_CR20","doi-asserted-by":"crossref","unstructured":"Kinneer C, Coker Z, Wang J, Garlan D, Le Goues C (2018) Managing uncertainty in self-adaptive systems with plan reuse and stochastic search. In: 13th Intl Symposium on Softw. Eng. for Adaptive and Self-Managing Systems, SEAMS\u201918, pp. 40\u201350","DOI":"10.1145\/3194133.3194145"},{"key":"1052_CR21","doi-asserted-by":"crossref","unstructured":"Mann Z\u00c1 (2016) Interplay of virtual machine selection and virtual machine placement. In: 5th European Conf. on Service-Oriented and Cloud Computing, ESOCC\u201916, vol. 9846, pp. 137\u2013151","DOI":"10.1007\/978-3-319-44482-6_9"},{"issue":"1","key":"1052_CR22","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1109\/TPDS.2017.2744627","volume":"29","author":"Z\u00c1 Mann","year":"2018","unstructured":"Mann Z\u00c1 (2018) Resource optimization across the cloud stack. IEEE Transactions on Parallel and Distributed Systems 29(1):169\u2013182","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"1052_CR23","doi-asserted-by":"crossref","unstructured":"Metzger A, Bayer A, Doyle D, Sharifloo AM, Pohl K, Wessling F (2016) Coordinated run-time adaptation of variability-intensive systems: an application in cloud computing. In: 1st Intl Workshop on Variability and Complexity in Software Design, VACE@ICSE 2016, pp. 5\u201311. ACM","DOI":"10.1145\/2897045.2897049"},{"key":"1052_CR24","doi-asserted-by":"crossref","unstructured":"Metzger A, Di Nitto E (2012) Addressing highly dynamic changes in service-oriented systems: Towards agile evolution and adaptation. In: Agile and Lean Service-Oriented Development: Foundations, Theory and Practice, pp. 33\u201346","DOI":"10.4018\/978-1-4666-2503-7.ch002"},{"key":"1052_CR25","doi-asserted-by":"crossref","unstructured":"Metzger A, Pohl K (2014) Software product line engineering and variability management: Achievements and challenges. In: Future of Software Engineering, FOSE\u201914, pp. 70\u201384","DOI":"10.1145\/2593882.2593888"},{"key":"1052_CR26","doi-asserted-by":"crossref","unstructured":"Metzger A, Quinton C, Mann Z\u00c1, Baresi L, Pohl K (2020) Feature model-guided online reinforcement learning for self-adaptive services. In: Intl Conference on Service-Oriented Computing (ICSOC 2020), LNCS, vol. 12571, pp. 269\u2013286. Springer","DOI":"10.1007\/978-3-030-65310-1_20"},{"key":"1052_CR27","doi-asserted-by":"crossref","unstructured":"Moustafa A, Ito T (2018) A deep reinforcement learning approach for large-scale service composition. In: Intl. Conf. on Principles and Practice of Multi-Agent Systems, pp. 296\u2013311","DOI":"10.1007\/978-3-030-03098-8_18"},{"key":"1052_CR28","doi-asserted-by":"crossref","unstructured":"Moustafa A, Zhang M (2014) Learning efficient compositions for QoS-aware service provisioning. In: IEEE Intl Conf. on Web Services, ICWS\u201914, pp. 185\u2013192","DOI":"10.1109\/ICWS.2014.37"},{"key":"1052_CR29","unstructured":"Nachum O, Norouzi M, Xu K, Schuurmans D (2017) Bridging the gap between value and policy based reinforcement learning. In: Advances in Neural Inform. Proc. Systems 12 (NIPS 2017), pp. 2772\u20132782"},{"key":"1052_CR30","doi-asserted-by":"crossref","unstructured":"Palm A, Metzger A, Pohl K (2020) Online reinforcement learning for self-adaptive information systems. In: E.\u00a0Yu, S.\u00a0Dustdar (eds.) Int\u2019l Conference on Advanced Information Systems Engineering, CAiSE\u201920","DOI":"10.1007\/978-3-030-49435-3_11"},{"key":"1052_CR31","unstructured":"Plappert M, Houthooft R, Dhariwal P, Sidor S, Chen RY, Chen X, Asfour T, Abbeel P, Andrychowicz M (2018) Parameter space noise for exploration. In: 6th Intl Conf. on Learning Representations, ICLR 2018. OpenReview.net"},{"key":"1052_CR32","doi-asserted-by":"crossref","unstructured":"Ramirez AJ, Cheng BHC, McKinley PK, Beckmann BE (2010) Automatically generating adaptive logic to balance non-functional tradeoffs during reconfiguration. In: 7th Intl Conf. on Autonomic Computing, ICAC\u201910, pp. 225\u2013234","DOI":"10.1145\/1809049.1809080"},{"key":"1052_CR33","doi-asserted-by":"crossref","unstructured":"Restuccia F, Melodia T (2020) DeepWiERL: Bringing deep reinforcement learning to the internet of self-adaptive things. In: IEEE INFOCOM 2020 \u2014 IEEE Conference on Computer Communications, pp. 844\u2013853. IEEE","DOI":"10.1109\/INFOCOM41043.2020.9155461"},{"key":"1052_CR34","doi-asserted-by":"crossref","unstructured":"Salehie M, Tahvildari L (2009) Self-adaptive software: Landscape and research challenges. TAAS 4(2)","DOI":"10.1145\/1516533.1516538"},{"key":"1052_CR35","doi-asserted-by":"crossref","unstructured":"Shaw R, Howley E, Barrett E (2021) Applying reinforcement learning towards automating energy efficient virtual machine consolidation in cloud data centers. Information Systems. early access","DOI":"10.1016\/j.is.2021.101722"},{"key":"1052_CR36","doi-asserted-by":"crossref","unstructured":"Siegmund N, Kolesnikov SS, K\u00e4stner C, Apel S, Batory D, Rosenm\u00fcller M, Saake G (2012) Predicting Performance via Automated Feature-interaction Detection. In: 34th Intl Conf. on Softw. Eng., ICSE\u201912, pp. 167\u2013177","DOI":"10.1109\/ICSE.2012.6227196"},{"key":"1052_CR37","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG (2018) Reinforcement Learning: An Introduction, 2nd edn. MIT Press, Cambridge, MA, USA","edition":"2"},{"key":"1052_CR38","first-page":"1633","volume":"10","author":"ME Taylor","year":"2009","unstructured":"Taylor ME, Stone P (2009) Transfer learning for reinforcement learning domains: A survey. J. Mach. Learn. Res. 10:1633\u20131685","journal-title":"J. Mach. Learn. Res."},{"key":"1052_CR39","doi-asserted-by":"crossref","unstructured":"Th\u00fcm T, Batory D, Kastner C (2009) Reasoning About Edits to Feature Models. In: 31st Intl Conf. on Softw. Eng., ICSE\u201909, pp. 254\u2013264","DOI":"10.1109\/ICSE.2009.5070526"},{"key":"1052_CR40","doi-asserted-by":"crossref","unstructured":"Th\u00fcm T, K\u00e4stner C, Erdweg S, Siegmund N (2011) Abstract features in feature modeling. In: 15th Intl Conf. on Software Product Lines, SPLC\u201911, pp. 191\u2013200","DOI":"10.1109\/SPLC.2011.53"},{"key":"1052_CR41","doi-asserted-by":"crossref","unstructured":"Van Der Donckt J, Weyns D, Quin F, Van Der Donckt J, Michiels S (2020) Applying deep learning to reduce large adaptation spaces of self-adaptive systems with multiple types of goals. In: 15th Intl Symp. on Softw. Eng. for Adaptive and Self-Managing Systems, SEAMS 2020. ACM","DOI":"10.1145\/3387939.3391605"},{"key":"1052_CR42","doi-asserted-by":"crossref","unstructured":"Wang H, Gu M, Yu Q, Fei H, Li J, Tao Y (2017) Large-scale and adaptive service composition using deep reinforcement learning. In: 15th Intl Conference on Service-Oriented Computing (ICSOC\u201917), pp. 383\u2013391","DOI":"10.1007\/978-3-319-69035-3_27"},{"key":"1052_CR43","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1016\/j.knosys.2019.05.020","volume":"180","author":"H Wang","year":"2019","unstructured":"Wang H, Gu M, Yu Q, Tao Y, Li J, Fei H, Yan J, Zhao W, Hong T (2019) Adaptive and large-scale service composition based on deep reinforcement learning. Knowledge-Based Systems 180:75\u201390","journal-title":"Knowledge-Based Systems"},{"key":"1052_CR44","doi-asserted-by":"crossref","unstructured":"Weyns D (2021) Introduction to Self-Adaptive Systems: A Contemporary Software Engineering Perspective. Wiley","DOI":"10.1002\/9781119574910"},{"key":"1052_CR45","unstructured":"Weyns D, et al. (2013) Perpetual assurances for self-adaptive systems. In: R.\u00a0de\u00a0Lemos, D.\u00a0Garlan, C.\u00a0Ghezzi, H.\u00a0Giese (eds.) Software Engineering for Self-Adaptive Systems III. Assurances - International Seminar, Dagstuhl Castle, Germany, December 15-19, 2013, Revised Selected and Invited Papers, Lecture Notes in Computer Science, vol. 9640, pp. 31\u201363. Springer"},{"key":"1052_CR46","doi-asserted-by":"crossref","unstructured":"Zhao T, Zhang W, Zhao H, Jin Z (2017) A reinforcement learning-based framework for the generation and evolution of adaptation rules. In: Intl Conf. on Autonomic Computing, ICAC, pp. 103\u2013112","DOI":"10.1109\/ICAC.2017.47"}],"container-title":["Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00607-022-01052-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00607-022-01052-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00607-022-01052-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,25]],"date-time":"2024-03-25T15:03:17Z","timestamp":1711378997000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00607-022-01052-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,1]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,4]]}},"alternative-id":["1052"],"URL":"https:\/\/doi.org\/10.1007\/s00607-022-01052-x","relation":{},"ISSN":["0010-485X","1436-5057"],"issn-type":[{"value":"0010-485X","type":"print"},{"value":"1436-5057","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,1]]},"assertion":[{"value":"9 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 December 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 March 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}