{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T06:28:28Z","timestamp":1774938508000,"version":"3.50.1"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T00:00:00Z","timestamp":1769904000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,2,2]],"date-time":"2026-02-02T00:00:00Z","timestamp":1769990400000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Universidad de Cadiz"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2026,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Models play a critical role in supporting Verification and Validation activities. However, they are often unavailable in practice, such as for legacy systems or third-party components. Model learning addresses this gap through two main approaches: passive learning, which infers models from existing execution traces, and active learning, which interacts with the system under learning (SUL) to generate more general models, but at higher computational cost. In this work, we propose a novel application of Transformer architectures to implicitly learn generic system models from execution traces alone, combining the efficiency of passive learning with the generality of active learning. We design and evaluate two Transformer-based architectures: one focused on exploitation, achieving over\n                    <jats:inline-formula>\n                      <jats:tex-math>$$\\varvec{95\\%}$$<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    valid trace generation; and another focused on exploration, generating approximately\n                    <jats:inline-formula>\n                      <jats:tex-math>$$\\varvec{60\\%}$$<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    novel, previously unseen traces. Our methods outperform state-of-the-art model learning techniques, demonstrating that Transformers can achieve results comparable to active learning while requiring significantly fewer resources.\n                  <\/jats:p>","DOI":"10.1007\/s10489-025-07066-0","type":"journal-article","created":{"date-parts":[[2026,2,2]],"date-time":"2026-02-02T06:11:20Z","timestamp":1770012680000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Using transformers to learn system models"],"prefix":"10.1007","volume":"56","author":[{"given":"Alfredo","family":"Ibias","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6802-2164","authenticated-orcid":false,"given":"Manuel","family":"M\u00e9ndez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9808-6401","authenticated-orcid":false,"given":"Manuel","family":"N\u00fa\u00f1ez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6773-205X","authenticated-orcid":false,"given":"Francisco","family":"Palomo-Lozano","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,2,2]]},"reference":[{"key":"7066_CR1","doi-asserted-by":"crossref","unstructured":"Magnani L, Bertolotti T (eds.) (2017) Handbook of Model-Based Science. Springer","DOI":"10.1007\/978-3-319-30526-4"},{"issue":"4","key":"7066_CR2","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1145\/2736348","volume":"58","author":"L Lamport","year":"2015","unstructured":"Lamport L (2015) Who builds a house without drawing blueprints? Commun ACM 58(4):38\u201341","journal-title":"Commun ACM"},{"issue":"5","key":"7066_CR3","doi-asserted-by":"publisher","first-page":"1525","DOI":"10.1007\/s10270-020-00855-w","volume":"20","author":"S Farshidi","year":"2021","unstructured":"Farshidi S, Jansen S, Fortuin S (2021) Model-driven development platform selection: four industry case studies. Softw Syst Model 20(5):1525\u20131551","journal-title":"Softw Syst Model"},{"key":"7066_CR4","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2022.111391","volume":"192","author":"A Ibias","year":"2022","unstructured":"Ibias A (2022) Using mutual information to test from Finite State Machines: Test suite generation. J Syst Softw 192:111391","journal-title":"J Syst Softw"},{"key":"7066_CR5","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2023.107173","volume":"158","author":"A Ibias","year":"2023","unstructured":"Ibias A, N\u00fa\u00f1ez M (2023) Squeeziness for non-deterministic systems. Inf Softw Technol 158:107173","journal-title":"Inf Softw Technol"},{"key":"7066_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2023.107263","volume":"162","author":"M M\u00e9ndez","year":"2023","unstructured":"M\u00e9ndez M, Benito-Parejo M, Ibias A, N\u00fa\u00f1ez M (2023) Metamorphic testing of chess engines. Inf Softw Technol 162:107263","journal-title":"Inf Softw Technol"},{"key":"7066_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2023.104426","volume":"165","author":"M N\u00fa\u00f1ez","year":"2023","unstructured":"N\u00fa\u00f1ez M, Hierons RM, Lefticaru R (2023) Implementation relations and testing for cyclic systems: adding probabilities. Robot Autonomous Syst 165:104426","journal-title":"Robot Autonomous Syst"},{"key":"7066_CR8","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-019-9212-z","volume":"15","author":"S Ali","year":"2021","unstructured":"Ali S, Sun H, Zhao Y (2021) Model learning: a survey of foundations, tools and applications. Front Comput Sci 15:155210","journal-title":"Front Comput Sci"},{"key":"7066_CR9","doi-asserted-by":"crossref","unstructured":"Beschastnikh I, Brun Y, Schneider S, Sloan M, Ernst MD (2011) Leveraging existing instrumentation to automatically infer invariant-constrained models. In: 19th ACM SIGSOFT symposium on the foundations of software engineering and 13th European software engineering conference, FSE\/ESEC\u201913, ACM pp 267\u2013277","DOI":"10.1145\/2025113.2025151"},{"key":"7066_CR10","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/0890-5401(87)90052-6","volume":"2","author":"D Angluin","year":"1987","unstructured":"Angluin D (1987) Learning regular sets from queries and counterexamples. Inf Computat 2:87\u2013106","journal-title":"Inf Computat"},{"key":"7066_CR11","doi-asserted-by":"crossref","unstructured":"Aichernig BK, Mostowski W, Mousavi MR, Tappler M, Taromirad M (2018) Model learning and model-based testing. In: Machine learning for dynamic software analysis: potentials and limits, Springer pp 74\u2013100","DOI":"10.1007\/978-3-319-96562-8_3"},{"issue":"2","key":"7066_CR12","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1145\/2967606","volume":"60","author":"FW Vaandrager","year":"2017","unstructured":"Vaandrager FW (2017) Model learning. Commun ACM 60(2):86\u201395","journal-title":"Commun ACM"},{"key":"7066_CR13","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.106041","volume":"121","author":"M M\u00e9ndez","year":"2023","unstructured":"M\u00e9ndez M, Merayo MG, N\u00fa\u00f1ez M (2023) Long-term traffic flow forecasting using a hybrid CNN-BiLSTM model. Eng Appl Artif Intell 121:106041","journal-title":"Eng Appl Artif Intell"},{"key":"7066_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2023.119078","volume":"640","author":"A Ibias","year":"2023","unstructured":"Ibias A, Varma VR, Capala K, Gherardini L, Sousa J (2023) SaNDA: a Small and iNcomplete DAtaset analyser. Inf Sci 640:119078","journal-title":"Inf Sci"},{"key":"7066_CR15","doi-asserted-by":"crossref","unstructured":"M\u00e9ndez M, Montero C, N\u00fa\u00f1ez M (2022) Using deep transformer based models to predict ozone levels. In: 14th Asian conference on intelligent information and database systems, ACIIDS\u201922, LNAI 13757, Springer pp 169\u2013182","DOI":"10.1007\/978-3-031-21743-2_14"},{"key":"7066_CR16","doi-asserted-by":"crossref","unstructured":"Almeida RD, Silva RM, Serrano LS, S.\u00a0Campos H, O.\u00a0Neves V (2023) Mock objects in software testing: An analysis of usage in open-source projects. In: Proceedings of the XXII Brazilian symposium on software quality, SBQS 2023, Brasilia, Brazil, November 7-10, 2023, ACM pp 72\u201379","DOI":"10.1145\/3629479.3629510"},{"key":"7066_CR17","doi-asserted-by":"publisher","first-page":"88008","DOI":"10.1109\/ACCESS.2023.3302356","volume":"11","author":"MS Farooq","year":"2023","unstructured":"Farooq MS, Omer U, Ramzan A, Rasheed MA, Atal Z (2023) Behavior driven development: A systematic literature review. IEEE Access 11:88008\u201388024","journal-title":"IEEE Access"},{"key":"7066_CR18","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: 31st Conf. on neural information processing systems, NIPS\u201917, pp 1\u201311"},{"issue":"4","key":"7066_CR19","doi-asserted-by":"publisher","first-page":"831","DOI":"10.2307\/412337","volume":"45","author":"PA Reich","year":"1969","unstructured":"Reich PA (1969) The finiteness of natural language. Language 45(4):831\u2013843","journal-title":"Language"},{"key":"7066_CR20","unstructured":"Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler DM, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language models are few-shot learners. In: 33rd Annual conf. on neural information processing systems, NeurIPS\u201920"},{"key":"7066_CR21","doi-asserted-by":"crossref","unstructured":"Oncina J, Garcia P (1992) Inferring regular languages in polynomial updated time. In: Pattern recognition and image analysis: Selected papers from the 4th Spanish Symposium, World Scientific pp 49\u201361","DOI":"10.1142\/9789812797902_0004"},{"key":"7066_CR22","doi-asserted-by":"crossref","unstructured":"Isberner M, Howar F, Steffen B (2014) The TTT algorithm: A redundancy-free approach to active automata learning. In: 5th Int. Conf. on Runtime Verification, RV\u201914, LNCS 8734, Springer pp 307\u2013322","DOI":"10.1007\/978-3-319-11164-3_26"},{"issue":"5","key":"7066_CR23","doi-asserted-by":"publisher","first-page":"729","DOI":"10.1093\/jigpal\/jzl007","volume":"14","author":"A Groce","year":"2006","unstructured":"Groce A, Peled D, Yannakakis M (2006) Adaptive model checking Logic J IGPL 14(5):729\u2013744","journal-title":"Adaptive model checking Logic J IGPL"},{"issue":"1\u20132","key":"7066_CR24","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1007\/s10994-013-5405-0","volume":"96","author":"F Aarts","year":"2014","unstructured":"Aarts F, Kuppens H, Tretmans J, Vaandrager FW, Verwer S (2014) Improving active mealy machine learning for protocol conformance testing. Mach Learn 96(1\u20132):189\u2013224","journal-title":"Mach Learn"},{"key":"7066_CR25","doi-asserted-by":"crossref","unstructured":"Pitt L, Warmuth MK (1989) The minimum consistent DFA problem cannot be approximated within any polynomial. In: 21st Annual ACM Symposium on Theory of Computing, STOC\u201989, ACM pp 421\u2013432","DOI":"10.1145\/73007.73048"},{"key":"7066_CR26","doi-asserted-by":"crossref","unstructured":"Kearns MJ, Valiant LG (1989) Cryptographic limitations on learning boolean formulae and finite automata. In: 21st Annual ACM symposium on theory of computing, STOC\u201989, ACM pp 433\u2013444","DOI":"10.1145\/73007.73049"},{"issue":"5","key":"7066_CR27","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-019-9212-z","volume":"15","author":"S Ali","year":"2021","unstructured":"Ali S, Sun H, Zhao Y (2021) Model learning: a survey of foundations, tools and applications. Front Comput Sci 15(5):155210","journal-title":"Front Comput Sci"},{"key":"7066_CR28","unstructured":"Xiao G, Southey F, Holte RC, Wilkinson DF (2005) Software testing by active learning for commercial games. In: 20th National conference on artificial intelligence, AAAI\u201905, AAAI Press \/ The MIT Press pp 898\u2013903"},{"key":"7066_CR29","doi-asserted-by":"crossref","unstructured":"Avellaneda F, Petrenko A (2019) Learning minimal DFA: taking inspiration from RPNI to improve SAT approach. In: 17th Int. Conf. on software engineering and formal methods, SEFM\u201919, LNCS 11724, Springer pp 243\u2013256","DOI":"10.1007\/978-3-030-30446-1_13"},{"key":"7066_CR30","doi-asserted-by":"crossref","unstructured":"Li N, Liu S, Liu Y, Zhao S, Liu M (2019) Neural speech synthesis with transformer network. In: 33rd Conference on artificial intelligence, AAAI\u201919, 31st Innovative Applications of Artificial Intelligence Conference, IAAI\u201919 and 9th Symposium on Educational Advances in Artificial Intelligence, EAAI\u201919, AAAI Press pp 6706\u20136713","DOI":"10.1609\/aaai.v33i01.33016706"},{"key":"7066_CR31","doi-asserted-by":"crossref","unstructured":"Li S, Sui X, Luo X, Xu X, Liu Y, Goh RSM (2021) Medical image segmentation using squeeze-and-expansion transformers. In: 30th Int. joint conf. on artificial intelligence, IJCAI\u201921, ijcai.org pp 807\u2013815","DOI":"10.24963\/ijcai.2021\/112"},{"key":"7066_CR32","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2023.110124","volume":"136","author":"H Ilyas","year":"2023","unstructured":"Ilyas H, Javed A, Malik KM (2023) AVFakeNet: A unified end-to-end Dense Swin Transformer deep learning model for audio-visual deepfakes detection. Appl Soft Comput 136:110124","journal-title":"Appl Soft Comput"},{"key":"7066_CR33","doi-asserted-by":"crossref","unstructured":"Fontes A, Gay G (2021) Using machine learning to generate test oracles: A systematic literature review. In: 1st Int. workshop on test oracles, TORACLE\u201921, ACM pp 1\u201310","DOI":"10.1145\/3472675.3473974"},{"key":"7066_CR34","doi-asserted-by":"crossref","unstructured":"Makondo W, Nallanthighal R, Mapanga I, Kadebu P (2016) Exploratory test oracle using multi-layer perceptron neural network. In: 5th Int. Conf. on advances in computing, communications and informatics, ICACCI\u201916, IEEE pp 1166\u20131171","DOI":"10.1109\/ICACCI.2016.7732202"},{"key":"7066_CR35","doi-asserted-by":"crossref","unstructured":"Jin H, Wang Y, Chen N, Gou Z, Wang S (2008) Artificial neural network for automatic test oracles generation. In: International Conference on Computer Science and Software Engineering, CSSE\u201908, IEEE Computer Society pp 727\u2013730","DOI":"10.1109\/CSSE.2008.774"},{"key":"7066_CR36","doi-asserted-by":"crossref","unstructured":"Mammadov T (2023) Learning program models from generated inputs. In: 45th Int. Conf. on Software Engineering: ICSE\u201923 Companion Proceedings, IEEE pp 245\u2013247","DOI":"10.1109\/ICSE-Companion58688.2023.00066"},{"key":"7066_CR37","unstructured":"ISO\/IEC JTCI\/SC21\/WG7, ITU-T SG 10\/Q.8 (1996) Information Retrieval, Transfer and Management for OSI; Framework: Formal Methods in Conformance Testing. Committee Draft CD 13245-1, ITU-T proposed recommendation Z.500. ISO \u2013 ITU-T"},{"key":"7066_CR38","unstructured":"Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: 3rd Int. Conf. on Learning Representations, ICLR\u201915"},{"key":"7066_CR39","doi-asserted-by":"crossref","unstructured":"Neider D, Smetsers R, Vaandrager FW, Kuppens H (2019) Benchmarks for automata learning and conformance testing. In: Margaria T, Graf S, Larsen KG (eds.) Models, Mindsets, Meta: The What, the How, and the Why Not? - Essays Dedicated to Bernhard Steffen on the Occasion of His 60th Birthday, Springer pp 390\u2013416","DOI":"10.1007\/978-3-030-22348-9_23"},{"key":"7066_CR40","unstructured":"Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Man\u00e9 D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Vi\u00e9gas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from https:\/\/www.tensorflow.org\/"},{"key":"7066_CR41","doi-asserted-by":"crossref","unstructured":"Isberner M, Howar F, Steffen B (2015) The open-source learnlib: A framework for active automata learning. In: 27th Int. Conf. on Computer Aided Verification, CAV\u201915, LNCS 9206, Springer pp 487\u2013495","DOI":"10.1007\/978-3-319-21690-4_32"},{"key":"7066_CR42","unstructured":"Merrill W, Tsilivis N (2022) Extracting finite automata from rnns using state merging. CoRR arXiv:2201.12451"},{"key":"7066_CR43","doi-asserted-by":"crossref","unstructured":"Lambeau B, Damas C, Dupont P (2008) State-merging DFA induction algorithms with mandatory merge constraints. In: 9th Int. colloquium on grammatical inference: Algorithms and applications, ICGI\u201908, LNCS 5278, Springer pp 139\u2013153","DOI":"10.1007\/978-3-540-88009-7_11"},{"key":"7066_CR44","doi-asserted-by":"crossref","unstructured":"Lang KJ, Pearlmutter BA, Price RA (1998) Results of the abbadingo one DFA learning competition and a new evidence-driven state merging algorithm. In: 4th Int. colloquium on grammatical inference: Algorithms and applications, ICGI\u201998, LNCS 1433, Springer pp 1\u201312","DOI":"10.1007\/BFb0054059"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-025-07066-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-025-07066-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-025-07066-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T05:00:37Z","timestamp":1774933237000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-025-07066-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2]]},"references-count":44,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,2]]}},"alternative-id":["7066"],"URL":"https:\/\/doi.org\/10.1007\/s10489-025-07066-0","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2]]},"assertion":[{"value":"18 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 December 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 February 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"All authors give their consent to publish this work.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}],"article-number":"71"}}