{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T12:16:46Z","timestamp":1771330606318,"version":"3.50.1"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2020,5,6]],"date-time":"2020-05-06T00:00:00Z","timestamp":1588723200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,5,6]],"date-time":"2020-05-06T00:00:00Z","timestamp":1588723200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2020,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n<jats:title>Context<\/jats:title>\n<jats:p>The testing of software-intensive systems is performed in different test stages each having a large number of test cases. These test cases are commonly derived from requirements. Each test stages exhibits specific demands and constraints with respect to their degree of detail and what can be tested. Therefore, specific test suites are defined for each test stage. In this paper, the focus is on the domain of embedded systems, where, among others, typical test stages are Software- and Hardware-in-the-loop.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Objective<\/jats:title>\n<jats:p>Monitoring and controlling which requirements are verified in which detail and in which test stage is a challenge for engineers. However, this information is necessary to assure a certain test coverage, to minimize redundant testing procedures, and to avoid inconsistencies between test stages. In addition, engineers are reluctant to state their requirements in terms of structured languages or models that would facilitate the relation of requirements to test executions.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Method<\/jats:title>\n<jats:p>With our approach, we close the gap between requirements specifications and test executions. Previously, we have proposed a lightweight markup language for requirements which provides a set of annotations that can be applied to natural language requirements. The annotations are mapped to events and signals in test executions. As a result, meaningful insights from a set of test executions can be directly related to artifacts in the requirements specification. In this paper, we use the markup language to compare different test stages with one another.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>We annotate 443 natural language requirements of a driver assistance system with the means of our lightweight markup language. The annotations are then linked to 1300 test executions from a simulation environment and 53 test executions from test drives with human drivers. Based on the annotations, we are able to analyze how similar the test stages are and how well test stages and test cases are aligned with the requirements. Further, we highlight the general applicability of our approach through this extensive experimental evaluation.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Conclusion<\/jats:title>\n<jats:p>With our approach, the results of several test levels are linked to the requirements and enable the evaluation of complex test executions. By this means, practitioners can easily evaluate how well a systems performs with regards to its specification and, additionally, can reason about the expressiveness of the applied test stage.<\/jats:p>\n<\/jats:sec>","DOI":"10.1007\/s10664-020-09815-w","type":"journal-article","created":{"date-parts":[[2020,5,6]],"date-time":"2020-05-06T21:02:46Z","timestamp":1588798966000},"page":"2809-2843","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["What am I testing and where? Comparing testing procedures based on lightweight requirements annotations"],"prefix":"10.1007","volume":"25","author":[{"given":"Florian","family":"Pudlitz","sequence":"first","affiliation":[]},{"given":"Florian","family":"Brokhausen","sequence":"additional","affiliation":[]},{"given":"Andreas","family":"Vogelsang","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,5,6]]},"reference":[{"key":"9815_CR1","volume-title":"Discrete-event system simulation","author":"J Banks","year":"2000","unstructured":"Banks J, Carson JS, Nelson BL, Nicol DM (2000) Discrete-event system simulation. Prentice Hall, Upper Saddle River"},{"key":"9815_CR2","doi-asserted-by":"publisher","unstructured":"Barmi ZA, Ebrahimi AH, Feldt R (2011) Alignment of requirements specification and testing: A systematic mapping study. In: IEEE International conference on software testing, verification and validation workshops. https:\/\/doi.org\/10.1109\/ICSTW.2011.58. IEEE","DOI":"10.1109\/ICSTW.2011.58"},{"issue":"6","key":"9815_CR3","doi-asserted-by":"publisher","first-page":"1809","DOI":"10.1007\/s10664-013-9263-y","volume":"19","author":"E Bjarnason","year":"2014","unstructured":"Bjarnason E, Runeson P, Borg M, Unterkalmsteiner M, Engstr\u00f6m E, Regnell B, Sabaliauskaite G, Loconsole A, Gorschek T, Feldt R (2014) Challenges and practices in aligning requirements with verification and validation: a case study of six companies. Empir Softw Eng 19(6):1809\u20131855. https:\/\/doi.org\/10.1007\/s10664-013-9263-y","journal-title":"Empir Softw Eng"},{"issue":"9","key":"9815_CR4","doi-asserted-by":"publisher","first-page":"796","DOI":"10.1109\/TSE.2003.1232285","volume":"29","author":"J Cleland-Huang","year":"2003","unstructured":"Cleland-Huang J, Chang CK, Christensen M (2003) Event-based traceability for managing evolutionary change. IEEE Trans Softw Eng 29(9):796\u2013810. https:\/\/doi.org\/10.1109\/TSE.2003.1232285","journal-title":"IEEE Trans Softw Eng"},{"key":"9815_CR5","doi-asserted-by":"publisher","unstructured":"Codeca L, Frank R, Engel T (2015) Luxembourg sumo traffic (lust) scenario: 24 hours of mobility for vehicular networking research. In: 2015 IEEE Vehicular Networking Conference (VNC), pp 1\u20138. https:\/\/doi.org\/10.1109\/VNC.2015.7385539","DOI":"10.1109\/VNC.2015.7385539"},{"key":"9815_CR6","doi-asserted-by":"publisher","unstructured":"Dalpiaz F (2018) Requirements data sets (user stories). Mendeley Data, vol v1. https:\/\/doi.org\/10.17632\/7zbk8zsd8y.1","DOI":"10.17632\/7zbk8zsd8y.1"},{"key":"9815_CR7","unstructured":"DIN (2012) Bahnanwendungen \u2013 Telekommunikationstechnik, Signaltechnik und Datenverarbeitungssysteme. DIN EN 50128 DIN Deutsches Institut f\u00fcr Normen e.V., Berlin, Germany"},{"key":"9815_CR8","doi-asserted-by":"publisher","unstructured":"Eckhardt J, Vogelsang A, Femmer H, Mager P (2016) Challenging incompleteness of performance requirements by sentence patterns. In: 2016 IEEE 24th International Requirements Engineering Conference (RE), pp 46\u201355. https:\/\/doi.org\/10.1109\/RE.2016.24","DOI":"10.1109\/RE.2016.24"},{"key":"9815_CR9","doi-asserted-by":"publisher","unstructured":"Galko C, Rossi R, Savatier X (2014) Vehicle-hardware-in-the-loop system for adas prototyping and validation. In: 2014 International conference on embedded computer systems: Architectures, Modeling, and Simulation (SAMOS XIV), pp 329-334. https:\/\/doi.org\/10.1109\/SAMOS.2014.6893229","DOI":"10.1109\/SAMOS.2014.6893229"},{"key":"9815_CR10","doi-asserted-by":"publisher","unstructured":"Gotel OCZ, Finkelstein CW (1994) An analysis of the requirements traceability problem. In: Proceedings of IEEE International Conference on Requirements Engineering, pp 94\u2013101, https:\/\/doi.org\/10.1109\/ICRE.1994.292398","DOI":"10.1109\/ICRE.1994.292398"},{"issue":"3","key":"9815_CR11","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1109\/52.896248","volume":"17","author":"CA Gunter","year":"2000","unstructured":"Gunter CA, Gunter EL, Jackson M, Zave P (2000) A reference model for requirements and specifications. IEEE Softw 17(3):37\u201343. https:\/\/doi.org\/10.1109\/52.896248","journal-title":"IEEE Softw"},{"key":"9815_CR12","doi-asserted-by":"publisher","unstructured":"Hasling B, Goetz H, Beetz K (2008) Model based testing of system requirements using uml use case models. In: 2008 1st international conference on software testing, verification, and validation, pp 367\u2013376. https:\/\/doi.org\/10.1109\/ICST.2008.9","DOI":"10.1109\/ICST.2008.9"},{"key":"9815_CR13","doi-asserted-by":"publisher","unstructured":"Huang W, Wang K, Yisheng L, Zhu F (2016) Autonomous vehicles testing methods review, pp 163\u2013168. https:\/\/doi.org\/10.1109\/ITSC.2016.7795548","DOI":"10.1109\/ITSC.2016.7795548"},{"key":"9815_CR14","unstructured":"ISO (2011) Road vehicles \u2013 Functional safety. ISO 26262 International Organization for Standardization, Geneva, Switzerland"},{"key":"9815_CR15","doi-asserted-by":"publisher","unstructured":"Jedlitschka A, Pfahl D (2005) Reporting guidelines for controlled experiments in software engineering. In: 2005 international symposium on empirical software engineering, 2005., pp 10 pp.\u2013 https:\/\/doi.org\/10.1109\/ISESE.2005.1541818","DOI":"10.1109\/ISESE.2005.1541818"},{"issue":"4","key":"9815_CR16","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1007\/s11334-014-0232-4","volume":"10","author":"M Kassab","year":"2014","unstructured":"Kassab M, Neill C, Laplante P (2014) State of practice in requirements engineering: contemporary data. Innovations in Systems and Software Engineering 10 (4):235\u2013241. https:\/\/doi.org\/10.1007\/s11334-014-0232-4","journal-title":"Innovations in Systems and Software Engineering"},{"key":"9815_CR17","doi-asserted-by":"publisher","unstructured":"Lopez PA, Behrisch M, Bieker-Walz L, Erdmann J, Fl\u00f6tter\u00f6d Y, Hilbrich R, L\u00fccken L, Rummel J, Wagner P, WieBner E (2018) Microscopic traffic simulation using sumo. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp 2575\u20132582. https:\/\/doi.org\/10.1109\/ITSC.2018.8569938","DOI":"10.1109\/ITSC.2018.8569938"},{"key":"9815_CR18","doi-asserted-by":"publisher","unstructured":"Lucassen G, Robeer M, Dalpiaz F, van der Werf J, Brinkkemper S (2017) Extracting conceptual models from user stories with visual narrator. Requir Eng, pp 1\u201320. https:\/\/doi.org\/10.1007\/s00766-017-0270-1","DOI":"10.1007\/s00766-017-0270-1"},{"key":"9815_CR19","doi-asserted-by":"publisher","unstructured":"Mavin A, Wilkinson P, Harwood A, Novak M (2009) Easy approach to requirements syntax (ears). In: 2009 17th IEEE International Requirements Engineering Conference, pp 317\u2013322. https:\/\/doi.org\/10.1109\/RE.2009.9","DOI":"10.1109\/RE.2009.9"},{"key":"9815_CR20","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1007\/s00766-003-0179-8","volume":"9","author":"L Mich","year":"2004","unstructured":"Mich L, Franch M, Novi Inverardi PL (2004) Market research for requirements analysis using linguistic tools. Requir Eng 9:40\u201356. https:\/\/doi.org\/10.1007\/s00766-003-0179-8","journal-title":"Requir Eng"},{"key":"9815_CR21","doi-asserted-by":"publisher","unstructured":"Murukannaiah PK, Ajmeri N, Singh MP (2017) Toward automating crowd re. In: 2017 IEEE 25th International Requirements Engineering Conference (RE), pp 512\u2013515. https:\/\/doi.org\/10.1109\/RE.2017.74","DOI":"10.1109\/RE.2017.74"},{"issue":"3","key":"9815_CR22","doi-asserted-by":"publisher","first-page":"140","DOI":"10.1109\/TSE.2006.22","volume":"32","author":"C Nebut","year":"2006","unstructured":"Nebut C, Fleurey F, Le Traon Y, Jezequel J (2006) Automatic test generation: a use case driven approach. IEEE Trans Softw Eng 32(3):140\u2013155. https:\/\/doi.org\/10.1109\/TSE.2006.22","journal-title":"IEEE Trans Softw Eng"},{"key":"9815_CR23","doi-asserted-by":"crossref","unstructured":"Protzmann R, Sch\u00fcnemann B, Radusch I (2017) Simulation of convergent networks for intelligent transport systems with vsimrti. Networking Simulation for Intelligent Transportation Systems: High Mobile Wireless Nodes, pp 1\u201328","DOI":"10.1002\/9781119407447.ch1"},{"key":"9815_CR24","doi-asserted-by":"publisher","unstructured":"Pudlitz F, Brokhausen F, Vogelsang A (2019a) Extraction of system states from natural language requirements. In: 27th IEEE International Requirements Engineering Conference (RE), https:\/\/doi.org\/10.14279\/depositonce-8717","DOI":"10.14279\/depositonce-8717"},{"key":"9815_CR25","doi-asserted-by":"publisher","unstructured":"Pudlitz F, Vogelsang A, Brokhausen F (2019b) A lightweight multilevel markup language for connecting software requirements and simulations. In: Knauss E, Goedicke M (eds) Requirements engineering: foundation for software quality. https:\/\/doi.org\/10.1007\/978-3-030-15538-4_11. Springer International Publishing, Cham, pp 151\u2013166","DOI":"10.1007\/978-3-030-15538-4_11"},{"issue":"1","key":"9815_CR26","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1109\/32.895989","volume":"27","author":"B Ramesh","year":"2001","unstructured":"Ramesh B, Jarke M (2001) Toward reference models for requirements traceability. IEEE Trans Softw Eng 27(1):58\u201393. https:\/\/doi.org\/10.1109\/32.895989","journal-title":"IEEE Trans Softw Eng"},{"key":"9815_CR27","unstructured":"RTCA (2012) Software considerations in airborne systems and equipment certification. DO- 178c, RTCA and EUROCAE"},{"issue":"5","key":"9815_CR28","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1002\/stvr.456","volume":"22","author":"M Utting","year":"2012","unstructured":"Utting M, Pretschner A, Legeard B (2012) A taxonomy of model-based testing approaches. Software Testing, Verification and Reliability 22(5):297\u2013312. https:\/\/doi.org\/10.1002\/stvr.456","journal-title":"Software Testing, Verification and Reliability"},{"key":"9815_CR29","doi-asserted-by":"publisher","unstructured":"Uusitalo EJ, Komssi M, Kauppinen M, Davis AM (2008) Linking requirements and testing in practice. In: IEEE International Requirements Engineering Conference (RE), pp 265\u2013270. https:\/\/doi.org\/10.1109\/RE.2008.30","DOI":"10.1109\/RE.2008.30"},{"key":"9815_CR30","doi-asserted-by":"crossref","unstructured":"Whalen MW, Rajan A, Heimdahl MP, Miller SP (2006) Coverage metrics for requirements-based testing. In: International Symposium on Software Testing and Analysis (ISTA), pp 25\u201336,","DOI":"10.1145\/1146238.1146242"},{"issue":"2","key":"9815_CR31","doi-asserted-by":"publisher","first-page":"1513","DOI":"10.1007\/s10270-017-0623-8","volume":"18","author":"D W\u00fcest","year":"2019","unstructured":"W\u00fcest D, Seyff N, Glinz M (2019) Flexisketch: a lightweight sketching and metamodeling approach for end-users. Software & Systems Modeling 18(2):1513\u20131541. https:\/\/doi.org\/10.1007\/s10270-017-0623-8","journal-title":"Software & Systems Modeling"},{"issue":"4","key":"9815_CR32","doi-asserted-by":"publisher","first-page":"366","DOI":"10.1145\/267580.267590","volume":"29","author":"H Zhu","year":"1997","unstructured":"Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4):366\u2013427. https:\/\/doi.org\/10.1145\/267580.267590","journal-title":"ACM Comput Surv"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09815-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-020-09815-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09815-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:22:46Z","timestamp":1620260566000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-020-09815-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,6]]},"references-count":32,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,7]]}},"alternative-id":["9815"],"URL":"https:\/\/doi.org\/10.1007\/s10664-020-09815-w","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,6]]},"assertion":[{"value":"6 May 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}