{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T10:06:53Z","timestamp":1778321213506,"version":"3.51.4"},"reference-count":42,"publisher":"Springer Science and Business Media LLC","issue":"7","license":[{"start":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T00:00:00Z","timestamp":1663891200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T00:00:00Z","timestamp":1663891200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Technische Hochschule Mittelhessen"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Domain-specific languages (DSLs) are a popular approach among software engineers who demand for a tailored development interface. A DSL-based approach allows to encapsulate the intricacies of the target platform in transformations that turn DSL models into executable software code. Often, DSLs are even claimed to reduce development complexity to a level that allows them to be successfully applied by domain-experts with limited programming knowledge. Recent research has produced some scientifically backed insights on the benefits and limitations of DSLs. Further empirical studies are required to build a sufficient body of knowledge from which support for different claims related to DSLs can be derived. In this research study, we adopt current DSL evaluation approaches to investigate potential gains in terms of effectiveness and efficiency, through the application of our DSL <jats:italic>Athos<\/jats:italic>, a language developed for the domain of traffic and transportation simulation and optimisation. We compare Athos to the alternative of using an application library defined within a general-purpose language (GPL). We specified two sets of structurally identical tasks from the domain of vehicle routing problems and asked study groups with differing levels of programming knowledge to solve the tasks with the two approaches. The results show that inexperienced participants achieved considerable gains in effectiveness and efficiency with the usage of Athos DSL. Though hinting at Athos being the more efficient approach, the results were less distinct for more experienced programmers. The vast majority of participants stated to prefer working with Athos over the usage of the presented GPL\u2019s API.<\/jats:p>","DOI":"10.1007\/s10664-022-10210-w","type":"journal-article","created":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T09:02:58Z","timestamp":1663923778000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["An empirical evaluation of a novel domain-specific language \u2013 modelling vehicle routing problems with Athos"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6400-9966","authenticated-orcid":false,"given":"Benjamin","family":"Hoffmann","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Neil","family":"Urquhart","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kevin","family":"Chalmers","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Guckert","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,9,23]]},"reference":[{"key":"10210_CR1","doi-asserted-by":"publisher","unstructured":"Baldacci R, Battarra M, Vigo D (2008) Routing a heterogeneous fleet of vehicles. In: The vehicle routing problem: latest advances and new challenges, operations research\/computer science interfaces, vol 43. Springer US, Boston, pp 3\u201327. https:\/\/doi.org\/10.1007\/978-0-387-77778-8_1","DOI":"10.1007\/978-0-387-77778-8_1"},{"key":"10210_CR2","doi-asserted-by":"publisher","unstructured":"Bari\u0161i\u0107 A, Amaral V, Goul\u00e3o M (2012a) Usability evaluation of domain-specific languages. In: 2012 eighth international conference on the quality of information and communications technology (QUATIC). IEEE, Piscataway, pp 342\u2013347. https:\/\/doi.org\/10.1109\/QUATIC.2012.63","DOI":"10.1109\/QUATIC.2012.63"},{"key":"10210_CR3","doi-asserted-by":"publisher","unstructured":"Bari\u0161i\u0107 A, Amaral V, Goul\u00e3o M, Barroca B (2012b) How to reach a usable dsl? Moving toward a systematic evaluation. Electronic Communication of The European Association of Software Science and Technology. https:\/\/doi.org\/10.14279\/tuj.eceasst.50.741.750","DOI":"10.14279\/tuj.eceasst.50.741.750"},{"key":"10210_CR4","doi-asserted-by":"publisher","unstructured":"Bari\u0161i\u0107 A, Amaral V, Goul\u00e3o M, Barroca B (2014) Evaluating the usability of domain-specific languages. In: Software design and development: concepts, methodologies, tools, and applications, IGI Global, pp 2120\u20132141. https:\/\/doi.org\/10.4018\/978-1-4666-4301-7.ch098","DOI":"10.4018\/978-1-4666-4301-7.ch098"},{"key":"10210_CR5","unstructured":"Basili VR (1994) Goal question metric paradigm. Encyclopedia of Software Engineering:528\u2013532. https:\/\/ci.nii.ac.jp\/naid\/10011278869\/en\/"},{"key":"10210_CR6","doi-asserted-by":"publisher","first-page":"755","DOI":"10.1007\/s11219-015-9291-5","volume":"24","author":"M Challenger","year":"2016","unstructured":"Challenger M, Kardas G, Tekinerdogan B (2016) A systematic approach to evaluating domain-specific modeling language environments for multi-agent systems. Software Quality Journal 24:755\u2013795. https:\/\/doi.org\/10.1007\/s11219-015-9291-5","journal-title":"Software Quality Journal"},{"issue":"18","key":"10210_CR7","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1002\/cpe.5802","volume":"33","author":"G Cordasco","year":"2021","unstructured":"Cordasco G, D\u2019Auria M, Negro A, Scarano V, Spagnuolo C (2021) Toward a domain\u2013specific language for scientific workflow\u2013based applications on multicloud system. Concurrency and Computation: Practice and Experience 33(18):66. https:\/\/doi.org\/10.1002\/cpe.5802","journal-title":"Concurrency and Computation: Practice and Experience"},{"key":"10210_CR8","unstructured":"do Nascimento LM, Viana DL, Am Neto PS, Martins DAO, Garcia VC, Meira SRL (2012) A systematic mapping study on domain-specific languages. In: Proceedings of the 7th international conference on software engineering advances (ICSEA\u201912), pp 179\u2013187"},{"key":"10210_CR9","doi-asserted-by":"publisher","unstructured":"Ewais A, de Troyer O (2014) A usability evaluation of graphical modelling languages for authoring adaptive 3d virtual learning environments. In: Proceedings of the 6th international conference on computer supported education - Volume 1, SCITEPRESS - Science and Technology Publications, Lda, Setubal, PRT, CSEDU 2014, pp 459\u2013466. https:\/\/doi.org\/10.5220\/0004947204590466","DOI":"10.5220\/0004947204590466"},{"key":"10210_CR10","unstructured":"Feldt R, Magazinius A (2010) Validity threats in empirical software engineering research - an initial survey. In: Proceedings of the 22nd international conference on software engineering and knowledge engineering (SEKE), vol 16, Redwood City, California, pp 374\u2013379"},{"key":"10210_CR11","doi-asserted-by":"publisher","unstructured":"Hermans F, Pinzger M, van Deursen A (2009) Domain-specific languages in practice: a user study on the success factors. In: Model driven engineering languages and systems: 12th international conference, MODELS 2009, Denver, CO, USA, October 4-9, 2009. Proceedings. Springer, pp 423\u2013437. https:\/\/doi.org\/10.1007\/978-3-642-04425-0_33","DOI":"10.1007\/978-3-642-04425-0_33"},{"key":"10210_CR12","doi-asserted-by":"publisher","unstructured":"Hoffmann B, Chalmers K, Urquhart N, Farrenkopf T, Guckert M (2018) Towards reducing complexity of multi-agent simulations by applying model-driven techniques. In: Advances in practical applications of agents, multi-agent systems, and complexity: the PAAMS Collection. Springer International Publishing, pp 187\u2013199. https:\/\/doi.org\/10.1007\/978-3-319-94580-4_15","DOI":"10.1007\/978-3-319-94580-4_15"},{"key":"10210_CR13","doi-asserted-by":"publisher","unstructured":"Hoffmann B, Guckert M, Chalmers K, Urquhart N (2019) Simulating dynamic vehicle routing problems with athos. In: ECMS 2019 proceedings, European council for modeling and simulation, vol 33, pp 296\u2013302. https:\/\/doi.org\/10.7148\/2019-0296","DOI":"10.7148\/2019-0296"},{"key":"10210_CR14","unstructured":"Hoffmann B, Urquhart N, Chalmers K, Guckert M (2020) Athos: an extensible dsl for model driven traffic and transport simulation. In: Modellierung 2020, Gesellschaft f\u00fcr Informatik e.V, Bonn, pp 141\u2013156"},{"key":"10210_CR15","doi-asserted-by":"publisher","first-page":"4205","DOI":"10.1007\/s10664-020-09872-1","volume":"25","author":"A Iung","year":"2020","unstructured":"Iung A, Carbonell J, Marchezan L, Rodrigues E, Bernardino M, Basso FP, Medeiros B (2020) Systematic mapping study on domain-specific language development tools. Empirical Software Engineering 25:4205\u20134249. https:\/\/doi.org\/10.1007\/s10664-020-09872-1","journal-title":"Empirical Software Engineering"},{"key":"10210_CR16","doi-asserted-by":"publisher","unstructured":"Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Guide to advanced empirical software engineering. Springer, London, pp 201\u2013228. https:\/\/doi.org\/10.1007\/978-1-84800-044-5_8","DOI":"10.1007\/978-1-84800-044-5_8"},{"issue":"4","key":"10210_CR17","doi-asserted-by":"publisher","first-page":"2206","DOI":"10.1007\/s10664-016-9483-z","volume":"22","author":"AN Johanson","year":"2017","unstructured":"Johanson AN, Hasselbring W (2017) Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: a controlled experiment. Empirical Software Engineering 22(4):2206\u20132236. https:\/\/doi.org\/10.1007\/s10664-016-9483-z","journal-title":"Empirical Software Engineering"},{"issue":"1","key":"10210_CR18","first-page":"43","volume":"13","author":"A Kahlaoui","year":"2008","unstructured":"Kahlaoui A, Abran A, Lefebvre \u00c9 (2008) Dsml success factors and their assessment criteria. Metrics News 13(1):43\u201351","journal-title":"Metrics News"},{"issue":"4","key":"10210_CR19","doi-asserted-by":"publisher","first-page":"1505","DOI":"10.1007\/s10270-013-0387-8","volume":"14","author":"G Kahraman","year":"2015","unstructured":"Kahraman G, Bilgen S (2015) A framework for qualitative assessment of domain-specific languages. Software & Systems Modeling 14(4):1505\u20131526. https:\/\/doi.org\/10.1007\/s10270-013-0387-8","journal-title":"Software & Systems Modeling"},{"key":"10210_CR20","doi-asserted-by":"publisher","first-page":"247","DOI":"10.2298\/CSIS1002247K","volume":"7","author":"T Kosar","year":"2010","unstructured":"Kosar T, Oliveira N, Mernik M, Pereira MJ, \u010crepin\u0161ek M, da Cruz D, Henriques P (2010) Comparing general-purpose and domain-specific languages: an empirical study. ComSIS\u2013Computer Science an Information Systems Journal 7:247\u2013264. https:\/\/doi.org\/10.2298\/CSIS1002247K","journal-title":"ComSIS\u2013Computer Science an Information Systems Journal"},{"key":"10210_CR21","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1007\/s10664-011-9172-x","volume":"17","author":"T Kosar","year":"2012","unstructured":"Kosar T, Mernik M, Carver JC (2012) Program comprehension of domain-specific and general-purpose languages: comparison using a family of experiments. Empirical Software Engineering 17:276\u2013304. https:\/\/doi.org\/10.1007\/s10664-011-9172-x","journal-title":"Empirical Software Engineering"},{"key":"10210_CR22","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1016\/j.infsof.2015.11.001","volume":"71","author":"T Kosar","year":"2016","unstructured":"Kosar T, Bohra S, Mernik M (2016) Domain-specific languages: a systematic mapping study. Information and Software Technology 71:77\u201391. https:\/\/doi.org\/10.1016\/j.infsof.2015.11.001","journal-title":"Information and Software Technology"},{"key":"10210_CR23","doi-asserted-by":"publisher","first-page":"2734","DOI":"10.1007\/s10664-017-9593-2","volume":"23","author":"T Kosar","year":"2018","unstructured":"Kosar T, Gaberc S, Carver JC, Mernik M (2018) Program comprehension of domain-specific and general-purpose languages: replication of a family of experiments using integrated development environments. Empirical Software Engineering 23:2734\u20132763. https:\/\/doi.org\/10.1007\/s10664-017-9593-2","journal-title":"Empirical Software Engineering"},{"key":"10210_CR24","doi-asserted-by":"publisher","unstructured":"Mayer T, Uhlig T, Rose O (2016) An open-source discrete event simulator for rich vehicle routing problems. In: 2016 IEEE 19th international conference on intelligent transportation systems (ITSC). IEEE, pp 1305\u20131310. https:\/\/doi.org\/10.1109\/ITSC.2016.7795725","DOI":"10.1109\/ITSC.2016.7795725"},{"issue":"4","key":"10210_CR25","doi-asserted-by":"publisher","first-page":"316","DOI":"10.1145\/1118890.1118892","volume":"37","author":"M Mernik","year":"2005","unstructured":"Mernik M, Heering J, Sloane AM (2005) When and how to develop domain-specific languages. ACM Comput Surv 37(4):316\u2013344. https:\/\/doi.org\/10.1145\/1118890.1118892","journal-title":"ACM Comput Surv"},{"key":"10210_CR26","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/s10489-006-6926-z","volume":"24","author":"B Ombuki","year":"2006","unstructured":"Ombuki B, Ross BJ, Hanshar F (2006) Multi-objective genetic algorithms for vehicle routing problem with time windows. Applied Intelligence 24:17\u201330. https:\/\/doi.org\/10.1007\/s10489-006-6926-z","journal-title":"Applied Intelligence"},{"issue":"1","key":"10210_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ejor.2012.08.015","volume":"225","author":"V Pillac","year":"2013","unstructured":"Pillac V, Gendreau M, Gu\u00e9ret C, Medaglia AL (2013) A review of dynamic vehicle routing problems. European Journal of Operational Research 225 (1):1\u201311. https:\/\/doi.org\/10.1016\/j.ejor.2012.08.015","journal-title":"European Journal of Operational Research"},{"key":"10210_CR28","doi-asserted-by":"publisher","unstructured":"Poltronieri I, Zorzo AF, Bernardino M, de Borba Campos M (2018) Usa-dsl: usability evaluation framework for domain-specific languages. In: Proceedings of the 33rd annual ACM symposium on applied computing, association for computing machinery, New York, USA, SAC \u201918, pp 2013\u20132021. https:\/\/doi.org\/10.1145\/3167132.3167348","DOI":"10.1145\/3167132.3167348"},{"key":"10210_CR29","doi-asserted-by":"crossref","unstructured":"Poltronieri I, Pedroso AC, Zorzo AF, Bernardino M, de Borba Campos M (2021) Is usability evaluation of dsl still a trending topic? In: Kurosu M (ed) Human-computer interaction. Theory, methods and tools. Springer International Publishing, Cham, pp 299\u2013317","DOI":"10.1007\/978-3-030-78462-1_23"},{"key":"10210_CR30","unstructured":"Pr\u00fcmper J (1999) Test it: Isonorm 9241\/10. In: Human-computer interaction \u2013 communication, cooperation, and application design. L. Erlbaum Associates Inc, New Jersey, USA, pp 1028\u20131032"},{"issue":"9","key":"10210_CR31","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1109\/TSE.2005.97","volume":"31","author":"D Sjoeberg","year":"2005","unstructured":"Sjoeberg D, Hannay JE, Hansen O, Kampenes VB, Karahasanovic A, Liborg NK, Rekdal AC (2005) A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31(9):733\u2013753. https:\/\/doi.org\/10.1109\/TSE.2005.97","journal-title":"IEEE Transactions on Software Engineering"},{"key":"10210_CR32","doi-asserted-by":"publisher","unstructured":"de Sousa LM, da Silva AR (2018) Usability evaluation of the domain specific language for spatial simulation scenarios. Cogent Engineering 5(1). https:\/\/doi.org\/10.1080\/23311916.2018.1436889","DOI":"10.1080\/23311916.2018.1436889"},{"key":"10210_CR33","doi-asserted-by":"publisher","unstructured":"ISO\/IEC 25010 (2011) Systems and software engineering. Systems and software quality requirements and evaluation (square). System and software quality models. https:\/\/doi.org\/10.3403\/30215101","DOI":"10.3403\/30215101"},{"issue":"5","key":"10210_CR34","doi-asserted-by":"publisher","first-page":"2907","DOI":"10.1007\/s10270-018-00705-w","volume":"18","author":"B Tekinerdogan","year":"2019","unstructured":"Tekinerdogan B, Arkin E (2019) Pardsl: a domain-specific language framework for supporting deployment of parallel algorithms. Software & Systems Modeling 18(5):2907\u20132935. https:\/\/doi.org\/10.1007\/s10270-018-00705-w","journal-title":"Software & Systems Modeling"},{"issue":"4","key":"10210_CR35","doi-asserted-by":"publisher","first-page":"569","DOI":"10.3758\/BF03193886","volume":"38","author":"DR Thorne","year":"2006","unstructured":"Thorne DR (2006) Throughput: a simple performance index with desirable characteristics. Behavior Research Methods 38(4):569\u2013573. https:\/\/doi.org\/10.3758\/BF03193886","journal-title":"Behavior Research Methods"},{"key":"10210_CR36","unstructured":"Tisue S, Wilensky U (2004) Netlogo: design and implementation of a multi-agent modeling environment. In: Proceedings of the Agent 2004 Conference, pp 161\u2013184"},{"key":"10210_CR37","volume-title":"The goal\/question\/metric method: a practical guide for quality improvement of software development","author":"R van Solingen","year":"1999","unstructured":"van Solingen R, Berghout E, Berghout EW (1999) The goal\/question\/metric method: a practical guide for quality improvement of software development. The McGraw-Hill Companies, London"},{"issue":"2","key":"10210_CR38","doi-asserted-by":"publisher","first-page":"653","DOI":"10.3758\/s13428-016-0721-5","volume":"49","author":"A Vandierendonck","year":"2017","unstructured":"Vandierendonck A (2017) A comparison of methods to combine speed and accuracy measures of performance: a rejoinder on the binning procedure. Behavior Research Methods 49(2):653\u2013673. https:\/\/doi.org\/10.3758\/s13428-016-0721-5","journal-title":"Behavior Research Methods"},{"key":"10210_CR39","doi-asserted-by":"publisher","unstructured":"Vandierendonck A (2018) Further tests of the utility of integrated speed-accuracy measures in task switching. Journal of Cognition 1(1). https:\/\/doi.org\/10.5334\/joc.6","DOI":"10.5334\/joc.6"},{"key":"10210_CR40","doi-asserted-by":"publisher","unstructured":"Welch P (2017) Developing a commercial dynamic vehicle routing system - a case study. https:\/\/doi.org\/10.13140\/RG.2.2.14915.30247","DOI":"10.13140\/RG.2.2.14915.30247"},{"key":"10210_CR41","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in software engineering","author":"C Wohlin","year":"2012","unstructured":"Wohlin C, Runeson P, H\u00f6st M, Ohlsson MC (2012) Experimentation in software engineering. Springer Berlin Heidelberg, Berlin"},{"key":"10210_CR42","doi-asserted-by":"publisher","first-page":"668","DOI":"10.3758\/BF03193587","volume":"34","author":"DJ Woltz","year":"2006","unstructured":"Woltz DJ, Was CA (2006) Availability of related long-term memory during and after attention focus in working memory. Memory & Cognition 34:668\u2013684. https:\/\/doi.org\/10.3758\/BF03193587","journal-title":"Memory & Cognition"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10210-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10210-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10210-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T02:14:16Z","timestamp":1668996856000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10210-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,23]]},"references-count":42,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["10210"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10210-w","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,23]]},"assertion":[{"value":"13 July 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 September 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All procedures performed in the presented study that involve human participants were in accordance with the ethical standards of the institutional and\/or national research committees.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}},{"value":"All authors have no conflict of interest to report.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}],"article-number":"180"}}