{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T04:13:32Z","timestamp":1781064812074,"version":"3.54.1"},"publisher-location":"New York, NY, USA","reference-count":43,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,9,22]],"date-time":"2024-09-22T00:00:00Z","timestamp":1726963200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"name":"Agencia Espa\u00f1ola de Investigaci\u00f3n","award":["PID2020-114615RB-I00\/AEI\/10.13039\/501100011033"],"award-info":[{"award-number":["PID2020-114615RB-I00\/AEI\/10.13039\/501100011033"]}]},{"name":"Luxembourg National Research Fund - PEARL program","award":["16544475"],"award-info":[{"award-number":["16544475"]}]},{"DOI":"10.13039\/501100006374","name":"Electronic Components and Systems for European Leadership","doi-asserted-by":"publisher","award":["101007350"],"award-info":[{"award-number":["101007350"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,9,22]]},"DOI":"10.1145\/3640310.3674093","type":"proceedings-article","created":{"date-parts":[[2024,9,30]],"date-time":"2024-09-30T13:38:07Z","timestamp":1727703487000},"page":"203-213","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["A DSL for Testing LLMs for Fairness and Bias"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5921-9440","authenticated-orcid":false,"given":"Sergio","family":"Morales","sequence":"first","affiliation":[{"name":"Universitat Oberta de Catalunya, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9639-0186","authenticated-orcid":false,"given":"Robert","family":"Claris\u00f3","sequence":"additional","affiliation":[{"name":"Universitat Oberta de Catalunya, Barcelona, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2418-2489","authenticated-orcid":false,"given":"Jordi","family":"Cabot","sequence":"additional","affiliation":[{"name":"Luxembourg Institute of Science and Technology, University of Luxembourg, Esch-sur-Alzette, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,9,22]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"crossref","unstructured":"Sarah Alnegheimish Alicia Guo and Yi Sun. 2022. Using natural sentence prompts for understanding biases in language models. In Human Language Technologies. ACL 2824--2830.","DOI":"10.18653\/v1\/2022.naacl-main.203"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-021-00084-x"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Christine Basta Marta R. Costa-Juss\u00e0 and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Gender Bias in NLP. ACL 33--39.","DOI":"10.18653\/v1\/W19-3805"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12559-021-09881-2"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594095"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600211.3604722"},{"key":"e_1_3_2_1_7_1","volume-title":"Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advances in NeurIPS 29","author":"Bolukbasi Tolga","year":"2016","unstructured":"Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advances in NeurIPS 29 (2016)."},{"key":"e_1_3_2_1_8_1","volume-title":"Marked personas: Using natural language prompts to measure stereotypes in language models. arXiv preprint arXiv:2305.18189","author":"Cheng Myra","year":"2023","unstructured":"Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. Marked personas: Using natural language prompts to measure stereotypes in language models. arXiv preprint arXiv:2305.18189 (2023)."},{"key":"e_1_3_2_1_9_1","volume-title":"Retrieved","author":"Comission European","year":"2019","unstructured":"European Comission. 2019. Ethics guidelines for trustworthy AI. Retrieved February 29, 2024 from https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/ethics-guidelines-trustworthy-ai"},{"key":"e_1_3_2_1_10_1","volume-title":"Proceedings of the 21st International Conference on Software Engineering (ICSE '99)","author":"Dalal S. R.","unstructured":"S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz. 1999. Model-based testing in practice. In Proceedings of the 21st International Conference on Software Engineering (ICSE '99). Association for Computing Machinery, 285--294."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445924"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-020-00011-6"},{"key":"e_1_3_2_1_13_1","volume-title":"Retrieved","author":"Union European","year":"2024","unstructured":"European Union. 2024. The artificial intelligence act. Retrieved February 29, 2024 from https:\/\/artificialintelligenceact.eu"},{"key":"e_1_3_2_1_14_1","volume-title":"Principled artificial intelligence: Mapping consensus in ethical and rights-based approaches to principles for AI","author":"Fjeld Jessica","year":"2020","unstructured":"Jessica Fjeld, Nele Achten, Hannah Hilligoss, Adam Nagy, and Madhulika Srikumar. 2020. Principled artificial intelligence: Mapping consensus in ethical and rights-based approaches to principles for AI. Berkman Klein Center Research Publication 2020-1 (2020)."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-018-9482-5"},{"key":"e_1_3_2_1_16_1","volume-title":"Smith","author":"Gehman Samuel","year":"2020","unstructured":"Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In EMNLP. ACL, 3356--3369."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cola.2023.101209"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.3233\/SSW220008"},{"key":"e_1_3_2_1_19_1","volume-title":"An ontology-based approach to engineering ethicality requirements. SoSyM","author":"Guizzardi Renata","year":"2023","unstructured":"Renata Guizzardi, Glenda Amaral, Giancarlo Guizzardi, and John Mylopoulos. 2023. An ontology-based approach to engineering ethicality requirements. SoSyM (2023), 1--27."},{"key":"e_1_3_2_1_20_1","volume-title":"The ethics of AI ethics: An evaluation of guidelines. Minds and machines 30, 1","author":"Hagendorff Thilo","year":"2020","unstructured":"Thilo Hagendorff. 2020. The ethics of AI ethics: An evaluation of guidelines. Minds and machines 30, 1 (2020), 99--120."},{"key":"e_1_3_2_1_21_1","volume-title":"An ontology for ethical AI principles. Semantic Web Journal","author":"Harrison Andrew","year":"2021","unstructured":"Andrew Harrison, Dayana Spagnuelo, and Ilaria Tiddi. 2021. An ontology for ethical AI principles. Semantic Web Journal (2021)."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00146-021-01267-0"},{"key":"e_1_3_2_1_23_1","volume-title":"AST '07","author":"Javed Abu Zafer","unstructured":"Abu Zafer Javed, Paul A. Strooper, and Geoffrey N. Watson. 2007. Automated generation of test cases using model-driven architecture. In AST '07. IEEE, 3--3."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0088-2"},{"key":"e_1_3_2_1_25_1","volume-title":"Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337","author":"Kurita Keita","year":"2019","unstructured":"Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337 (2019)."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Qinghua Lu Liming Zhu Xiwei Xu Jon Whittle David Douglas and Conrad Sanderson. 2022. Software engineering for responsible AI: An empirical study and operationalised patterns. In ICSE-SEIP. ACM 241--242.","DOI":"10.1145\/3510457.3513063"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-021-09563-w"},{"key":"e_1_3_2_1_28_1","volume-title":"Operationalising AI ethics: Barriers, enablers and next steps","author":"Morley Jessica","year":"2021","unstructured":"Jessica Morley, Libby Kinsey, Anat Elhalal, Francesca Garcia, Marta Ziosi, and Luciano Floridi. 2021. Operationalising AI ethics: Barriers, enablers and next steps. AI & Society (2021), 1--13."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2009.30"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.416"},{"key":"e_1_3_2_1_31_1","volume-title":"A semantic framework to support AI system accountability and audit","author":"Naja Iman","unstructured":"Iman Naja, Milan Markovic, Peter Edwards, and Caitlin Cottrill. 2021. A semantic framework to support AI system accountability and audit. In ESWC. Springer, 160--176."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpa.2024.100619"},{"key":"e_1_3_2_1_33_1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 1, Article 140 (jan 2020), 67 pages.","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Emily Sheng Kai-Wei Chang Premkumar Natarajan and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In EMNLP-IJCNLP. ACL 3407--3412.","DOI":"10.18653\/v1\/D19-1339"},{"key":"e_1_3_2_1_35_1","volume-title":"Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri\u00e0 Garriga-Alonso, et al.","author":"Srivastava Aarohi","year":"2022","unstructured":"Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri\u00e0 Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022)."},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2871196"},{"key":"e_1_3_2_1_37_1","volume-title":"Retrieved","author":"House The White","year":"2023","unstructured":"The White House 2023. Executive order on the safe, secure, and trustworthy development and use of artificial intelligence. Retrieved February 29, 2024 from https:\/\/www.whitehouse.gov\/briefing-room\/presidential-actions\/2023\/10\/30\/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence"},{"key":"e_1_3_2_1_38_1","volume-title":"Retrieved","author":"UNESCO.","year":"2021","unstructured":"UNESCO. 2021. Recommendation on the ethics of artificial intelligence. Retrieved February 29, 2024 from https:\/\/unesdoc.unesco.org\/ark:\/48223\/pf0000380455"},{"key":"e_1_3_2_1_39_1","volume-title":"BiasAsker: Measuring the bias in conversational AI system. arXiv preprint arXiv:2305.12434","author":"Wan Yuxuan","year":"2023","unstructured":"Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, and Michael Lyu. 2023. BiasAsker: Measuring the bias in conversational AI system. arXiv preprint arXiv:2305.12434 (2023)."},{"key":"e_1_3_2_1_40_1","unstructured":"Laura Weidinger John Mellor Maribeth Rauh et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021)."},{"key":"e_1_3_2_1_41_1","volume-title":"Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876","author":"Zhao Jieyu","year":"2018","unstructured":"Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876 (2018)."},{"key":"e_1_3_2_1_42_1","volume-title":"Advances in Neural Information Processing Systems","volume":"36","author":"Zheng Lianmin","year":"2023","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 46595--46623."},{"key":"e_1_3_2_1_43_1","volume-title":"Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 4: Tutorial Abstracts). 9--16","author":"Zhiheng Xi","year":"2023","unstructured":"Xi Zhiheng, Zheng Rui, and Gui Tao. 2023. Safety and ethical concerns of large language models. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 4: Tutorial Abstracts). 9--16."}],"event":{"name":"MODELS '24: ACM\/IEEE 27th International Conference on Model Driven Engineering Languages and Systems","location":"Linz Austria","acronym":"MODELS '24","sponsor":["Johannes Kepler University, Linz, Austria","SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS"]},"container-title":["Proceedings of the ACM\/IEEE 27th International Conference on Model Driven Engineering Languages and Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640310.3674093","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3640310.3674093","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T23:51:58Z","timestamp":1755906718000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640310.3674093"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,22]]},"references-count":43,"alternative-id":["10.1145\/3640310.3674093","10.1145\/3640310"],"URL":"https:\/\/doi.org\/10.1145\/3640310.3674093","relation":{},"subject":[],"published":{"date-parts":[[2024,9,22]]},"assertion":[{"value":"2024-09-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}