{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T06:50:59Z","timestamp":1782283859029,"version":"3.54.5"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T00:00:00Z","timestamp":1777593600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T00:00:00Z","timestamp":1778457600000},"content-version":"vor","delay-in-days":10,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100013003","name":"Universit\u00e0 degli Studi di Cagliari","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100013003","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2026,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Modern data architectures increasingly rely on decentralized data products described through semi-structured artifacts, such as YAML descriptors, to support scalability, interoperability, and governance. Ensuring that these descriptors comply with organizational and regulatory policies remains a challenging and labor-intensive task, largely due to the natural-language nature of policies and the heterogeneity of descriptor schemas. In this paper, we investigate a model-based approach to evaluating natural-language policies over semi-structured data product descriptors expressed in YAML, leveraging Large Language Models (LLMs) as reasoning components. We formalize the policy-descriptor compliance task as a binary decision problem and propose an LLM-based microservice architecture that combines structured prompt engineering with deterministic output constraints to support automated policy verification. To evaluate the approach, we construct an augmentation-based benchmark comprising 4,000 policy-descriptor pairs derived from 40 manually annotated seed combinations. The benchmark is designed to assess model robustness to policy paraphrases and semantically equivalent descriptor rewrites. We evaluate four open-source instruction-tuned LLMs under identical conditions and report results in terms of accuracy, precision, recall, F1-score, and execution time. Experimental results show that LLMs can effectively support policy verification in semi-structured environments, achieving up to 72% accuracy. However, we observe a consistent conservative bias toward non-compliance, as well as systematic failure modes involving conjunctions, bidirectional constraints, and conditionally inapplicable rules. Our analysis highlights both the potential and the current limitations of LLM-driven governance, and suggests that expressing policies as explicit procedural checks substantially improves validation reliability.<\/jats:p>","DOI":"10.1007\/s10994-026-07038-6","type":"journal-article","created":{"date-parts":[[2026,5,11]],"date-time":"2026-05-11T08:17:39Z","timestamp":1778487459000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["LLM-Driven Compliance Checking for Natural-Language Policies over Data Product Descriptors"],"prefix":"10.1007","volume":"115","author":[{"given":"Francesco","family":"Simbola","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diego","family":"Reforgiato Recupero","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniele","family":"Riboni","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martina","family":"Salis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,11]]},"reference":[{"key":"7038_CR1","doi-asserted-by":"crossref","unstructured":"Abiteboul, S. (1997). Querying semi-structured data. In Proceedings of the 6th international conference on database theory. ICDT \u201997, pp. 1\u201318. Springer.","DOI":"10.1007\/3-540-62222-5_33"},{"key":"7038_CR2","doi-asserted-by":"publisher","DOI":"10.1007\/s11301-025-00545-1","author":"D Acev","year":"2025","unstructured":"Acev, D., Biyani, S., Rieder, F., Aldenhoff, T. T., Blazevic, M., Riehle, D. M., & Wimmer, M. A. (2025). Systematic analysis of data governance frameworks and their relevance to data trusts. Management Review Quarterly. https:\/\/doi.org\/10.1007\/s11301-025-00545-1","journal-title":"Management Review Quarterly"},{"key":"7038_CR3","unstructured":"AI, M. (2024a). Mistral-7B-Instruct-v0.3-GGUF. https:\/\/huggingface.co\/lmstudio-community\/Mistral-7B-Instruct-v0.3-GGUF"},{"key":"7038_CR4","unstructured":"AI, M. (2024b). Mistral-nemo-instruct-2407. https:\/\/lmstudio.ai\/models\/mistralai\/mistral-nemo-instruct-2407"},{"key":"7038_CR5","doi-asserted-by":"publisher","first-page":"185291","DOI":"10.1109\/ACCESS.2024.3512434","volume":"12","author":"D Al-Turki","year":"2024","unstructured":"Al-Turki, D., Hettiarachchi, H., Medhat Gaber, M., Abdelsamea, M. M., Basurra, S., Iranmanesh, S., Saadany, H., & Vakaj, E. (2024). Human-in-the-loop learning with llms for efficient rase tagging in building compliance regulations. IEEE Access, 12, 185291\u2013185306. https:\/\/doi.org\/10.1109\/ACCESS.2024.3512434","journal-title":"IEEE Access"},{"key":"7038_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.aei.2023.102288","volume":"59","author":"T Beach","year":"2024","unstructured":"Beach, T., Yeung, J., Nisbet, N., & Rezgui, Y. (2024). Digital approaches to construction compliance checking: Validating the suitability of an ecosystem approach to compliance checking. Advanced Engineering Informatics, 59, Article 102288. https:\/\/doi.org\/10.1016\/j.aei.2023.102288","journal-title":"Advanced Engineering Informatics"},{"key":"7038_CR7","doi-asserted-by":"publisher","first-page":"12859","DOI":"10.1109\/ACCESS.2025.3528941","volume":"13","author":"YA Bena","year":"2025","unstructured":"Bena, Y. A., Ibrahim, R., Mahmood, J., Al-Dhaqm, A., Alshammari, A., Nasser, M., Nura Yusuf, M., & Ayemowa, M. O. (2025). Big data governance challenges arising from data generated by intelligent systems technologies: A systematic literature review. IEEE Access, 13, 12859\u201312888. https:\/\/doi.org\/10.1109\/ACCESS.2025.3528941","journal-title":"IEEE Access"},{"issue":"4","key":"7038_CR8","doi-asserted-by":"publisher","DOI":"10.1016\/j.jik.2024.100598","volume":"9","author":"BMV Bernardo","year":"2024","unstructured":"Bernardo, B. M. V., Mamede, H. S., Barroso, J. M. P., & dos Santos, V. M. P. D. (2024). Data governance & quality management\u2013innovation and breakthroughs across different fields. Journal of Innovation & Knowledge, 9(4), Article 100598. https:\/\/doi.org\/10.1016\/j.jik.2024.100598","journal-title":"Journal of Innovation & Knowledge"},{"key":"7038_CR9","doi-asserted-by":"crossref","unstructured":"Bhathena, H., Joshi, A., & Singh, P. (2023). An efficient method for natural language querying on structured data. In S. Sitaram, B. Beigman\u00a0Klebanov, & J. D. Williams (Eds.) Proceedings of the 61st annual meeting of the association for computational linguistics (Volume 5: Industry Track), pp. 322\u2013331. Association for Computational Linguistics. https:\/\/doi.org\/10.18653\/v1\/2023.acl-industry.31 . https:\/\/aclanthology.org\/2023.acl-industry.31\/","DOI":"10.18653\/v1\/2023.acl-industry.31"},{"key":"7038_CR10","doi-asserted-by":"publisher","unstructured":"Buthelezi, M. P., Van Der\u00a0Poll, J. A., & Ochola, E. O. (2016). Ambiguity as a barrier to information security policy compliance: A content analysis. In 2016 International conference on computational science and computational intelligence (CSCI), pp. 1360\u20131367. https:\/\/doi.org\/10.1109\/CSCI.2016.0254","DOI":"10.1109\/CSCI.2016.0254"},{"issue":"7","key":"7038_CR11","doi-asserted-by":"publisher","DOI":"10.3390\/buildings14071983","volume":"14","author":"N Chen","year":"2024","unstructured":"Chen, N., Lin, X., Jiang, H., & An, Y. (2024). Automated building information modeling compliance check through a large language model combined with deep learning and ontology. Buildings, 14(7), Article 1983. https:\/\/doi.org\/10.3390\/buildings14071983","journal-title":"Buildings"},{"key":"7038_CR12","volume-title":"Data mesh: Delivering data-driven value at scale","author":"Z Dehghani","year":"2022","unstructured":"Dehghani, Z., & Fowler, M. (2022). Data mesh: Delivering data-driven value at scale. O\u2019Reilly Media."},{"key":"7038_CR13","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpa.2025.100802","volume":"27","author":"T G\u00f3rski","year":"2026","unstructured":"G\u00f3rski, T. (2026). Software architecture description in original software publications. Software Impacts, 27, Article 100802. https:\/\/doi.org\/10.1016\/j.simpa.2025.100802","journal-title":"Software Impacts"},{"key":"7038_CR14","doi-asserted-by":"crossref","unstructured":"Gupta, V., Kandoi, P., Vora, M., Zhang, S., He, Y., Reinanda, R., & Srikumar, V. (2023). TempTabQA: Temporal question answering for semi-structured tables. In H. Bouamor, J. Pino, & K. Bali (Eds.) Proceedings of the 2023 conference on empirical methods in natural language processing, pp. 2431\u20132453. Association for Computational Linguistics. https:\/\/doi.org\/10.18653\/v1\/2023.emnlp-main.149 . https:\/\/aclanthology.org\/2023.emnlp-main.149\/","DOI":"10.18653\/v1\/2023.emnlp-main.149"},{"key":"7038_CR15","doi-asserted-by":"publisher","unstructured":"Hosseini, M. B., Heaps, J., Slavin, R., Niu, J., Breaux, T. (2021). Ambiguity and generality in natural language privacy policies. In 2021 IEEE 29th international requirements engineering conference (RE), pp. 70\u201381. https:\/\/doi.org\/10.1109\/RE51729.2021.00014","DOI":"10.1109\/RE51729.2021.00014"},{"key":"7038_CR16","unstructured":"Ibba, A., Reforgiato\u00a0Recupero, D., & Alonso, R. (2025). Enabling natural language access to bim models with ai and knowledge graphs."},{"issue":"6","key":"7038_CR17","doi-asserted-by":"publisher","first-page":"2053","DOI":"10.1007\/s10994-022-06259-9","volume":"112","author":"G Jaimovitch-L\u00f3pez","year":"2023","unstructured":"Jaimovitch-L\u00f3pez, G., Ferri, C., Hern\u00e1ndez-Orallo, J., Mart\u00ednez-Plumed, F., & Ram\u00edrez-Quintana, M. J. (2023). Can language models automate data wrangling? Machine Learning, 112(6), 2053\u20132082. https:\/\/doi.org\/10.1007\/s10994-022-06259-9","journal-title":"Machine Learning"},{"issue":"3","key":"7038_CR18","doi-asserted-by":"publisher","DOI":"10.1016\/j.giq.2020.101493","volume":"37","author":"M Janssen","year":"2020","unstructured":"Janssen, M., Brous, P., Estevez, E., Barbosa, L. S., & Janowski, T. (2020). Data governance: Organizing data for trustworthy artificial intelligence. Government Information Quarterly, 37(3), Article 101493. https:\/\/doi.org\/10.1016\/j.giq.2020.101493","journal-title":"Government Information Quarterly"},{"key":"7038_CR19","doi-asserted-by":"crossref","unstructured":"Jiang, H., & Bouabdallah, A. (2017). Towards a json-based fast policy evaluation framework. In H. Panetto, C. Debruyne, W. Gaaloul, M. Papazoglou, A. Paschke, C. A. Ardagna, & R. Meersman (Eds.), On the move to meaningful internet systems. OTM 2017 Conferences (pp. 22\u201330). Springer.","DOI":"10.1007\/978-3-319-69459-7_2"},{"key":"7038_CR20","unstructured":"Khan, I. (2024). Creating code with codex, pp. 287\u2013301."},{"issue":"1","key":"7038_CR21","doi-asserted-by":"publisher","first-page":"148","DOI":"10.1145\/1629175.1629210","volume":"53","author":"V Khatri","year":"2010","unstructured":"Khatri, V., & Brown, C. V. (2010). Designing data governance. Communications of the ACM, 53(1), 148\u2013152. https:\/\/doi.org\/10.1145\/1629175.1629210","journal-title":"Communications of the ACM"},{"key":"7038_CR22","doi-asserted-by":"publisher","unstructured":"Kiyavitskaya, N., Zeni, N., Breaux, T. D., Ant\u00f3n, A. I., Cordy, J. R., Mich, L., & Mylopoulos, J. (2007). Extracting rights and obligations from regulations: toward a tool-supported process. In Proceedings of the 22nd IEEE\/ACM international conference on automated software engineering. ASE \u201907, pp. 429\u2013432. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/1321631.1321701 .","DOI":"10.1145\/1321631.1321701"},{"key":"7038_CR23","unstructured":"Liu, X., Li, H., & Zhu, X. (2023). A GPT-based Method of Automated Compliance Checking through Prompt Engineering. Technical report, Cardiff University ORCA repository. Accessed: 2025-12-07. https:\/\/orca.cardiff.ac.uk\/id\/eprint\/159694\/1\/A%20GPT-based%20method%20of%20Automatic%20Compliance%20Checking%20through%20prompt%20engineering_20230417001851.pdf"},{"key":"7038_CR24","unstructured":"OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., & Zoph, B. (2024). GPT-4 technical report. https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"7038_CR25","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2025.127330","volume":"278","author":"R Ren","year":"2025","unstructured":"Ren, R., Ma, J., & Zheng, Z. (2025). Large language model for interpreting research policy using adaptive two-stage retrieval augmented fine-tuning method. Expert Systems with Applications, 278, Article 127330. https:\/\/doi.org\/10.1016\/j.eswa.2025.127330","journal-title":"Expert Systems with Applications"},{"key":"7038_CR26","unstructured":"Research, I. (2024). granite-3.2-8b-instruct-GGUF. https:\/\/huggingface.co\/ibm-research\/granite-3.2-8b-instruct-GGUF"},{"key":"7038_CR27","doi-asserted-by":"crossref","unstructured":"Shen, X., Barlacchi, G., Del\u00a0Tredici, M., Cheng, W., & Gispert, A. (2022). semiPQA: A study on product question answering over semi-structured data. In S. Malmasi, O. Rokhlenko, N. Ueffing, I. Guy, E. Agichtein, & S. Kallumadi (Eds.) Proceedings of the Fifth Workshop on e-Commerce and NLP (ECNLP 5), pp. 111\u2013120. Association for Computational Linguistics, Dublin, Ireland. https:\/\/doi.org\/10.18653\/v1\/2022.ecnlp-1.14 . https:\/\/aclanthology.org\/2022.ecnlp-1.14\/","DOI":"10.18653\/v1\/2022.ecnlp-1.14"},{"key":"7038_CR28","doi-asserted-by":"crossref","unstructured":"Sondhi, P., & Zhai, C. (2014). Mining semi-structured online knowledge bases to answer natural language questions on community qa websites. In\u00a0Proceedings of the 23rd ACM international conference on conference on information and knowledge management","DOI":"10.1145\/2661829.2661968"},{"key":"7038_CR29","unstructured":"Sun, J., Luo, Z., & Li, Y. (2025). A compliance checking framework based on retrieval augmented generation. In O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, & S. Schockaert (Eds.) Proceedings of the 31st international conference on computational linguistics, pp. 2603\u20132615. Association for Computational Linguistics, Abu Dhabi, UAE. https:\/\/aclanthology.org\/2025.coling-main.178\/"},{"key":"7038_CR30","unstructured":"Team, D. (2024). deepseek-r1-0528-qwen3-8b. https:\/\/lmstudio.ai\/models\/deepseek\/deepseek-r1-0528-qwen3-8b"},{"key":"7038_CR31","doi-asserted-by":"publisher","unstructured":"Waltinger, U., Tecuci, D., Olteanu, M., Mocanu, V., & Sullivan, S. (2013). Usi answers: Natural language question answering over (semi-) structured industry data. In Proceedings of the AAAI conference on artificial intelligence 27, pp. 1471\u20131478. https:\/\/doi.org\/10.1609\/aaai.v27i2.18985","DOI":"10.1609\/aaai.v27i2.18985"},{"key":"7038_CR32","unstructured":"Wang, Y., Cai, C., Xiao, Z., & Lam, P. E. (2025). LLM access shield: Domain-specific LLM framework for privacy policy compliance. https:\/\/arxiv.org\/abs\/2505.17145"},{"key":"7038_CR33","doi-asserted-by":"publisher","unstructured":"Weber, I. (2024). Large language models are pattern matchers: Editing semi-structured and structured documents with chatgpt. https:\/\/doi.org\/10.18420\/AKWI2024-001","DOI":"10.18420\/AKWI2024-001"},{"key":"7038_CR34","unstructured":"Ying, H., & Sacks, R. (2024). From automatic to autonomous: A large language model-driven approach for generic building compliance checking. EasyChair Preprint 15364, ??? (EasyChair, 2024)"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-026-07038-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-026-07038-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-026-07038-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T06:35:37Z","timestamp":1782282937000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-026-07038-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5]]},"references-count":34,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,5]]}},"alternative-id":["7038"],"URL":"https:\/\/doi.org\/10.1007\/s10994-026-07038-6","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5]]},"assertion":[{"value":"7 January 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 March 2026","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 March 2026","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 May 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests that could have influenced the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to Participate"}},{"value":"All participants provided explicit consent for the anonymous use and publication of their data for scientific research purposes.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for Publication"}}],"article-number":"116"}}