{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,24]],"date-time":"2026-08-24T09:34:23Z","timestamp":1787564063864,"version":"build-2736575974"},"reference-count":0,"publisher":"Privacy Enhancing Technologies Symposium Advisory Board","issue":"3","license":[{"start":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T00:00:00Z","timestamp":1719792000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["PoPETs"],"abstract":"<jats:p>Website privacy policies are often lengthy and intricate. Privacy assistants assist in simplifying policies and making them more accessible and user-friendly. The emergence of generative AI (genAI) offers new opportunities to build privacy assistants that can answer users\u2019 questions about privacy policies. However, genAI\u2019s reliability is a concern due to its potential for producing inaccurate information. This study introduces GenAIPABench, a benchmark for evaluating Generative AI-based Privacy Assistants (GenAIPAs). GenAIPABench includes: 1) A set of curated questions about privacy policies along with annotated answers for various organizations and regulations; 2) Metrics to assess the accuracy, relevance, and consistency of responses; and 3) A tool for generating prompts to introduce privacy policies and paraphrased variants of the curated questions. We evaluated 3 leading genAI systems\u2014ChatGPT-4, Bard, and Bing AI\u2014using GenAIPABench to gauge their effectiveness as GenAIPAs. Our results demonstrate significant promise in genAI capabilities in the privacy domain while also highlighting challenges in managing complex queries, ensuring consistency, and verifying source accuracy.<\/jats:p>","DOI":"10.56553\/popets-2024-0081","type":"journal-article","created":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T14:57:15Z","timestamp":1719327435000},"page":"336-352","source":"Crossref","is-referenced-by-count":9,"title":["GenAIPABench: A Benchmark for Generative AI-based Privacy Assistants"],"prefix":"10.56553","volume":"2024","author":[{"given":"Aamir","family":"Hamid","sequence":"first","affiliation":[{"name":"Univ. of Maryland Baltimore County"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hemanth Reddy","family":"Samidi","sequence":"additional","affiliation":[{"name":"Univ. of Maryland Baltimore County"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Primal","family":"Pappachan","sequence":"additional","affiliation":[{"name":"Portland State University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tim","family":"Finin","sequence":"additional","affiliation":[{"name":"Univ. of Maryland Baltimore County"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roberto","family":"Yus","sequence":"additional","affiliation":[{"name":"Univ. of Maryland Baltimore County"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"35752","published-online":{"date-parts":[[2024,7]]},"container-title":["Proceedings on Privacy Enhancing Technologies"],"original-title":[],"deposited":{"date-parts":[[2024,6,25]],"date-time":"2024-06-25T15:06:43Z","timestamp":1719328003000},"score":1,"resource":{"primary":{"URL":"https:\/\/petsymposium.org\/popets\/2024\/popets-2024-0081.php"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7]]},"references-count":0,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["10.56553\/popets-2024-0081"],"URL":"https:\/\/doi.org\/10.56553\/popets-2024-0081","relation":{},"ISSN":["2299-0984"],"issn-type":[{"value":"2299-0984","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7]]}}}