{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T04:32:17Z","timestamp":1773808337660,"version":"3.50.1"},"reference-count":0,"publisher":"Association for the Advancement of Artificial Intelligence (AAAI)","issue":"42","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["AAAI"],"abstract":"<jats:p>Accurate jailbreak evaluation is critical for LLM red team testing and jailbreak research. Mainstream methods rely on binary classification (string matching, toxic text classifiers, and LLM-based methods), outputting only \"yes\/no\" labels without quantifying harm severity. Emerged multi-dimensional frameworks (e.g., Security Violation, Relative Truthfulness and Informativeness) use unified evaluation standards across scenarios, leading to scenario-specific mismatches (e.g., \"Relative Truthfulness\" is irrelevant to \"hate speech\"), undermining evaluation accuracy. To address these, we propose SceneJailEval, with key contributions:\n(1) A pioneering scenario-adaptive multi-dimensional framework for jailbreak evaluation, overcoming the critical \"one-size-fits-all\" limitation of existing multi-dimensional methods, and boasting robust extensibility to seamlessly adapt to customized or emerging scenarios.  \n(2) A novel 14-scenario dataset featuring rich jailbreak variants and regional cases, addressing the long-standing gap in high-quality, comprehensive benchmarks for scenario-adaptive evaluation.  \n(3) SceneJailEval delivers state-of-the-art performance with an F1 score of 0.917 on our full-scenario dataset (+6% over SOTA) and 0.995 on JBB (+3% over SOTA), breaking through the accuracy bottleneck of existing evaluation methods in heterogeneous scenarios and solidifying its superiority.<\/jats:p>","DOI":"10.1609\/aaai.v40i42.40866","type":"journal-article","created":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T03:37:36Z","timestamp":1773805056000},"page":"35553-35561","source":"Crossref","is-referenced-by-count":0,"title":["SceneJailEval: A Scenario-Adaptive Multi-Dimensional Framework for Jailbreak Evaluation"],"prefix":"10.1609","volume":"40","author":[{"given":"Lai","family":"Jiang","sequence":"first","affiliation":[]},{"given":"Yuekang","family":"Li","sequence":"additional","affiliation":[]},{"given":"Xiaohan","family":"Zhang","sequence":"additional","affiliation":[]},{"given":"Youtao","family":"Ding","sequence":"additional","affiliation":[]},{"given":"Li","family":"Pan","sequence":"additional","affiliation":[]}],"member":"9382","published-online":{"date-parts":[[2026,3,14]]},"container-title":["Proceedings of the AAAI Conference on Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/download\/40866\/44827","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/download\/40866\/44827","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T03:37:37Z","timestamp":1773805057000},"score":1,"resource":{"primary":{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/view\/40866"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,14]]},"references-count":0,"journal-issue":{"issue":"42","published-online":{"date-parts":[[2026,3,17]]}},"URL":"https:\/\/doi.org\/10.1609\/aaai.v40i42.40866","relation":{},"ISSN":["2374-3468","2159-5399"],"issn-type":[{"value":"2374-3468","type":"electronic"},{"value":"2159-5399","type":"print"}],"subject":[],"published":{"date-parts":[[2026,3,14]]}}}