In jailbreak-robustness research, safety is typically evaluated using an LLM-as-judge approach. However, such evaluations are sensitive to the benchmark's grading procedure and only capture observed behavior on specific attacks, failing to reveal the hidden fragility of the underlying safety mechanisms.
This paper proposes JADR (Jacobian Assessment of Danger Recognition), a protocol that measures a model's internal representation through Jacobian space (J-space, a recently proposed workspace of verbalizable concepts) before generating the first response token. For each prompt and layer, we record the top-k J-space tokens, which are grouped into six behavioral scenario axes and compared between a danger sample based on StrongREJECT and a safe control drawn from XSTest and OKTest.
The method does not rely on an external judge model: the computation runs entirely locally on the activations of the model under evaluation, allowing us to compare different models and modifications of a single model—particularly quantization and fine-tuning—on the same terms. The final comparison is based on the proposed SafetyAUC metric, complemented with bootstrap confidence intervals.
The protocol is applied to six models (Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-Uncensored-4B, Qwen3-SafeRL-4B, Gemma 2 9B) across three weight-representation regimes—BF16, INT8, and INT4—and checked against an independent behavioral evaluation with the StrongREJECT grader. The metric statistically separates models with strong versus weak internal safety mechanisms and captures substantively different effects across quantization regimes.
Blogger's Review: The proposed JADR protocol offers a novel approach to assessing safety by quantifying a model's internal representation, which is particularly significant for comparing models across different quantization levels. This method not only enhances our understanding of model fragility but also provides new avenues for future safety evaluations. Overall, this research demonstrates how to effectively assess and compare the safety of different models without relying on external models.