Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On

One-line takeaway

Safety benchmarks measure training policy definitions, not model capability, and are highly sensitive to label definitions.

安全基准实际上衡量的是训练时所采用的‘危险’定义，而非模型的真实能力，且其分数受标签定义显著影响，存在严重污染和混淆。

Key points

Safety benchmarks do not evaluate model capability but reflect the training policy's definition of 'unsafe'. The study shows that benchmark scores vary dramatically based on label definitions, with significant contamination from training data. Performance differences across models are driven by policy labels rather than detection ability, and validation aligns more with annotator policy than actual safety detection.

• Safety benchmarks reflect training policy definitions, not model capability.
• Scores vary 23x across models due to label definition differences.
• Minor parameter changes shift benchmark scores dramatically.
• Validation accuracy correlates with annotator policy, not detection ability.

Written only from the official arXiv abstract by an open-source model, then checked automatically (every number and name must appear in the abstract). The paper is authoritative.