Theoretical Limits of Language Model Alignment

A covariance-based estimator from base model samples enables practical prediction. The study proves that reward hacking increases with reward error and decreasing KL penalty, and that reward ensembling mitigates it. Empirical results show best-of-NNN approaches theoretical limits in safety and summarization tasks, while PPO and GRPO fall significantly short.

Summarised from the original article by an open-source model, then checked automatically: every number and English name must appear in the source. The original article is authoritative.