Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

One-line takeaway

Pretrained transformers use little depth to follow in-context references (13 base models reliably track only 1.4–3.6 lines); a rank-8 LoRA at one early layer, with all weights frozen, lifts Qwen3-8B from 15.5% to 99% exact accuracy on 24-line chains.

研究发现预训练 Transformer 只用了少量深度去追踪上下文引用，13 个基座模型只能可靠跟随 1.4–3.6 行；在一个早期层加秩 8 的 LoRA（其余权重冻结），Qwen3-8B 在 24 行引用链上的精确准确率从 15.5% 升到 99%。