arXiv 2609.36585 · cs.AI · 2026-09-29
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
Zehao Jin, Ruixuan Deng, Junran Wang
机构待核实
在最新榜 #264 本站收录于 2026-10-26
一句话要点
研究发现预训练 Transformer 只用了少量深度去追踪上下文引用，13 个基座模型只能可靠跟随 1.4–3.6 行；在一个早期层加秩 8 的 LoRA（其余权重冻结），Qwen3-8B 在 24 行引用链上的精确准确率从 15.5% 升到 99%。
Pretrained transformers use little depth to follow in-context references (13 base models reliably track only 1.4–3.6 lines); a rank-8 LoRA at one early layer, with all weights frozen, lifts Qwen3-8B from 15.5% to 99% exact accuracy on 24-line chains.
阅读 arXiv 原文 PDF Hugging Face 讨论 代码仓库 项目页
主题与学科
同主题：大语言模型
- Jeva: System One Models from First Principles ▲371
- Post-Training Leaves Behavioral Shadows on Unrelated Decisions ▲273
- OmniEdu: Open Foundation Models for Learning and Teaching ▲240
- PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking ▲163
- MemT-1.0: Towards Memory-Native Evolution ▲143
同主题：推理
- LoopVL: Recurrent Visual Intelligence 2026-09-29 · ▲417
- LEV: Thinking Fast and Deep with Looped Decision Models 2026-10-14 · ▲337
- Thinking Inertia: LLMs Keep Thinking When Told Not To 2026-10-08 · ▲241
- Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence 2026-09-28 · ▲224
- On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics 2026-09-28 · ▲205
数据说明
元数据来自 arXiv API（已核实）；点赞来自 Hugging Face 每日论文；引用来自 Semantic Scholar。最近刷新 2026-10-26。
要点由本站根据摘要自写，不转载摘要原文，请阅读原文。