arXiv 2610.22428 · cs.CL · 2026-10-15
Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models
Sai Kiran Bathula
Affiliation unverified
In latest · #392 Added on 2026-10-25
One-line takeaway
In small LMs the embedding dominates (168M of Gemma 3 270M's 268M params); the authors replace it with a frozen 16 MB table of 512-bit codes and retrain to ask what a learned embedding actually buys.
小模型里词表最占参数，如 Gemma 3 270M 的嵌入占 168M。作者把它换成一张冻结的 16 MB 512 比特编码表，同样数据从头训练对比，追问可学习嵌入到底值不值。
Read on arXiv PDF Hugging Face discussion Project page
Topics & disciplines
More on LLMs
- Jeva: System One Models from First Principles ▲371
- Post-Training Leaves Behavioral Shadows on Unrelated Decisions ▲273
- OmniEdu: Open Foundation Models for Learning and Teaching ▲240
- PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking ▲163
- MemT-1.0: Towards Memory-Native Evolution ▲143
More on Efficient training & inference
- MJev-Spatial: System One Spatial Intelligence via Native Attention Matching 2026-10-15 · ▲556
- The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation 2026-09-29 · ▲511
- VideoJev: System One Model for Video Understanding and Interaction 2026-10-13 · ▲375
- LEV: Thinking Fast and Deep with Looped Decision Models 2026-10-14 · ▲337
- Scaling Properties of Same-Family On-Policy Distillation 2026-09-26 · ▲327
About this data
Metadata from the arXiv API (verified); upvotes from Hugging Face Daily Papers; citations from Semantic Scholar. Last refreshed 2026-10-26.
The takeaway is written by AIMap from the abstract; the abstract itself is not reproduced — read the original.