Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models

一句话要点

小模型里词表最占参数，如 Gemma 3 270M 的嵌入占 168M。作者把它换成一张冻结的 16 MB 512 比特编码表，同样数据从头训练对比，追问可学习嵌入到底值不值。

In small LMs the embedding dominates (168M of Gemma 3 270M's 268M params); the authors replace it with a frozen 16 MB table of 512-bit codes and retrain to ask what a learned embedding actually buys.