// ◉ Multimodal & Vision
VALL-E (Neural Codec Language Model TTS)
Speech as discrete codec tokens generated by a language model – clones a voice from seconds of audio.
VALL-E 神经编解码语言模型 TTS aka Neural Codec LM / 语音 token 2023 Intermediate extended
IN PLAIN WORDS
A neural audio codec turns speech into discrete tokens, and TTS becomes language modelling: continue the tokens given text plus a short audio prompt. The paper clones a speaker from a 3-second recording, keeping emotion and acoustic environment. The codec-tokens-plus-LM recipe became the mainstream for speech generation.
Used in: Zero-shot voice cloning and TTS; codec-token + LM speech generation
TIMELINE · prerequisites → this → what builds on it
01Original paper
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chengyi Wang, Sanyuan Chen, Yu Wu, et al. · 2023
arXiv ↗ ✓ Verified · arXiv API
02Where it sits
Learn the left column first; the right column builds on this one.
Then you can learn
- Nothing in the atlas builds on it yet
Related:Whisper
Left: prerequisites · right: builds on this · dashed: related. Full map →
03Learning resources
Official docs, explainers, courses and reference code. All links checked.
04Where the course teaches it
For AI / data
- Verification: verified, checked 2026-10-22.
- JSON: /api/learn/algorithms.json (id:
valle)