RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

One-line takeaway

RubatoCodec enables efficient, high-fidelity speech representation with adaptive event framing and streaming decoding, improving reconstruction and intelligibility.

RubatoCodec通过事件级预算和自适应分帧机制，在流式解码中实现语音的高效、高质量表示，显著提升重建保真度与可懂度，且支持生成器自主学习事件速率。

Key points

RubatoCodec introduces a budgeted adaptive speech codec with event-level control and streaming decoding. It dynamically partitions speech into variable-duration events using feature and spectral changes, without learned boundaries. Each event is encoded with discrete tokens and explicit duration. Experiments show superior reconstruction quality and intelligibility over fixed-rate methods at low rates, especially on English and Japanese benchmarks. The generated speech maintains high fidelity and aligns well with text in TTS tasks, demonstrating effective integration with streaming generation.

• RubatoCodec adapts event duration based on spectral and feature changes without learned boundaries.
• It enables streaming decoding by processing one event at a time with discrete tokens and explicit durations.
• Outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates.
• Speech generator learns utterance-level event budget without explicit rate constraints.

Written only from the official arXiv abstract by an open-source model, then checked automatically (every number and name must appear in the abstract). The paper is authoritative.