// Train & fine-tune
tokenizersv0.23.3
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Open source Maintained For developers
IN ONE PARAGRAPH
tokenizers is 💥 Fast State-of-the-Art Tokenizers optimized for Research and Production.
Good for: Developers and teams who want to train & fine-tune.
01How to start
- 1Read the official quick-start (link above).
- 2Install from the official package registry:
pip install tokenizersnpm install tokenizers - 3Try it on a small example before wiring it into a project.
02Alternatives & comparison
Compare side by side → LlamaFactory Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024) ★ 76k transformers 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training. ★ 168k datasets 🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools ★ 22k gpt-neox An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries ★ 7.5k argilla Argilla is a collaboration tool for AI engineers and domain experts to build high-quality datasets ★ 5.1k