// Read docs & web pages
crawl4aiv0.9.4
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.
Open source Maintained For agents
IN ONE PARAGRAPH
crawl4ai is open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.
Good for: Developers who want their agent (Claude Code, Codex, Cursor…) to read docs & web pages.
01How to start
- 1Read the official README for requirements and permissions.
- 2Install it from the official package registry:
pip install crawl4ai - 3Tell your agent the command exists (e.g. in AGENTS.md / CLAUDE.md) so it knows when to call it.
02Alternatives & comparison
Compare side by side → firecrawl Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥 ★ 194k Scrapling 🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev ★ 89k Scrapegraph-ai Python scraper based on AI ★ 32k ragflow RAGFlow is a leading open-source Retrieval-Augmented Generation (RAG) engine that fuses cutting-edge RAG with Agent capabilities to create a superior context layer for LLMs ★ 93k docling Get your documents ready for gen AI ★ 69k