// I want to solve · Developers / Enterprise IT / Researchers
📄 Build RAG and process documents
Turn PDFs and web pages into clean text, choose vector stores and rerankers, debug poor retrieval, and give agents long-term memory.
4 steps4 alternatives4 stacks · 3 problemschecked 2026-10-22
How to do it
- 01Turn documents into clean text: MarkItDown for simple files, Docling / MinerU for complex PDFs
- 02Chunk by headings or semantics and keep source and page numbers
- 03Combine vector and keyword search, then add a reranker
- 04Build 50 gold Q&A pairs and evaluate with Ragas
What it costs
Free start
Parsing, vector stores and eval tools all have open-source versions
Pay as you go
Embeddings and answers bill per token
Team / enterprise
Hosted vector DBs bill by storage and queries
Prices change; the vendor’s official page wins. Subscriptions compared · API prices
Alternatives: which to pick
✓ Good fit
- Document Q&A, search and summarisation
✗ Not a good fit
- Very small corpora: just use long context
Ready-made stacks
All stacks Local private knowledge baseBeginner Documents never leave your machine: a local model answers and a local vector store retrieves. Good for individuals and small teams turning PDFs and notes into a Q&A knowledge base. 4 tools Enterprise RAG knowledge baseAdvanced Team-scale document Q&A: deep document parsing, a scalable vector database, one model gateway, plus tracing and quality evaluation. 6 tools Personal knowledge base Capture notes, web pages and PDFs into a searchable second brain that AI can answer from. 3 tools Legal document workflow Summarize, compare and search contracts/case files (human review required). 4 tools
Software that helps
All software📄 Document parsing & web scraping🧲 Retrieval, vector DBs & embeddings🧠 Memory & context
firecrawl Document parsing & web scrapingv2.11.502 ★ 194k Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥 Open source APISDK MarkItDown Document parsing & web scrapingv0.1.9 ★ 191k Microsoft tool converting office documents to Markdown for LLMs. Open source CLIPythonSDK ragflow Document parsing & web scrapingnightly ★ 93k RAGFlow is a leading open-source Retrieval-Augmented Generation (RAG) engine that fuses cutting-edge RAG with Agent capabilities to create a superior context layer for LLMs Open source API PaddleOCR Document parsing & web scrapingv3.7.0 ★ 91k Turn any PDF or image document into structured data for your AI. A powerful, lightweight OCR toolkit that bridges the gap between images/PDFs and LLMs. Supports 100+ languages. Open source SDKCLI crawl4ai Document parsing & web scrapingv0.9.4 ★ 86k Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Open source SDKAPI MinerU Document parsing & web scrapingmineru-4.0.11-released ★ 82k Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows. Open source CLIAPI docling Document parsing & web scrapingv2.137.0 ★ 69k Get your documents ready for gen AI Open source CLISDKMCP pandoc Document parsing & web scraping3.12.1 ★ 47k Universal markup converter Open source CLI marker Document parsing & web scrapingv2.0.0 ★ 41k Convert PDF to markdown + JSON quickly with high accuracy Open source CLI LightRAG Document parsing & web scrapingv1.5.7 ★ 40k [EMNLP2025] LightRAG: Simple and Fast Retrieval-Augmented Generation Open source SDKAPI searxng Document parsing & web scrapingv1.0.0 ★ 39k SearXNG is a free internet metasearch engine which aggregates results from various search services and databases. Users are neither tracked nor profiled. Open source API graphrag Document parsing & web scrapingv3.3.0 ★ 36k A modular graph-based Retrieval-Augmented Generation (RAG) system Open source CLISDK postgres Document parsing & web scrapingREL_18_6 ★ 22k Mirror of the official PostgreSQL GIT repository. Note that this is just a *mirror* - we don't work with pull requests on github. To contribute, please see https://wiki.postgresql.org/wiki/Submitting_a_Patch Open source SQL olmocr Document parsing & web scrapingv0.4.27 ★ 20k Toolkit for linearizing PDFs for LLM datasets/training Open source CLI unstructured Document parsing & web scraping0.27.33 ★ 16k Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding. Open source SDKAPI FlagEmbedding Document parsing & web scrapingv1.4.2 ★ 12k Retrieval and Retrieval-augmented LLMs Open source SDK reader Document parsing & web scraping ★ 12k Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/ Open source API PyMuPDF Document parsing & web scraping1.28.2 ★ 11k PyMuPDF is a high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents. Open source SDK trafilatura Document parsing & web scrapingv2.3.1 ★ 7.0k Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML Open source CLISDK Poppler Document parsing & web scraping Open spec Pinecone Document parsing & web scraping Closed / commercial Exa Document parsing & web scraping Closed / commercial Tavily Document parsing & web scraping Closed / commercial Brave Search API Document parsing & web scraping Closed / commercial
Problems & solutions
All problems- ChoiceWhich PDF/document parser to use for RAGRAG quality depends heavily on parsing. Popular open-source parsers have different strengths; compare them on a sample of your own documents. We have not run a unified test yet.
- How-toRAG retrieves the wrong passages: how to debugFirst find which stage fails: parsing, chunking, retrieval or generation. Measure with a fixed question set, then improve one stage at a time.
- ChoiceGiving agents memory across sessions: memory layer or rules file?For coding agents a project rules file is often enough; apps for end users that need long-term memory may need a dedicated memory layer.
Needs in this scenario
-
DEV-03RAG parsing, chunking and retrieval are inaccurate
Developers · People search for: best pdf parser for rag; docling vs markitdown; rag retrieval accuracy
-
DEV-10Context management: lossy compaction and no memory across sessions
Developers · People search for: agent memory layer; claude code context compaction; mem0 vs letta
Learn
All tutorials- LlamaIndex 文档：Understanding
LlamaIndex · RAG · English
- RAG_Techniques
Nir Diamant · RAG · English
- Anthropic: Contextual Retrieval
Anthropic · RAG · English
- OpenAI Cookbook
OpenAI · RAG · English
- openai/openai-cookbook
OpenAI · RAG · English
- Claude Cookbooks
Anthropic · RAG · English
- Gemini API Cookbook
Google · RAG · English
- LangChain RAG 教程
LangChain · RAG · English
- Pinecone Learning Center
Pinecone · RAG · English
- Weaviate Academy
Weaviate · RAG · English