crawl4ai
Developed by unclecode
Crawl4AI is an open-source, LLM-friendly web crawler and scraper that converts web pages into clean, structured Markdown for RAG, agents, and data pipelines. It features LLM-driven structured extraction, dynamic crawling, stealth mode, session management, and a Dockerized API server, making large-scale web extraction highly efficient and cost-effective.
- LLM-Friendly Markdown
- AI-Driven Structured Extraction
- Dynamic Crawling & Stealth Mode
- Deep Crawl & Crash Recovery
- Dockerized API Deployment
webdesktop