Crawl4AI

Crawl4AI is an async Python crawler (Playwright-based, also usable via a CLI or a Docker/FastAPI server) built specifically for feeding LLMs and RAG pipelines rather than general-purpose scraping. Its markdown generator applies heuristic content filtering (a pruning filter and a BM25 filter keyed to a query) to strip nav/boilerplate before the page ever reaches an LLM, and it ships three extraction paths — CSS/XPath schemas, LLM-based extraction against a Pydantic schema, and a table-specific LLM extractor with automatic chunking for large tables. Version 0.7.0 added 'adaptive crawling', which stops crawling a site once an information-sufficiency threshold is met rather than exhausting a fixed page budget, and 0.8.0 added crash-recoverable deep crawls (BFS/DFS/best-first with resumable state). It needs no API key or account to use at all — everything runs on infrastructure you control.

79,195 GitHub stars and 8,206 forks as of 2026-08-23 (github.com/unclecode/crawl4ai). The PyPI package has been downloaded 18.5 million times all-time and 1.64 million times in the trailing 30 days (pepy.tech/projects/crawl4ai, checked today); the Docker image has 3.21 million pulls (Docker Hub API). Development is active day-to-day, not just at release time: the repo shows commits pushed as recently as 2026-08-20 and pull requests merged as recently as 2026-08-22, against a steady release cadence (v0.9.0 on 2026-06-18, v0.9.1 on 2026-07-08, v0.9.2 on 2026-07-15). This is a genuinely live, high-traffic project, not an abandoned star-farm.

If you self-host the Docker deployment (rather than the plain pip library), know that it has twice shipped a critical, unauthenticated remote-code-execution flaw with a maximum CVSS score of 10.0. CVE-2026-26216 (disclosed Jan 2026, fixed in v0.8.0) let an unauthenticated caller smuggle Python code into the /crawl endpoint's hooks parameter and run arbitrary system commands via exec(). CVE-2026-57572 (disclosed mid-2026, fixed in v0.9.0) let an unauthenticated caller inject Chromium launch switches through browser_config.extra_args to spawn an attacker-controlled process — a partial denylist fix in v0.8.9 didn't fully close it. Both are documented candidly in the project's own SECURITY.md, which also lists a third fixed deserialization RCE (v0.8.1) and commits to a 48-hour acknowledgment and severity-tiered fix timeline. v0.9.0 (2026-06-18) changed the default posture: the Docker API now requires authentication and binds to loopback unless a token is explicitly configured, rather than shipping open by default. (Sources: github.com/advisories/GHSA-5882-5rx9-xgxp, ionix.io/threat-center/cve-2026-57572, github.com/unclecode/crawl4ai/blob/main/SECURITY.md)

The library and Docker image are free under Apache 2.0 — no seat fees, no usage caps, no account required. The project's only current monetization is GitHub Sponsors, with published tiers from $5/month (support the project) to $2,000/month (a 'Data Infrastructure Partner' tier with dedicated support) — these buy sponsor recognition and, at higher tiers, direct maintainer access, not product features (github.com/sponsors/unclecode). A hosted 'Crawl4AI Cloud API' is announced as a closed beta with no public pricing; access is by application through a Google Form, and the vendor's own framing — 'drastically more cost-effective than any of the existing solutions' — is an unverified marketing claim with no numbers behind it yet.

Crawl4AI is led by Hossein Tohidi ('Unclecode' on GitHub/X), who is also the founder of Kidocode, an ed-tech company that appears in Crawl4AI's own sponsor list as a Gold-tier sponsor (github.com/unclecode/crawl4ai README). We found no evidence of institutional (VC) funding behind Crawl4AI itself, and no public information on team size beyond the maintainer and a small number of named contributors thanked in SECURITY.md. This is, as far as we can establish, a single-founder open-source project rather than a funded company — which is consistent with its extremely active commit history but also means there is no established institutional continuity plan if the maintainer stops.

The README states that using Crawl4AI 'requires' one of two attribution methods — a badge or a text credit — in your own documentation or product. The Apache 2.0 license itself (github.com/unclecode/crawl4ai/blob/main/LICENSE) only obligates preserving existing copyright and license notices when you redistribute the licensed source; it does not require a public 'powered by' badge on a downstream product built with the library. Teams with legal review processes should treat this as the maintainer's request, not a license term, when deciding whether to comply.

We could not find neutral, non-vendor-authored comparisons of Crawl4AI. Most comparison content found in search (Apify's blog, Bright Data's blog, Firecrawl's own 'alternatives' page, spider.cloud's benchmark post) is published by a competing scraping vendor and should be read as competitor marketing, not independent testing. We found no rating on Trustpilot, G2, or a comparable independent review site.