Search, Crawl, or Extract: Which Web Data API Does Your Agent Actually Need?

Three different jobs wearing one label

"Web data API" gets used for three genuinely different products, and picking the wrong one is the most common way teams waste a quarter's budget on this stack. A search API takes a question and returns ranked, relevant sources — it's a research tool. A crawl API takes a URL (or a whole site) and returns clean text or structured data from it — it's an ingestion tool. An extract API takes a URL and a question about one page and returns just that field, in a way that keeps working after the site's HTML changes — it's a monitoring tool. Agents that need "the current price of this SKU" want extract. Agents that need "what are people saying about this topic" want search. Agents building a RAG index from a documentation site want crawl. Buying the wrong shape means paying crawl-API prices for a job a $50/month search plan would have done, or the reverse.

Pricing makes this worse before it makes it better: search APIs meter per call, crawl APIs meter per credit (where one credit is roughly one page, except when it isn't), and Apify meters compute units that have no fixed relationship to either. None of these numbers are comparable at face value. Below, they're normalized to the same usage level so they actually are.

Job one: find something — Tavily, Exa, or You.com

At 10,000 search calls a month — a reasonable volume for an agent doing ongoing research rather than one-off lookups — the three land within 60% of each other: You.com's Search API is $5 per 1,000 calls ($50 total), Exa charges $7 per 1,000 requests for up to 10 results ($70 total), and Tavily is $0.008 per credit with basic search costing 1 credit ($80 total on pure pay-as-you-go; the first 1,000 credits a month are free on every plan).

The prices converge; the architectures don't. Exa maintains a continuously-refreshed index across specialized verticals — code, financial filings, company data, legal records — and is built for natural-language queries a keyword search engine handles badly ("companies selling AI voice agents to dental practices"). Tavily's pitch is the single call: search, scrape, filter, and rank collapsed into one request built specifically for feeding an LLM's context window, plus an optional inline answer field for cross-agent handoffs. You.com bundles a third product, a Research API at $6.50 per 1,000 calls that runs a multi-step search-and-synthesis pipeline and reports ranking first on the DeepSearchQA benchmark — a claim from You.com, not one we've independently verified. If your agent needs breadth across obscure verticals, Exa's index is the differentiator; if it needs the fewest API calls per research task, Tavily's single-call design wins; if price per call is the only variable that matters, You.com is cheapest today.

Job two: turn pages into clean text — Firecrawl, Apify, or Crawl4AI

At 5,000 pages a month, Firecrawl's Hobby plan is a clean $19/month ($16 billed annually) for exactly 5,000 credits, where one credit equals one page on a basic scrape. That's the whole calculation — Firecrawl's pricing is genuinely per-page.

Apify's is not, and that's worth saying plainly rather than working around: the platform meters compute units (RAM-hours), proxy bandwidth, and storage separately, and the actual cost of "5,000 pages" depends entirely on which of its thousands of pre-built Actors you run and whether the target site needs residential proxies (from $7–$8/GB) instead of datacenter ones (from $0.6–$1/IP). The same 5,000-page job can cost anywhere from a few dollars to several times Firecrawl's flat rate. What Apify buys instead of a predictable per-page price is breadth — a marketplace of ready-made scrapers for specific sites (Amazon, LinkedIn, Google Maps) that would otherwise mean writing and maintaining your own extraction logic.

Crawl4AI is the free option in the literal sense: Apache-2.0 licensed, 83,800+ GitHub stars, and a commit pushed within the last day as of this writing — this is an actively maintained project, not an abandoned star magnet. Self-hosted, it costs nothing per page; you pay in your own compute and, if the target requires it, your own proxy budget. Crawl4AI is building a hosted Cloud API, but it's in closed beta with no public pricing yet, so today the choice is genuinely "run it yourself" or pick a hosted vendor.

Job three: pull one field from a page that keeps changing — AgentQL

AgentQL's pitch is narrower and, for the right job, better than either of the above: instead of a CSS selector that breaks the next time a site redesigns its checkout page, you write a natural-language query ("the current price"), and AgentQL's model re-locates that field regardless of markup changes. Starter is pay-as-you-go — 50 free calls a month, then $0.02/call. Professional is $99/month flat, including 10,000 calls, then $0.015/call beyond that.

The crossover between the two plans lands almost exactly at 5,000 calls a month: below that, Starter's metered pricing is cheaper; above it, Professional's flat rate wins, and it comes with 10,000 calls of headroom before per-call charges resume. If your agent's job is "watch this one field on 200 competitor pricing pages and alert me when it changes," that's roughly 6,000 calls a month checking daily — comfortably inside Professional's flat $99.

The question none of these vendors answer: can you still reach the page

Every comparison above assumes the target page is reachable. That assumption got measurably worse through 2025 and 2026. On July 1, 2025 — a date Cloudflare's own blog calls "Content Independence Day" — Cloudflare changed its default to block declared AI crawlers site-wide unless the crawler pays. A month later, Cloudflare reported that over two and a half million websites had already opted to fully disallow AI training crawlers through its managed robots.txt feature or its managed AI-bot block rule — available to every customer, including the free tier. Alongside the block, Cloudflare shipped pay per crawl: a crawler now gets HTTP 402 Payment Required with a crawler-price header instead of content, and either pays the quoted price or gets nothing. It's still in private beta, but it formalizes something that used to be informal: a growing share of the web now meters access to AI systems the same way it meters an API.

That specific block list — GPTBot, ClaudeBot, Bytespider, Amazonbot, Google-Extended, Meta's crawler, and a handful of others — targets crawlers declared for AI model training, not the search, crawl, and extract vendors compared above; a Firecrawl or Apify scraper making a request on a customer's behalf is a different, generic bot as far as that rule is concerned. But it still runs into the broader machine-learning bot-detection system behind it, the same system Cloudflare used to catch Perplexity rotating its user agent and source IPs after its declared crawler got blocked. None of the seven vendors above publish a success rate against Cloudflare-protected sites specifically. What they publish instead is a tell: Apify prices residential proxies at 8–13x the cost of datacenter proxies specifically because datacenter IPs are, in its own pricing copy, "ideal for scraping less-protected websites" — meaning the harder, better-protected ones need the expensive tier. Firecrawl bundles "anti-bot" handling into every scrape as a named, unpriced, unmeasured feature. Neither is a number you can budget against; both are an admission that reachability, not just format or price, is now part of what you're buying.

Search APIs are less exposed to this than crawl APIs, and it's worth understanding why rather than just noting it. Exa's index is "refreshed continuously" rather than fetched fresh on every query, so a page it already indexed keeps answering search requests even if the origin later blocks Exa's crawler going forward — the damage shows up as staleness, not failure. A crawl API fetching a specific URL at request time has no such buffer: if the target blocks it today, the request fails today. That's a real advantage for search-shaped work, and a real reason not to reach for a crawl API when a search API would answer the question.

What to actually buy

Pick by job, not by brand: Tavily if the priority is one clean API call per research task, Exa if the priority is index breadth across specialized data, You.com if price per call is what decides it. Firecrawl for predictable per-page crawling costs, Apify when you need one of its thousands of pre-built site-specific scrapers and can tolerate variable pricing, Crawl4AI if the volume justifies running your own infrastructure for zero marginal cost. AgentQL when the job is one field, not a whole page, and the target's HTML changes often enough that a selector-based scraper would need constant maintenance. And treat every price on every one of these pages as provisional against a harder constraint: whether the page will still answer at all once enough of the web has switched its default to no.