Skip to content
AgentsIndex

DeepEval

Python test framework that runs LLM app and agent evals as Pytest-style unit tests, locally or in CI.

Best for
AI engineers who need repeatable tests for agents, RAG, and tool-using workflows.
GitHub stars
18.9k

What we checked

Pages last read
Facts checked
3 facts

From 3 sources

AI answers
Too few read

58 buyer-question answers read, November 2026

At a glance

Tags
Self-hosted

Code

Last push
15 October 2026

For developers

Works with
LangChain, LlamaIndex, CrewAI, OpenAI Agents, LangGraph, PydanticAI and 24 more

DeepEval in our research

All research

Study

AI Tool Pricing Index, November 2026

Pricing

Free only

29 of the 361 tools we checked are free with no paid plan.

Edition price, checked 6 November 2026.

Pricing pages of the AI tools, November 2026 editionas of n = 361 tools checked

What is DeepEval?

DeepEval is an LLM evaluation framework for AI engineers that turns model behavior into repeatable tests, traces, and scored runs. It supports Unit testing for LLMs, LLM-as-a-Judge, 50+ research-backed metrics, Native conversational evals, and Multi-modal by default, while tracing across LangChain, LlamaIndex, LangGraph, OpenTelemetry, and Vercel AI SDK.

DeepEval's homepage (checked 12 Oct 2026) lists Visa, Walmart, Toyota, Mastercard, NVIDIA, Mercedes-Benz, Siemens, and AXA as customers, and it runs locally in your own environment.

Still actively maintained

DeepEval's GitHub repository (confident-ai/deepeval) is not archived and received a commit on the day of this check (12 October 2026). The latest PyPI release, 4.2.8, shipped 2 October 2026, continuing a roughly weekly cadence of point releases through the second half of 2026 (4.2.4 on 22 Sep, 4.2.5 on 22 Sep, 4.2.6 on 24 Sep, 4.2.7 on 29 Sep). The project has contributions from 308 people, and it still ships under the Apache-2.0 license recorded in its LICENSE.md. The 'needs update' flag on this listing reflects our own re-check lapsing since May, not the project going quiet.

What the framework does well

DeepEval's core idea is pytest-native testing for LLM output: you write assert_test() calls using metrics like GEval or TaskCompletionMetric the same way you'd write any other unit test, and deepeval test run wires that into CI/CD. Confident AI's own plan-comparison page — the one place the vendor states a concrete metric count — lists 30+ single-turn and 15+ multi-turn DeepEval metrics, covering RAG (faithfulness, contextual precision/recall), agents (task completion, tool correctness, plan adherence), multi-turn conversations, MCP tool use, and multimodal output. Tracing integrations cover LangChain, LlamaIndex, LangGraph, OpenTelemetry, and the Vercel AI SDK, and evaluation runs locally against a judge model of your choice, including local models via Ollama, LM Studio, or vLLM. On PyPI, the package pulled 2.58 million downloads in the last month — a stronger, independently checkable usage signal than any developer count the vendor states.

Free framework, priced platform

DeepEval itself costs nothing: Apache-2.0, pip install deepeval, and your evaluation data stays on your own machine — the only thing that leaves is anonymous PostHog telemetry (event names and metric names, no PII), and that's opt-out. The paid product sits one layer up. Confident AI, built by the same team, adds shared dashboards, dataset versioning, and production monitoring on top of the same metrics: Free is $0 forever but caps you at 2 seats, 1 project, and 5 test runs a week; Starter is $200/month; Team is $2,000/month and is where SOC 2, SSO, and custom RBAC appear; Enterprise is custom. That Starter-to-Team jump is ten times the price for the compliance features a regulated buyer may need from day one, so it's worth checking which tier you'd actually land on before treating $200 as the number. Startups in the YC network, or anywhere with under $5M raised, can apply for a grant through Confident AI's own pricing page.

Who it says uses it

DeepEval's homepage (checked 12 October 2026) displays logos for Visa, Walmart, Toyota, Mastercard, EY, Mercedes-Benz, NVIDIA, AXA, Siemens, and CVS Health. DeepEval's own docs separately describe it as 'one of the most used' LLM eval frameworks 'in the world,' citing 20 million-plus daily evaluations — a vendor figure we have no independent way to confirm. On the Confident AI side specifically, three customers are named with attributed quotes on the pricing page — Finom, Amdocs, and Humach — plus an unnamed 'Fortune 500 medical device company.'

License and data handling

The repository's LICENSE.md and GitHub's own license detection both show Apache-2.0, with no separate commercial clause on the code that we could find — self-hosting for production use isn't gated behind a paid tier. The OSV vulnerability database returns no published advisories for the deepeval PyPI package as of this check. On the hosted side, Confident AI lists SOC 2 as a Team-tier ($2,000/month) feature rather than something every plan carries, which is worth checking directly if compliance is the reason you're evaluating the paid product.

DeepEval price and terms

Terms

Free plan
YesYesFree Apache 2.0 open-source library
The terms in full

runs locally. LLM-judge metrics need your own model provider key (e.g. OPENAI_API_KEY), billed by that provider.

For AI agents

Developer docs

Where these prices come from

deepeval.com, checked

All plans and limits

PlanPriceMain limits
DeepEvalFreeNone stated

Read from deepeval.com.

What we checked on DeepEval

YesPartlyNo

What it costs to startYes

Free to use

Anyone can pip install the Apache 2.0 framework and run evaluations without a sales call or an account.

In a newly created virtual environment, run: pip install -U deepeval

deepeval.com/docs/getting-startedChecked

How you build with itYes

Public API

DeepEval is used as a documented Python library and CLI (deepeval test run), with metrics and test cases called from Python code; it is a code SDK rather than a hosted REST API.

Pytest-native evals that run in CI/CD or as Python scripts. Iterate locally, on your own environment, on your own criteria.

deepeval.comChecked

Questions about DeepEval

How much does DeepEval cost?
DeepEval is free to use. From deepeval.com, checked 6 November 2026.
Does DeepEval have an API or an MCP server?
DeepEval has an API. From deepeval.com, checked 6 November 2026.
Is DeepEval open source, and can you host it yourself?
DeepEval is open source. We checked github.com/confident-ai/deepeval/blob/main/LICENSE.md ourselves on 15 November 2026. You can host DeepEval yourself.
What have we checked on DeepEval?
3 facts, under these headings: What it costs to start, how you build with it.
How do we check DeepEval?
We hold 3 facts on DeepEval, quoted from 3 pages. Facts last checked 15 November 2026. Price last checked 6 November 2026. Last change found 15 November 2026.

How we check every listing

Sources and changes

Last full check

Sources on deepeval.com2

What changed2

  • What it does
  • What it does

Repository

Licence
Apache-2.0
Stars
18,891
Last push
15 October 2026

DeepEval price history

We have read the price once so far, on 6 November 2026: free.

PricePlanFirst seenLast seenReadingsSource
Free6 November 20266 November 20261Pricing page, read 6 November 2026

From our research on 6 November 2026. A date is when we read the page, not when the price changed.

Similar in Agent tools and integrations

See all in Agent tools and integrations

Is DeepEval yours?

Free, and the page then carries the verified owner mark. You confirm it with an email at deepeval.com. Claim this listing

Report an error if a fact is wrong.