# DeepEval

> Source: https://agentsindex.ai/deepeval

Python test framework that runs LLM app and agent evals as Pytest-style unit tests, locally or in CI.

- Best for: AI engineers who need repeatable tests for agents, RAG, and tool-using workflows.

Starting price

Free

From [deepeval.com](https://deepeval.com/), checked 6 November 2026

- MCP server Not stated
- API Yes From deepeval.com, 6 November 2026 [deepeval.com](https://deepeval.com/)
- Open source Yes Checked by us github.com/confident-ai/deepeval/blob/main/LICENSE.md, 15 November 2026 [Checked by us](https://github.com/confident-ai/deepeval/blob/main/LICENSE.md)
- Self-host Yes

- GitHub stars: 18.9k

[Docs](https://deepeval.com/docs/getting-started) [GitHub](https://github.com/confident-ai/deepeval)

What we checked

- Pages last read: 15 November 2026
- Facts checked: 3 facts · From 3 sources
- AI answers: Too few read · 58 buyer-question answers read, November 2026

[How we check](https://agentsindex.ai/methodology) [Editorial policy](https://agentsindex.ai/editorial-policy#not-for-sale)

[Visit DeepEval](https://agentsindex.ai/go/deepeval) [Compare](https://agentsindex.ai/compare?cat=agent-tools-and-integrations&products=deepeval)

## At a glance

- Category: [Agent tools and integrations](https://agentsindex.ai/categories/agent-tools-and-integrations)
- Tags: Self-hosted

### Code

- Last push: 15 October 2026

### For developers

- Works with: LangChain, LlamaIndex, CrewAI, OpenAI Agents, LangGraph, PydanticAI and 24 more

## DeepEval in our research

[All research](https://agentsindex.ai/blog)

Study 19 October 2026 – 16 November 2026

### [AI Tool Pricing Index, November 2026](https://agentsindex.ai/data/ai-tool-pricing-index)

Pricing

Free only

29 of the 361 tools we checked are free with no paid plan.

Edition price, checked 6 November 2026.

Pricing pages of the AI tools, November 2026 edition as of 16 November 2026 n = 361 tools checked

## What is DeepEval?

DeepEval is an LLM evaluation framework for AI engineers that turns model behavior into repeatable tests, traces, and scored runs. It supports Unit testing for LLMs, LLM-as-a-Judge, 50+ research-backed metrics, Native conversational evals, and Multi-modal by default, while tracing across LangChain, LlamaIndex, LangGraph, OpenTelemetry, and Vercel AI SDK.

DeepEval's homepage (checked 12 Oct 2026) lists Visa, Walmart, Toyota, Mastercard, NVIDIA, Mercedes-Benz, Siemens, and AXA as customers, and it runs locally in your own environment.

## Still actively maintained

DeepEval's GitHub repository (confident-ai/deepeval) is not archived and received a commit on the day of this check (12 October 2026). The latest PyPI release, 4.2.8, shipped 2 October 2026, continuing a roughly weekly cadence of point releases through the second half of 2026 (4.2.4 on 22 Sep, 4.2.5 on 22 Sep, 4.2.6 on 24 Sep, 4.2.7 on 29 Sep). The project has contributions from 308 people, and it still ships under the Apache-2.0 license recorded in its LICENSE.md. The 'needs update' flag on this listing reflects our own re-check lapsing since May, not the project going quiet.

## What the framework does well

DeepEval's core idea is pytest-native testing for LLM output: you write assert_test() calls using metrics like GEval or TaskCompletionMetric the same way you'd write any other unit test, and `deepeval test run` wires that into CI/CD. Confident AI's own plan-comparison page — the one place the vendor states a concrete metric count — lists 30+ single-turn and 15+ multi-turn DeepEval metrics, covering RAG (faithfulness, contextual precision/recall), agents (task completion, tool correctness, plan adherence), multi-turn conversations, MCP tool use, and multimodal output. Tracing integrations cover LangChain, LlamaIndex, LangGraph, OpenTelemetry, and the Vercel AI SDK, and evaluation runs locally against a judge model of your choice, including local models via Ollama, LM Studio, or vLLM. On PyPI, the package pulled 2.58 million downloads in the last month — a stronger, independently checkable usage signal than any developer count the vendor states.

## Free framework, priced platform

DeepEval itself costs nothing: Apache-2.0, `pip install deepeval`, and your evaluation data stays on your own machine — the only thing that leaves is anonymous PostHog telemetry (event names and metric names, no PII), and that's opt-out. The paid product sits one layer up. Confident AI, built by the same team, adds shared dashboards, dataset versioning, and production monitoring on top of the same metrics: Free is $0 forever but caps you at 2 seats, 1 project, and 5 test runs a week; Starter is $200/month; Team is $2,000/month and is where SOC 2, SSO, and custom RBAC appear; Enterprise is custom. That Starter-to-Team jump is ten times the price for the compliance features a regulated buyer may need from day one, so it's worth checking which tier you'd actually land on before treating $200 as the number. Startups in the YC network, or anywhere with under $5M raised, can apply for a grant through Confident AI's own pricing page.

## Who it says uses it

DeepEval's homepage (checked 12 October 2026) displays logos for Visa, Walmart, Toyota, Mastercard, EY, Mercedes-Benz, NVIDIA, AXA, Siemens, and CVS Health. DeepEval's own docs separately describe it as 'one of the most used' LLM eval frameworks 'in the world,' citing 20 million-plus daily evaluations — a vendor figure we have no independent way to confirm. On the Confident AI side specifically, three customers are named with attributed quotes on the pricing page — Finom, Amdocs, and Humach — plus an unnamed 'Fortune 500 medical device company.'

## License and data handling

The repository's LICENSE.md and GitHub's own license detection both show Apache-2.0, with no separate commercial clause on the code that we could find — self-hosting for production use isn't gated behind a paid tier. The OSV vulnerability database returns no published advisories for the deepeval PyPI package as of this check. On the hosted side, Confident AI lists SOC 2 as a Team-tier ($2,000/month) feature rather than something every plan carries, which is worth checking directly if compliance is the reason you're evaluating the paid product.

Our take

Check whether your workflow needs shared observability in Confident AI, since that integration is the listed path for dashboards and regression tracking. If you need a dashboard-first setup from day one, verify that before adopting the local-first workflow.

## DeepEval price and terms

### Terms

- Free plan: Yes Free Apache 2.0 open-source library

The terms in full

runs locally. LLM-judge metrics need your own model provider key (e.g. OPENAI_API_KEY), billed by that provider.

### For AI agents

- API: Yes [deepeval.com, 6 November 2026](https://deepeval.com/)

[Developer docs](https://deepeval.com/docs/getting-started)

Where these prices come from

[deepeval.com](https://deepeval.com/), checked 6 November 2026

### All plans and limits

| Plan | Price | Main limits |
| --- | --- | --- |
| DeepEval | Free | None stated |

Read 6 November 2026 from [deepeval.com](https://deepeval.com/).

## What we checked on DeepEval

Yes Partly No

What it costs to start Yes

### Free to use

Anyone can pip install the Apache 2.0 framework and run evaluations without a sales call or an account.

> In a newly created virtual environment, run: pip install -U deepeval

Installation section [deepeval.com/docs/getting-started](https://deepeval.com/docs/getting-started) Checked 5 November 2026

How you build with it Yes

### Public API

DeepEval is used as a documented Python library and CLI (deepeval test run), with metrics and test cases called from Python code; it is a code SDK rather than a hosted REST API.

> Pytest-native evals that run in CI/CD or as Python scripts. Iterate locally, on your own environment, on your own criteria.

Homepage, 'Unit testing for LLMs.' section [deepeval.com](https://deepeval.com/) Checked 6 November 2026

How you build with it Yes

### Open source

Apache-2.0 licence in its public repository

> Apache License Version 2.0

LICENSE.md in github.com/confident-ai/deepeval [github.com/confident-ai/deepeval/blob/m…](https://github.com/confident-ai/deepeval/blob/main/LICENSE.md) Checked by us, 15 November 2026

## Questions about DeepEval

- How much does DeepEval cost?: DeepEval is free to use. From deepeval.com, checked 6 November 2026.
- Does DeepEval have an API or an MCP server?: DeepEval has an API. From deepeval.com, checked 6 November 2026.
- Is DeepEval open source, and can you host it yourself?: DeepEval is open source. We checked github.com/confident-ai/deepeval/blob/main/LICENSE.md ourselves on 15 November 2026. You can host DeepEval yourself.
- What have we checked on DeepEval?: 3 facts, under these headings: What it costs to start, how you build with it.
- How do we check DeepEval?: We hold 3 facts on DeepEval, quoted from 3 pages. Facts last checked 15 November 2026. Price last checked 6 November 2026. Last change found 15 November 2026.

[How we check every listing](https://agentsindex.ai/methodology)

## Sources and changes

Last full check 6 November 2026

Sources on deepeval.com 2

- Read 6 November 2026 · [/](https://deepeval.com/)
- Read 5 November 2026 · [/docs/getting-started](https://deepeval.com/docs/getting-started)

What changed 2

- 15 November 2026 What it does
- 6 November 2026 What it does

Repository

- Licence: Apache-2.0
- Stars: 18,891
- Last push: 15 October 2026

## DeepEval price history

We have read the price once so far, on 6 November 2026: free.

| Price | Plan | First seen | Last seen | Readings | Source |
| --- | --- | --- | --- | --- | --- |
| Free |   | 6 November 2026 | 6 November 2026 | 1 | [Pricing page, read 6 November 2026](https://deepeval.com/) |

From our research on 6 November 2026. A date is when we read the page, not when the price changed.

## Similar in Agent tools and integrations

[See all in Agent tools and integrations](https://agentsindex.ai/categories/agent-tools-and-integrations)

- [ERIUS PHONE](https://agentsindex.ai/eriusphone) Rented cloud Android phones, each in its own container, that keep an agent's apps and sign-ins between sessions. From about $44 a month
- [TinyFish](https://agentsindex.ai/tinyfish) Web search, page fetching, cloud browsers and goal-driven web automation for AI agents, with search and fetch free. Free plan, then $0.002 per minute
- [Smithery](https://agentsindex.ai/smithery) Public MCP server registry, now run by Arcade, listing direct vendor MCP URLs after its proxy shut down. Free plan
- [UI Discovery](https://agentsindex.ai/uidiscovery) A library of real websites, iOS apps and dashboards with each one's palette, fonts and original build prompt. From $20 a month

## Is DeepEval yours?

Free, and the page then carries the verified owner mark. You confirm it with an email at deepeval.com. [Claim this listing](https://agentsindex.ai/claim/deepeval?from=owner-line)

[Report an error](mailto:hello@agentsindex.ai?subject=Report%20an%20error%3A%20DeepEval%20(deepeval)&body=What%20is%20wrong%20on%20the%20page%2C%20and%20where%20on%20the%20product) if a fact is wrong.
