Skip to content
AgentsIndex

Ragas

Open-source LLM evaluation for repeatable experiments and tracked results.

Best for
AI teams who need systematic evaluation loops for prompts, RAG systems, and agents.
GitHub stars
16k

What we checked

Pages last read
Facts checked
3 facts

From 4 sources

AI answers
Too few read

58 buyer-question answers read, November 2026

At a glance

Tags
Free, Open source

Code

Last push
24 February 2026

For developers

Works with
Arize, LangSmith, Amazon Bedrock, Google Gemini, OCI Gen AI, AG-UI and 17 more

Ragas in our research

All research

Study

AI Tool Pricing Index, November 2026

Pricing

Free only

29 of the 361 tools we checked are free with no paid plan.

Edition price, checked 14 November 2026.

Pricing pages of the AI tools, November 2026 editionas of n = 361 tools checked

What is Ragas?

Ragas is an open-source LLM evaluation library for AI teams that need repeatable experiments across prompts, RAG systems, workflows, and agents. It combines Experiments-first approach, Ragas Metrics, dataset management, result tracking, and Test Data Generation, and integrates with LangChain, LlamaIndex, Haystack, LangGraph, Amazon Bedrock, Arize, LangSmith, Google Gemini, and OCI Gen AI. Public references include Atomicwork, Pinecone, Weaviate, Qdrant, Deepset Haystack, Mixedbread.ai, LangChain, and OpenAI.

A Python library that scores LLM apps with LLM judges

Ragas is a library you pip install and call from code. Its introduction describes moving teams "from \"vibe checks\" to systematic evaluation loops" by combining LLM-graded metrics with repeatable experiments. It also generates synthetic test sets from your documents by building a knowledge graph and writing single-hop and multi-hop questions, so a team without a labeled eval set can start from its own corpus.

The library is free, and the judge model's tokens are the real bill

What costs money is the model doing the grading. The cost analysis guide states that "By default, Ragas does not calculate the usage of tokens for evaluate()"; you get a token count only after wiring in a TokenUsageParser, and a ready parser is shown for OpenAI. The only commercial offer on the site is help: the docs home says "Book a slot or drop us a line: founders@vibrantlabs.com", with no figure attached.

An import issue with Langchain dependencies broke Ragas for months

The release published on November 14, 2026 addresses it directly: "import ragas works again", with the VertexAI imports now optional. Its notes still list known issues, including that "Claude models reached through LiteLLM still receive Ragas' default sampling parameters".

The docs disagree with themselves on dependencies and templates

The Phoenix (Arize) integration tutorial goes further back: it installs "ragas<0.1.1", "llama-index<0.10.0" and "openinference-instrumentation-llama-index<1.0.0", yet its own code later imports from llama_index.core. The same tutorial says "Let's generate 100 data points using Ragas" while the code beneath it sets TEST_SIZE = 25. The agent metrics and tutorials exist; the scaffolding command for them does not yet.

Vibrant Labs maintains it while describing another focus

Ragas is published by Vibrant Labs, renamed on GitHub from explodinggradients to vibrantlabsai. The docs home adds that the team is "compressing this knowledge into a product to replace vibe checks with eval loops", but no such product is described anywhere on the site. Support is community-run: the community page points to a Discord server and GitHub issues, and the latest release notes include a change titled "chore: remove office hours cal".

Telemetry is on by default and eval data goes to your LLM provider

Because the library runs in your own environment, your eval data goes only to the LLM provider you configure, not to a Ragas server. The Phoenix (Arize) tutorial notes that "Ragas by default uses OpenAI models under the hood, but you're free to use any model of your choice", and states that the notebook "requires an OpenAI API key". For seeing what the judge sends, the integrations page lists Arize Phoenix and LangSmith as tracing tools that can be used "to get the traces of the evaluator LLMs".

It suits Python teams, not buyers who want a hosted eval product

Ragas fits an engineering team that already writes Python, wants RAG, agent or SQL metrics it can run in CI or a notebook, and is willing to pay its own model provider for judge calls. It does not fit a team without a developer: there is no web app, no hosted dashboard and no vendor contract.

Ragas price and terms

Terms

Free plan
YesYesOpen-source library, free to install and run
The terms in full

LLM calls made during evaluation are billed by the model provider you configure.

For AI agents

Developer docs

Where these prices come from

docs.ragas.io/en/stable/getstarted/install, checked

All plans and limits

PlanPriceMain limits
RagasFreeNone stated

Read from docs.ragas.io/en/stable/getstarted/install.

What we checked on Ragas

YesPartlyNo

What it costs to startYes

Free to use

The whole library installs free from PyPI with pip and is Apache-2.0 licensed; evaluations still call an LLM you supply and pay for, such as OpenAI.

To get started, install Ragas using pip with the following command:

docs.ragas.io/en/stable/getstarted/inst…Checked

How you build with itYes

Public API

Ragas is used through its documented Python API, with a References section covering evaluate(), metrics, LLMs, embeddings and test set generation; it is a library API, not a hosted REST service.

API documentation and technical details for diving deeper.

docs.ragas.io/en/stableChecked

Questions about Ragas

How much does Ragas cost?
Ragas is free to use. From docs.ragas.io/en/stable/getstarted/install, checked 14 November 2026.
Does Ragas have an API or an MCP server?
Ragas has an API. From docs.ragas.io/en/stable, checked 14 November 2026.
Is Ragas open source, and can you host it yourself?
Ragas is open source. We checked github.com/vibrantlabsai/ragas/blob/main/LICENSE ourselves on 15 November 2026.
What have we checked on Ragas?
3 facts, under these headings: What it costs to start, how you build with it.
How do we check Ragas?
We hold 3 facts on Ragas, quoted from 4 pages. Facts last checked 15 November 2026. Price last checked 14 November 2026. Last change found 15 November 2026.

How we check every listing

Sources and changes

Last full check

What changed2

  • What it does
  • What it does

Repository

Licence
Apache-2.0
Stars
16,035
Last push
24 February 2026

Ragas price history

We have read the price once so far, on 14 November 2026: free.

PricePlanFirst seenLast seenReadingsSource
Free14 November 202614 November 20261Pricing page, read 14 November 2026

From our research on 14 November 2026. A date is when we read the page, not when the price changed.

Similar in Agent tools and integrations

See all in Agent tools and integrations

Is Ragas yours?

Free, and the page then carries the verified owner mark. You confirm it with an email at docs.ragas.io. Claim this listing

Report an error if a fact is wrong.