# Inspect

> Source: https://agentsindex.ai/inspect-ai

Open-source Python framework for evaluating LLMs and agents, from the UK AI Security Institute.

Inspect is a Python framework for writing and running large language model evaluations, built from datasets, solvers, scorers and agents. It ships over 200 pre-built benchmarks, a web log viewer, sandboxing and tool calling across more than 20 model providers.

- Best for: Inspect AI is best for AI eval engineers who need a flexible framework for benchmarking models and agents.

Starting price

Free

From [github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE), checked 27 October 2026

- MCP server Not stated
- API Not stated
- Open source Yes
- Self-host Yes

- GitHub stars: 3k

[Docs](https://inspect.aisi.org.uk/) [GitHub](https://github.com/UKGovernmentBEIS/inspect_ai)

What we checked

- Pages last read: 27 October 2026
- Facts checked: 1 fact From 2 sources
- AI answers: Not named in any of 62 Buyer questions, our November check

[How we check](https://agentsindex.ai/methodology) [Editorial policy](https://agentsindex.ai/editorial-policy#not-for-sale)

[Visit Inspect](https://agentsindex.ai/go/inspect-ai) [Compare](https://agentsindex.ai/compare?cat=agent-tools-and-integrations&products=inspect-ai)

## At a glance

- Categories: [Agent tools and integrations](https://agentsindex.ai/categories/agent-tools-and-integrations) [Agent frameworks](https://agentsindex.ai/categories/agent-frameworks)
- Tags: Open source, Self-hosted

### Code

- Last push: 15 October 2026

### For developers

- Works with: OpenAI, Anthropic, Google, Grok, Mistral, HF and 18 more

## What it actually is

Inspect is a pip-installable Python library (`pip install inspect-ai`) plus a CLI (`inspect eval`, `inspect view`) for defining evaluations as three composable pieces — a dataset, a solver (how the model produces an answer, from a single `generate()` call to a full ReAct agent loop), and a scorer (how the answer is graded). It was built and open-sourced by the UK AI Security Institute (then AI Safety Institute) in May 2024, and development is now shared with Meridian Labs, a 501(c)(3) nonprofit the AISI team spun out to steward the project alongside sibling tools (Inspect Scout for transcript analysis, Inspect Petri for alignment auditing).

## What it's good at

It's a real, load-bearing piece of infrastructure, not a hobby project. A companion repo to UKGovernmentBEIS/inspect_ai, inspect_evals, holds 200+ pre-built benchmark implementations spanning coding (SWE-bench, HumanEval, LiveCodeBench), cybersecurity (Cybench, CyberGym, GDM Dangerous Capabilities), safety/scheming (AgentHarm, StrongREJECT, WMDP, Agentic Misalignment), science, math, and multimodal tasks — that count is corroborated directly on the vendor's own evals listing page. It has built-in support for 24+ model providers (OpenAI, Anthropic, Google, Mistral, AWS Bedrock/SageMaker, Azure, Groq, Together, local inference via vLLM/Ollama/SGLang, and more), and a genuine third-party ecosystem has grown around it: METR built and open-sourced 'Hawk', a Kubernetes-based platform for running Inspect evals at scale in AWS; the Vector Institute, Arcadia Impact, and independent contributors maintain sandbox backends (Podman, Modal, Daytona, Proxmox) and analysis plugins; and IBM Research cited Inspect AI as one of three eval harnesses (alongside HELM and lm-eval-harness) it built automated translators for in its EveryEvalEver benchmarking-standardization project. Commit activity is current as of this check (last push within the last day) and the CLI ships a VS Code extension with about 11,000 installs.

## Maturity and stability

PyPI still classifies the package as 'Development Status:: 4 - Beta', and the project is on a 0.3.x version scheme (current release 0.3.260 as of this check) with no 1.0 yet, despite roughly 113 million cumulative PyPI downloads and multi-year production use by AISI itself. Inspect ships frequent point releases (hundreds of 0.3.x tags), which is normal for an actively developed framework but means buyers building long-lived internal tooling on top of it should pin versions and expect occasional breaking changes between releases rather than treating it as a stable public API.

## Security disclosure and scope

The repo publishes a formal SECURITY.md (contact: security@meridianlabs.ai) with a stated 3-business-day acknowledgment window and a 30-day target for fixes. It's explicit about scope: sandbox escapes, secrets leaking into logs, and log-viewer vulnerabilities are in scope; running untrusted model-generated code inside a sandbox, or using the (explicitly unsandboxed) 'local' sandbox mode, is called out as expected behavior, not a vulnerability — worth knowing if your eval design assumes stronger isolation than Docker provides by default. No published security advisories exist on the repo as of this check.

## Governance and who's behind it

Ownership is split between a UK government body (UKGovernmentBEIS, i.e. AI Security Institute, under the Department for Science, Innovation and Technology) and Meridian Labs, a US 501(c)(3) nonprofit. This is unusual for a widely-used dev tool: it means the roadmap answers to a state safety institute's priorities (the evals catalog leans heavily toward dangerous-capability, cyber, and alignment benchmarks) rather than a commercial vendor's, and there is no pricing or support tier to buy — everything is free and community-supported via GitHub issues and a public Slack.

## Inspect price and terms

### Terms

- Free plan: Yes The framework itself is unrestricted under MIT · The terms in full · running evaluations requires your own model provider keys, so inference is billed by the provider.

Where these prices come from

[github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE), checked 27 October 2026

### All plans and limits

| Plan | Price | Main limits |
| --- | --- | --- |
| Inspect | Free | None stated |

Read 27 October 2026 from [github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE).

## What we checked on Inspect

How you build with it Yes

### Open source

The whole framework is developed in public on GitHub and released under the MIT license, which permits commercial use, modification and redistribution without fee.

> Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to…

LICENSE file, MIT License grant paragraph [github.com/UKGovernmentBEIS/inspect_ai/…](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE) Checked 27 October 2026

## Inspect alternatives

- [Dify](https://agentsindex.ai/dify) · Open-source platform for building agents, agentic workflows and RAG pipelines, on cloud or self-hosted. · Free, then $59 a month Checked 22 Oct · Freemium · [Visit](https://agentsindex.ai/go/dify)
  - MCP server
  - API
  - Open source
  - Self-host
- [TinyFish](https://agentsindex.ai/tinyfish) · The team behind TinyFish runs this listing · One web layer for agents: free search and fetch, plus hosted browsers and goal-driven web automation. · Free, then $0.002 per minute Checked 11 Nov · ISO 27001 · [Visit](https://agentsindex.ai/go/tinyfish)
  - MCP server
  - API
  - Open source
  - Self-host
- Inspect and its 2 alternatives · All 2 publish a price.
  | Inspect | Free | 27 Oct |
  | --- | --- | --- |
  | [Dify](https://agentsindex.ai/dify) | Free, then $59 a month | 22 Oct |
  | [TinyFish](https://agentsindex.ai/tinyfish) | Free, then $0.002 per minute | 11 Nov |

## Questions about Inspect

- How much does Inspect cost?: Inspect is free to use. From github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE, checked 27 October 2026.
- Is Inspect open source, and can you host it yourself?: It is open source. You can host it yourself.
- What have we checked on Inspect?: 1 fact, under one heading: How you build with it.
- How do we check Inspect?: We hold 1 fact on Inspect, quoted from 2 pages. Facts and price last checked 27 October 2026.

[How we check every listing](https://agentsindex.ai/methodology)

## Sources and changes

Last changed 24 March 2026

Sources 2

- Read 27 October 2026 · [github.com](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/LICENSE) /UKGovernmentBEIS/inspect_ai/blob/main/LICENSE Repository
- Read 26 October 2026 · [inspect.aisi.org.uk](https://inspect.aisi.org.uk/llms.txt) /llms.txt

Repository

- Licence: MIT
- Stars: 3,013
- Last push: 15 October 2026

## Similar in Agent tools and integrations

[See all in Agent tools and integrations](https://agentsindex.ai/categories/agent-tools-and-integrations)

- [ERIUS PHONE](https://agentsindex.ai/eriusphone) A dedicated virtual Android phone per AI agent, with Google Play apps and sign-ins that persist between sessions. From about $45 a month
- [ClineFlow](https://agentsindex.ai/clineflow) Git-native project memory and handoffs for coding agents. Free plan
- [UI Discovery](https://agentsindex.ai/uidiscovery) Library of real website and app UI sections, queryable by AI assistants over MCP and a REST API. Free plan, then $15 a month
- [AgentMail](https://agentsindex.ai/agentmail) Email inbox API that gives each AI agent its own address, threads, webhooks and MCP server. Free plan, then $20 a month

## From our research

[All research](https://agentsindex.ai/blog)

- [Data table](https://agentsindex.ai/data/research-agents-security) 31 Oct 2026 Research agents: SOC 2, ISO 27001, GDPR and SSO
- [Data table](https://agentsindex.ai/data/sales-agents-security) 6 Nov 2026 Sales agents: SOC 2, ISO 27001, GDPR and SSO
- [Data study](https://agentsindex.ai/data/what-ai-cites-on-ai-tools) November 2026 91% of pages cited by AI on AI tools weren’t product sites

## Is this yours?

Claim it free, then see who reads this page and carry the verified owner mark. You confirm it with an email at inspect.aisi.org.uk.

[Claim this listing](https://agentsindex.ai/claim/inspect-ai?from=owner-line) [Get Featured](https://agentsindex.ai/claim/inspect-ai?from=owner-line&next=featured)

[Report an error](mailto:hello@agentsindex.ai?subject=Report%20an%20error%3A%20Inspect%20(inspect-ai)&body=What%20is%20wrong%20on%20the%20page%2C%20and%20where%20on%20the%20product) if a fact is wrong.