Evals

EvalsPaid

Datadog LLM Observability

LLM and agent observability within the Datadog platform.

AR✓ Attested

Arthur

Enterprise AI monitoring, evaluation, and governance platform.

Maxim AI

End-to-end agent simulation, evaluation, and observability platform.

Lakera

Runtime security guardrails and red teaming for GenAI applications.

Portkey

AI gateway with built-in observability, guardrails, and governance.

Laminar

Open-source observability for AI agents with eval-driven debugging.

AgentOps

Observability and debugging platform for AI agents.

AT✓ Attested

Athina AI

Collaborative platform for building, testing, and monitoring LLM features.

LangWatch

Open-source platform for testing, evaluating, and monitoring AI agents.

EvalsContact for pricing

Patronus AI

AI evaluation research company behind Lynx, GLIDER, and FinanceBench.

OpenCompass

Open-source LLM evaluation platform covering 100+ datasets.

lm-evaluation-harness

EleutherAI's unified framework for benchmarking generative language models.

Inspect

Open-source framework for frontier AI evaluations from the UK AI Security Institute.

promptfoo

Open-source LLM evals and automated red teaming.

ML✓ Attested

MLflow

Open-source AI platform with GenAI tracing, evaluation, and monitoring.

Opik

Open-source LLM observability and evaluation platform by Comet.

OpenLIT

OpenTelemetry-native, open-source GenAI observability platform.

Traceloop

LLM reliability platform built on the open-source OpenLLMetry SDK.

HoneyHive

Agent observability and evaluation platform for production AI.

Galileo

AI evaluation and observability platform with production guardrails.

Fiddler AI

Enterprise AI observability, guardrails, and governance platform.

Evidently

Open-source evaluation, testing, and monitoring for LLM and ML systems.

NeMo Guardrails

NVIDIA's toolkit for programmable guardrails on LLM applications.

Guardrails AI

Open-source framework for validating and guarding LLM inputs and outputs.

Giskard

Automated testing and red teaming for LLM agents.

TruLens

Open-source evaluation and tracing for LLM apps and agents.

RA✓ Attested

Ragas

Open-source evaluation framework for RAG and LLM pipelines.

Confident AI

Cloud evaluation and observability platform built by the DeepEval team.

DeepEval

Open-source evaluation framework for LLM apps, agents, and RAG.

OpenAI Evals

Framework and open registry of benchmarks for evaluating LLMs.

PromptLayer

Prompt management with built-in evaluations and observability.

Helicone

Open-source AI gateway and LLM observability platform.

W&B Weave

LLM tracing and evaluation toolkit from Weights & Biases.

Braintrust

Evaluation and observability platform for AI agents and LLM products.

Arize AX

Enterprise observability and evaluation platform for AI agents.

Arize Phoenix

Open-source tracing and evaluation for LLM apps and agents.

Langfuse

Open-source LLM engineering platform for tracing, evals, and prompt management.

LangSmith

Agent and LLM observability platform from the LangChain team.