If you need open-source evals and tracing, Opik is the first tool to look at here. Look at LangSmith for LangChain and LangGraph tracing, and at Weights and Biases Weave for ML teams adding LLM traces. The other 5 tools in the list cover narrower or pricier cases.
| Product | Ideal for | Price from | Free version | |
|---|---|---|---|---|
| Open-source evals and tracing | See website | Yes | Visit | |
| LangChain and LangGraph tracing | See website | Yes | Visit | |
| ML teams adding LLM traces | See website | Yes | Visit | |
| Open-source ML and LLM tracing | Free | Yes | Visit | |
| AI gateway with observability | See website | Yes | Visit | |
| Open-source tracing and evals | See website | Yes | Visit | |
| Proxy-based cost and latency tracking | See website | Yes | Visit | |
| Prompt evals and experimentation | See website | Yes | Visit |
Ainotely compared 8 llm observability tools. The goal is a practical shortlist: published prices, free plan facts, best-fit use cases and honest limits for every entry.

Ideal for: Open-source evals and tracing
For open-source evals and tracing, Opik is a natural candidate. Opik, from Comet, provides open-source tracing, evaluation, and prompt management for LLM and agent apps.
Limits to weigh: Enterprise deployment needs planning.
On top of that, the community is smaller than Langfuse, which can matter for examples and integrations. Opik still spans the central use cases well, and its open-source model fits teams that want control over their data. Pricing for Opik is not stated here, so ask the vendor for current figures. Opik includes a free tier.

Ideal for: LangChain and LangGraph tracing

Pick LangSmith if this describes you: LangChain and LangGraph tracing. Evaluation tools and datasets help teams test prompts before they ship, not only after problems appear.
Pricing and self-hosting also favor cloud use for most teams. This comparison lists no price for LangSmith; ask the vendor. A free plan is available with LangSmith.
The main drawbacks: Self-hosting needs an enterprise plan.

Ideal for: ML teams adding LLM traces

Weights and Biases Weave is designed for ML teams adding LLM traces. Teams that already track models, datasets, and experiments in Weights and Biases can add LLM traces without adopting a separate system.
As for the details, the fit is most solid when ML experiment tracking is already central to how your team works. Weights and Biases Weave does not show a starting price in this comparison, so check the vendor site for current pricing. Weights and Biases Weave has a free plan.
Watch-outs: LLM observability is part of a wider ML system.

Ideal for: Open-source ML and LLM tracing
Arize Phoenix is designed for open-source ML and LLM tracing. Arize Phoenix hands teams an open-source way to trace LLM apps and inspect retrievals, embeddings, and agent steps.
Teams that want managed evals, alerts, and collaboration may need the wider Arize system or another tool beside it. On cost, Arize Phoenix starts like this: Free. Arize Phoenix has a free plan.
Trade-offs to know: Setup for production evals takes more work than a hosted eval tool.

Ideal for: AI gateway with observability
Portkey serves AI gateway with observability. Routing, fallbacks, and caching help control cost and reliability, while request logs provide a clear view of usage by model, key, and feature.
More specifically, Portkey merges an AI gateway with observability, which fits teams using multiple models or providers. No starting price is listed for Portkey here; confirm pricing on the vendor site. Portkey offers a free plan.
The main drawbacks: Small single-model apps may not need a gateway.

Ideal for: Open-source tracing and evals
Langfuse was built for open-source tracing and evals. It traces LLM calls, tool calls, and agent steps, handles prompt versions, and runs evaluations on production traces. You gain control, but you also own upgrades, storage, and scaling.
Trade-offs to know: Self-hosting layers on operations work.
Cloud hosting removes that burden for smaller teams. Worth adding: since the core is open source, teams with data residency needs can self-host and keep traces inside their own infrastructure. Check the vendor site for Langfuse pricing; none is listed in this comparison. Langfuse offers a free plan.
Ideal for: Proxy-based cost and latency tracking
Where Helicone fits: Proxy-based cost and latency tracking. Helicone appeals to teams that want visibility quick. Worth adding: by routing LLM requests through a proxy, it logs prompts, completions, costs, and latency across providers without a major rewrite.
That view alone solves a typical problem: knowing which model, feature, or customer is driving the token bill. Helicone is less of an evaluation product than Braintrust or Langfuse, though it handles entry-level scoring and monitoring. Teams that require deep evals may pair it with another tool. Helicone shows no starting price here, so confirm the cost on the vendor site. You can start on Helicone without paying.
Trade-offs to know: Deeper eval workflows are less developed than in eval-first tools.

Ideal for: Prompt evals and experimentation
Braintrust works best for prompt evals and experimentation. That workflow fits product and engineering teams that iterate on prompts every week.
Drawbacks to consider: Tracing alone is less of a focus than evals and experiments.
On top of that, teams with high prompt churn, agent behavior changes, or quality regressions will get the most from it. No starting price is listed for Braintrust here; confirm pricing on the vendor site. Free access exists with Braintrust.
Ainotely looked at what each vendor publishes: pricing, plan limits, feature coverage for llm observability tools and stated drawbacks. No hands-on testing is claimed. Figures are shown as the vendors publish them. Billing terms, seat minimums and add-ons can change the real cost. Order on the page follows fit for the typical buyer first, then price transparency and free access. Your own requirements should decide the final pick.
Not all of them. Opik, LangSmith, Weights and Biases Weave and others publish no starting price, so expect a quote request for those. The rest, Arize Phoenix included, show a starting price, though seats and billing terms still change the final cost.
The order follows typical fit for llm observability tools, with Opik first and Braintrust last, then price clarity and free access. It is editorial judgement, not a score, so another team could rank the tools differently.
Self-hosting fits teams with data residency needs, strict privacy rules, or existing infrastructure skills. For everyday use, cloud hosting fits smaller teams that want quicker setup and less maintenance, even if traces leave their network.
For a small prototype, entry-level logging may be enough. Once the chatbot serves real users, takes care of varied inputs, or uses retrieval and tools, observability becomes handy for debugging, cost control, and quality checks.
No single tool suits every team. Opik comes first because it fits open-source evals and tracing, while LangSmith suits LangChain and LangGraph tracing and Weights and Biases Weave suits ML teams adding LLM traces. Match the tool to your own size, budget and workflow before deciding.
LLM observability is the practice of tracing, measuring, and evaluating LLM applications in production. It handles prompts, model calls, tool calls, retrievals, latency, token cost, and output quality.