// How-to · Evaluate and monitor AI quality
Self-hosted LLM observability: trace every call and agent step
Recording every model and tool call’s input, output, latency and tokens as traces is what lets you debug. Open-source options can be self-hosted so data stays on your network.
verified 2026-10-102 official sources
Scenario: Evaluate and monitor AI quality →
Symptoms
- When a multi-step or multi-agent task fails you cannot tell which step broke
- You do not know which feature uses the most tokens
Common causes
- Only final results are logged, not intermediate steps
- Each service logs in a different format and nothing links up
How to fix
- Pick a self-hosted platform: Langfuse (official Docker Compose and other deployment options), Arize Phoenix and others
- Add its SDK to your app, or instrument with OpenTelemetry-based OpenLLMetry
- Follow the OpenTelemetry GenAI semantic conventions so you can switch back-ends later
- Teams already on OpenTelemetry can forward everything through the Collector
- Traces may contain user data and secrets; configure redaction and retention
Official references
Compiled from official documentation and public discussions on 2026-10-10; not yet reproduced in our own environment. Follow the official docs if they differ.