AI Dev Skills
Observability & Monitoring
What is it?
Tracking every LLM call, prompt, response, latency, cost and quality metric in production. Observability gives you full visibility into what your AI system is actually doing at runtime.
Why it matters for AI PMs
You can't improve what you can't measure. Cost overruns, quality regressions, and silent failures are completely invisible without observability. Every production AI incident investigation starts here.
The 2026 landscape
Langfuse, Phoenix, and OpenLIT are the leading open source tools. OpenTelemetry is becoming the standard tracing protocol. The space has consolidated significantly in 2025.
What strong coverage looks like
Having 3+ observability repos signals a team that takes production AI seriously. They are monitoring costs, tracking prompt versions, and running LLM-as-judge evaluations on live traffic.
Your library coverage (0 repos)
No repos in this skill area yet.
Key concepts to know
- •Traces and spans for LLM calls
- •LLM-as-judge evaluation
- •Cost per token tracking
- •Prompt versioning and A/B testing
- •Latency percentiles (p50, p95, p99)