The Observability Gap
Most AI agent deployments measure what is easy: response latency, token cost, error rate. These are necessary but insufficient. An agent can be fast, cheap, and error-free while still giving users wrong answers. You need domain-specific reliability metrics.
Metric 1: Task Completion Rate
Did the agent actually complete what the user asked? We classify each conversation as complete, partial, or abandoned. Complete means the user got a usable answer and did not immediately rephrase or escalate. We target 85%+ completion rate.
Metric 2: Tool Call Correctness
For agents that call external tools (APIs, databases, calculators), what fraction of tool calls used the right tool with the right parameters? We log every tool call and sample 5% for manual review. Current benchmark: 94%.
Metric 3: Escalation Rate
How often does the agent say "I cannot help with this" or transfer to a human? Some escalation is healthy ÔÇö the agent should know its limits. Too much means the agent is not capable enough. Too little means it is overconfident. We target 8ÔÇô12% escalation rate.
Metric 4: Retrieval Precision
For RAG-based agents: of the chunks retrieved, what fraction were actually relevant to the query? We measure this via LLM-as-judge: a separate model evaluates retrieved chunks against the query. Below 70% precision triggers a retrieval pipeline review.