Evaluating agentic systems is a multi-dimensional challenge that requires a multi-layered framework. A robust evaluation system must verify both functional and non-functional requirements. The system should verify whether an agent is fulfilling its core objectives, while performing efficiently, safely and with low latency. Furthermore, the system must account for scalability while ensuring the agent's underlying reasoning remains transparent and traceable. Those different evaluation layers allow, to evaluate the model both from the technical and strategic business perspectives.
In this article I will focus on following evaluation layers:
- Reliability and quality
- Performance
- Costs
- Safety and ethics measures
- Auditability and explainability
- Persistence and memory
Reliability and quality metrics
Most important questions from the functional system point of view:
- Goal alignment. Is the agent fulfilling the primary objective?
- Accuracy. Is it doing the task accurately, correctly?
- Consistency. Does it produce similar quality and content across multiple runs with the same input?
- Groudness. Is the agent not hallucinating?
- Constraints adherence. Does the agent stick to the specific length, format, tone requirements?
This layer focuses on evaluating, whether the agentic system does what it was designed to do and how well it does it?
While specific metrics vary by use case, several tools and techniques are industry standards. We can leverage libraries like Ragas for pre-defined specific metrics or employ LLM-as-a-Judge to evaluate complex agentic reasoning and actions. These techniques should be paired with a validation dataset. The dataset can consist of hand labeled or LLM labeled real world examples or can be fully synthetic, created by the LLM. For the format and output integrity validation, the Pydantic models can be used.
Example metrics:
Blue score
Measures word-level similarity between the agent’s response and a reference text. It ensures that the statements made are not false positives, relative to the source/reference answer. In that aspect the BLUE score focuses on answer precision. It is strongly connected with verifying the groundness of the response with respect to facts or source documents.
Pros: | Cons: |
|---|---|
|
|
Rouge Score
Similar to the BLUE score, but focuses on the missing facts (recall). It verifies whether the response captures all aspects present in the reference response or retrieved documents context.
Pros: | Cons: |
|---|---|
|
|
Embeddings similarity
It is used to quantify how similar the query and reference text, answer, fact are. The text is first converted to the vector representation using the embedding model and then for example cosine similarity can be computed between two numerical vectors to quantify the similarity.
Pros: | Cons: |
|---|---|
|
|
Groundness/faithfulness
Measures, whether the agent is hallucinating while making the response or whether it stays true with respect to provided information, context and facts. The popular metrics used to quantify groundess are BLUE and ROGUE scores or LLM-as-a-Jude. The last one is especially desired, due to its ability to capture semantics, intent and synonyms and does not rely on exact word matching.
Accuracy
Accuracy quantifies the goal achievement. We assign binary value 0 or 1, based on whether the agent failed or succeeded on the task or decimal value between 0-1 on partial hits. It might be that the agent completed the task partially. In that case we need to assign the value between 0-1. For accuracy assessment we can use LLM-as-a-judge in most situations. Sometimes we can rely on user feedback (thumb up, thumb down), task success rate (resolved tickets) or click through ratio for recommendation agents.
Recall
Recall measures an agent 's ability to not miss things. In recall we focus on monitoring the false negative cases. Recall tells us how many times the agent misses the task or user intent and does not take an action, when it should. It’s a measure of agent reactivity, ability to take actions, not missing the tool use and not missing any facts.
In retrieval tasks recall@k is often used, to verify if the relevant information exists within the top k retrieved chunks. If the agent is given multiple tasks it can also be used to quantify how many tasks were missed.
Precision
Precision is complementary to Recall. It focuses on false positives. It measures how often an agent took the wrong action, used a wrong tool or made a positive statement, when it shouldn’t.
F1 score
F1 score combines recall and precision into a single value. One value makes it easier to compare agent and prompt versions between runs.
F1 = 2*recall*precision/(recall+precision)
@k
Used to quantify how many attempts or retrieved items are needed to find a correct answer. For agents, this helps determine if we need to increase the retrieval window or allow for multiple reasoning loops to reach a successful outcome.
Trajectory metrics
In a multi step reading system, we can use LLM-as-a-Judge to verify the model reasoning and flow correctness.
Answer relevance
Measures how much of the response actually addresses the original prompt. Here similarly to groundness we can use BLUE, ROUGE, embedding similarity or LLM-as-a-Judge to quantify the relevance.
Negative Constraint Adherence
Agents are usually bad at negative constraints over long trajectories. For example, "Do not mention Brand X". In how many conversations or after how many turns, the agent stops to follow negative constraint adherence?
Thoughts on Reliability and quality metric
In agentic systems we naturally want to evaluate final agent response correctness. However, if we want to track the bug, or improve on the final score, the final response metric does not give us a good understanding of what and where to improve. Agentic systems are usually multi step processes. In order to have a full understanding of the agentic system quality we should evaluate the final and each intermediate step. Only this approach highlights spots for improvements.
Performance evaluation
The agentic system has to be fast and scalable. We might want to monitor following metrics:
Example metrics
Average execution time (latency)
The average time from the request to the agentic system response. It is crucial for live systems, and less but still important for background executed agents.
Throughput
How many requests per second can the agent handle?
Average Steps per Task (Trajectory Length)
How many API calls, tool calls does the agent make per task? The agent might be requested to complete one or more tasks in one request. If it calls irrelevant tools, or with wrong parameters, or if it finds itself in a loop calling the same APIs, then the agent is not efficient, even if it completes the task in a pre-defined time.
Time per Step (TPS)
How much time does the agent spend per task? It helps to track down bottlenecks and optimize the longest performing tasks.
Tool-Call Success Rate
Does the agent call tools with correct parameters?
Tools accuracy, recall, precision, concordance index
Are the correct tools called? Are those tools called in the correct order?
Tool-Call Fallback Rate
Measures how often the agent successfully switches to an alternative tool or a "safer" model when its primary path fails.
Self-Correction Success Rate
When the agent receives an error (e.g., a 404 from a search tool or a syntax error in code), does it fix the query and try again, or does it crash?
Mean Time to Recovery (MTTR - Agentic)
How many "re-planning" steps does it take for the agent to get back on track after a failed action?
Context length
How the agent manages the context. Does it blow up the context or keep it reasonably short.
Scaling
Horizontal: How many parallel Agent Workers can your infrastructure spin up before the API rate limits kick in?
Time to First Token
The classical metric for LLMs evaluation. Can be used to indirectly measure agent performance, through the speed of the LLM, the agent is relaying on.
Autonomous Completion Rate (ACR)
Percentage of tasks finished without any human intervention.
Cache Utilization Rate
How much of the context was served from the cache.
Metrics specific for self-hosted solutions
GPU profiling
GPU profiling allows finding latency and throughput optimisation opportunities. It can help identify bandwidth bottlenecks, paralyzation opportunities or communication overhead in multi-GPU clusters.
KV-cache usage patterns
KV-cache management is a big challenge in self-hosting solutions. The more we cache, the faster and cheaper the LLMs response generation is. On the other hand cache takes space of the expensive VRAM on GPUs. Therefore we would like to cache only most frequently used prompts or prompts that are most likely to be reused soon. We may also use a few KV-cache optimisation techniques including cache pruning, encoding or optimizations coming from model architectures.
Great tools for self-hosting solutions profiling are NVIDIA Nsight Systems profiling solutions and Prometheus with Grafana for Kubernetes deployments, with combination with Nvidia TensorRT for optimisation.
Thoughts on performance evaluation
Most of those metrics can be calculated based on agent traces. We can use Langfuse, W&B Weave or native cloud provider tools, like for example Agent Development Kit to gather agent traces step by step. Then we can use those traces, to compute times per span, find loops or use LLM-as-a-Judge to evaluate trajectories.
Costs metrics
Example metrics:
Average Cost per Task
How much is the single agent task completion?
Average Cost per successful task
Compared to the previous metric, we divide the total cost of N runs, by the number of successfully completed tasks, not by N.
Cache Hit Rate
How much of the requests/tool calls or prompts can we reuse from cache. In case of prompt caching we focus on how many tokens can we reuse from cache and in terms of tools, request caching, we focus on reusing the same requests.
Cost per tool execution
How much does the tool execution cost? Helps to track the most expensive tools and optimize them.
Planning efficiency
Does the agent choose the most optimal trajectory? We can use LLM-as-a-Judge, to verify the agent trajectory.
Cost of failure
How expensive are wrong or failed agent actions. It also includes the loss of potential incomes, because of lost customers.
Token Efficiency Ratio
How many tokens does the agent require on average to complete the task? This metric should be normalized by the task complexity to provide measurable results.
Safety and ethics measures
While performing actions, agents need to take special care about sensitive data, customer data and be resilient to attacks.
Example metrics:
Attack Success Rate (ASR)
Adversarial Attacks like:
- Competitor redirect. Make the agent inject some data or redirection instructions, to recommend for example competitors services.
- Data exfiltration. Reveal customer data.
- Denial of service. Make an agent inject some data into memory, so service will fail on using these resources.
- Harmful content. Make agents violate their safety guidelines, ethical standards, or legal requirements, for example answering in a hate language, hallucinating and giving 90% promo code, returning unsafe links to the user.
Those attacks usually rely on prompt injection to fool the agent and to convince it to make harmful actions, reveal customer data or inject unsafe content into the agent managed resources.
Percentage of PII detected and anonymized
What % of PII were automatically masked? The good practice is to let agents use anonymized data, so there is no risk of agents using it in a harmful way. Standard customer PII to detect and anonymize includes: emails, names, purchase history, passwords, account numbers, phone numbers, ID numbers and so on. If an agent must perform operations on this data, sensitive information should only be decrypted at the final step, ideally post-execution. Both encryption and decryption must be deterministic and handled programmatically, rather than relying on LLM decision-making.
Percentage of policy violations
Severity score of the agent violating the company policy. We can use LLM-as-a-Judge to evaluate agent actions and responses against the policy documents.
Over-Privileged Access Rate (OPAR)
How often the agent has access to confidential data that were not necessary for the successful completion of the specific task. For example accessing a person ID number, when only a ZIP code was needed is a violation.
,where \ is a set difference. The perfect score is 0, while everything greater than zero indicates higher security risk
Unintended Harmful Behaviors
How often the agent is impolite, rude, and makes inappropriate questions? Does it detect and de-escalate user frustration, or does it aggravate the user through repetitive or robotic responses? Does the agent can adapt to the user mood, tone and style, capture user intent and preferences? For non user/customer facing agents, do they take proper care of companies resource allocation while taking actions or pushing results to db? Does the agent saved content meet ethical and safety measures? Does the generated or saved content inadvertently violate safety standards (e.g., generating biased comparisons or toxic summaries) even without an explicit adversarial attack?
Tool Call Authorization Error Rate
How often does the agent attempt to call a function outside its current scope or permissions and the access is successfully rejected?
Intent-Permission Alignment Score
Even if an agent has permission to a database, did the user's intent authorize that specific use? For example delete_order() might be a valid tool for the agent, but it falls outside the Intent Scope of the current session. Measure the percentage of tool calls where the Action matches the User's Original Request.
Goal Hijacking / Indirect Prompt Injection
Ability to ignore malicious instructions embedded in third-party content (e.g., websites the agent researches).
Human-in-the-Loop (HITL) Bypass Rate
Did the agent ever perform a restricted action without triggering the required human approval?
Thoughts on Safety and ethics measures
Great framework for AI Agents guardrailing and safety evaluation is the NVIDIA Nemo Agentic Toolkit. It wraps up any agentic code and adds a configuration layer for any pre-execution, prompts safety evaluation, PII masking and prompt injection detection. It also provides powerful tools for red-teaming testing, with predefined main attack strategies and customisable options for creating own test attacks.
Auditability and explainability
Being able to understand and explain agents' action is one of the most important aspects for debugging, optimising agents performance and meeting legal requirements. There are already multiple tools on the market specialised in agents tracing and explainability. Among others: Langfuse, LangSmith, Braintrust, Weights & Biases (W&B) Weave, Galileo AI, Arize Phoenix, Maxim AI.
Trajectory-Level Explanations
Tracing is important for understanding agent actions and trajectory, but it does not reveal reasoning behind taking certain actions. The agent should be instructed to provide a reasoning behind each action. Reasoning is then logged along traces and can be further analysed on a case-by-case basis or with LLM-based aggregation strategies.
Source-to-Output Mapping
The agent can be also asked to provide a link back to the specific chunk of text in the retrieved document or search result, to prove it claims.
Tool Data Lineage
If the agent uses multiple sources to calculate and return or create a new value, the data source and middle transformation steps should be tracked, along with the code and version of the interpreter used. Lineage can be implemented as a part of the data platform using for example OpenLineage.
Immutable Audit Logs
For legal purposes, the critical agent actions (like publishing a post or deleting data) should be logged in a tamper-resistant manner. Tools like:
- Fiddler AI: Specialized governance platform that provides SOC 2 Type II audit trails
- Braintrust (Enterprise version): Provides comprehensive, immutable traces that link every decision to a specific version of code, prompt, and dataset.
Ensure logs are tamper resistant.
Moreover Microsoft Agent Governance Toolkit (Open Source), released in early 2026, provides runtime security governance.
Tool Models Explainability
Another important aspect is the explainability of tools used by the agents, especially if the tool uses AI models to make decisions or predictions. For tabular or vision models we can use Shap library or FoxAI, an open source project to explain AI underlying decisions. It has already proven in multiple projects its ability to troubleshoot, identify and fix bugs, biases and bottlenecks in dataset-model pipelines as well as boost users' trust in model predictions. In case of tools depending on LLMs reasoning, it’s best to instruct the model in a prompt to provide the reasoning behind the decision and log those reasonings using agent tracing tools.
Persistence & Memory
Redundancy Rate
Does the agent ask the user for information it was already given in a previous session? Give the agent the same task across 5 different sessions and measure how much the recalled facts vary.
Retrieval Relevance (Long-term Memory)
Recall@k for evaluating the retrieval performance: does the feathered context contain all required facts to answer the solve the prompt?
Context Precision
Out of all information chunks gathered (web, databases, etc.) how many were actually used in the final reasoning and answer?
Context Loss Rate
Does the agent start ignoring initial instructions as the conversation grows?
Memory Growth Rate
If we are saving past experience, user preferences, at some point the DB stored facts rate should saturate, which indicates that we learned most about the task/user. Does it actually saturate in tests or live monitoring scenarios?
Grounding
Does the agent ground its response based on information from memory or does it hallucinate?
For memory evaluation we can use both regular Langfuse traces and LLM-as-a-Judge, as well as Ragas for specific metrics and LangGraph Stateful Memory, for short term memory evaluation.
Online vs Offline evaluation
Agents should be evaluated before production rollout in an offline meaner and while on the production in the online meaner. The offline evaluation should be border and deeper in terms of datasets, metrics and strategies used, to ensure agents quality in all layers of the multi-layer evaluation systems. The online evaluation should be lightweight, to not add much additional latency. Often the online evaluation is run on the subset of the real traffic to ensure robust agent performance in real world and detect potential performance degradations as soon as possible. Online evaluation additionally allows for user feedback gathering, which is a powerful learning and self improvement signal for AI Agents.
Overall agent performance
Evaluating an agent system requires a multi-layer evaluation system, to ensure the agent is reliable, costly effective, time constrained, auditable, explainable and efficient. Holistic approach allows us to build and deploy robust agents that will provide real business value. Apart from the metrics and approaches presented in this article, specific agentic benchmarks is a great approach to evaluate your solution against SOTA agentic systems.
Reviewed by Michał Zaręba
