Grafana RCA Workbench: Reconstructing the Failure, Not Just Spotting the Symptom

An alert fires. Latency is too high. The error rate is increasing. An SLO is burning through its error budget. A Kubernetes pod keeps restarting.

We know that something is wrong. But what exactly?

This sounds like a simple question, but an alert usually shows only a symptom. The service that triggered the alert is not necessarily the component that caused the problem.

A frontend error might originate in an application service. The application service might fail because PostgreSQL has reached its connection limit. The database might be overloaded because a recent application change is leaking connections.

Everything is connected, but the alert shows only one piece of the failure chain.

This is where Grafana RCA workbench can help. It puts the affected services, infrastructure, insights, and telemetry in one investigation context.

How incident investigations usually look

Let's assume that we receive an alert about high latency in the product catalog service.

A typical investigation might look like this:

1. Open the application dashboard.
2. Find the correct service.
3. Set the incident time range.
4. Check errors and latency.
5. Open the Kubernetes dashboard.
6. Check whether the pods are restarting.
7. Search the application logs.
8. Find an errored trace.
9. Check the database dashboard.
10. Ask the frontend team whether users are affected.

Each step makes sense. The problem is that every step might require a different dashboard, query, browser tab, or even a different tool. We need to copy the time range between screens, remember the correct labels, and know enough about the architecture to decide where to look next.

During a production incident, this takes time. And usually, time is the one thing we do not have.

Starting directly from the alert

A Grafana Cloud notification can be configured to include a link to RCA workbench. Grafana documents this flow for Knowledge Graph notifications and SLO alerts using an Alertmanager template. The link can carry information about the affected entity and the relevant time range.

• Configuration guide

Instead of starting with an empty dashboard, we start from the signal that informed us about the problem.

After clicking the link, we are taken directly to the RCA workbench.

This is a small but important detail. We do not have to search for the affected service, select the environment, or manually reconstruct the incident window. We can start investigating immediately.

What does the RCA workbench show?

RCA workbench puts the selected services and infrastructure components on a shared timeline.

It combines insights and gives us access to the associated:

• application metrics,
• infrastructure metrics,
• frontend telemetry,
• database metrics,
• logs,
• traces,
• profiles,
• and entity-specific dashboards.

The timeline is the most important part.

Root cause analysis is not only about listing everything that failed. We need to determine the order in which it failed.

RCA workbench displays insights from related entities over time, helping us see which problem appeared first and which problems followed. We can also move from the timeline to metrics, logs, traces, profiles, dashboards, and the entity graph without rebuilding the investigation context.

• RCA workbench documentation

For example, we might see the following sequence:

1. A deployment or configuration change occurs.
2. Database connections begin to rise.
3. One application endpoint becomes slower.
4. Application instances start restarting.
5. The frontend begins returning errors.
6. The latency alert fires.

If we looked only at the alert, we might blame the application service. Looking at the whole sequence shows that the application alert might be only the last visible symptom of a larger failure.

How does Grafana know which services are connected?

RCA workbench is part of the Grafana Cloud Knowledge Graph. The Knowledge Graph discovers entities such as services, pods, nodes, workloads, databases, and frontend applications. It also builds relationships between them — for example, which pod runs a service, which node hosts the pod, and which services communicate with each other. It continuously generates insights about errors, saturation, failures, anomalies, and configuration changes.

The discovery process is metrics-first. Metrics and their labels drive entity discovery, relationship mapping, and insight generation. Logs enrich the investigation, but they do not drive discovery. Traces are not required to activate the Knowledge Graph, although they provide richer request-level context.

• Getting started guide

So the practical rule is straightforward:

The more complete your telemetry is, the more complete your root cause analysis can be.

Grafana cannot correlate a service that it cannot discover or inspect.

Investigating the incident layer by layer

There is no single investigation order that works for every incident. However, it is useful to start with broad signals and gradually move towards more detailed evidence. Let's go through an example.

Step 1: Check the timeline

The first step is to look at the sequence of events.

Which insight appeared first?

Was there a deployment, feature flag update, or configuration change shortly before the failures began? Did the database become saturated before or after the application started returning errors? Were all services affected simultaneously, or did the issue propagate from one service to another?

At this point, we are not proving the root cause yet. We are building a hypothesis. For example:

A configuration change caused a problem in the product catalog service, which then exhausted database connections and affected the frontend.

Now we need to validate or reject that hypothesis.

Step 2: Check application metrics

Application metrics tell us whether the alerted service is behaving abnormally. For a service, we usually begin with the RED signals:

• request rate,
• error rate,
• duration.

Suppose the latency alert fired for the product catalog service.

When we open its application metrics, we might discover that the problem is limited to the /ListProducts operation. Other endpoints are still working normally.

This already narrows the problem. Instead of investigating the whole service, we can focus on one operation and its dependencies. We can now ask more precise questions:

• Did traffic to this endpoint increase?
• Did the errors begin after a deployment?
• Is the operation failing inside the service?
• Is it waiting for a downstream dependency?
• Are successful and failed requests following different paths?

Metrics tell us what changed. They usually do not tell us why.

Step 3: Check the frontend impact

A backend service can be unhealthy without causing a major customer-facing issue. It can also appear only slightly degraded while breaking the most important user journey in the application. For example, errors in the product catalog service might affect:

• the home page,
• product search,
• product details,
• the cart,
• checkout.

When Grafana Frontend Observability is configured, frontend applications can become part of the investigation. From a Frontend Observability dashboard, we can also open the affected entity in RCA workbench.

• Frontend integration guide

This answers an important question:

Is this only a technical anomaly, or are users unable to complete a business operation?

The answer influences the priority of the incident.

A rarely used internal endpoint returning errors is one situation. Customers being unable to purchase products is another.

Step 4: Check Kubernetes and infrastructure metrics

Suppose RCA workbench shows that the application pod is repeatedly restarting. A common first hypothesis is an out-of-memory error. But a common hypothesis is not necessarily the correct one. We can open the Kubernetes view and check:

• memory usage,
• memory limits,
• CPU usage,
• restart counts,
• pod status,
• container failures.

If memory usage is well below the configured limit, we can probably rule out an OOM kill. This is an important part of root cause analysis. We are not only looking for data that confirms our theory. We should also look for data that proves it wrong. At this stage, we know that the pod is crashing, but we still do not know why.

Step 5: Check the database

The product catalog service communicates with PostgreSQL, so the database is the next logical layer. The database metrics might show that active connections are repeatedly approaching the configured maximum.

This is a useful finding, but it does not prove that the database caused the incident. The same metric can support several explanations:

• traffic increased unexpectedly,
• the connection pool is incorrectly configured,
• application connections are not being released,
• restarting application instances repeatedly reconnect,
• another service is consuming the available connections,
• the database connection limit is too low.

The timeline helps us distinguish between these cases.

If database saturation begins before anything changes in the application, the database might be the initiating component.

If saturation begins immediately after a deployment or feature flag change, the application might be responsible for exhausting the database.

The database can be unhealthy and still not be the root cause.

Step 6: Inspect traces

Metrics show that requests are failing. Traces show where they are failing. By opening an errored trace, we can follow a request through the system and inspect every operation along its path.

For example:

The trace might show that most of the request time is spent waiting for PostgreSQL. It might also contain an error on the database span. We can also compare successful and failed traces. Suppose every failed trace contains the /ListProducts span, while other operations complete successfully. This is much stronger evidence than a general service-level error metric.

We now know:

• which operation is failing,
• which dependency it calls,
• and where the error appears in the request path.

But we still need the exact error message.

Step 7: Inspect logs

Logs usually contain the lowest-level explanation.

Because the service and time range are already selected, we can inspect logs without constructing a new query from scratch.

The application logs might contain a message such as:

Shortly afterwards, we might see:

Now we can connect the signals:

1. A code or configuration change modified the database access path.
2. Connections were not handled correctly.
3. PostgreSQL reached its connection limit.
4. Requests to /ListProducts started failing.
5. The application encountered an unhandled error and restarted.
6. Frontend pages depending on the product catalog returned errors.
7. The latency alert fired.

This is no longer only a correlation visible on a dashboard. It is a root cause hypothesis supported by:

• the event timeline,
• application metrics,
• frontend signals,
• Kubernetes metrics,
• database metrics,
• traces,
• and logs.

Using Grafana Assistant

The manual investigation is useful because it shows how the failure propagated. During a real incident, however, we might want to reach the same conclusion faster.

From RCA workbench, we can click Analyze RCA Workbench to open Grafana Assistant with the current investigation context.

The Assistant understands:

• selected entities and their relationships,
• active insights,
• the selected time range,
• timeline patterns,
• environment and namespace filters.

It can query metrics, logs, and traces, form hypotheses, explain relationships between failures, and recommend the next investigation steps.

We can ask questions such as:

We can then ask follow-up questions and narrow the investigation without manually navigating through every signal.

There is one important distinction.

The Assistant generates a hypothesis. It does not remove the need to verify the evidence. An AI-generated explanation might direct us to the correct service, trace, or log message much faster. The final decision — especially one involving rollback, failover, or production configuration — should still be based on the underlying telemetry.

Which Grafana platforms support RCA workbench?

At the time of writing, the documented RCA workbench experience is a Grafana Cloud capability delivered through the Grafana Cloud Knowledge Graph.

However, starting with Grafana v13, Grafana Assistant can be installed in a self-managed Grafana deployment and connected to a Grafana Cloud Assistant backend. However, this does not make the Cloud Knowledge Graph or RCA workbench available as a native self-managed feature.

Summary

An alert tells us that something is wrong. It rarely tells us what happened first.

Grafana RCA workbench helps reconstruct the failure by placing related services, infrastructure components, and insights on one timeline. In our example, the investigation moved through several layers:

1. The alert identified the visible symptom.
2. The timeline showed the sequence of events.
3. Application metrics narrowed the problem to one endpoint.
4. Frontend telemetry showed the customer impact.
5. Kubernetes metrics ruled out an out-of-memory failure.
6. Database metrics showed connection saturation.
7. Traces identified the failing dependency.
8. Logs exposed the concrete application error.
9. Grafana Assistant could analyze the same context and propose the most likely failure chain.

The main advantage is not another dashboard. It is the connection between dashboards, entities, and signals. Instead of proving that each individual component is innocent, we can follow the evidence through the system and identify what actually needs to be fixed.

Reviewed by Grzegorz Kocur