Key Takeaways• Root-cause confidence and action confidence are separate: knowing what caused an incident doesn’t tell you which response should run. • Trust in the diagnosis rises with full-stack visibility and historical evidence that you’ve reached the cause, not another correlated signal. • Before automating a response, teams need to weigh production risk, business impact, reversibility, human approval, and whether the fix worked. • The observability and remediation history teams build now becomes the context future agents need to evaluate options instead of just following rules. |
You can trust the root cause and still not be ready to automate the response.
Knowing what caused the problem doesn’t tell you which response is appropriate. The right response still depends on the conditions around that incident.
Is it production? What’s the business impact? What happens to the user experience? Has this response worked before? Can you roll it back if it goes wrong?
That’s the difference between root-cause confidence and action confidence.
Before AIOps moves from diagnosis into action, it needs enough technical and operational context to know that this response makes sense for this incident.
This article builds on my earlier article on observability maturity, which explains what needs to be in place before teams move toward self-healing AIOps.
CPU and memory running hot tells you there’s a problem.
It doesn’t tell you what caused it.
A lot of teams stop there. CPU goes high, RAM goes high, so they call the infrastructure team at 2 am.
We see this when a code or configuration change gets pushed into production. Database calls increase, CPU and memory spike, and infrastructure starts running hot.
You can restart that server 13 times, but if the code issue is still there, the problem will keep happening.
The question I always ask is: Do we have all the data from all the impacted systems?
You need to consider the full stack. If an application isn’t working properly, how many systems does it touch? Can you see the web server, the backend database, load balancers, firewalls, networking, and everything in between?
If you’re observing at least 90% of that data, the confidence meter goes up. If you can only see the web server but not the backend database, or you can’t see beyond the load balancer, the confidence meter goes down.
Then validate the root cause against history.
When this happened before, what did you see? That may be in the data, or it may be tribal knowledge from someone who remembers the same problem happening last week or last month.
That history gives you another way to validate the root cause and increase your confidence in the diagnosis.
Finding the root cause doesn’t mean you’re ready to automate the response.
You also need to know whether that response actually fixes the problem. Has this happened before? When it did, what fixed it?
I start with what I call manual automation.
Write the script, but trigger it manually when the issue happens. See whether it actually resolves the problem before you let it run on its own. Then ask, “What happens if the action goes wrong?” And, “Can you quickly roll it back?”
If you can’t prove the response works, or you can’t easily reverse it, I’d be hesitant to fully automate it.
Even when you trust the root cause and the remediation, you still have to decide whether that response makes sense for this incident.
The same response can carry very different risk depending on where it runs and what it affects.
Start by understanding the stakes:
Those conditions determine how cautiously you should handle the response.
You also need the business context.
Take an e-commerce site. The login, checkout, and credit card service may be working. But if product search or adding products to the cart fails 40% of the time, a working checkout has little value.
That’s where looking only at the technical systems can get you into trouble. You need to understand what the user is experiencing and what matters to the business.
You also need to know what normal looks like.
If a system regularly runs at 70 or 80% on the 1st, 15th, or 30th of the month, that may be expected behavior. The same spike on the 10th or 20th may indicate an anomaly. So, you need enough history to compare week over week, month over month, and eventually quarter over quarter.
Then decide when a human needs to be involved:
That depends on how much you trust the process and what happens if the response is wrong.
At Arctiq, we start with the technical teams and ask who else needs to be involved.
Who gets called at 2:00 in the morning when the application goes down? That may be the application owner.
But who gets asked how much revenue was lost during the outage? That may be somebody completely different.
These questions help surface the people who own the technical problem and the people who own its business impact. You need both perspectives before you can turn that judgment into automation.
The technical team knows how the application behaves. The business owner knows the total cost of an outage. If you’re going to automate the response, you need that knowledge in the process.
We surface that technical, business, and tribal knowledge and turn it into the Ansible playbooks and guardrails the automation can follow.
We ask:
Take a restart, for example. You don’t want a script retrying the same failed action forever because that’s what you told it to do. Maybe it tries two or three times. If it keeps getting the same error, stop and trigger the on-call team because now you have an exception.
You also have to define what a successful response looks like.
Take the checkout example: Dynatrace identifies the JVM heap issue, and we can map that diagnosis to the right Ansible playbook. Ansible takes the workload out of the load balancer, restarts the service, and puts it back.
But that doesn’t mean you’ve fixed the problem.
Dynatrace still needs to check the system health afterward. In this case, response time dropped from around eight seconds to one or two seconds, and the error rate returned to baseline.
The final step is proving the action worked, not just proving that the automation ran.
You can’t jump straight to autonomous operations.
You need the monitoring, observability, root-cause history, automation, business KPIs, and the teams around it first. Otherwise, you’re asking the system to make decisions without the context it needs.
Today, most remediation is still rule-based. If this happens, run this playbook.
Agentic AIOps starts to change that. Now the system may have to decide:
Very few companies are letting agents trigger those changes today. Most of what I’ve seen is still being tested in lower environments while the system learns, with a human approving the action.
It needs history to make those decisions:
You have to collect that data before you need it.
We’ve seen customers take existing Dynatrace data, root-cause analysis, and log history and feed it into agents. Instead of waiting six or eight months for the agent to learn the environment from scratch, they get useful context within weeks.
But that only works if the data is already there and you trust it.
If you’re still moving from basic monitoring into observability, you still have to build that history first. The organizations that have already been collecting it are in a much better position when they’re ready to move into more autonomous operations.
Before moving toward automated remediation, understand your applications' current maturity. Use Arctiq's 15-point Autonomous Operations Readiness Assessment to identify gaps across observability, intelligence, and automation, and determine where to focus next.
For organizations ready to take the next step, connect with Arctiq to explore how Dynatrace and Red Hat Ansible Automation Platform can support automated remediation and autonomous operations.