What Has to Be in Place Before an AI Agent Gets Production Access

Recently, AWS introduced mandatory peer review for production access following the Cost Explorer incident, which exposed what can go wrong when access controls and governance fail.

Misconfigured access controls and human error gave an AI agent elevated access and contributed to a 13-hour production disruption. They fixed it afterward, and that sequence matters for AI agents. Teams rush to connect AI agents to business infrastructure, running them inside production systems, internal workflows, and operational tools.

The assumption organizations are operating under is that they can connect the agent now and sort out the controls later. That assumption breaks the moment an agent has broad permissions because these systems aren’t fully baked AND they operate on probability(rather than facts and wisdom) in decision making.

Once they get access, they can misunderstand the request, make the wrong decision, and cause damage quickly. To protect your organization, you need guardrails in place before broad access reaches production.

AI Agent Access Creates a New Failure Mode

Recently, an AI agent tasked with managing email cleanup began deleting messages despite being instructed to wait for approval before taking action.

Prompts and approval steps can help to a degree, but they do not solve the core problem. They evaluate actions one at a time, without the full context behind them. On paper, a single deletion may look harmless and so may the next one; however, by the time you realize those small approvals are part of a larger destructive sequence, the agent may have already deleted something important.

Humans judge singular incidents but the system acts in totality. It’s like sparks around a fire pit. One spark doesn’t look like a problem but then another lands. And another. Judge each one in isolation, and you miss the larger fire starting to catch.

Put the Controls Where They Can’t Be Bypassed

Governance can’t depend on the agent or the LLM policing itself. The moment an agent gets production access, it goes beyond a prompt or approval problem to an infrastructure problem. We don’t know how these systems interpret or manipulate their own controls, so the control point has to exist outside the context window of the agent’s thought process.

The broader issue is that there is no single, universally adopted standard for agent security, authorization, auditing, governance, and observability across this space. Everything is still preliminary studies or analysis. And the code ships so fast that even if standards existed, teams would be naive to assume they could be implemented consistently. So architecturally as security professionals, we have to solve this externally.

Kubernetes is the practical way to do it. It gives you an isolated environment and a governed way to control how things flow in and out.

Even if security sits on both sides of the model, the hard question remains: how do you monitor internal context? That is where APM-like solutions still need to catch up.

You need to monitor both the immediate action and the full context of what the system is trying to do, much like application performance monitoring of web apps and services today.

That’s very different from running an agent flat on a Windows machine or Linux host, where it can end up with access to the whole file system, host ports, and anything else its permissions allow. Still, isolation by itself is not enough because you have to govern how it communicates with other systems.

Teams are trying to put agents into real systems, thinking they’re doing it in a controlled environment. They’re not. It’s like lighting a fire on open ground and assuming you can run for water if it gets out of hand.

Kubernetes gives you something closer to a fire pit with a fire extinguisher in your hand. The workload is isolated, traffic is governed, and the kill switch is close to the source.

In enterprise environments, you could use a platform like Red Hat OpenShift. If the system starts doing something it shouldn’t do, you can stop that workload without shutting everything else down.

The same logic applies to service-to-service traffic:

• Route traffic through a controlled gate, (i.e. AgentGateway)
• Enforce identity, authorization, and audit outside the agent
• Use network policies to limit what the agent can talk to
• Keep the kill switch close enough to stop bad behavior fast

The Minimum Bar for Any AI Agent with Production Access

If you’re going to give an agent production access, there has to be a minimum bar; not a vague idea of governance. You need actual controls in place before this goes live. If you accept the earlier point that controls must live outside the runtime, this is the minimum checklist that follows from it.

Verify What’s Running

First, make sure the binaries are the ones you think they are, they come from the source you think they come from, and those are the only models allowed to run. Then, protect the workload at runtime.

What you need to ask:

• What does normal look like?
• What services does it usually access?
• What files does it touch?
• What ports are normally open?

If it starts doing something outside that range, kill switch it or at least audit log it and alert on it. The goal is to keep individual components from becoming runaway rogues.

Tooling to consider:

• JFrog AI Catalog for validating models and controlling what can be pulled
• Sysdig/NeuVector for runtime monitoring, behavior analysis, and enforcement when workloads move outside expected behavior

Control What AI Agents Can Touch

Identity has to carry through the environment. Authorization has to decide what can talk to what. The whole process has to be audited along the way. The agent shouldn’t talk directly to whatever it wants. It should have to route through a governed control point.

For higher-risk actions, add human approval. The request context gets sent to a person, the person decides whether to approve it. If approval doesn’t come back in time, it gets denied.

Then, like with web applications, put a context aware firewall (similar to a WAF) that’s responsible for prompt guarding input. It can help catch prompt injection patterns, block unwanted requests, and reduce the chance of sensitive data being exposed.

Tooling:

• Red Hat OpenShift as the enterprise Kubernetes platform for running these controls in a governed, supportable environment
• Keycloak for JWT tokens and identity
• OPA Gatekeeper for policy enforcement and deciding who can talk to what
• AgentGateway for routing, auth checks, audit, and prompt guarding
• Create an Envoy ext_authz as the custom hook for human approval flows (more on this in a later post)

Make AI Agents Stoppable and Traceable

You need observability for context, not just what the agent is trying to do, but why. Sadly, that part is still developing, but it matters because you need the full context behind the action (not just the action itself).

If something goes wrong, you need an audit trail of every step along the way:

• Who initiated the original query?
• What was asked?
• What was the full context of the conversation?
• What systems did it try to access?
• What systems did it actually access?
• Did human approval trigger?
• Was it approved or denied?
• What context did the approver see?
• How did the permissions and tokens get handed over?

You want the who, what, where, why, and how.

Tooling:

• Dynatrace for context-aware observability as that capability develops
• Sysdig/NeuVector for pausing, stopping, or restarting workloads when behavior moves outside the normal range

That’s the practical baseline for production access.

Without those pieces, and without a Kubernetes environment that can actually enforce them, you’re just playing with dynamite.

AI operational governance starts with the right architecture. Connect with Arctiq to explore how Kubernetes, observability, policy enforcement, and runtime controls can help organizations operationalize AI more securely and responsibly.